I think the Sicilian aspect played a big role in Frankie's decision.

1) Frankie was actually born there,and was, at heart, an old-school Mafioso.

2) When he saw his brother, I think it made him realize how far he had fallen from the values of the Tradition he had been a part of.

3) I think it was more his brother's shame than anything else that was communicated in that brief glance.

4)Knowing that he ( Frankie) would forever be known in Palermo as a rat,and that Vincenzo would be ostracized by everyone in his world,not only during his lifetime ,but the lifetimes of his children and grandchildren as well,was unacceptable to 5 Angels.
Don't forget,Sicily was the land of Vendetta.There was bloodshed between families over things like the theft of a pig committed by long dead ancestors.

Also Frankie says that his brother could have had his own Family,so I assume that the concepts of Honor, Family, and Omerta were very well ingrained in him, and Frankie's betrayal would have destroyed everything he lived for and believed in.