In a draft of the script, it was implied that Frankie had a mistress and child (or maybe just a child, I forget) back in Sicily, and it was hinted that it would not be good for them if he was a rat. Now, I'm not sure if the implication was that they'd kill them (as they don't kill "civilians"), but more likely that they'd be dishonored and disgraced, where if he did the "right" thing, then the Corleones would make sure they had money and were taken care of.

Even in the movie it's hinted that if he testifies, he's bringing ruin to his name and his family (for breaking Omerta and being a traitor; his family would be disgraced) - his brother was brought to the trial to illustrate it. If Frankie had testified, his brother would see the Pentangeli family name disgraced. That's why the sight of his brother was enough to make him recant the statement; he couldn't bear to do that to his family. It's also why he committed suicide at the end - to wash away the disgrace of nearly breaking the "Code" (the "like the Roman Empire" discussion)