I agree with everyone here who sees Barzini as the big threat if Vito had said yes to Sol's deal:

I believe that, logically, Sol would have approached Barzini first with his deal. Sol would have seen that Barzini--the would-be capo di tutti cappi in waiting--would have the most to gain from the money he'd make with Sol on drugs because it would help him push past Vito. Barzini probably told him that Vito would never make a deal if he knew that Barzi was Sol's partner, so he said, "Go to Tattaglia. He's a pimp--Vito's not afraid of him. Tell Tatt anything you want, but I'll be your silent partner."

So, if Vito had said yes, his capital and his police and political protection for Sol would be making Barzi stronger, putting him in a better position to move against Vito. Equally if not more important, Vito would have the problem of controlling greed within his own family. The novel says that Sonny was hot for the deal because he saw it as a way to get out from under his father's thumb. Vito was already doubtful about Sonny as his successor because of Sonny's hot-headedness. Add to that, greed, the threat to the family's police/political allies, and dealing with sharpies like Sol and (hidden) Barzi, Sonny would bring grief to the family sooner rather than later.