As many have pointed out, I think it's pretty clear from the novel and the film that Fredo was had limitations that the family members recognized. Keep in mind that from the book and the film, we learn that the only son in whom Vito really wanted to invest was Michael. Remember how he allowed sonny to become involved in the family business under Clemenza's tutelige, but Clemenza saw what the Don really wanted, and used Sonny primarily as a bodyguard for his father? Also, remember how the Don let Sonny handle the situation with the furnace repair guys and how the Don was no satisified with the way Sonny handled it. On the other hand, the Don expresses surprise at Fredo's talent for running a hotel, but almost in the same breath disdains his son's usefulness ("What for? My wife can cook our meals.") and his unnatural proclivity for sex. Mike, onthe other hand, the Don wants to invest extensively in Micahel's future whether as part of hte family or as a poitician.

Regardless of Michael's reason and rationalizations, he could walk away from it all at any time without endangering his blood family. He chose not to. He wanted, instead, to leave it all behind (perhaps) on his own terms and that kept his family in danger.