I reckon Vito had his finger on the pulse, his ear to the ground and knew everything

It is fleshed out in the novel, he knew Fredo was amusing himself with young girls more than a grown man should and getting slapped in public, by Greene

No doubt he wouldn't have been happy about Fredo expressing an opposing opinion in front of others.
Never tell anybody outside the family what you're thinking again

Vito didn't consider Fredo anything more than taking care of some Mickey Mouse night club somewhere! and picking somebody up at the airport! but he should have especially after the Vegas outburst Vito was slippin'