In the movie we see a lot of "business scenes". And since Fredo is not into the Family activities, he never walks into Don Vito's office.

But it's interesting that they're not seen together more than a couple of scenes.

And when Vito learns that Mike is now into the hot action he says something like "Santino: I knew he had to pass thru all of this. Fredo: well... you know... Fredo..."