I think Vito knew that Fredo had serious limitations, but he never figured that Fredo would be dumb enough to do what he ended up doing. Fredo was included in the Sollozzo meeting, and the Don fully understood that Fredo would never dare say a word during the meeting (if he even understood what was going on). At that point Sonny was the heir apparent, and if you look at how wveryone was seated in the room, Santino has a very prominent position, while Fredo is less prominent than Tom. It is Santino who shoots his mouth off, and who ghets rebuked.

I think Vito figured that Fredo could just be a kind of hanger on, and sort of a non entity. I believe Vito knew Fredo was limited because in his speech to Michael at the end of the movie he tells Michael he knew Santino would have to "go through" all of this and that he never expected that Michael would have to. When he mentions Fredo (who you would think would have been #2 to Sonny (when in fact Tom was) the Don simply says "Fredo was....well...." and he moves on. This tells us the Don knew well that Fredo could never have been the head of the family.