The Don's lack of affection for Fredo may have planted the seed of what Michael winds up doing later on.
The lack of affection may have come as late as hearing of Fredo's exploits in Vegas, but it may have occurred sooner. It becomes apparent when, in the book, Carlo is moved onto the mall. Tom questions it, and Vito says to him: "I seem to be short on sons."
Santino may have been dead, and Michael exiled in Sicily, but Fredo was very much alive and capable of returning to New York to be with his family. Vito didn't want him back. When Michael eventually returned, Fredo's insignificance to the Don rubbed off on Michael, who then also saw no worth in Fredo.
Lack of affection is a much too strong appraisal. Vito is disappointed in Fredo. Fredo was originally sent out to Vegas to begin checking it out for its business potential. Vito later instructed Tom to put together a team to inform him on what was going on out there and on Fredo also. Tom did not question Vito about it. He asked Vito if Carlo should be part of the team to which Vito said no. He then instructed that Connie and Carlo be moved into the mall to be near Mama.
