The Don's lack of affection for Fredo may have planted the seed of what Michael winds up doing later on.
The lack of affection may have come as late as hearing of Fredo's exploits in Vegas, but it may have occurred sooner. It becomes apparent when, in the book, Carlo is moved onto the mall. Tom questions it, and Vito says to him: "I seem to be short on sons."
Santino may have been dead, and Michael exiled in Sicily, but Fredo was very much alive and capable of returning to New York to be with his family. Vito didn't want him back. When Michael eventually returned, Fredo's insignificance to the Don rubbed off on Michael, who then also saw no worth in Fredo.