I've always felt that UNTIL the boathouse, Michael might have been willing to let Fredo go. It was his good-hearted brother led astray by that manipulative Roth and his Sicilian messenger boy. He would ostracize him, never see him or speak to him, but I think that when he asks about Fredo after his return from Cuba, he was sincere in what he said.
After hearing the depth of Fredo's bitterness, anger and, most of all, envy, Michael knew that Fredo was a danger. While everyone's initial reaction to Fredo's betrayal was that "poor Fredo" with his "good heart" had been lied to. After hearing Fredo's rant, Michael is shaken and realizes that his brother had held back all this bile for years, and that Roth had taken advantage of it. Michael hadn't even been aware of it.