He knew. Up until that point he was under the impression that all was forgiven. But when Connie calls Anthony away, he realizes that he's in an inescapable situation. Alone in a boat with Al Neri, and the rest of the family gone from the compound. Twice before, Mike had used last-minute changes of travel plans to get his victim alone for the killing (Tessio and Carlo). When Mike once again makes a last-minute change of plans leaving Fredo alone with the Corleones' top enforcer, he must have known that was it. I believe Fredo was smaht enough to figure that out. Not like everybody says, like dumb. He was smaht, and he knew.