Could be, Ack. Another possible explanation: Fredo wasn't in the room at the time, so Tom was being very focused in his statement.
The most telling example, IMO, occurs right at the beginning of the film: Fredo isn't present in the Don's office during any of the conferences on Connie's wedding day. He's busy getting drunk--a sure sign of weakness that the Don would disapprove of.