Nope, disagree. Michael knew that Frankie was "old school" and that the honor thing was important to him. His brother's presence was a reminder of what the Corleone family was to him.

Yes, his brother was intimidated by his surroundings, but I think he looked more upset with Frankie than the Corleone Family.