A real man can both spend time with his family AND have a mistress. Vito may have frowned about mistresses, but he reserved his harsh criticism for a man that doesn't spend time with his family (he wanted Sonny to hear that). Note Vito's only mild retort to Sonny: "playing comedy with that young girl."

"A refusal is not the act of a friend"