Re: the original question.

I think that Tom probably did have a mistress. As someone said, he's in Vegas, has access to brothels, and Vito is long dead.

While we may believe that Vito was such a huge influence on Tom (and he probably was), remember that it was Sonny who was his best friend, Sonny who saved him from a life of poverty and blindness, and it was really with Sonny that he felt a brotherhood. If Sonny and Fredo were closer to him in his early years, I would think it quite likely that he had a mistress.