Pictures in the 1940s and 1950s shows these guys mostly insuits often with fedora type hats
Even earlier
And yes,they probably wanted to fit it with the businessmen and politicians and other upper crust people rather that rely on committing street crimes

Back then if they were together at a crap game or social club,they probably dressed more casually but not in ripped jeans an a T-Shirt or a baseball cap turned backward

Somehow they managed to dress well and still not get exposed as being part of a criminal society for many years

I think rich people including politicians these days often are associating with people dressed like bums as long as they have some kind of reputation or fame or wealth and can influence large followings.