My take on it is that most humor is offensive to somebody or another.
While this is arguably true - that someone has to be the butt of a joke - it's not true that every joke contributes to the marginalisation of a group of people according to their gender and/or ethnicity.
A Charlie Chaplin gag in which a policeman falls on his arse is less harmful and offensive than a rape joke, for instance, for reasons that I hope are obvious.