Laws are there to ensure that worst case scenarios won't happen. As for your assumption that someone less qualified would get ahead of a white male, I refer you to Frank's post.
Diversity is not the excuse. It's normal to assume there are bright talents among minorities and among women who would be discriminated against if not for affirmative action.
So a less qualified woman or minority should automatically be put ahead of a more qualified man/white just so we can make sure there is no racism? What a bunch of bunk. You're entire argument is based on assumption, which is a big reason why the 5 justices punted on this case. There's too much assuming in regards to race by people who act and think it's still the 1960's because it benefits them.
But it's not based on qualification alone, not by a long shot. Studies have shown that legacy students are twice as likely to be admitted as students without legacy status because if the school rejects the legacy student, the parent's good will is likely to evaporate as will the gifts. Some lawmakers are even trying to make legacy admissions illegal, for they do, in fact, result in less qualified students being admitted over more qualified students.
Yet another reason affirmative action is still very much necessary; not just along racial and gender lines, but socioeconomic lines as well.
That's simply a reason to do away with "legacy" as a qualification too, since that also has nothing to do with qualifications. By supporting affirmative action, all you're doing is going the other route in some kind of misguided attempt to "balance" the playing field. You're adding one wrong on top of another. No matter how you try to justify it, nothing excuses the average white person who, despite being more qualified, gets passed over for a spot at a university or a company by a less qualified minority and/or woman.