If the people of the Louisiana 1st Congressional district want Steve Scalise to represent them in Congress, they can have him. But after the revelation that Rep. Scalise (R) spoke to a group organized by former Ku Klux Klan leader and all-time racist gadabout David Duke in 2002, the party should dump him as majority whip. Not because the liberal media demands it, but because it is just political common sense. And decent.

A number of conservatives have raged at the idea that Scalise should resign his leadership position in the party. They offer a pu-pu platter of arguments. Scalise worked on the political campaign of his cousin Ben Bagert against Duke in 1990. They say that Scalise does not have a pattern of saying racist things. That the GOP in Louisiana had a difficult time rehabilitating and purging figures that supported Duke's run for governor against Edwin Edwards, a known crook. They say he's also been critical of Duke in other contexts. Scalise chose to speak on tax reform at that 2002 event, not something like phrenology. And, after all, sometimes politicians get roped into speaking at events they don't understand.

Scalise also received a robust character defense from Representative Cedric Richmond, a black Democrat from New Orleans. Richmond told New Orleans' Times-Picayune, "Steve and I have worked on issues that benefit poor people, black people, white people, Jewish people. I know his character." Governor Bobby Jindal also came out in Scalise's defense.

And further, many conservative activists believe a terrible double-standard is at work. Scalise only spoke to this group once; the deceased Robert Byrd was allowed to continue for decades as a leading voice in the Democrats' Senate caucus. Byrd had been a recruiter for the Klan, and wrote letters in the 40s declaiming, "Rather I should die a thousand times, and see Old Glory trampled in the dirt never to rise again, than to see this beloved land of ours become degraded by race mongrels." Some point to more recent instances of the left or the media tolerating divisive and corrupt figures so long as they are Democrats, people like Rev. Al Sharpton. Others point to the president associating with left-wing radicals. Why should we play by the media's double-standard, they say?

Well, I can't think of a more unattractive pose than arguing that the Democrats have awful standards and the GOP should sink to them. Or confronting a controversy about palling around with David Duke and making your first response turning to the media refs and asking for a freebie. Yes, the Democrats tolerated Robert Byrd. He eventually earned 100 percent voting ratings from the NAACP. And by the time he died, his party was pulling in 90 percent of the black vote nationally. When black voters almost uniformly perceive you as the party opposed to their interests, you just can't put an one-time honored speaker at a David Duke event in party leadership.

For the good of the party and the country, Republicans need to reach out to America's oldest minority, the group whose emancipation was a motivation for many of the GOP's founders. Black voters, just as much as white exurban voters, deserve competition for their vote, in the cities just as much as in the South. If a Republican majority whip was discovered to have once spoken to a group created by Anton LaVey, founder of the Church of Satan, he'd be dropped from leadership before Richard Land could hit up a friend on speed dial. Keeping Scalise in leadership confirms that the GOP doesn't care about its perception among black voters. It's an own goal.

Why is he even worth defending? Scalise is not a exactly a unique voice or personality in the party. He's not a great policy entrepreneur, or a strategic thinker. He doesn't have an inside track on attracting new blocs of voters to the GOP. In other words, he brings nothing to the table that justifies excusing the 2002 event as just a matter of poor judgement, or incompetent staffing. Scalise is basically a standard issue-conservative distinguished only by his ability to play politics within his own caucus. There are quite literally about 200 guys just like him in the Republican rank and file. They are just like him except for one thing: They don't have a David Duke oopsie to explain for the rest of their lives.

In 1997, Senator Byrd in one of his many many apologies for his history as a Klansman said he had advice for young people wanting to join politics. "Be sure you avoid the Ku Klux Klan. Don't get that albatross around your neck. Once you've made that mistake, you inhibit your operations in the political arena." It seems funny now, to think that in 1997, anyone needed advice to avoid the Klan. But Steve Scalise needed to avoid a Klansman in 2002. It is an albatross around his neck. And once he made that mistake, his operations in politics should be inhibited.