This is a very good article by Sheldon Richman, editor of The Freeman magazine. Here are some snippets making Sheldon’s point:

…a libertarian who holds his or her philosophy out of a conviction that all men and women are (or should be) equal in authority and thus none may subordinate another against his or her will (the most
common justification) ? that libertarian would naturally object to even
nonviolent forms of subordination.  Racism is just such a form (though
not the only one), since existentially it entails at least an
obligatory humiliating deference by members of one racial group to
members of the dominant racial group…Seeing fellow human beings locked into a servile role ? even if that role is not explicitly maintained by force ? properly, reflexively summons
in libertarians an urge to object…Another, related, libertarian reason to oppose nonviolent racism is
that it all too easily metamorphoses from subtle intimidation into
outright violence. Even in a culture where racial ?places? have long
been established by custom and require no coercive enforcement, members
of a rising generation will sooner or later defiantly reject their
assigned place and demand equality of authority. What happens then? It
takes little imagination to envision members of the dominant race ?
even if they have professed a ?thin? libertarianism to that point ?
turning to physical force to protect their ?way of life.?…a libertarian campaign
against racism in public accommodations should take the form of
boycotts, sit-ins, and the like, rather than assault and destruction of
property… it follows that State action is also
beyond the pale, since government is force. Hence the libertarian
objection to government bans on segregation in privately owned places…It would be a mistake, however, to think that ruling out government
action would severely limit the scope of protest…lunch counters throughout the American south were being desegregated
years before passage of the 1964 Act. How so? Through sit-ins,
boycotts, and other kinds of nonviolent, nongovernmental
confrontational social action.

I would also add that an ideology of racism is inherently collectivist in its world view. Racisim strips people of their individual identity and judges them as part of a group or collective. On the other hand, an ideology of true anti-racism, i.e. an ideology that judges people as individuals and not as members of a race (and we can include gender or sexual orientation) is inherently an individualist ideology. Racism and collectivism go had-in-hand.