Actually, this happens all the time. I've known people who grow cotton for profit in the South, and even if you're black, you will get called a racist nearly daily. "Cotton is racist" is a daily thing you will hear. And has been for like forty years.
It's pretty amazing, and I can't believe it took this long to make it to civilization.
Edit: as an aside, strangely, even if you're a white man who owns a little gulf based sugarcane plantation staffed only by black people, no one will ever make the connection even jokingly. No idea why. It was at least as iconic as cotton historically.
shrug