Why do people care so much about skin color?

I mean if we all turned our skin inside out it would be pretty much the same