I am slightly confused about the phrase "Beauty is skin deep"?

You know how people always say that beauty is skin deep? In other words, how you look does not correlate with who you are on the inside. However, i believe the hardships people face can sometimes how through their appearance. For example, think about alcoholics. They age at a faster rate and it clearly shows. You know what i'm saying? The same goes to people who do drugs. So why do people say that how you look on the outside does not reflect who you are in the inside? It's just confusing me.
I never understood that phrase properly.