Five O’clock Shadow

I have only heard this phrase as referred to a beard. I.e., one would shave in the morning and by 5pm you'd see it start to grow back, like a shadow on the face.

Yes, exactly, it's not any kind of figurative expression, it is really quite literal: if you shave in the morning, by 5 o'clock you'll get a fresh growth on your face that looks a bit like a shadow.

If it looks like a shadow, but isn't actually, then it is a figurative expression.

Damn, right you are again. :-)

It's not a beard, but a scruffy shadowy look one receives if they shave in the morning but get a "shadow" at night. I have friends who work in jobs and have to shave twice a day if they have a shadow that comes in by the afternoon to keep that crisp look.

It occurs by the end of the work day (5pm), but it also reminds me of the light shadows cast by the sun by that time in the afternoon, but doubt there's much truth to that logic.