I just wear one when I have to OR want to. And I absolutely hate gloves, never wear them. Blue jeans, t-shirt and tennis shoes most of the time.
I always found it to be humorous that so many people feel the need to worry about what someone else wears.