I’m so with you on this. Two years ago I took my entire wardrobe, put it in a box to donate and sent it off. I then replaced it with the same items to make it simple. At the time I was struggling with depression and doing whatever I could to unclutter my mind and it seemed to help a lot. Do I miss those clothes? Not really. Fashion is a sense of identity but I identify myself in so many other ways that clothing was merely a materialistic function for that.
I’m not sure if anyone notices. No one has ever said anything to me but if they did I don’t feel a need to explain myself.