I’m not sure if it’s beneficial or not so beneficial to allow how others will perceive your writing as an obstacle to hold you back. I think it’s with everything in life the old “you can’t make everyone happy.” But here’s the thing, the ones that do enjoy your writing will continue to follow your writing and that’s all that really matters anyway.