Spiritual Paul used to say, 'It's better to be a God-pleaser than a people-pleaser, because there's only one of Him.'
It's possible people will think ill of one, with or without cause. But equally, some people will think well of one when they should frankly be taking a dimmer view. It's swings and roundabouts.
It's deadly to speculate on what other might think about one or to dwell on what they actually say.
Sift criticism for useful information to make course adjustments, then drop it completely.
One might appear to be serving others but one is really serving God.
God is already pleased and requires only reasonable sincerity and diligence of application.
That's it. Do that and one is home dry.
Keep the eye off other people and onto God.