Show me a person who says, "I don't care what other people think about me," and I'll show you a liar.
We all care, just in different ways. There's no one who doesn't care. If someone just absolutely insists that they're somehow the exception, plaster flyers all over town stating that they're a twice-convicted serial killer/sex offender and see if they care about the way the neighbors react.
Trust me, they'll care. And they should.
Because your professional and personal reputation may be the most important thing you have. If someone knocked my professional integrity, work ethic, or sense of purpose towards work, I would care quite a bit, given the importance of reputation in my field. I would want to nip whatever was being said in the bud before it took root.
So to reiterate, yes, I would care.
Qualitative statements that can't be proven right or wrong are a little bit different, I'll admit. For instance, let's say someone said, "No one can be that friendly all the time. He must be fake." (If you're wondering, this has been said about me...more than once).
That's a hard one to respond to, and honestly I don't really get all that worked up over it. No one can define what makes someone 'fake' and there's nothing that could be said, or maybe even done, to challenge that opinion.
But if someone said something factually false (i.e. 'I heard that guy plays Nintendo in his office and takes a three-martini lunch every day at the Officers' Club') I would feel an absolute compulsion to find the source and shut him up.
To not care about something like that would be downright irresponsible.
So as a small step forward, let's all just start by admitting that we do -- and should -- care what other people say about us, and then we can start to diverge when we get down to the specific nature of the speech in question.