No, he's right. Trying to bully either actual Nazis or just a normal person a leftie accused of Nazism, can reinforce the idea in both accused/accusors mind. It will also never disuade a Nazi.

You're cementing the idea "he's a nazi" in your mind. His mind received your attack and was hurt (in some small way, I'm sure he'll be fine). A damaged mind has crazy ideas like "all black/white/Jewish people are bad"