If he means it hatefully as is implied by his comment then yes. Otherwise it's just a statement that any reasonable person would interpret as racist. Given that his comment is racist with or without the slur I don't see any reason to give him the benefit of the doubt.

I really don't see why you want to argue that his comment isn't racist.