So whereas the South African Bantus arrived in South Africa a few centuries before the whites did

No, they didn't. They arrived more or less at the same time. Up until then, it had been inhabited by the Khoi and San peoples. Who the Bantu then largely genocided.