I think he means that he does not hate the slaveowner as a person, but he hates what slavery has made him become. Slavery in itself is bad, but it also caused the existence of slaveowners and their harsh behaviors. He wants to say that without slavery, he loves him as a human being, but he hates slavery and what slavery has turned him into
