“The mere holding of slaves, therefore, is a condition having per se nothing of moral character in it, any more than the being a parent, or employer, or ruler.”