“The idea that the country should be led by white men goes back to antebellum slaveholders, who argued that the world was naturally divided between working drudges and elite leaders, who directed their workers and used the wealth the workers produced to promote progress.”