“Father had notions about manhood suffrage, public schools, the education and the elevation of the masses, and the gradual emancipation of the slaves, that did not suit the uncompromising views of people in places like Richmond.”