“Affirmative action was designed to recognize the uniquely difficult journey of African-Americans. This policy was justifiable and understandable, even to those who came from white cultural groups that had also suffered in socio-economic terms from the Civil War and its aftermath.”