“The right of the judge to inflict punishment gives him both power and opportunity to oppress the innocent; yet none but crazy men will from thence determine that it is best to have neither a legislature nor judges.”