“There is surely no contradiction in saying that a certain section of the community may be quite competent to protect the persons and property of the rest, yet quite unfit to direct our opinions, or to superintend our private habits.”