“But when his mistress for the time being was a woman in society, or at least one whose birth was not so lowly, nor her position is so irregular that he was unable to arrange for her reception in 'society,' then for her sake he would return to it, but only to the particular orbit in which she moved or into which he had drawn her.”