“But I must think that an address to his majesty to remove one of his servants, without so much as alleging any particular crime against him, is one of the greatest encroachments that was ever made upon the prerogatives of the crown.”