“Some of the biggest battles you face as a minister are, in the nicest possible way, with Whitehall and internally with civil servants, as opposed to your political battles in the chamber.”