“Free movement of labour was never meant to be an unqualified principle, irrespective of how it might have worked on the ground. We do need to see action taken in relation to negotiation with the EU. [The government is] taking a fundamental look at some of the rules that allow unrestricted immigration.”