“Leaving the single market, making communities poorer and more alienated, is not the way to deal with public concerns about immigration, most of which comes from outside the E.U.”