Their unemployment is 5.4%. Not sure about the British, but when Alabama passed laws to kick out its illegal immigrants, farmers couldn't find anyone to work the fields. Alabama's unemployment is 5.8%.
When I hear a lot of this talk of jobs, it seems like people are looking for factory work that really isn't coming back to the industrialized world for a variety of reasons. If there was real evidence that native-born Western citizens were looking at low level service jobs with a serious eye, I don't think we would be having a lot of these discussions.
And, even if we said that these low paying jobs were the issue, it's throwing the baby out with the bathwater, because the French bankers are being kept out along with the Polish plumbers- assuming that the Brexiteers get their wish. Because leaving the EU does not impact the fact that many Caribbeans and South Asians immigrate to the UK as well.
I don't know of many societies in the history of the world for whom growing insular produced real results. On the other hand, there are a lot of examples where looser borders and more trade produced fantastic results that benefited the middle class.