More border police are needed to tackle "highly sophisticated" trafficking gangs targeting the UK, a think tank has warned.