Slower global growth would surely hurt, plus Britain is a key U.S. trading partner and biggest foreign investor