German-speaking refugees dragged British culture into the 20th century. But that didn’t go down well in Stepney or Stevenage, says William Cook