Fans said the same thing after referee Bill Vinovich apologized to Josh Allen