Fans shared the same concern about Tom Brady