Josh Allen was 'in a panic' over vulgar tweets, says 'family is hurting'