Jim Brown quickly abandoned his plan to fight Muhammad Ali