Rays veteran says 1 thing could lead him to leave team