We study the quantum phase transition in an atomic Bose gas near a Feshbach resonance in terms of the renormalization group. This quantum phase transition is characterized by an Ising order parameter. We show that in the low temperature regime where the quantum fluctuations dominate the low-energy physics this phase transition is of first order because of the coupling between the Ising order parameter and the Goldstone mode existing in the bosonic superfluid. However, when the thermal fluctuations become important, the phase transition turns into the second order one, which belongs to the three-dimensional Ising universality class. We also calculate the damping rate of the collective mode in the phase with only a molecular Bose-Einstein condensate near the second-order transition line, which can serve as an experimental signature of the second-order transition.
