What are the "masses"?

Newtonian inertial mass is based Newtonian mechanics, which is invalid, to an extent, as per Einstein's relativity. It is still valid for most application in common everyday scenarios. It is defined as the mass that gives resistance to acceleration. It is the mass that will come in $ F = ma $.

Rest mass is the mass that an object has at rest. It does not include the mass(/energy) that it gains due to velocity. This mass does not change irrespective of the environment, the velocity, etc.

Relativistic mass is the mass that the object has in total, which includes the rest mass, and in addition the mass it has in virtue of energy, including kinetic energy. The total mass is given by

$$ m_{rel} = frac{m_r}{sqrt{1-frac{v^2}{c^2}}} $$

where $m_r$ is the rest mass and v is the speed with which it is moving. Derivation to this formula uses more of relativity.

However, it is this mass (relativistic mass) that is to be used in all formulae, when the velocities are large. At small velocities, the denominator is approximately 1, and hence the relativistic mass and rest mass are nearly the same. When the velocities are almost that of the speed of light, the denominator is nearly 0, and hence the relativistic mass approaches infinity. Henci it becomes increasingly hard to accelerate past that.

Hope this helps.

The mass that most theoretical physicists used is 'invariant mass', defined by $m^2=E^2-{bf p}^2$ in natural units with c=1. This avoids confusion arising from 'relativistic mass' which is different in different directions and different Lorentz systems.