What causes rapid nitrogen inversion?

In the second row of the periodic table, elements have relatively small differences between the size their $mathrm s$- and $mathrm p$-orbitals. Therefore, the orbitals of $ce{NR3}$ can go from $mathrm {sp}^3$ to $mathrm {sp}^2$ with relatively little energy increase, so an amine can become planar and then reorient with the inverted stereochemistry. The same occurs with carbanions. I'm not certain, but I'm guessing oxonium cations do this, as well, and I would even guess that it would happen more quickly due to oxygen's smaller size.

The reason we don't see this with $ce{NF3}$ is because orbitals with more $mathrm p$ character are more affected by induction, so the highly electronegative fluorine atoms would lose electron density during inversion, which would be energetically unfavorable (to the point that it doesn't occur readily).

Chiral phosphines do not invert because the size difference between $mathrm s$- and $mathrm p$-orbitals for larger atoms is greater, so converting $ce{PR3}$ $mathrm{sp}^3$ orbitals to $mathrm {sp}^2$ would require much contraction and, therefore, more energy than for the analogous amine.