What are the red, white, blue (and green) numbers?

Some numbers have been partitioned into $4$ groups. You must figure out how the numbers in each group are related, and what separates the groups.

Red: $1, 3, 4, 9, 13, 14, 15, 19$

White: $5, 7, 10, 12, 17, 20, 22, 24$

Blue: $2, 6, 8, 11, 18, 21, 25, 26$

Green: $16, 23$

One convoluted explanation involves giving $4$ different polynomials and then saying that the red numbers are all roots of the first one, the green numbers are roots of the second one, and so on. However, I guarantee you that the same phenomenon is behind all four groups. If your answer involves $4$ different explanations, it’s wrong.