Choice of representation $R$ in the $rm SU(2)$ Yang-Mills action $frac{1}{g^2} mathrm{Tr}_{R} (Fwedge star F)$

The action should be invariant under the action of gauge group, therefore, you have to construct something, that when considering the action of gauge group remains unchanged.

Assume that the field stregth tensor $F$ transform under some representation of $G$: $$ F Rightarrow U^{i_1 ldots i_n}_{j_1 ldots j_m} F $$ To make this more clear, let us list several simple examples: $$ phi^i Rightarrow U_{j}^{i} phi^j - text{fundamental representation} $$ $$ phi_i Rightarrow U_{i}^{j} phi_j - text{antifundamental representation} $$ $$ phi_i^{j} Rightarrow U_{i}^{a} phi_a^{b} U_{b}^{j} - text{adjoint representation} $$ After multiplying $F$ and $F^{*}$ and taking of the trace, there has to remain and object without free indices. The operation of multiplication and $text{Tr}$ contracts equal number of lower and upper indices, so the only consistent way to make singlet is to have $m = n$. $$ text{Tr} F wedge F^{*} = text{Tr} U^{i_1 ldots i_m}_{j_1 ldots j_m} U^{j_1 ldots j_m}_{k_1 ldots k_m} F wedge F^{*} = text{Tr} U^{i_1 ldots i_m}_{j_1 ldots j_m} U^{j_1 ldots j_m}_{i_1 ldots i_m} F wedge F^{*} $$ The written by far expression by far doesn't correspond to irreducible representation in general - i.e, one can symmetrize (antisymmetrize) over $i_a$ or $i_j$. I may be wrong, but it seems that the general action of the aforementioned form is some irreducible representation of tensor product of several adjoint representations.