What stops miners from manipulating "target" difficulty in the block header?

The difficulty target in the block header has to match the currently required difficulty. This value is derived and checked by each node individually from the timestamps and difficulty statements of previous blocks. Then, the block hash of a new block has to adhere to this given difficulty target. If either of the two requirements is not met, other nodes will reject a block as invalid.

So, if a miner ignored either of the rules, they'd of course be able to do that, but then their blocks are simply not accepted by the Bitcoin network.