Region Proposal Network - How to subsampling 256 fg/bg anchors

I am trying to understand training process of RPN. I have problem with creating mini batches of 256 anchors. If features map has shape 18×25=450 and every position has 9 anchors it is 4050 potential anchors. Output class shape will be 18x25x18 and output regression shape will be 18x25x72. How to select only 256 anchors? I read that we have to select 128 fg and 128 bg randomly. If we label fg anchors with [1,0] and bg anchors with [0,1], how to label anchors which should be ignore? With [0,0]? I don’t think [0,0] will prevent backpropagation of loss over anchors which should be ignored.

For example, if we have 9 anchors for same position on features map and only 1 of 9 anchors is fg, while others are bg (based on iou calculations). My question is how to discard 7 other anchors, ie. how to make 50:50 fg:bg ratio (in this example 1fg, 1bg and 7 discarded anchors)? If we can’t change the output shape, we have to somehow discard redundant bg anchors. We don’t want RPN to backpropagate loss over all of 9 anchors. It will be biased to bg anchors.

Please correct me if I am wrong and tell me what is wrong.