Any ideas on how to segment a 2D vector field?

I am given a 2D vector field and a ROI out of which I sample a random number of n FLOAT vectors of the form (x, y, dx, dy). What could be good ideas to classify each of these vectors in any of two classes? (e.g. foreground/background in the case of optic flow). The challenge is over all with the variable input size (from 0 to n elements) for all possible given ROIs and maybe not having a given discrete structure to accommodate the vectors.