How to identify segment/object that is anomaly using computer vision

I have an OpenCV script that can pull out the shape of different objects in an image.
For example, here is the output from one script where it calculated the outline of 4 different shapes[
and here is the original image

I want to automatically identify if one of the objects is different from the others. In the given example, the object in the bottom right is smaller than the others, so it is an anomaly. In some cases there will be no anomalies.
I am trying to figure out the best way to do this. What I’m thinking is that I can calculate the width and height of each object. Then I can calculate the average width and height. Then if any of the objects are more than 10% off the average height or width I could mark them as anomalies.

This seems like a janky / hardcoded solution though so I am wondering if there is a better way to do this?