Is a PostGIS raster column more efficient than a geometry column in this scenario?

We have several different PostGIS tables that have a timestamp column and a geometry column. The geometry column contains polygons that are queried by a GeoServer instance to be served for display.

The same information is stored as vector points for each coordinate in "analysis" tables. The only other use case we have is querying these analysis tables for single point values, at specific timestamps.

The data is loaded with GDAL from raster files, and is always global. My question is, would there be any benefit to switch the pipeline to store the rasters in raster columns (either in-db or out-db) instead?

I’ve seen advice against using raster columns if not performing relevant computations, but the official PostGIS documentation doesn’t offer any information on the drawbacks of using them.