I am building a REST API which would power a front end as well as other 3rd party apps and hence I want it to be as "standard" as possible. Right now, I am trying to stick to HATEOAS. The only place I am struggling is pagination.
The authorization layer of our application is centralised. Multiple other apps use the centralised auth service and so my API needs to use the same. This gives rise to obvious problems in pagination, namely:
Right now, I am using an ad-hoc solution that fetches all records from the database, according to the API filters, and then the authorization layer, filters the unauthorized records and then another layer (lets call it the "Pagination Layer") filters according to the page parameters.
This works for now as our dataset is relatively less but I don’t think this will scale well. What are my options?
There are a few things I have thought about but have no idea how good of an idea they are:
For an efficient solution you need to be able to put the authorization and pagination constraints into the database query, and have the proper indexes for those aspects. Anything else will potentially overfetch an enormous amount of data. How big an issue this is depends entirely on the scale and characteristics of your data.
Can you translate the information you get from the authorization layer into a filter on the database query? Ideally something like getting the information "User A can access Projects X,Y and Z" and translate that into filters on your query.
If you cannot do that and have to pass every single result to it to know whether it is visible, you will always have some pathologically slow scenarios. For example if you have 1 million items, and your current user is allowed to view 10 of them, you might have to push the entire million items through the authorization layer just to get 10 results. How big of a problem this is depends heavily on the specifics of your application.
If you cannot push all these concerns to the datbase, which I assume is the case from your description, I think something like your solution 2 is the only reasonable way to handle this. You essentially need an internal pagination layer that fetches a bunch of results, passes them through the authorization layer and provides them to the rest of your application. Your externally-visible pagination layer then needs to request internally pages results until it has enough to fulfill the request.
This has the issue I mentioned above with potentially pathological queries in terms of performance, but I see no way to avoid that with these restrictions. There is also no fundamental issue with querying specific pages in this way, it's just expensive as you have to also query all pages before. But that is a general issue with pagination unless you can use advanced methods like keyset pagination.
If it is possible, you can also simply avoid providing the option to query specific pages. So you'd only provide a "next" link in each paginated response. This gives you the largest flexibility in designing your pagination, but obviously restricts what the client can do.
Correct answer by Mad Scientist on December 18, 2020
10 Asked on December 21, 2020 by hawk
1 Asked on December 18, 2020 by sayak-mukhopadhyay
3 Asked on December 18, 2020 by yadiv
1 Asked on December 18, 2020 by jaguar
5 Asked on December 17, 2020 by tau
3 Asked on December 15, 2020 by p-paul
2 Asked on December 10, 2020 by oren
4 Asked on December 4, 2020 by imperialgendarme
1 Asked on December 3, 2020 by user-agent
0 Asked on December 1, 2020 by codeinaire
3 Asked on November 27, 2020 by snail-cadet
0 Asked on September 25, 2020 by pinman
3 Asked on September 11, 2020 by d33tah
3 Asked on September 9, 2020 by sinaesthetic
1 Asked on August 3, 2020 by leeish
Get help from others!