Software Engineering Asked by Sayak Mukhopadhyay on December 18, 2020
I am building a REST API which would power a front end as well as other 3rd party apps and hence I want it to be as "standard" as possible. Right now, I am trying to stick to HATEOAS. The only place I am struggling is pagination.
The authorization layer of our application is centralised. Multiple other apps use the centralised auth service and so my API needs to use the same. This gives rise to obvious problems in pagination, namely:
Right now, I am using an ad-hoc solution that fetches all records from the database, according to the API filters, and then the authorization layer, filters the unauthorized records and then another layer (lets call it the "Pagination Layer") filters according to the page parameters.
This works for now as our dataset is relatively less but I don’t think this will scale well. What are my options?
There are a few things I have thought about but have no idea how good of an idea they are:
For an efficient solution you need to be able to put the authorization and pagination constraints into the database query, and have the proper indexes for those aspects. Anything else will potentially overfetch an enormous amount of data. How big an issue this is depends entirely on the scale and characteristics of your data.
Can you translate the information you get from the authorization layer into a filter on the database query? Ideally something like getting the information "User A can access Projects X,Y and Z" and translate that into filters on your query.
If you cannot do that and have to pass every single result to it to know whether it is visible, you will always have some pathologically slow scenarios. For example if you have 1 million items, and your current user is allowed to view 10 of them, you might have to push the entire million items through the authorization layer just to get 10 results. How big of a problem this is depends heavily on the specifics of your application.
If you cannot push all these concerns to the datbase, which I assume is the case from your description, I think something like your solution 2 is the only reasonable way to handle this. You essentially need an internal pagination layer that fetches a bunch of results, passes them through the authorization layer and provides them to the rest of your application. Your externally-visible pagination layer then needs to request internally pages results until it has enough to fulfill the request.
This has the issue I mentioned above with potentially pathological queries in terms of performance, but I see no way to avoid that with these restrictions. There is also no fundamental issue with querying specific pages in this way, it's just expensive as you have to also query all pages before. But that is a general issue with pagination unless you can use advanced methods like keyset pagination.
If it is possible, you can also simply avoid providing the option to query specific pages. So you'd only provide a "next" link in each paginated response. This gives you the largest flexibility in designing your pagination, but obviously restricts what the client can do.
Correct answer by Mad Scientist on December 18, 2020
1 Asked on October 29, 2021 by hh54188
2 Asked on March 4, 2021
2 Asked on February 28, 2021 by stevanity
architecture development environment enterprise architecture microservices
1 Asked on February 26, 2021 by toshakins
1 Asked on February 23, 2021 by enlico
0 Asked on February 19, 2021 by newbiesoftwarer
0 Asked on February 17, 2021 by leonardo
2 Asked on February 16, 2021 by mcarthey
5 Asked on February 14, 2021 by noisecapella
1 Asked on February 12, 2021 by tlp
0 Asked on February 11, 2021 by lee-alderdice
4 Asked on February 9, 2021 by miguel-stevens
1 Asked on February 8, 2021 by jcode
1 Asked on February 7, 2021 by alec-sanger
1 Asked on February 5, 2021 by programer
1 Asked on February 1, 2021 by jojomatik
0 Asked on February 1, 2021 by brandon-gibson
search engine social networks web applications web crawler web development
2 Asked on February 1, 2021
1 Asked on January 31, 2021 by simon-h
1 Asked on January 31, 2021 by scottie
Get help from others!
© 2023 AnswerBun.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP