TransWikia.com

Feedme Pagination by Token

Craft CMS Asked by JMKelley on December 11, 2020

Has anyone used a pagination token when paginating Feedme? I currently use Firestore and they don’t use a full pagination URL but a token you append onto your next URL.

Initial URL
https://firestore.googleapis.com/v1/projects/YOUR_PROJECT_ID/databases/(default)/documents/cities/LA

Pagination URL
https://firestore.googleapis.com/v1/projects/YOUR_PROJECT_ID/databases/(default)/documents/cities/LA?pageToken=Ab7e2991jej91od727

In the Response

"nextPageToken": "AFTOeJwxKBJYSXXPpKyhOmB4_-SifKCDzUH0mhJchn1u5hknl3iutUKaPez8OVyR99PZM2014ZU1EpZt4GnUHwHBnwM0P1B9Z-yDhTyqTrmsVt4SIffOXCyxq1Yl7iZQJ0c1ltGHpuV9KWgUH6mvl6rHdCA"

2 Answers

I'm the one who posted that bug report regarding cursor pagination. ? I just wanted to share a simple version of the code I'm using in case it's useful for anyone. I'm using Airtable so you can see I'm only running the function if the feed source is an Airtable URL.

For reference, I'm getting an offset value like abcdefg from the API and I'm changing it to https://my-api-url.com/feed1/abcdefg to get the next page of results.

You can run this from a module like the one you can generate at https://pluginfactory.io/

use craftfeedmeeventsFeedDataEvent;
use craftfeedmeservicesDataTypes;
use yiibaseEvent;


class Module extends yiibaseModule
{
    public function init()
    {
      Craft::setAlias('@modules', __DIR__);
      if (Craft::$app->getRequest()->getIsConsoleRequest()) {
          $this->controllerNamespace = 'modulesconsolecontrollers';
      } else {
          $this->controllerNamespace = 'modulescontrollers';
      }

      parent::init();

      /* Changes the Airtable API pagination key on the fly */

      Event::on(DataTypes::class, DataTypes::EVENT_AFTER_FETCH_FEED, function(FeedDataEvent $event) {
        $response = $event->response;
        $feedUrl = $event->url;
        
        /* Only run if this is an Airtable feed with a response */
        if ( strpos($feedUrl, "airtable") && isset($response['data']) ) : 
          $data = json_decode($response['data']);
          
          /* We only need to update the event if there's an offset key */
          if ( isset($data->offset) ) {

            /* Break the URL up into it's components */
            $url = parse_url($feedUrl);
            $baseUrl = $url['scheme'] . '://' . $url['host'] . $url['path'];
            parse_str($url['query'], $params);

            /* Update the current query with the new offset value */
            $params['offset'] = $data->offset;

            /* Rebuild the pagination URL */
            $data->offset = $baseUrl . '?' . http_build_query($params, $enc_type = PHP_QUERY_RFC3986);

            /* Return the data to the event */
            $response['data'] = json_encode($data);
            $event->response = $response;
          }
        endif;
      });
    }
}

Correct answer by Dalton Rooney on December 11, 2020

First, you'll want to make sure you're on the latest FeedMe release as a bug around this was recently fixed: https://github.com/craftcms/feed-me/issues/694

You might also need a plugin that hooks into the EVENT_AFTER_FETCH_FEED event to do some offset transformation based on the tokens passed in.

Answered by Brad Bell on December 11, 2020

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP