How to Get the data from multiple pages.?

Hello Klipfolio Gurus,

How you get the data from a API which returns data in multiple pages and depending upon the page_num parameter in the API query. 

I do not want to create like 15 different datasource for my 15 pages and use them in the Klip as there is limit on the number of datasources per formula. Is there a way around this? 

How would I manipulate data from these different pages in the Klip?

Thank you for your excellent support as always! :)


  • Avatar
    Keely Davison Official comment

    You can use our dynamic data sources with variables and the MAP function to get a set of pages from an api that requires paging.  This approach requires the api to return information on the total number of pages or entries returned by the query.  This approach can be used if you have a limited number of pages to return.  We typically recommend less than 10 pages.

    Here are links to dynamic data sources, variables and MAP documentation.




    Here is a specific example based on the service desk.com

    Create 2 datasources:
    1. a static datasource to return the first batch of data. This is your existing query with ?per_page =100&status=pending,open. From this data source, the 'total_entries' field tells you how many records are returned by your query, although the API can only return 100 at a time.

    2. a dynamic data source  query that assigns a "page" variable to the page parameter. It will look mostly like your static data source

    You will need to create the "page" variable and assign it a valid value (e.g., 2) before you create the dynamic data source.

    Then your formula would be something like (where total_entries is a field retrieved from your static data source):

    MAP ( NUMBERFORMAT ( CUMULATIVE( REPEAT( 1, CEILING( total_entries/100 ) ) ) ),
    <reference fields in the dynamic datasource> )

    The MAP function creates a dynamic data source instance for each page value. For example, if total_entries = 629 records, CUMULATIVE( REPEAT( 1, CEILING ( total_entries /100 ) ) ) would return 1, 2, 3, 4, 5, 6, 7, so 7 dynamic datasource instances would be created.

  • 3
    The Data Girl

    Thanks Keely,

    This is so cool, I will try this solution. U rock!

  • 1
    Contractor Publishing

    Thanks Keely so much - this is really very well explained! 

    I've followed all your steps and almost there in solving this - but not able to understand the entire thing within a klip and its execution basically - so some follow up questions

    1. How many data sources are you finally loading in the klip? Static or dynamic or both of them? 

    2. Where is the final query with "MAP" exactly added?

  • 0
    Arnie Chaudhuri

    Thank you so much Keely.

    Whatever the questions posted by Contractor Publishing I also would like to know them 

    Could you please help us to understand them completely.

  • 0
    David Georgy

    I would also like to know the answer to "Contractor Publishing"'s post - particularly "Where is the final query with "MAP" exactly added?"

    Could anyone from the support team please assist with this?  I work for Terem Technologies and am building a dashboard


    Thank you,

    David Georgy

  • 0
    Janice Janczyn

    Hi David,

    Please see Data Sources With Multiple Pages for details. If you have further questions, please email support@klipfolio.com and let us know what kind of datasource you are using.

    Thank you,

Please sign in to leave a comment.