How can we help?

New post
0

Pagination in a Rest API - MATFLAT

Avatar
MARCELO ROCHA
None
Follow

Hi ! 

I'm new here. Even so I have background in database, data analysis and reporting in different client tools, I'm unable to use the MAPFLAT formula and I really did not find any tutorial or step-by-step to guide me.

Please, if someone could give some directions, I would really appreciate it.

My scenario

I need to connect to REST API, using a GET method  to obtain a JSON file with pagination. Each call/page brings me 100 itens. The webservice has a fixed number up to 200 pages.

My URL for the query is: https://bling.com.br/b/Api/v2/notasfiscais/page=1/json&apikey= xxxxxxxxxxx


After create my static connection, I created a variable in Klip with the name page. Then I created a dynamic connection, using this URL https://bling.com.br/b/Api/v2/notasfiscais/page={props.page}/json&apikey=


I tried to create a MAPFLAT formula in the KLIP but I did not succeed. What should I do to loop all 200 pages and generate a single file in Klipfolio.

Does anybody knows why it is done in the Klip Enviroment not in connection or modelling tiers?

Thank you very much,

M.Penner



 

 

Date Votes

1 comment

  • 0
    Avatar
    Adam Doogan-Smith

    Hi Marcelo!

    Unfortunately if you are trying to loop through 200 pages of an API that will not work within Klipfolio. Within Klipfolio, long-running queries (pagination) have a maximum of 80 seconds to complete at which time if the query is not complete the data source stops. This means that a data source can approximately return about 20 pages from a single dynamic data source before time-out issues become present.

    To test this on your end, perhaps try looping through the first 10 pages of the API and see if that returns data. If that does try 15, then 20, etc. However, at this time if you are trying to calculate on 200 pages of API data, you'll need to process that data outside of Klipfolio and maybe aggregate it before bring it in to create a dashboard with.

    Hope this helps!

    Adam

Please sign in to leave a comment.