0

Klipfolio Datasource: Pagination LinkedIn, Facebook and Twitter

kxell 2001

March 21, 2019 09:11

None

Follow

Hello,

Does anyone have already experience with dynamic data sources e.g. pagination for linkedIn, Facebook and Twitter?

I tried it for LinkedIn, but somehow I'm stuck. I used the explanations in following Links.

https://support.klipfolio.com/hc/en-us/articles/216183237-BETA-Working-with-dynamic-data-sources

https://support.klipfolio.com/hc/en-us/community/posts/208219638-How-to-Get-the-data-from-multiple-pages-

https://support.klipfolio.com/hc/en-us/community/posts/209530928-Klipfolio-Datasource-Zendesk-API-Pagination-

 

So first, i created a static data source with following Query URL:

https://api.linkedin.com/v1/companies/{id}/updates?format=json?count=100

where i get the total number of pages in the constant _totalpages (which is in my case 400+)

Then a dynamic data source with a klip variable pagenum = 1;

https://api.linkedin.com/v1/companies/{id}/updates?format=json&count=100&start={props.pagenum}

Then i used the formula in the table: MAP (CUMULATIVE ( REPEAT ( 1, ( CEILING ( _totalpages/100)))) ,"pagenum", JOIN(id) )

But somehow, instead of getting the data like pagination e.g.

https://api.linkedin.com/v1/companies/{id}/updates?format=json&pages=1

https://api.linkedin.com/v1/companies/{id}/updates?format=json&pages=2

https://api.linkedin.com/v1/companies/{id}/updates?format=json&pages=3

https://api.linkedin.com/v1/companies/{id}/updates?format=json&pages=4

etc.

i get 

https://api.linkedin.com/v1/companies/{id}/updates?format=json&pages=1

exactly two times

Any ideas?

Best regards

• Facebook
• Twitter
• LinkedIn

3 comments

• 0
  
  Janice Janczyn March 21, 2019 14:15

  Hello,

  Evaluate the first parameter in your MAP to verify it's returning the expected values, that is 1, 2, 3, 4....  If _totalpages gives the total number of items returned, you need to divide by the number of items/page to calculate the number of pages. For example, if your query returns a total of 420 items at 100/page, you'll need to get 5 pages of data; this is what the CEILING ( _totalpages/100) calculates. But if _totalpages is the number of pages, you do not need to divide by 100, however, MAP/MAPFLAT cannot process 400 pages: it will time out so you'll need to find a way to filter/limit the amount of data being requested. Typically we recommend a maximum of 15-20 items in a MAP/MAPFLAT although in some cases it's possible to exceed this.

  A few other things to note:

  • the LinkedIn API v1 has recently been deprecated, you now must use LinkedIn API v2
  • not all services support pagination and some services, such as Twitter, use cursor-based pagination which Klipfolio does not support. 
  • as referred to above, we now have the MAPFLAT function which supports returning more than 1 item for each value in the 1st parameter, so instead of

                 MAP (CUMULATIVE ( REPEAT ( 1, ( CEILING ( _totalpages/100)))) ,"pagenum", JOIN(id) )

            you should use (note the JOIN is no longer needed in the 3rd parameter): 

                 MAPFLAT (CUMULATIVE ( REPEAT ( 1, ( CEILING ( _totalpages/100)))) , "pagenum", id )

   

  Thanks,
          Janice

• 0
  
  kxell 2001 March 21, 2019 15:09

  Hello Janice,

  thnx for the fast response. I have another question for my understandings how Klipfolio works. If I understand it correctly the API request gives me only 100 items per request. Does the dynamic data source try to get all items with one requests? or does it manually start a request if I start the paging on the dashboard? or am i completely wrong?

  best regards

• 0
  
  Janice Janczyn March 21, 2019 16:53

  Hi,

  A dynamic data source query returns only the data for the specific variable settings. Following your example, that is only one page at a time. This is why the MAPFLAT is needed: to query all the pages one at a time and combine them into a single result.

  Thank you,
              Janice