How can we help?

New post
0

Support 'Pagination' for JIRA Data Sources

Avatar
Haddon Fisher
None
Follow

Atlassian puts some low limits on the number of issues which can be returned through their API's, making it hard to pull out and pivot across large datasets. Their work-around is to use the 'startAt' parameter to pull multiple sets and then combine them, however there is no easy way to do this within Klipfolio; I believe you would need to set up multiple data sources and then map them back together. I reviewed the 'dynamic data source' documentation but I do not think this would be able to do what is needed here.

Given that this would impact any JIRA Cloud user, it would be a really nice feature to be able to configure a 'keep pulling until complete' option which would look at the 'total' field (which contains the total number of results in the search) and automatically increase the 'startAt' value until all of the issues are returned.

I realize that there could be latency or other processing implications, but I'd rather pull less frequently than need to build a separate data source for every 100 issues.

Date Votes

2 comments

  • 0
    Avatar
    Meggan King

    Hi Haddon 

    You can paginate through JIRA using our dynamic data sources. This is how we paginate through APIs - either by page number, offset or startAt. As you know, you can use startAt and maxResults to get more than the default number.

    https://yourCompany.atlassian.net/rest/api/2/search?jql=(project = "yourProject")&startAt=0&maxResults=100

    If you take your query, you can make it dynamic and iterate through all the data by adjusting the startAt value. Have you worked with dynamic data sources before? We have a great article here: https://support.klipfolio.com/hc/en-us/articles/216183237-BETA-Working-with-dynamic-data-sources and you would use MAP/MAPFLAT to paginate https://support.klipfolio.com/hc/en-us/articles/216182487-MAP-and-MAPFLAT (Advanced use cases)

    Your JIRA response will need to use the total value in order to calculate the startAt option and know when to stop.

     

    Please try using our dynamic data sources to get all the data in a single data source and let us know if you run into issues. You can send an email to support@klipfolio.com

    Thanks,

    Meggan

  • 0
    Avatar
    Haddon Fisher

    Hi Megan,

    I just heard back from Atlassian - it turns out that they lowered the API limit from 1,000 to 100!

    I read through the documentation on dynamic data sources; it's a little confusing, but I am going to give it a crack. However, I still would suggest a potential new feature to support this more easily, as this is a major pain point for JIRA admins and being able to support this 'natively' would be a big feather in your cap :)

    I'll chat back if I have any questions about dynamic datasources.

    Thanks!

Please sign in to leave a comment.