How can we help?

New post
3

Extend Data Source Refresh Interval Options

Avatar
Hannah Funk
None
Follow

Hello,

We are currently running into an issue with a third-party API burning too many credits, because all klips have a max data refresh window of 24 hours. I submitted a ticket to Klipfolio asking if there's a way to extend the data source refresh beyond 24 hours. Ideally, we’d only like to update the data source once a week or once a month. Joshua Cohen-Collier said this was not part of the roadmap, but advised to open a feature request. 

The only current workaround for this is to create a third-party script to execute a Klipfolio point, which unfortunately just creates one more potential failure point with our analytics setup.

Has anyone else had issues with this or thoughts?

Thanks in advance!

Godfrey Team 

Date Votes

7 comments

  • 1
    Avatar
    Leann Lewis

    We haven't had any issues with the refresh exhausting API call tokens BUT we have had need for more robust scheduling features. One example of a use case for us: we will query a large database table once in the morning after the table is updated (which also occurs once daily). We want our managers to be able to look at their dashboards in the morning and see metrics and data current to yesterday.

    The problem is if I tweak that data source/build a new one at any point the new "refresh time" becomes the moment I edited/created it. So if I edit at 2pm CST the data in the Klip isn't updated until the following afternoon unless we:

    • Make a point to "edit" the data source again in the morning to "fix" the schedule
    • Create a custom GScript (which we have for a few of these but I dislike the additional layer of complexity to an already complex system as well)
    • Set the refresh interval to a higher frequency (which is a waste of server resources since you'd be querying the exact same data set repeatedly through the day; on multiple tables for multiple data sources of similar design)

    It would be great if we had the flexibility to set a custom refresh schedule from the Klip itself. A start time for the refresh call and a recurrence interval of minutes, hours, days or weeks for example. 

  • 1
    Avatar
    Scott Lawrence

    Hi Leanne and Leann,

    Thank you both for raising your request in the community!  It is great to see the detail about your use case and some of the efforts you have been going through to get what you are looking for.

    What I am reading from both is a request to have more flexible scheduling options for data source refreshing to have more control over API credits being used (Leanne, can you confirm which service you are running into the API credit use with?) and more control over when the refresh schedule time is set:

    • longer frequency options
    • set for specific times of day 

    Does this capture the context you are asking about?

    Thanks,

     

    Scott.

  • 0
    Avatar
    Leann Lewis

    Thank you, Scott. I think that captures the essence of the requests.

  • 0
    Avatar
    Scott Lawrence

    Great!  I'll add both of you to a feature request and your use cases to help with prioritization.

    Cheers,

    Scott.

  • 1
    Avatar
    Boris Faitout

    Still no further refresh option... But I need this too, so I join the discussion to back the request.

  • 0
    Avatar
    Meggan King

    Hi Boris,

    Until we have this functionality, there is another option. 

    If you are comfortable using our Klipfolio API, you could control the data source refresh based on your custom schedule. You would need to set your data sources to refresh never, and then use the refresh endpoint for those data sources. The best way would be through a script. Google suite of tools has the ability to write scripts and schedule those scripts to run at specific times. 

    Here is a link to our data source refresh API query:

    https://apidocs.klipfolio.com/reference#section-post-datasource-instancesidrefresh

    Example:

    POST https://app.klipfolio.com/api/1/datasource-instances/0123456789abcdef0123456789abcdef/@/refresh

    kf-api-key: <your API key>

     

    Thanks

    Meggan

     

  • 0
    Avatar
    Kevin Languasco

    Hello,

    We're having a similar problem with Snowflake, with the Klipfolio refreshes consuming a lot of credits unless we have more precise control.

    Has there been any developments in the last few years to allow for this? Or is using a script to call the API still the best practice?

     

    Thanks!

    Kevin

Please sign in to leave a comment.