How can we help?

New post
2

Monitoring Failing Data Sources Table

Avatar
Adam Doogan-Smith
None
Follow

Occasionally data sources will fail for one reason or another. It could be that the permissions or the password for a token has changed, your token has expired and needs to be re-authorized or a setting within your database has changed. If you have multiple dashboards it could be time consuming to check every one to see if the red exclamation mark is in the top right corner. However, you can create a klip that tracks all the data sources in your account. If any data source begins to fail you can quickly determine which data source is it and remedy the situation.

To build this monitoring klip, you will use the Klipfolio API. First, you can generate an API key by clicking on your name in the top right->Account->General Information->Edit->Generate New API Key.

Next, you will create a custom data source to pull data for the data sources in your account with the /datasource-instances endpoint. The exact query is:

https://app.klipfolio.com/api/1.0/datasource-instances?limit=100

*Note that the maximum amount of records the /datasource-instances endpoint returns in one call is 100. If you have more than 100 data sources, you will need to utilize pagination to return all data sources. Here is documentation on how to paginate through multiple pages in Klipfolio.

Once the data source is complete, you can now create a table to list all your data sources. In one column, you can have the data source name, in another the data source id and in the final, the fail count. You can sort the column that has the fail count highest to lowest. You could also filter the fail count column to only show records that are greater than zero, though in this example I have not. If any data source is failing, its refresh_fail_count record will be greater than zero. You can then identify the failing data sources quickly and take their data source id to investigate why the data source is failing. Once the data source begins to refresh properly, the refresh_fail_count will reset to zero.

Hopefully this helps you manage your data sources!

-Adam

Date Votes

3 comments

  • 0
    Avatar
    Carlos Kynäslahti

    Hi,

    Thanks for this tutorial. Hopefully one day Klipfolio will have out-of-the-box style data source monitoring (Klip template) so we don't have to build or own.

    Does this Klipfolio API GET data source work essentially just like any other data source or does it 'bypass' whatever might be in the data source refresh queue? I'm using a dynamic data source which might return a result which is too large for Klipfolio to handle and then the data source would fail. I'm wondering how fast would my dashboard let the user know the data source is failing using this API technique? If it takes a few minutes, that might be an issue.

  • 0
    Avatar
    Adam Doogan-Smith

    Hi Carlos,

    This endpoint works the same as every other GET operation and will return all data source instances in your account including the ones in the data source refresh queue. Unless you have over ~1000 data sources within your account, then this technique should not fail. I say ~1000 because there is 80 second timeout with API queries and this usually occurs around 10 pages of data. It could also be more or less depending on how much information is being passed and this can't be determined until running the queries and seeing the timeout point. If you have more than 100 data source instances, you will have to paginate through the multiple pages of data source instances.

    The dashboard will be as fast as your data source refresh to let users know which data sources are refreshing/not-refreshing. The connector for the Klipfolio API allows for the data source to be refreshed as quick as every minute, so in theory the klip/dashboard you build would be updated every minute.

    -Adam

     

  • 0
    Avatar
    Caileigh Simpson
    Hi There!

    I wanted to let you know that a new feature has been released in which data source monitoring has been made easier. 

    If there's a data source error or errors, a red icon will display in the side nav beside Data Sources. If you click the icon you will go to the Data Source list page, where you can find red icons beside the the offending data source (may be more than one).  Click the error icon  to go to that particular data source details page. If there is red text there you can click it to see the error and attempt resolve it. 

    Please let me know if you have any feedback after using the feature!

    Thank you, 

    Caileigh

    (Product Manager)

Please sign in to leave a comment.