Skip to main content

Understanding data source refresh intervals, failures, and errors

Updated

This article contains answers to the following questions:

About data source refresh intervals

Klips attempts multiple times to connect to a data source before disabling it. The re-check interval is designed to ensure that the maximum amount of time is given to retrieving new data and to enable errors to be cleared.

Refresh attempts use the following intervals:

  1. After one failure, Klips attempts to refresh again every 5 minutes until success.
  2. After three failures, Klips attempts to refresh again every 30 minutes until success.
  3. After five failures, Klips attempts to refresh again every 6 hours until success.
  4. After 20 failures, the data source is disabled.

Modifying the queue for refresh settings

You can manually place a data source in the queue for refresh or change its refresh interval settings.

To manually queue a data source for refresh:

  1. In the left navigation sidebar, click the 3-dot menu beside Data Sources and select Data Source Library.
    If you don’t see this option, go to the bottom of the left navigation sidebar and click your Account Name > Data Sources instead.
  2. Select the checkbox(es) for the data source(s) you want to manually refresh.
    Note: You can only manually queue raw, unmodelled data sources for refresh. For modelled data sources, select the data source from the list, click the link beside Uses Data Source (to access the unmodelled data source it is based on), then click Queue for Refresh on the About this Data Source page.
  3. Click Queue for Refresh.

To change the refresh interval for a data source:

  1. In the left navigation sidebar, click the 3-dot menu beside Data Sources and select Data Source Library.
    If you don’t see this option, go to the bottom of the left navigation sidebar and click your Account Name > Data Sources instead.
  2. Select the data source you want to modify.
  3. For a modelled data source, under Connections, at Uses Data Source, click the link to the raw, unmodelled data source. Go to step 4.
  4. Under Connection, at Last Refresh, click View Refresh Info.
  5. Under Refresh Information, click Edit and select an alternate refresh interval from the drop-down beside Queue for Refresh.
  6. Click Save.

Investigating data source failures

If there is a red icon beside Data Sources in the left navigation bar, that means one or more of your data sources are failing. (If you don’t see Data Sources in the navigation bar, go to the bottom of the sidebar and click your Account Name >  Data Sources instead.) To access your list of failed data sources in the Data Source Library, click the red icon.

To view error details or to reconfigure the data source connection, click the red exclamation mark beside the Data Source Name.

If it’s a modelled data source, under Connections, click the link beside Uses Data Source (as any changes need to be made in the raw, unmodelled data source).

In a raw, unmodelled data source, depending on the red warning text, at Refresh Status, click Reconfigure or Re-enable.

Finding data source error logs

If a data source is failing or disabled, you can consult the error log to help diagnose the problem.

To find the error log for a data source:

  1. Click the red icon beside Data Sources in the left navigation sidebar to open your Data Source Library.
    If you don’t see this option, go to the bottom of the left navigation sidebar and click your Account Name > Data Sources instead.
  2. Select the relevant data source.
  3. For modelled data sources, under Connections, at Uses Data Source, click the link to the raw, unmodelled data source. Go to step 4.
  4. Under Connection, at Last Refresh, click View Refresh Info to display the Error Log.

See the following section to learn more about how to use the Error Log.

Analyzing the error log

The following information about a data source is available in the error log:

  • The last Refresh Date/Time
  • The length of time (Duration) to refresh the data
  • The current size of the data source
  • The result of refreshing the data source. (Click the link for error information.)
  • The kind of data source
  • Whether the refresh was successful or not (Updated?)

Note: Data in the error log is purged weekly and is current for the last ten minutes.

Troubleshooting data sources that fail to refresh

  • Some APIs, like Google Analytics, impose a limit on the number of requests per profile, number of concurrent requests per profile, and the number of tokens per Google user account. For more information on the Google Analytics API, see Limits and Quotas on API requests.
     
  • Some OAuth tokens expire and must be re-authorized on a regular basis. For example, a Facebook OAuth token must be re-authorized every two months. Go here for more information.
     
  • Some APIs impose a limit to the size of file you can upload.
     
  • Some data sources, such as SQL data sources inside your firewall, might require you to add the Klipfolio dashboard IP addresses to your allow list. For more information, see Allow list IP addresses.
     
  • API limits can affect the refresh interval for services. For example, due to the API limits set by Google Analytics and Adobe Analytics, the shortest refresh interval that can be selected for these services in Klipfolio is every 15 minutes.
     
  • A query might be incorrect. For example, a Google Analytics query returns a "Connection Service" error if there is a line break at the end of a query.
     
  • A file name might be incorrect. For example, if a file in Dropbox has been replaced with a file using a different file name.
     
  • To ensure all of our customers experience optimum performance, we apply fair use limits. As such, there’s a maximum number of rows of data that can be ingested when creating and refreshing Klips. Most customers don’t approach these limits but, if you do, the ability to refresh your data may be impacted.

Learn more

Related articles