Data sources refresh based on a specified frequency rate, from every 15 minutes to every 4 hours (depending on your pricing plan). You can select an alternate refresh rate for any number of data sources in your account as long as your selection fits into your current plan limit. To attain more frequent refresh rates, you can upgrade to a different plan.

This article includes:

• Automated refresh
• Changing the refresh frequency rate for instant metrics
• Changing the refresh frequency rate for custom metrics
• Manual refresh
• Data refresh failures

Automated refresh

Data sources are automatically added to the refresh queue based on their specified refresh frequency rate. When refreshed, the data in the raw (unmodelled) data source is automatically updated, as well as the data in all of its associated modelled data sources and metrics.

Note: Data sources that are connected to data from uploaded files cannot be set to automatically refresh. If the data in the file changes and you want the data source to include the new data, you must manually re-upload the file.

The time it takes for a data source to refresh depends on:

• The number of data sources already in the queue.
• The number of data sources in your account being refreshed simultaneously.
• The time it takes for the source system to respond to the query defined in the data source.
• Whether or not the refresh needs to be deferred in order to avoid exceeding the API limits for the service.

Note: Data sources are refreshed on a first-in-first-out order of priority. This ensures that resources are made available to active users. The only exception is deferred refreshes. When a refresh is deferred past the scheduled refresh time frame, the deferred refresh request is automatically added to the front of the queue.

Changing the refresh frequency rate for instant metrics

When you add instant metrics, we create and model your data sources for you. To ensure the integrity and accuracy of your instant metrics, note that the underlying data in these modelled data sources is managed by Klipfolio PowerMetrics and cannot be edited. However, you can update their refresh rate frequency.

To change the refresh rate for an instant metric:

1. In the left navigation sidebar, click Metrics to display the Metric List page.
2. Select the instant metric you want to edit.
3. Click the 3-dot menu in the top navigation bar and click Edit.
4. Under Data > Refresh rate, click the drop-down beside your current refresh frequency rate and select a different one from the list of available options.
   
   Note: The choices available to you depend on your pricing plan. If you want to access more frequent refresh rates, click Upgrade to move to a different plan. Note that this change will also impact related instant metrics that use the same service.
5. Click Save metric.

Changing the refresh frequency rate for custom metrics

With custom metrics, you create and manage all associated raw and modelled data sources.

To change the refresh rate for a custom metric:

1. In the left navigation sidebar, click Metrics to display the Metric List page.
2. Select the custom metric you want to edit.
3. Click the 3-dot menu in the top navigation bar and click Edit.
4. Click the link under Data to open the associated modelled data source details page.
5. On the modelled data source details page, click the link beside Uses data source to open the details page for the raw (unmodelled) data source.
6. Beside About this Data Source, click Edit Properties.
7. Beside Queue for Refresh, click the drop-down and select a different refresh frequency rate from the list of available options.
   
   Note: The choices available to you depend on your pricing plan. If you want to access more frequent refresh rates, click Upgrade to move to a different plan.
8. Click Save.

Manual refresh

As mentioned above, data sources refresh automatically based on a schedule but you can also manually queue data sources to be refreshed outside of that schedule.  Doing so can help you see updates to your metrics between automatic refreshes and troubleshoot refresh errors.

Manually queuing for refresh forces the data source to refresh and resets the automatic refresh counter. For example, if you plan includes a refresh frequency rate of every 4 hours and you manually refresh it 2 hours after it was last auto-refreshed, it will refresh automatically again 4 hours after the manual refresh. 

To manually queue a data source to be refreshed:

1. Click Data Sources in the left navigation sidebar to open your list of data sources. If you don’t see this option, go to the bottom of the left navigation sidebar and click your Account Name > Data Sources instead.
2. Select the raw (unmodelled) data source from your list of data sources to open its details page.
3. Under Connection, at Queue for Refresh, click Queue for Refresh.

Note: You can only manually queue raw, unmodelled data sources for refresh. For modelled data sources, select the data source from the list, click the link beside Uses Data Source (to access the unmodelled data source it is based on), then click Queue for Refresh on the About this Data Source page.

Data refresh failures

There are various reasons why data sources may fail to refresh and actions you can take to get them up and running again.

This section includes:

• About data refresh intervals
• Investigating data source failures
• Finding data source error logs
• Analyzing the error log
• Troubleshooting refresh failures

About data refresh intervals

Klipfolio attempts multiple times to connect to your data before disabling refresh. Refresh intervals are designed to make sure the maximum amount of time is given to retrieving new data and to enable errors to be cleared.

We attempt to refresh your data using the following intervals:

• After one refresh failure, Klipfolio attempts to refresh again every 5 minutes until successful.
• After three refresh failures, Klipfolio attempts to refresh again every 30 minutes until successful.
• After five refresh failures, Klipfolio attempts to refresh again every 6 hours until successful.
• After 20 refresh failures, the data source is disabled.

Investigating data source failures

If there is a red icon beside Data Sources in the left navigation bar, that means one or more of your data sources are failing. (If you don’t see Data Sources in the navigation bar, go to the bottom of the sidebar and click your Account Name >  Data Sources instead.)

To filter the list of data sources to only show the ones that are failing, in the upper-right corner of the data source list page, click the drop-down and select Failed Data Sources. (See below.)

To view error details or to reconfigure the data source connection, click the red exclamation mark beside the name of the failing data source. (See below.)

If it’s a modelled data source, under Connections, click the link beside Uses Data Source (as any changes need to be made in the raw, unmodelled data source).

In a raw, unmodelled data source, depending on the red warning text, at Refresh Status, click Reconfigure or Re-enable.

Finding data source error logs

If a data source is failing or disabled, you can consult the error log to help diagnose the problem.

To find the error log for a data source:

1. Click the red icon beside Data Sources in the left navigation sidebar to open your list of data sources. If you don’t see the above option, go to the bottom of the left navigation sidebar and click your Account Name > Data Sources instead.
2. Select the relevant data source.
3. For modelled data sources, under Connection, at Uses Data Source, click the link to the raw, unmodelled data source. Go to step 4.
4. Under Connection, at Last Refresh, click View Refresh Info to display the Error Log.

See the following section to learn more about how to use the Error Log.

Analyzing the error log

The following information about a data source is available in the error log:

• The last Refresh Date/Time
• The length of time (Duration) to refresh the data
• The current size of the data source
• The result of refreshing the data source. (Click the link for error information.)
• The kind of data source
• Whether the refresh was successful or not (Updated?)

Note: Data in the error log is purged weekly and is current for the last ten minutes.

Troubleshooting refresh failures

Note that service API limits can impact queue refresh times:

• Some APIs, like Google Analytics, impose a limit on the number of requests per profile, number of concurrent requests per profile, and the number of tokens per user account. For more information on the Google Analytics API, see Limits and Quotas on API Requests. Issues are most commonly seen when you have a large number of queries set to refresh in close succession. Klipfolio monitors the number of API calls for each OAuth token and profile. Klipfolio may defer a new refresh request from executing for a short period of time in order to prevent an account from violating an enforced limit. To help avoid this issue, when setting data source refresh rates, make sure the rate complies with limitations set by the service you’re connecting to.

Here are some possible causes of data refresh failure:

• OAuth tokens for some services expire and must be reauthorized on a regular basis. For example, a Facebook OAuth token must be reauthorized every two months. Learn more about OAuth authentication. Learn more about managing your account connections.
  
  
• Some APIs impose a limit on the size of file you can upload.
  
  
• Some data sources, such as SQL data sources inside your firewall, require you to add the Klipfolio dashboard IP addresses to your allow list. For more information, see Allow list IP addresses.
  
  
• API limits can affect the refresh interval for services. For example, due to the API limits set by Google Analytics and Adobe Analytics, the shortest refresh interval that can be selected for these services in Klipfolio is every 15 minutes.
  
  
• A query might be incorrect. For example, a Google Analytics query returns a "Connection Service" error if there is a line break at the end of a query.
  
  
• A file name might be incorrect. For example, if a file in Dropbox has been replaced with a file using a different file name.
  
  
• To ensure all of our customers experience optimum performance, we apply fair use limits. As such, there’s a maximum number of rows of data that can be ingested when creating and refreshing metrics. Most customers don’t approach these limits but, if you do, the ability to refresh your data may be impacted.