Download Feeds from Advertisers for your Adspace

Before you start

To find out more about product data feeds, have a go at this article in the knowledge base. Please note that advertisers do not necessarily supply product data feeds and if they do, they might not grant all partners access to them. If you should find that – after following these instructions – you cannot find the product data feed, please contact your advertiser.

Getting started

Navigate to Creatives -> Product data feeds in the main menu.

Select an Adspace. You will then see the product feeds you are allowed to download for that particular feed.

Configure your feed

To start working with a product feed, click on the icon next to the product feed you need. If you do so, you will see a panel looking more or less like this:

This panel allows you to configure the data feed and export a file that is completely adjusted to your needs.

Columns

First, select the columns of the products you require: you can add and remove columns using the buttons in the middle of the panel. The window on the left represents the available data fields, the window on the right represents the fields you have selected. By using the buttons on the far right (first, move up, move down, last), you can define the order in which the data will be displayed in the file.
If you do not select any column, then the export will automatically contain all columns of the data feed.

Sub-IDs

Next, there is the section that allows for additional configuration of the tracking URL. Here you can specify subIds appended to your tracking Link. These subIds will be added as parameter to the tracking URLs. By doing this you can easily sub-segment traffic that originates from the product data feed (e.g. to distinguish traffic from connected public networks).

Format

Further below, you can define other characteristics of your file:
- the format (CSV, or XML)
- the csv-delimiter (SEMICOLON, COMMA, or TAB)

Preview

Setting parameter preview to true allows you to download a feed with a vastly reduced file. Use this for testing purposes. File downloads with the parameter preview set to true will not consume any quotas.

Compression

the compression type (ZIP or GZ)

Note: The Ingenious Platform does not support uncompressed feeds. Feeds quickly become large and exceed hundreds of megabytes and more. Because the content of feeds is always textual (csv, xml), compression algorithms can reduce the file size significantly. Compressed feeds save bandwidth, but also are faster to download.

Your configuration will affect the URL you see at the bottom of the page. When you are done, you can just copy this URL and paste it into your own application, or download the product data directly using the download button.

Download your feed efficiently and with low latency

To retrieve the newest products their pricing from your advertisers, you have to consider the following topics:

quotas and rate-limiting: “You should not load a feed too often.”
conditional requests: “You should load the feed only if it’s changed.”

The following sections describe how you deal with both topics:

Quotas and Rate-Limiting

Different types of API requests to Ingenious are subject to different rate limits. To ensure the quality and to better protect our Product Data Feed system integrity, we have the following limitations

Maximum of 5 concurrent download requests per 1 minute per partner
Maximum of 3 download requests per 1 hour per feed per adspace for compressed feeds
Maximum of 1 download requests per 1 hour per feed per adspace for uncompressed feeds

In case you exceed these quotas, the response of the feed download will return a 429 Too Many Requests HTTP status code.

$ curl -I "YOUR-DOWNLOAD-LINK"
> HTTP/1.1 429 Too Many Requests
> Date: Tue, 15 Mar 2022 14:55:57 GMT

In this case you have the following options how to react:

your implementation should back-off and reduce its frequency of loading data. However, this will lead to a higher latency for getting new product data.
you implement the “conditional request” logic describe below

Conditional requests

The Ingenious Feed response returns an ETag header, and also a Last-Modified header. You can use the values of these headers to make subsequent requests to this feed using the If-None-Match and If-Modified-Since headers, respectively. If the resource has not changed, the server will return a 304 Not Modified HTTP status code.

Note: Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.

$ curl -I "YOUR-DOWNLOAD-LINK"
> HTTP/1.1 200 OK
> Date: Tue, 15 Mar 2022 14:59:17 GMT
> ETag: "b9bb4a36d73f69f44ae8588b15950651"
> Last-Modified: Tue, 15 Mar 2022 12:39:44 GMT
> Cache-Control: no-cache

Store the ETag (in the example "b9bb4a36d73f69f44ae8588b15950651") and/or Last-Modified (in the example Tue, 15 Mar 2022 14:59:17 GMT) value(s) on your side.

You now can check/poll regularly if the file changed:

Using If-None-Match and providing the ETag value you stored:

$ curl -I "YOUR-DOWNLOAD-LINK" -H 'If-None-Match: "b9bb4a36d73f69f44ae8588b15950651"'
> HTTP/1.1 304 Not Modified
> Date: Tue, 15 Mar 2022 15:01:01 GMT
> ETag: "b9bb4a36d73f69f44ae8588b15950651"
> Last-Modified: Tue, 15 Mar 2022 12:39:44 GMT
> Cache-Control: no-cache

Or using If-Modified-Since and providing the Last-Modified value you stored:

$ curl -I "YOUR-DOWNLOAD-LINK" -H 'If-Modified-Since: Tue, 15 Mar 2022 14:59:17 GMT'
> HTTP/1.1 304 Not Modified
> Date: Tue, 15 Mar 2022 15:02:18 GMT
> ETag: "b9bb4a36d73f69f44ae8588b15950651"
> Last-Modified: Tue, 15 Mar 2022 12:39:44 GMT
> Cache-Control: no-cache

Once the feed has been updated by the advertiser, the feed download will return the compressed feed file with an HTTP status code 200.

Check for changed feeds via API

You can check for updates on all your feeds regularly via the API point /creatives/productdata/findFeeds. Here are the important fields of information in the response regarding update status of the feed:

lastFileHash - Hash of the last version of the feed. Save this in your database. If this hasn’t changed since you last updated the feed, an update is not necessary.
lastUpdatedAt - The time when the feed last changed. If this date is older than your last download date, an update is not necessary.

Example Request:

curl -X POST "{YOUR_API_DOMAIN}/creatives/productdata/findFeeds?adspaceId={YOUR_ADSPACE_ID}&page=0&rowsPerPage=100" \
 -H "Accept: application/json" \
 -H "x-api-key: {YOUR_API_KEY}" \
 -H "Content-Type: text/plain"

Example Response:

{
  "page": 0,
  "totalPages": 1,
  "entries": [
    {
      "feedId": "1234",
      "name": "Example Feed",
      "advertiserId": "9876",
      "advertiserName": "Example Advertiser",
      "lastUpdatedAt": "2022-03-17T11:55:59.761+00:00",
      "lastModifiedAt": "2022-01-17T10:12:44.121+00:00",
      "numberOfProducts": 12844,
      "lastFileHash": "637df52981d762c284d3db9a588carc7",
      "updateInterval": "0 0,12 * * *",
      "nextImportScheduledAt": "2022-03-18T11:00:00.000+00:00",
      "importUrl": null,
      "contentType": null,
      "compression": null,
      "status": null,
      "partnerRestrictionEnabled": null,
      "allowedPartners": null,
      "lastAction": null,
      "lastActionStatus": null,
      "lastActionMessage": null
    }
  ],
  "totalEntries": 1
}

Responses are paged. Make sure to process all the pages to not miss out on a feed.

Debugging and Testing

When you want to integrate a feed for the first time, you may want to download it multiple times without having to concern yourself with rate-limits. For this use-case, you can use the preview mode to get a version of your feed that is vastly reduced in file size. Using preview mode will not consume any quota.

Error handling

If your request cannot be processed, you typically receive an error response like this:

{
	"errors": [
		{
			"code": "CRE_UNSUPPORTED_COLUMN",
			"description": "Requested columns that do not exists in processed file of feed (123): MY_COLUMN"
		}
	]
}

Here is a list of the most common error codes and how to best react to them:

Error Code	Reaction

Error Code	Reaction
`CRE_UNSUPPORTED_COLUMN`	You requested a column that doesn’t exist. Go to the UI and check the feed mapping again in the download page. This problem typically occurs when the Advertiser changed the available columns on the feed.