Howto - Smart Data Feed

Introduction

Instead of querying data from our system, we are able to actively deliver data to the customer's system. This has the advantage that every new or modified item is directly pushed to the client. We call it the "Smart Data Feed." This page describes how to set up this smart data feed.

Requirements

To be able to receive data pushed by our systems, a Webhook Client is needed. It can be any service that is able to receive and process an HTTPS POST request. As we have to be able to call the service, it has to be publicly available via the internet.

The client has to be able to process requests asynchronously as multiple requests can occur in parallel, and timeouts might cause the subscription getting disabled. For example, during an Initial Load phase it is possible that the client receives several thousand requests per minute, as the requests are not throttled on our side.

The request will be secured via HTTPS - more to follow.

In every request you will find additional data within the request headers. Most interesting are the ID (atrify-subscription-id) and name (atrify-subscription-name) of the subscription which has triggered this request and a trace id (atrify-trace-id) which can help us to identify problems.

If a request cannot be delivered to the webhook, and we receive a non-successful response code, we will retry the request a configured number of times. If the request continues to fail after the retries, or we receive a 4xx response code, we will disable the subscription.

There are two ways to re-enable a disabled subscription:

• You can use the updateSubscription endpoint and set the enabled flag to true. Missed events during the disabled period will not be sent with this solution.
• You can create a new subscription, set the start date of that subscription to the time when the old subscription was disabled, and delete the old subscription. With this approach, you will receive all missed events since the old subscription was disabled.

We currently do not support authentication for webhooks. However, it is possible to use authentication mechanisms configured within the webhook URL itself.

Subscriptions

Subscriptions enable you to subscribe to diverse updates including notifications of new items as well as results of GDSN deliveries. Item updates are delivered to you after filtering them according to permissions and subscription criteria. All examples need authorization and are written for Insomnium (HowTo Insomnium).

To specify the different types of subscriptions, the attribute type has to be set. If not set, the "item" subscription is the standard type. For all other attributes of the different types, please see the API of the subscriptions service.

When you start Insomnium with the subscriptions collections loaded, the base use cases are covered and you can start using the collection right away.

Item subscriptions

Workflows

On every subscription you need to define a workflow to process the data before export. Currently, there are three workflows defined. Depending on the workflow, the webhook client has to accept the specific content type.

Workflow	Content type
key	text/plain
atrify	application/xml
cin	application/xml

Child items won't be included by default, you can include them by setting the includeChildren attribute to true (only useful for workflows cin and atrify). Including child items can make the XML validatable.

For more information about the attributes of a subscription, see the API specification.

Plain subscription

A "plain" subscription enables you to get all item updates from the time of the subscription's creation on.

The result could look like this JSON:

{
    "id": "9d2d45fd-bc9d-3b3d-a6d8-1c01f56a7d14",
    "owner": "1WorldSync-demo",
    "name": "demo subscription",
    "target": "{{target}}",
    "type": "item",
    "startDate": null,
    "workflow": "key"
}

Filter subscription

A filter subscription not only gets you item updates but also prefilters specific use cases for you. Currently, the following filters are implemented:

• suppliers
• supplierItemIds

In this example we use the filter suppliers.

The result could look like this JSON:

{
    "name": "test subscription initial",
    "suppliers": ["12345678"],
    "target": "{{target}}",
    "type": "item",
    "workflow": "key"
}

Filtered subscriptions are the recommended way to test if your configured webhooks are receiving the item updates we send. This ensures that only a small set of item updates will be sent to you. If you successfully receive the updates through your configured webhook URL, you can either update the newly created subscription to fit your needs or delete it and create a new one.

Subscription with Initial Load

You might want to build up an item base but do not want to wait until you receive an update for every single item. Here you can make use of the Initial Load by defining a start date in the past. This will trigger the sending of all items from that start date onwards. As this can be several million items, ensure your webhook client is set up for that.

In this example that start date is 01.01.2020:

The result could look like this JSON:

{
    "id": "9d2d45fd-bc9d-3b3d-a6d8-1c01f56a7d14",
    "owner": "1WorldSync-demo",
    "name": "demo subscription",
    "target": "{{target}}",
    "type": "item",
    "enabled": true,
    "startDate": "2020-01-01T00:00:00.000000Z",
    "workflow": "key"
}

Getting information about the Initial Load

Once you have created an Initial Load for a subscription, you may use the initialdelivieries resource to get information about the progress or suspend/resume a running initial load.

The first step is to get the id of the Initial Load associated with your subscription, in the following example we get some Initial Load with the id 8fb0b915-56df-4e4b-82a5-2233b9558816:

Using this id, you are able to examine the Initial Load in more detail:

This gives you the status about the Initial Load:

Field	Meaning
id	The id of the Initial Load
status	An easy to read status string, as follows:
	- Searching items in progress: The Initial Load still gathers the items that match the subscription, as soon as all the items were found it will start to send them to the target configured in the subscription.
	- Searching items in progress, suspend for delivery requested: The Initial Load still gathers the items that match the subscription. The user has requested to suspend the Initial Load, so that no item will be sent to the target of the subscription unless the initial delivery has been resumed (see section below)
	- Searching items completed, delivery starts soon: This should be visible only in seldom cases, the Initial Load has fetched all items and is about to start sending all the items to the target of the subscription in the next few seconds
	- Delivery of items in progress: The Initial Load is currently sending the items to the target of the subscription.
	- Delivery of items suspended: The user has requested to suspend the Initial Load, so no items will be sent unless the user decides to resume the Initial Load (see section below)
	- Completed: The Initial Load has been completed.
suspended	If false, the Initial Load is running, if true, the Initial Load has been suspended by user request. A suspended Initial Load can be resumed again, but as long it is suspended, it sends no new events to the target (due to technical reasons, already created events but physically not sent events might still reach the target of the subscription). See also section below
totalItems	Contains the total number of items the Initial Load will send to the target. This number will increase as long as the Initial Load is gathering the items (so the state is "Searching items in progress..."), but will be stable once it starts delivering the items
doneItems	Contains the number already successfully sent to the target. Due to technical reasons this number might be temporarily higher than the number of items physically reached the subscriptions target
toBeDoneItems	Number of items not being sent to the target yet
failedItems	Number of items that could not be sent to the target due to some severe errors. If this number is not 0, the customer should contact the support.

Suspending an Initial Load

You might wish to suspend an Initial Load once it is created. To do so, you may use a request like the following:

All you have to specify in the payload is the id of the Initial Load and the new value of the suspend flag. You are free to set this flag from false to true or vice versa how often you like.

GdsnFeedback subscriptions

When you are sending data to our GDSN delivery service, the datapool results can be sent to you via a subscription.

{
    "owner": "1WorldSync-demo",
    "name": "GDSN feedback demo subscription",
    "target": "{{target}}",
    "type": "gdsnfeedback",
    "suppliers": ["6521265132432"]
}

This will create a subscription which will send all datapool results related to the specified supplier to the specified target URL.

View subscriptions

To view the existing subscriptions, you can use the subscriptions endpoint with or without an id. Without a subscription id you will get a list of all relevant subscriptions.

The result could look like this:

{
    "id": "9d2d45fd-bc92-3b3d-a6d8-1c01f56a7d14",
    "owner": "1WorldSync-demo",
    "name": "demo subscription 3",
    "suppliers": null,
    "supplierItemIds": null,
    "target": "{{target}}",
    "type": "item",
    "enabled": true,
    "updateTrace": null,
    "updateReason": null,
    "startDate": "2021-01-01T00:00:00.000000Z",
    "workflow": "key",
    "createdAt": "2021-07-06T04:00:50.315241Z",
    "modifiedAt": "2021-07-06T04:00:52.005241Z",
    "patchToken": 1,
    "notificationEmails": []
}

Delete Subscriptions

To delete a subscription, you have to know the id of the subscription. In the example below, the last subscription created with Insomnium will be deleted. Of course, you can replace the variable by any subscription id.

If successful, the API will return a 204.

Notifications

You can get a notification via e-mail if your subscription got disabled, e.g. due to your target HTTP endpoint not being reachable. For that, add the notificationEmails attribute to your subscription JSON, passing an array of e-mail addresses to send the notification to:

{
    "name": "demo subscription",
    "target": "{{target}}",
    "workflow": "key",
    "notificationEmails": ["subscriptions@yourCompany.com"]
}