Disruptive Technologies Disruptive Developers

Data Connectors

A Data Connector is how you integrate Sensor data into your cloud.

Data Connectors can be setup without writing any code and provides a scalable and reliable way to consume Sensor Events in real-time.

By sending Events as JSON over HTTPS POST requests, no matter if you are using Microsoft Azure, Amazon AWS, Google Cloud, IBM Cloud or host your services elsewhere, you will have your first integration up in no time.

Data Connectors are as easy as:

  1. A Sensor sends out an Event
  2. Our cloud sends the Event to your cloud
  3. Your cloud acknowledges that the event was received

At-Least-Once Guarantee

Too keep your code lean, we promise an at-least-once Event delivery guarantee.

The Data Connector is similar to a webhook, but with an added delivery guarantee. Every Event received by our cloud is put in a dedicated, per-Data Connector, queue. No Event is ever dropped from this queue until it has been acknowledged by your cloud.

If your cloud does not acknowledge the Event, it will be resent until successfully delivered. The pause between each resend attempt will gradually increase to reduce the load on the receiving side. This means that even if your cloud would go down for a moment, you would still receive the Event when it is back up again.

Low Latency

Data Connectors give you the lowest possible end-to-end latency from the Sensor to your cloud.

Our cloud will create one HTTPS POST request for every Event we receive. Because we do not introduce artificial wait states to aggregate Events, latency is kept as low as possible no matter how many Sensors you have.

High Scalability

Data Connectors scales.

To combine low latency with scale, our cloud will never wait for one HTTPS POST request to complete before sending the next. A new Event will always instantly be sent to your cloud, running as many HTTPS POST requests in parallel as there are Events at any given moment.


Using Data Connectors

Using Data Connectors entails setting it up in Studio and handling the incoming request in your cloud.

Setting up a Data Connector

Data Connectors are configured on a per-Project basis. To create one, your User (or Service Account, if you’re using one) have to be a Project Administrator. To modify one, it’s enough to be a Project Developer.

In Studio, if you open a Project, you can find them in the left menu under Data Connectors. Here you can see all Data Connectors, create new Data Connectors, configure existing ones and delete ones you don’t need any more.

In the API, you may find them under the /projects/{projec}/dataconnectors endpoint. See the Data Connector API in the API Reference for a more detailed description on how to set them up programatically.

A Project can have several Data Connectors, and all Events are sent to all active Data Connectors. All enabled (more on that later) Events from all sensors and Cloud Connectors in that Project will be send over an active Data Connector.

Configuration

Parameter Description
Connector Display Name A human-readable name for the Data Connector.
Endpoint URL (Mandatory) The URL that Events will be sent to.
Connection Status If the Data Connector is active or not.
Secret Used to sign and verify the authenticity and integrity of a Event.
Events to Forward The type of Events that should be sent to the Endpoint URL.
Include Label Data What Sensor Labels should be attached to each Event.

Connector Display Name

Used to identify the Data Connector in Studio.

Can be used to describe where it is pointing, or what the purpose of it is, such as “Azure EventHub” or “Primary Sensor Integration”.

Endpoint URL

The URL to which the Data Connector will POST Events.

The Endpoint URL has to be a, HTTPS, HTTP over TLS, endpoint. It cannot be an unencrypted HTTP only endpoint, or creating the Data Connector will fail.

The Endpoint URL may contain Basic Auth. information in the URL. This would be configured as https://user:password@server.corporation.com/sensorData, where user and password is the username and password for your servers login.

Connection Status

If the Data Connector is enabled or not.

See Disabling a Data Connector for details.

Secret

The secret to sign the request with.

See Verify Secret for details.

Events to Forward

Select what Events to send to the Endpoint URL.

See Listening for Requests to see an example of an Event.

Include Label Data

Select what Sensor Labels you want to embed into each Event from the Sensors.

For Labels to be included, they need to be added to Sensors in the first place. In a way, the Include Label Data parameter acts like a filter, only the Labels listed here will be added to Events.

Special labels in Studio

When a name is set in Studio, that devices will be listed in Studio using that name instead of the Device ID. In the API, that name is stored in a label called name. Similarly, the description field in Studio is stored in a label called description.

To include these fields with each event, make sure to add them to the Include Label Data list.

Including custom labels can also be useful if you, for example, have a Label with a custom ID or context information of your Sensors that you want included with each Event.

Using a Data Connector

Your cloud has the following four responsibilities:

  1. Your cloud listens for POST requests on a dedicated HTTPS endpoint, e.g. https://server.corporation.com/sensorData.
  2. (optional) Your cloud verifies the requests Secret signature
  3. Your cloud processes the Event
  4. Your cloud replies with an 200 OK acknowledgement

1. Listening for Requests

Using any type of HTTP server, or cloud function, you need to have a web-server listening on the Endpoint URL that you have configured the Data Connector to point to.

We will send one HTTPS POST request to that URL with an Event encoded as a JSON. This could look like:

{
  "event": {
    "eventId": "bboqciu55u1g00c0g9n0",
    "targetName": "projects/bbbk89v86c6000c19pbg/devices/bapo55k1hbj000f5l0ig",
    "eventType": "touch",
    "data": {
      "touch": {
        "updateTime": "2018-05-08T13:29:47.543486780Z"
      }
    },
    "timestamp": "2018-05-08T13:29:47.543483560Z"
  },
  "labels": {}
}

timestamp field deprecated

Please use the updateTime of the event instead for information on when the event was triggered.

To see in detail how all of the Events could look like, go to the API reference, scroll down after the main API, expand the Models section and take a look at the Event model.

The Endpoint URL should be able to handle multiple POST requests in parallel, as the Data Connector will send one POST request per Event.

If you have configured a Secret on your Data Connector, then you can use this to verify that a request originated from your Data Connector.

The secret is a symmetric key, so it needs to be known both on the Data Connector and the endpoint listening for incoming requests.

Adding a Secret to the Data Connector will add a HTTP Header to each request called x-dt-signature. This contains a JWT which in turn contains a signed checksum of the body.

By first verifying the validity of the JWT and then comparing the checksum with that of the body, you know that the integrity of the payload is intact, and that the request originated from your Data Connector.

Please see a snippet of how verifying it could look like in JavaScript:

...
const jwt = require('jsonwebtoken')
const crypto = require('crypto')

var token = request.headers["x-dt-signature"];
var jwtPayload=jwt.verify(token,SECRET)
var hash = crypto.createHash('sha1');
hash.update(requestBody);
var validRequest = (jwtPayload.checksum == hash.digest('hex'))
...

3. & 4. Handle Request and Response

When you receive an Event, you would typically perform some processing on it, save it to a database or maybe an internal event queue. It is all depends on the use-case of course, and is totally up to the implementation of the request endpoint.

A typical request-reply flow would go something like this:

  1. You receive a HTTPS POST Event
  2. You process the data. This could mean saving the data, in the format of your liking, into a database or maybe forwarding it to the next service inside your cloud.
  3. You reply with a 200 OK response.

What is important is that the request should never return an HTTP 200 OK response before you are certain you are done with it. That is because when our cloud receives that 200 OK, the Event will be taken off our Event queue and checked off as received by you.

It is also important to note that the at-least-once guarantee is just that. An Event could be delivered more than once during some circumstances and it is up to your cloud to handle this.

Monitoring and Modifying a Data Connector

All Data Connectors contain some statistics of the health of the integration the last 24 hours.

The Success count shows how many time Events have successfully been sent over to, and been acknowledge by, the Endpoint URL.

The Error count shows how many times a Data Connector has either times out trying to deliver an Event to the Endpoint URL, or the Endpoint URL has returned a status code other than 200 OK.

The 99 percentile latency gives an indication of how long it takes from the Data Connector sends an Event to the Endpoint URL until it gets a 200 OK reply.

Manually Disabling a Data Connector

The most important thing to understand about disabling a Data Connector is that the at-least-once guarantee no longer applies.

When a Data Connector is disabled, it will drop any queued Events it might have and will stop queueing up any new Events.

When the Data Connector is enabled again, it will once again operate as normal, with an at-least-once guarantee on any new incoming Events.

Automatically Disabled Data Connector

Data Connectors have an at-least-once guarantee to allow for robust cloud-to-cloud integration. If the receiving endpoint would be offline for a short maintenance window, or even a prolonged outage, the integration will automatically recover since events are resent until received.

However, Data Connectors will be automatically disabled if:

  1. Event delivery have failed for 120 hours with no successful deliveries.
  2. There has been 18000 failed delivery attempts the last hour (5 per second).

When the Data Connector is automatically disabled, it will drop any queued Events it might have and will stop queueing up any new Events.

When the Data Connector is enabled again, it will once again operate as normal, with an at-least-once guarantee on any new incoming Events.


Data Connector vs. Stream API

The API provides two ways of consuming real-time data:

  1. Data Connectors
  2. Streams

The most important distinction between these two is that only Data Connectors provide an at-least-once guarantee.

Data Connectors are the only reliable way to do cloud-to-cloud integration.

The stream API (../devices:stream and ../devices/{device}:stream) should be seen as a convenience API to allow your clients to listen to real-time data directly from our API, in the cases were you do not need the events to pass through your cloud.

When to use Stream API

The stream API is intended for short-lived connections directly between a client and our API.

Some good examples are the “Identify a sensor” feature and the real-time updates in Studio. Both of these are implemented using the stream API. In these use-cases, it is not mission critical if the browser would drop the connection and an animation is not played.

The protocol that the stream API uses, SSE - Server Sent Events, has solid use-cases but it does not support an acknowledgement scheme. Because of this, there is no way to know if an Event was received by a client or not. This makes it a bad match for reliable cloud-to-cloud integrations.

Because the stream API is so easy to setup and use on a local developer machine behind a firewall, it might be tempting to use it to do cloud-to-cloud integrations as well, but this is not what it was designed to do.

The figure above illustrates how the Partner Cloud opens up a stream on the stream API, and our cloud then sends events on this stream. This is very convenient, but there is no mechanism for acknowledging that an event was received. This means that there is no way for our cloud to know if your cloud received a specific event.

In contrast, the above figure illustrates how the Data Connector works, where the Partner Cloud acknowledges the reception of each individual event. This allows our cloud to know with 100% certainty that your cloud received the event.

Only use the stream API for short-lived connection directly between a client and the API.