Disruptive Technologies Disruptive Developers

Polling the Events API

For the majority of server-to-server integration use-cases, our API offers an easy-to-setup, scalable HTTPS Data Connector with an at-least-once guarantee.

There might be cases where polling the API periodically might be preferred instead, such as:

  • Integrating with legacy batch based APIs, where calling the API with one event at a time is too costly.
  • Integrating with an application behind a strict firewall, where opening the port needed for a Data Connector might not be possible.

This article will cover some best practices when polling our events API and a code example.

Summary of the Events API

All events for the last 30 days for each sensor is available at:

https://api.disruptive-technologies.com/v2/projects/PROJECT-ID/devices/DEVICE-ID/events

Key parameters for polling the Events API are:

  • start_time and end_time. This is the span from which to fetch events. If these parameters are not set, it will default to fetching the last 24h.
  • event_types, a list of the type of events to fetch.

Also, from the API documentation:

Indexing delay
There is a delay from when an event is received by our servers until they are indexed and available via this endpoint. This is typically 1-2 seconds, but can be up to 10 seconds.

For full documentation, see the Events API documentation.


Polling best practices

Some best practices for polling the Events API are:

  1. Always use the start_time and end_time to retrieve events.

    Ever new poll interval, the call to the Event API should ensure that:

    • start_time is set to the previous end_time minus the worst case indexing delay, 10 seconds.
    • end_time is set to the current time.

    By doing this you avoid:

    • Loosing events between calls - which can happen if you would always set start_time to be e.g. 5min into the past instead of using the old end_time, or because of indexing delays.
    • Fetching more data than you need - which is often the default if you don’t specify a timespan.
  2. Respect the Retry-After header in case of a 429 - Too Many Requests.

  3. Only fetch the type of events that you are interested in by using event_types


Code example

Below is a python code example which will:

  1. Fetch all sensors in a given project once.
  2. Poll the Events API of all sensors at a fixed interval and print a summary to the console.
  3. Supports pagination of both devices, for large projects, and for events, for long intervals.
  4. Respects the Retry-After header in case of a 429 - Too Many Requests response status code.

Prerequisites

To try out this code example you need to have Python 3 installed.

Create a new folder for this example open a command line console in this folder, run the following 3 lines:

1
2
3
python3 -m venv venv
source venv/bin/activate
pip3 install requests
  1. setup a standard Python 3 virtual environment (or venv),
  2. activate this environment and finally
  3. install the required packages via the built in Python package manager.

Keep this console window open as we will use it to run the example in a moment.

The source

Using your favorite editor, create a new file called poll-event-api.py with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
import time
import os
import requests
import json
import pprint
import datetime

MAX_INDEXING_DELAY_SECONDS=10

def requests_get_with_retry(url, **kwargs):
  max_retries = 10

  for _ in range(max_retries):
    response = requests.get(url, **kwargs)

    # Retry on status 429, Too Many Requests, after Retry-After seconds up to max_retries times
    if response.status_code == 429 and 'Retry-After' in response.headers:
      time.sleep(response.headers['Retry-After'])
    # Return immediately on any other status code
    else:
      break

  return response

def get_project_devices(project_id, auth):
  sensor_device_filter="device_types=touch&device_types=temperature&device_types=proximity"
  project_devices_url="https://api.disruptive-technologies.com/v2/projects/{}/devices?{}".format(project_id,sensor_device_filter)

  # Get all devices in project - with support for paganation
  devices = []
  nextPageToken = ''
  while True:
    raw_response = requests_get_with_retry(project_devices_url + "&page_token={}".format(nextPageToken), auth=auth)
    if raw_response.status_code != 200:
      print("Failed to access API with HTTP status code {}".format(raw_response.status_code))
      quit()

    response = raw_response.json()
    devices = devices + response["devices"]

    nextPageToken = response["nextPageToken"]
    if nextPageToken == '':
      break
  return devices

def get_device_events(device_name, start_time, end_time, auth):
  sensor_event_filter="event_types=touch&event_types=proximity&event_types=temperature&event_types=networkStatus"
  device_events_url = "https://api.disruptive-technologies.com/v2/{}/events?{}&start_time={}&end_time={}".format(device_name, sensor_event_filter, start_time, end_time)

  # Get all events from device - with support for paganation
  events = []
  nextPageToken = ''
  while True:
    raw_response = requests_get_with_retry(device_events_url + "&page_token={}".format(nextPageToken), auth=auth)
    if raw_response.status_code != 200:
      print("Failed to access API with HTTP status code {}".format(raw_response.status_code))
      quit()
    
    response = raw_response.json()
    events = events + response["events"]
    
    nextPageToken = response["nextPageToken"]
    if nextPageToken == '':
      break
  return events

def main():
  # Use Basic Auth. enabled Service Account Key ID and secret, and project ID, from environment
  username=os.environ.get('DT_SERVICE_ACCOUNT_KEY_ID')
  password=os.environ.get('DT_SERVICE_ACCOUNT_KEY_SECRET')
  project_id=os.environ.get('DT_SENSOR_PROJECT_ID')
  poll_interval_minutes=os.environ.get('DT_POLL_INTERVAL_MINUTES')
  
  # Start with an empty last-seen
  devices_last_seen = {}

  # Fetch all sensors in project, with paganation, once
  devices = get_project_devices(project_id, auth=(username, password))

  # Start with end_time being poll_interval_minutes back in time from now (UTC)
  end = datetime.datetime.utcnow() - datetime.timedelta(minutes=int(poll_interval_minutes))  

  while True:
    # Update start_time and end_time timestamps
    start = end - datetime.timedelta(seconds=MAX_INDEXING_DELAY_SECONDS)
    end = datetime.datetime.utcnow()
    start_time = "{:%Y-%m-%dT%H:%M:%SZ}".format(start)
    end_time = "{:%Y-%m-%dT%H:%M:%SZ}".format(end)

    print("Fetching all events between {} and {}...".format(start_time, end_time))

    # Get all events from all sensors between start_time and end_time
    for device in devices:
      events = get_device_events(device["name"], start_time, end_time, auth=(username, password))

      print("\t got {} events from {}".format(len(events),device['name']))

    print("Done! Waiting for {} minutes...".format(poll_interval_minutes))

    # Wait for poll_interval_minutes
    time.sleep(60*int(poll_interval_minutes))

if __name__ == "__main__":
  main()

Running the code

Before running the application, you need to set a few environmental variables:

DT_SERVICE_ACCOUNT_KEY_ID - The Key ID of the Service Account used to access the Project

DT_SERVICE_ACCOUNT_KEY_SECRET - The Secret of Key ID of the Service Account used to access the Project

DT_SENSOR_PROJECT_ID - The Project to poll

DT_POLL_INTERVAL_MINUTES - At what interval to poll, in minutes

When running the code below, it will assume that these environment variables are set.

Run the code on the command line as follows:

python ./poll-event-api.py

You should see output similar to:

Fetching all events between 2019-03-22T14:50:51Z and 2019-03-22T14:55:51Z...
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b5rj9el7rihk942p49p0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6m6s4d7rihhhm5omis0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6m6s4l7rihhhm5omku0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6roh657rihmn9oji66g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b6roh657rihmn9oji83g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/b9ngr9l7rihg93n9puqg
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bchonod7rihjtvdmd40g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bchoqbl7rihkg46592b0
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bchoqbt7rihkg46594m0
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktcl7rihjbm0406vg
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktcl7rihjbm040760
	 got 2 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktd57rihjbm040bn0
	 got 12 events from projects/bccgtp8en7inbeit5fbg/devices/bdokte57rihjbm040pmg
	 got 11 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040q80
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040s80
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdokted7rihjbm040sag
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktel7rihjbm040uj0
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/bdoktel7rihjbm04103g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvh5krle0009efg4g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvjdkrle0009efg50
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubccgvktkrle0009efg5g
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emube3a3aavpkgg00au2oog
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emube8odsegse4g0082e4vg
	 got 0 events from projects/bccgtp8en7inbeit5fbg/devices/emubfrtbunlfkr000eoequ0
Done! Waiting for 5 minutes...