Getting Visibility Data pushed into your Inhouse System
- ser_hcskb_api
- Faruk Mercan
Webhooks Visibility API allows pushing real-time transport data back to the registered consumer. Transport data includes information about the transport status changes, vehicle status, estimated time of arrival (ETA), transport related warnings etc.
Security
The Webhooks Visibility API uses an API key to authenticate requests. This API Key must be provided and used for authenticating webhook calls by the consumer side.
The authentication of the requests is performed using Bearer Authentication. The API key used by Visibility Hub webhook calls is the Bearer token.
Webhook target endpoints must be served over HTTPS. Transporeon Visibility Hub does not support unencrypted HTTP endpoints or any other protocols.
Request Retry Logic
HTTPS requests are retried for up to 4h after the first failure. Time between retries increases exponentially (up to maximum of 10 minutes between requests) until either HTTPS status code 2XX is received or 4h is exceeded. Webhook should be rejected only if there is an internal server issue - and otherwise we expect a 2XX response code.
When retry period of 4h is exceeded, the whole webhook is automatically disabled until further action is taken. If it happens, please contact the support once the problem is solved on your side (the consumer side) so that the Transporeon Visibility Hub can confirm the issue (by looking at the logs) and reactivate the webhook.
In case of 3xx and 4xx response status HTTPS requests are retried up to 1h after the first failure. When retry period of 1h is exceeded the webhook will be disabled.
The retry mechanism uses exponential backoff strategy, which starts from 100ms and increases gradually to 10 minutes. The interval between each request might change in the future without further notice.
When the target endpoint fails to receive the data, Visibility Hub queues up to 10 updates per transport before starting to internally discard the updates (ie: only the 10 latest updates are kept). This means if the webhook target endpoint becomes available after a certain period, then it's possible that some updates will be missed, which were generated during the downtime. The 10 updates can be any combination of the three main webhook triggers: tour-update (e.g. time-slot changed, stop changed, ..), tour-status-update (e.g. stop-level or tour-level events such as loading arrived/departed), eta-update (happening every 15 minutes up to every 1 minutes close to the stop).
SLA Requirements
Visibility Hub assumes high availability on webhook consumer side, as transport level events are pushed out immediately (without intentional delay) 24/7 as they happen. It is recommended to use some kind of message queue system (e.g. Kafka, Rabbit MQ) on consumer side to accept updates asynchronously and with a minimal delay. Using messaging system also helps to recover data in case it has been processed wrong initially and supports debugging efforts.
Backward and Forward Compatibility of APIs
The implementation of the webhook API endpoint must be forward-compatible for future additions of new fields. It is expected that receiving side accepts and silently ignores those new fields, without responding with an error.
Incompatible changes are agreed upon with integration partners beforehand before changing the API. Incompatible changes are:
Removal of mandatory fields or critical fields
Addition of new critical fields
Removal of any existing webhook topic type
Idempotency
Visibility Hub guarantees delivery of events at least once, for enabled webhooks with relevant topic subscription. Same update can be sent more than once under certain circumstances. You can handle these duplications by making your event processing idempotent so duplicate events would not have any additional effects. To support idempotency, each event is identified by a unique event ID. In case the webhook becomes disabled, then the data generated in the real-time is not queued, but simply ignored.
Events Ordering
At single transport level, Real Time Visibility (RTV) updates are sent out in the order they happened in Visibility Hub for this specific transport, so ordering in context of single transport is deterministic. For example, the events A, B and C happened for transport T in the sequence A,C,B. These transport status then also go out to the webhook in sequence A,C,B. The next update for specific transport is not pushed out until the previous update completes successfully (or exhausts its retries). Ordering of events across all transports is not deterministic, e.g. it is possible that transport A was updated before transport B, but webhook is sent out for transport B first.
Feature Parity with Visibility Hub Pull APIs
For the webhook there is a corresponding pull API endpoint served by Transporeon Visibility Hub. Therefore, for example if webhook implementor misses some transport status update calls (due to a bug, infra failure etc.), then pull API endpoints can be used retroactively for receiving the latest transport status. This is also the expected behavior after webhook becomes disabled due to the consumer service being down.
Re-receiving Failed Events
Missed transport updates can be retrieved over corresponding pull API endpoint. Visibility Hub currently does not keep track of failed webhook updates and resending of missed updates is not supported apart from the Request retry logic mentioned above.
Getting started
Webhook Endpoint
At the moment Visibility Hub does not offer a subscription API yet. Therefore, enabling webhooks is a manual work that must be done by the Visibility Hub engineering team.
The consumer side needs to
provide the URL and API-key (Bearer token) to the responsible project team, so that the configuration can be set up.
list the topics that the webhook will be processing and sending.
Available Topics
There's currently just one topic available for Transporeon clients: transport-rtv-update that contains the current state of the transport.
Event Structure
At the moment only JSON body is supported for the callbacks.
Every event sent to consumer will have identical base structure or event container. It is only the topic specific payload structure that differs per topic.
{
entity_id: (String),
topic: (String),
event_id: (String),
payload: {...}, // Actual event specific data.
source_ts: (String),
company_id: (String),
delivery_attempts: (Integer),
demo: (Boolean),
}The response fields meaning is defined in the API specification.
Below there is a full payload example with the event container and the payload within it.
Example JSON Payload (for topic: transport-rtv-update)
{
"entity_id": "708520151",
"topic": "transport-rtv-update",
"event_id": "7b6bd95f-14aa-479f-9071-c8e02f137a60",
"source_ts": "2023-08-09T04:14:22.173Z",
"company_id": "12314",
"delivery_attempts": 1,
"demo": false,
"payload": {
"transport_id": "708520151",
"transport_number": "932229150908032020-S",
"state": {
"label": "tracked",
"is_finished": false,
"is_working": true
},
"tracking_start_time": "2023-08-09T02:14:22.173Z",
"transport_sharing_url": "https://app.sixfold.com/go/14243gafe141f29fad8daf3a8163321e39245e787be21ba80a45206cdf6",
"vehicle_telemetry": {
"timestamp": "2023-08-09T08:11:30.990Z",
"position": {
"lat": 58.88099,
"lng": 25.54764
},
"heading": 94.92620116882966
},
"vehicle_data": {
"license_plate_number": "047TEST"
},
"custom_fields": {
"container_id": {
"value": "12-D",
"visibility": "creator"
}
},
"deliveries": [
{
"delivery_id": "d-1",
"loading": {
"location": {
"location_id": "lid",
"gate_id": "gid",
"gate": "gate",
"name": "Sawmill GMBH",
"booking_location_name": "Booking",
"address": {
"city": "Tallinn",
"street_address": "Telliskivi 1"
}
},
"rtv_info": {
"status": "arrived"
}
},
"unloading": {
"location": {
"address": {
"city": "Tartu",
"street_address": "Uus 1"
}
},
"rtv_info": {
"status": "en_route",
"eta_checked_at": "2023-08-09T07:44:44.467Z",
"eta": "2023-08-09T09:44:44.467Z",
"estimated_remaining_distance": 50000,
"eta_status": "on_time",
"eta_source": "internal"
}
},
"delivery_sharing_url": "https://app.sixfold.tech/go/token-value-1-d-1"
},
{
"delivery_id": "d-2",
"loading": {
"location": {
"address": {
"city": "Tallinn",
"street_address": "Telliskivi 1"
}
},
"rtv_info": {
"status": "en_route",
"eta_checked_at": "2023-08-09T07:44:44.467Z",
"eta": "2023-08-09T17:44:44.467Z",
"estimated_remaining_distance": 650000,
"eta_status": "delayed",
"eta_source": "internal"
}
},
"unloading": {
"location": {
"address": {
"city": "Tartu",
"street_address": "Uus 1"
}
},
"rtv_info": {
"status": "unvisited"
}
}
}
],
"temperatures": [
{
"timestamp": "2023-08-09T08:11:50.000Z",
"reefer_id": "default",
"sensor_readings": [
{
"sensor_id": "main",
"compartment_id": "default",
"temperature": 20
}
]
}
]
}
}