Skip to content
← API documentation

Specification

INTD bucket format v1

The format is open on purpose: confidentiality and integrity come from encrypting and signing the request, not from keeping the layout secret. A fixed structure and CRC32 let the server throw away junk cheaply, before anything is written to storage.

Stable Big-endian CRC-32/IEEE X25519 · HKDF · AES-256-GCM

Container

OffsetSizeFieldValue
0 4 MAGIC 0x49 0x4E 0x54 0x44 (“INTD”)
4 1 VERSION 0x01
5 1 FLAGS 0x00, reserved
6 8 TIMESTAMP uint64 BE, unix ms — the base for the deltas
14 4 EVENTS_COUNT uint32 BE
18 4 BLOB_LENGTH uint32 BE, length of EVENTS_BLOB
22 N EVENTS_BLOB the events, back to back
22+N 4 CRC32 uint32 BE, CRC-32/IEEE of EVENTS_BLOB

The total size is exactly 26 + N bytes; trailing bytes are an error.

Event

SizeFieldValue
1 TYPE 0x01 pageview, 0x02 search
4 TS_DELTA uint32 BE, ms after TIMESTAMP
2 + var DOMAIN uint16 length ≤ 255, UTF-8
2 + var URL uint16 length ≤ 2048, UTF-8
2 + var QUERY uint16 length ≤ 500, 0 when absent
1 + var LANG uint8 length ≤ 20, BCP 47
1 DEVICE_TYPE 0x00 unknown, 0x01 desktop, 0x02 mobile, 0x03 tablet, 0x04 bot

Strings are truncated on a UTF-8 character boundary. The server rejects a bucket when a length exceeds the limit, a string is not valid UTF-8, the event count disagrees with the header, or the CRC32 does not match.

What the server does

  • The domain is recomputed from the URL (eTLD+1); the client value is not used.
  • Tokens, passwords, e-mail addresses and card numbers are stripped from the URL and the query.
  • Events older than 7 days, and events more than 10 minutes in the future, are dropped.
  • The country is resolved from the connection IP, and the installation identifier is hashed separately for each extension.

Encryption and signing

envelope 
sessionKey   = random(32)
iv           = random(12)
ciphertext   = AES-256-GCM(sessionKey, iv, bucket)
ephemeral    = X25519 — a fresh pair per bucket
shared       = X25519(ephemeral.private, extension.publicKey)
derivedKey   = HKDF-SHA256(IKM = shared, salt = iv, info = "intentd-bucket-v1", 32)
encryptedKey = AES-256-GCM(derivedKey, iv, sessionKey)   // 48 bytes
signature 
X-Signature = hex(HMAC-SHA256(secret,
  "intentd-v2|bucket|" + X-Timestamp + "|" + X-API-Key + "|" + hex(SHA-256(body))))

The scheme gives no forward secrecy if the extension private key is later compromised, and does not protect against whoever holds a copy of the bundle: repackaging is countered by per-installation sessions, the extension ID check and the server-side checks.

Test vectors

bucket (hex) 
// One search event: base 1758000000000, delta 7,
// domain a.io, URL https://a.io/?q=x, query x, language en, desktop
494e5444 01 00 0000019950f72c00 00000001 00000025
02 00000007 0004 612e696f 0011 68747470733a2f2f612e696f2f3f713d78 0001 78 02 656e 01
cee33ee2
hkdf 
HKDF-SHA256(IKM = 0x11 × 32, salt = 0x22 × 12, info = "intentd-bucket-v1", L = 32)
=> 4a359b8818fc073f6f81a3cb4acd9065302e285fbef0d35d2ff71fba107e6056
Talk to us