ON THIS PAGE

Communicating with Fostrom over MQTT

MQTT is a lightweight messaging protocol that is ideal for IoT applications. It is a publish/subscribe protocol that allows devices to send messages to each other. Fostrom supports MQTT 3.1.1 over TCP/TLS and Secure WebSockets and requires the device to send packets with specific parameters.

The device can send the payload in two serialization formats: JSON and MsgPack.

To use JSON, the CONNECT packet's client_id field must be set to JSON.

To use MsgPack, the CONNECT packet's client_id field must be set to MSGPACK.

Connecting to Fostrom

A TCP/TLS connection can be opened at device.fostrom.dev at port 8883. The MQTT client should then send a specifically crafted CONNECT packet.

The following URL can be used to connect from an MQTT client:

mqtts://device.fostrom.dev:8883

Note that Fostrom does not support any unencrypted connection. You must use TLS.

WebSockets

Fostrom also supports MQTT over Secure WebSockets. The URL for connecting to Fostrom over WebSockets is:

wss://device.fostrom.dev

The WebSockets connection should be opened with the mqtt protocol header:

Sec-WebSocket-Protocol: mqtt

The CONNECT Packet

The MQTT CONNECT packet carries the following fields, and they should be set accordingly:

  • client_id - The client ID usually exists to identify a device, but we do that through the username field. Instead, set the client_id to "MSGPACK" to use MsgPack as the serialization format. Otherwise, set it to "JSON".

  • username - The username follows the following format: <fleet_id>::<device_id>. A Fleet ID is 8 characters long, and a Device ID is 10 characters long. They are separated by two colons ::, essentially creating a 20-character string.

  • password - The Device Secret is used as the password. The Device Secret starts with FOS- with 32 characters after it (totaling 36 characters).

  • keep_alive - The keep alive interval in seconds. If keep_alive is less than 11 seconds, Fostrom assumes the MQTT connection should be terminated after sending a response packet. This is useful for devices that don't need a persistent connection to Fostrom. If you want a persistent connection, set keep_alive to 30 seconds or higher (preferably 30-60 seconds).

  • clean_session - This flag is ignored by Fostrom.

  • will - You cannot set a Will in the connect packet at this time. The connection will fail if a will is specified.

Once you send a connect packet, either a CONNACK (0) -> ACCEPTED approving the connection will be returned, or one of two errors will be returned and the connection will be closed:

  • CONNACK (3) -> SERVER UNAVAILABLE - This is a transient error, the device should retry connecting after a short delay. Devices should use an exponential backoff strategy when trying to reconnect in face of network or server errors.

  • CONNACK (5) -> UNAUTHORIZED - This is a fatal error, retrying with the same credentials will not help. The device should not attempt to reconnect.

Once the device is connected, it can send subsequent packets to Fostrom.

Note that devices should use an exponential backoff strategy when trying to reconnect in face of network or server errors.

CONNECT Packet:

 

-> CONNECT

    client_id: "MSGPACK" | "JSON"

    username: "<fleet_id>::<device_id>" # total 20 characters

    password: "<device_secret>"         # starts with `FOS-`, 36 characters

    keep_alive: 0 | 30 | 60

    clean_session: true

    will: nil



<- CONNACK

    code: 0 | 3 | 5

    # 0 = ACCEPTED

    # 3 = SERVER UNAVAILABLE

    # 5 = UNAUTHORIZED

Heartbeats

Fostrom shows whether a device is connected or not, based on heartbeats and open sockets.

MQTT has a PINGREQ packet that acts as a heartbeat. The Fostrom server responds with a PINGRESP packet. It is normal for devices to connect to Fostrom and send a PINGREQ packet, and then disconnect. This is useful for devices that don't need a persistent connection to Fostrom but to indicate that they are functioning normally.

-> PINGREQ



<- PINGRESP

There is also another way to send a heartbeat by publishing a packet. This exists because many MQTT clients do not support the user manually triggering a PINGREQ packet. If you need manual control over when the heartbeat should be sent, you can use the publish method, as documented below.

Publishing to Fostrom

Fostrom ignores the dup and retain flags in PUBLISH packets. It supports QoS 0 and QoS 1, however we highly recommend using QoS 1 for all packets. There is only one topic a device can publish to: fostrom. The client should send a packet id between 1 and 65535 for QoS 1 packets. Fostrom does not support QoS 2 at this time. The payload should be an object/map, and needs to be serialized with your configured serializer (MsgPack or JSON).

-> PUBLISH

    topic: "fostrom"

    qos: 0 | 1        # we recommend QoS 1

    id: 1-65535       # only for QoS 1

    payload: <msgpacked_payload | json_encoded_payload>



<- PUBACK (for QoS 1 only)

    id: <packet_id>

Heartbeats through Publishing

To send a heartbeat by publishing a packet, the payload should be:

{

  "type": "heartbeat"

}

Send a Datapoint

To send a datapoint, you need to send the following fields in the payload:

{

  "type": "datapoint",

  "name": "<packet-schema-name>",

  "payload": {

    ...fields

  }

}

Send a Message

To send a message, you need to send the following fields in the payload:

{

  "type": "msg",

  "name": "<packet-schema-name>",

  "payload": {

    ...fields

  }

}

Receiving Messages from Fostrom

Step 1: Subscribe to topic fostrom

MQTT mandates that a client should subscribe to a topic before a server can send a PUBLISH packet to that topic. The MQTT spec mandates that a client should terminate the connection if it receives a PUBLISH packet on a topic it has not subscribed to. Since many MQTT clients are written to adhere to the spec, Fostrom requires that a client subscribe to the topic fostrom before Fostrom can send messages to the device.

Once a device sends a SUBSCRIBE packet for topic fostrom, Fostrom will send a SUBACK confirming the subscription.

-> SUBSCRIBE

    topic: "fostrom"

    qos: 1

    id: 1-65535



<- SUBACK

    id: <packet_id>

    code: 1             # 1 = SUCCESS with QoS 1

Step 2: Receive Push Notifications for New Mail

Once subscribed, whenever new mail arrives in the device's mailbox, Fostrom will automatically send a push notification to inform the device. This notification contains:

{

  "new_mail": true

}

The device should send a PUBACK to acknowledge receipt of this notification. This notification is purely informational - the device can then decide when to fetch the actual mail content.

Step 3: Fetch Mail from Mailbox

To retrieve mail from the mailbox, the device must explicitly request it by publishing a fetch request:

{

  "type": "next_mail"

}

Or, to fetch only the mail header without the payload (useful for checking mail metadata before downloading full content):

{

  "type": "next_mail",

  "include_payload": false

}

Fostrom will respond with a PUBLISH packet containing the mail details:

{

  "type": "mail",

  "request_txn_id": <txn_id_from_your_request>,

  "mailbox_size": <current_mailbox_size>,

  "mail": {

    "id": "<mail_id>",

    "name": "<packet-schema-name>",

    "payload": {

      ...fields

    }

  }

}

Note

The mail_id is always a UUIDv7 string with dashes, exactly 36 characters long.

If requesting header-only (with include_payload: false), the response will not include the payload field:

{

  "type": "mail",

  "request_txn_id": <txn_id_from_your_request>,

  "mailbox_size": <current_mailbox_size>,

  "mail": {

    "id": "<mail_id>",

    "name": "<packet-schema-name>"

  }

}

If the mailbox is empty, the response will be:

{

  "type": "mail",

  "request_txn_id": <txn_id_from_your_request>,

  "mailbox_size": 0

}

Flow Summary

 

# Initial subscription

-> SUBSCRIBE

    topic: "fostrom"

    qos: 1

    id: 1-65535



<- SUBACK

    id: <packet_id>

    code: 1



# When new mail arrives (push notification)

<- PUBLISH

    topic: "fostrom"

    qos: 1

    id: 1-65535

    payload: {"mailbox_size": 3, "id": "00000000-0000-7000-8000-000000000000"}



-> PUBACK

    id: <packet_id>



# Device requests next mail

-> PUBLISH

    topic: "fostrom"

    qos: 1

    id: 100

    payload: {"type": "next_mail"}



<- PUBACK

    id: 100



# Server sends mail content

<- PUBLISH

    topic: "fostrom"

    qos: 1

    id: 1-65535

    payload: {

      "type": "mail",

      "request_txn_id": 100,

      "mailbox_size": 2,

      "mail": {

        "id": "00000000-0000-7000-8000-000000000000",

        "name": "command",

        "payload": {...}

      }

    }



-> PUBACK

    id: <packet_id>

Acknowledging, Rejecting or Requeueing Messages

It is crucial for the device to either acknowledge, reject, or requeue any message that it receives from Fostrom. This tells Fostrom how to handle the message in the mailbox. To perform these operations, the device should send a PUBLISH packet on the topic fostrom with the appropriate payload.

Acknowledging Messages

To acknowledge a message (remove it from the mailbox as successfully processed):

{

  "type": "ack_mail",

  "id": "<mail_id>"

}

Rejecting Messages

To reject a message (remove it from the mailbox as unable to process):

{

  "type": "reject_mail",

  "id": "<mail_id>"

}

Requeueing Messages

To requeue a message (put it back in the mailbox for later processing):

{

  "type": "requeue_mail",

  "id": "<mail_id>"

}

All three operations will receive a PUBACK response if successful. If the operation fails (e.g., invalid mail ID), the connection will be closed.

Important Notes

Error Handling

The MQTT Specification mandates immediately closing the connection if a malformed packet is received. Hence, Fostrom will close the connection abruptly if any malformed payload is sent, such as:

  • Incorrectly serialized payloads (invalid JSON or MsgPack)
  • Incorrect packet types or missing required fields
  • Publishing or subscribing to topics other than fostrom
  • Packets larger than 64 KB

Unsubscribe Support

Although not commonly used, the device can send an UNSUBSCRIBE packet for the topic fostrom, at which point Fostrom will send an UNSUBACK packet confirming the removal of the subscription. Once unsubscribed, Fostrom will stop sending any push notifications for new mail arrivals. The device can subscribe again at any time to resume receiving notifications.

Streaming Decoder

Fostrom implements a robust streaming decoder to handle chunked packets and multiple packets concatenated together. Some MQTT clients such as MQTT.js write bytes in small chunks, which works perfectly with Fostrom. This means you don't need to worry about packet fragmentation at the network level.

Transaction IDs

When using QoS 1 (recommended), always use unique transaction IDs between 1 and 65535 for your PUBLISH packets. The server will also use transaction IDs in this range for its PUBLISH packets to you. These IDs help correlate PUBACK responses with their corresponding PUBLISH packets.