ucLog wire protocol — version 1¶

Status: Phase 0 draft. This is the normative specification for every ucLog implementation: firmware, the native hub/GUI/TUI (Rust), the web and VS Code versions (TypeScript), and the Python library. Decisions referenced as D# are in docs/plans/README.md §1.

• Executable reference: reference/ucref.py, Python standard library only.
• Test vectors: vectors/, generated by reference/gen_vectors.py and checked by reference/check_vectors.py.

Where this text and the reference disagree, that is a bug: raise it.

Conventions

• All integers are little-endian unless stated otherwise.
• u32/i32 are 4 bytes; u64/i64 are 8 bytes.
• Hex bytes are written as 0x.. or as space-separated pairs.
• "Device" is the firmware. "Host" is whatever is directly attached to it (a client or the hub).
• MUST/SHOULD/MAY are used as in RFC 2119.

1. Layers¶

From the top (what the hosts read) to the bottom (the wire):

Every link carries the same byte stream. A hub forwards whole frames unchanged (§8).

2. Framing¶

2.1 Wire encoding¶

A frame carries one payload (0 to 1444 bytes; §2.4). On the wire it is:

00  COBS(payload)  00

• COBS is Cheshire & Baker's Consistent Overhead Byte Stuffing. The encoded form contains no 00 bytes.
• If the last block is a full 0xFF block that ends exactly at the end of the payload, it is not followed by a 0x01 code byte. The decoder infers the implicit trailing zero. See vectors cobs.json "254 no zero" and "508 no zero".
• The leading 00 MUST always be sent. It resynchronises the receiver, and it is the UART wake byte (§2.3).
• The trailing 00 ends the frame. Back-to-back frames therefore produce … 00 00 ….

2.2 Receiving (frame scanner)¶

• Split the byte stream on 00. Each non-empty run between delimiters is one encoded frame.
• Empty runs are ignored. They are normal: they are the gap between frames, and pulses.
• Bytes received before the first 00 after a link opens are dropped. They belong to a frame whose start was missed.
• Maximum encoded frame size: 1450 bytes between the delimiters (§2.4). A longer run is discarded up to the next 00 and reported as a framing error (§7). Receivers MAY accept longer runs (--max-frame), but senders MUST NOT send them.
• A run that is not valid COBS is discarded and reported as a framing error (§7). Invalid means: a code byte runs past the end of the run.
• An encoded frame 01 decodes to an empty payload, which is a pulse (§2.3). It is not an error.

2.3 Pulses, liveness and UART wake¶

• Pulse: a lone 00, or 00 01 00 (an empty payload). Both forms MUST be accepted. Senders SHOULD use the lone 00.
• Sending: every sender (device, host, hub, client) MUST send something at least every 0.5 s while connected. When nothing else is pending, that is a pulse.
• Liveness: a peer is alive while bytes arrive. No bytes for more than 2 s means the peer is not responding. That covers UART unplug and device reset, which are otherwise invisible to the host. Hosts show "device not responding" and keep the link open. Hub ↔ client links MAY use a longer client timeout (hub default 5 s) to tolerate network jitter.
• UART wake (device behaviour): after 2 s without receiving a pulse, the device MAY power down UART RX and wake on the next RX edge. The first byte(s) after such an idle are therefore sacrificial. Senders handle this as follows:
  • Every frame starts with 00, so losing it is harmless.
  • After any TX idle longer than 2 s (including a freshly opened link), a sender MUST send a pulse and wait ≥ 50 ms before the first real frame. USB-UART converters buffer data, so the wake edge can reach the target well after the host wrote it.
  • The hub pulses the device continuously (D6), so the device stays awake while a hub runs.
• Pulses received by a hub from its client are dropped (not forwarded to the device). The hub pulses the device itself. Pulses received from the device MAY be forwarded to the client; at minimum, the hub MUST send the client a pulse every 0.5 s while the device is alive.

2.4 Size limit¶

A frame on the wire, both 00 delimiters included, is at most 1452 bytes. That is the UDP payload of an unfragmented datagram on a 1500-byte Ethernet MTU with IPv6 (1500 − 40 − 8); IPv4 leaves more room. One frame therefore always fits in one datagram (§4.4), whatever the link.

The legacy device code used a 1500-byte port buffer (about one Ethernet MTU); it now uses 1443. Senders MUST NOT exceed these sizes on any link, so that a hub can forward any frame to a UDP client. Port protocols size their messages accordingly (semihost READ §5.5.2, FW update block-size §5.4).

3. Frame header¶

3.1 Payload byte 0¶

b0 = payload[0]
type = b0 & 3          0 = log, 1 = memory log, 2 = reserved, 3 = port frame
port = b0 >> 2         (type 3 only) 0..63

The target digit of a log frame is (addr >> 20) & 0xF. It selects the logdata (§6) when a device has several cores or images that log over one link.

A type-0/1 payload shorter than 4 bytes is an error (§7).

3.2 Two-byte peek (hub)¶

The hub routes frames without decoding them. Given an encoded frame e (no delimiters):

e[1] is the first payload byte in every case except the two above. With b0, compute type/port as in §3.1. The peek reads at most e[0..2]. Vectors: peek_mux.json (peek_only for the one-byte runs).

3.3 Log frames¶

• addr & ~3 is the key into the logdata format table (§6.1).

• args holds the format arguments, packed with no alignment, in the order of the format's argument-kind list (§6.1):

  Kind	Bytes	Notes
  int32 / uint32 / pointer / sym / enum	4
  int64 / uint64 / double	8
  string	UTF-8 + 00 terminator
  bytes	the rest of the frame	the last kind of the list
  - Type 1: ptr (u32) then raw bytes (§6.2). Its kinds list is not used.

4. Links¶

4.1 Serial (UART bridge or USB CDC-ACM)¶

• 1 000 000 baud, 8N1, no flow control.

• Discovery filter (D1); users can override it, and pick from a list when there is more than one match:

  Family	VID:PID
  STLINK	0483:374B 0483:374E 0483:374F 0483:3752 0483:3753 0483:3754
  FTDI	0403:6001 0403:6010 0403:6011 0403:6014 0403:6015
  Zephyr CDC-ACM	2FE3:*
  - STLINK workaround: when a single write has len % 8 == 0, write len-1 bytes, wait at
  least 10 ms, then write the last byte. This is needed on older STLINK firmware. Keep it until it
  has been re-tested on current firmware. Vectors: links.json → stlink.

4.2 BLE¶

• Service and characteristic UUID 8683b8f3-0000-1000-8000-00805f9b34fb. Scan filter = the service UUID.
• Device → host: notifications. Their concatenated values form the byte stream.
• Host → device: writes to the characteristic. Any chunking is allowed (D18). Hosts write the largest chunk the platform allows (ATT MTU − 3, or long writes). No user setting.

4.3 HID¶

• VID 0x2FE3, interface 1. The firmware's report descriptor MUST use a vendor-defined usage page, otherwise WebHID refuses the device.
• Device → host: input report ID 0xA1, 63 data bytes. Hosts use interrupt IN reads on every OS. Polling GET_REPORT is only a fallback.
• Host → device: output report ID 0xA2, 63 data bytes. A byte stream is cut into 63-byte chunks, and the last chunk is zero padded. The padding bytes are 00 delimiters and therefore harmless (§2.2).
• Vectors: links.json → hid_out, hid_in_example.

4.4 UDP¶

• Each datagram carries one or more whole frames, each 00-delimited. There is no reassembly across datagrams. A datagram is at most 1452 bytes (§2.4); senders pack frames up to that size.
• No fragmentation. Senders set Don't Fragment (IPv4 DF; for IPv6, IPV6_DONTFRAG so the sending host does not fragment either). If a datagram is refused as too large (EMSGSIZE, ICMP "Fragmentation Needed" or ICMPv6 "Packet Too Big"), the sender reports an error and does not retry with smaller datagrams. UDP is not supported on paths with an MTU below 1500 bytes (VPNs, some tunnels and PPPoE links); use TCP or WebSocket to a hub there.
• Used for network devices (device side) and for a hub's optional UDP client (D12).

4.5 Hub links (TCP, WebSocket)¶

These carry exactly the §2 byte stream. Admission and session rules are in §8.

• TCP: a raw byte stream. A raw client's first byte MUST be 00, which lets the hub tell it apart from TLS (0x16) and WebSocket (G) on the same port. The hub closes a connection whose first byte is anything else, so that a web page's HTTP POST cannot reach the device.
• WebSocket: binary messages. Each message carries one or more whole frames. Text messages are a protocol error.

5. Ports¶

5.1 Port table¶

Ports not listed are reserved. Receivers show the first frame on an unhandled port as an error record, then only count further frames (§7).

5.2 Port 62 — device info¶

A CBOR map, sent by the device at boot, when a host returns after more than 2 s of silence (§2.3), and on request. Keys:

Every port-62 frame starts a new decoding context: the device may have reset or changed image. Hosts re-resolve logdata when app_hash changes. Vectors: device_info.json.

Request (F7). A host MAY send an empty port-62 frame (00 02 FB 00 on the wire); the device answers with its device info. A host that returns within 2 s (a hub restarted, or a client reconnecting directly) gets no device info otherwise, so it cannot select logdata.

• A hub sends the request when it opens a device, after the wake gap (§2.3). The reply is handled like any other port-62 frame: it is cached and goes into the backlog.
• A direct client (serial, HID, BLE, UDP) sends it on connect. A client connected to a hub does not; the hub replays the device info (§8).
• The device sends the device info for every request, even if it has just sent it. Hosts ignore a repeated device info with the same app_hash.
• A non-empty port-62 payload from the host is reserved; devices ignore it.
• Firmware without F7 ignores the request, or logs "no handler for port 62". The host carries on without the device info (-l selects the logdata).

5.3 Port 0 — commands¶

• Host → device: a CBOR map with at least cmd (text), plus arguments; or a text line ending in \r.
• Device → host: CBOR replies (rsp = the command name) or text.
• Command names are application-defined. Clients provide configurable buttons; there is no fixed list.

5.4 Ports 59 and 3 — FW update¶

Messages (CBOR maps), host → device:

• The device chooses block-size so that every frame stays within §2.4. A block-size of at most 1024 does so on both ports (port 3 adds inner COBS, CRC and delimiters).
• Messages with an event key are progress information and are ignored by the state machine.
• A reply with an error key fails the update; the error text is shown.
• The image is the signed .bin file, sent from offset 0 in block-size chunks; the last chunk may be shorter.
• Integrity (D22): the image carries an RFC 8032 (Ed25519) signature in its header, which the device verifies before it accepts the update. A corrupted transfer therefore fails at fw-schedule (or at boot). It can never be accepted. That is why port 59 has no CRC.

Encoding:

Port selection:

1. Send set-time on port 59. If a valid reply arrives within 1 s, use port 59 for the whole update.
2. Otherwise send set-time on port 3. If a valid reply arrives, use port 3.
3. Otherwise fail with "device does not support FW update".

Hosts MAY cache the chosen port per device serial number for the session. Vectors: fw_update.json.

5.5 Port 63 — semihost syscalls¶

The op set is the ucLog semihost protocol implemented by the legacy host (uclog/protocol/semihost.py in uclog-client), plus the time ops in §5.5.3. It is not ARM semihosting: the opcode numbers differ (for example, legacy op 2 is ISATTY, while ARM 0x02 is SYS_CLOSE). Moving to ARM semihosting is a later, separate transition (D32). It will need its own op set or port, because the numbers clash.

5.5.1 General¶

• Request (device → host): u8 op ‖ arguments.
• Reply (host → device): op-specific bytes, on port 63.
• One request is outstanding at a time. The device waits for the reply.
• Every request MUST get exactly one reply. The one exception is WRITE with bit 31 of the fid set. A host that cannot service a request replies with an error (§5.5.4); it never stays silent. If a client disconnects with a request outstanding, the hub replies EIO on its behalf.

5.5.2 File and stdio ops¶

All replies start with i32 r, i32 errno. On success, errno = 0. File system access is sandboxed to a host-chosen directory (D15, §5.5.5).

• fids are host-side handles in a per-session table. Host file descriptors are never exposed. 0/1/2 are stdin/stdout/stderr.

• OPEN flags use the newlib values:

  Flag	Value	Flag	Value
  O_RDONLY	0	O_EXCL	0x800
  O_WRONLY	1	O_SYNC	0x2000
  O_RDWR	2	O_NONBLOCK	0x4000
  O_APPEND	8	O_NOCTTY	0x8000
  O_CREAT	0x200	O_CLOEXEC	0x40000
  O_TRUNC	0x400	O_NOFOLLOW	0x100000
  		O_DIRECTORY	0x200000

  Flags the host OS lacks are ignored. - stdio:

  Implementation	stdout/stderr (WRITE 1/2)	stdin (READ 0)
  Interactive client	shown in the terminal view	waits for a line typed in the terminal (a trailing \r becomes \n)
  Hub --syscalls	discarded; reply with the full length	returns 0 bytes (EOF)
  No stdin available	—	returns 0 bytes

5.5.3 Time ops (D21, D29, D31)¶

These ops take no arguments. They always succeed and never set errno, so the reply is just the value:

The counter sources are:

When a hub is in the path, it answers all five time ops itself (D31), even with a client connected. They are never forwarded. Vectors: semihost.json → time_replies.

5.5.4 Errors¶

• Error reply: i32 -1, i32 errno, using newlib errno numbers (D25), regardless of the host OS. Common values: ENOENT 2, EIO 5, EBADF 9, EACCES 13, EEXIST 17, EINVAL 22, ENOSPC 28, ENOSYS 88, ENOTEMPTY 90, ENAMETOOLONG 91.
• Fixed-size replies are zero padded: an error reply for FSTAT is 48 bytes (-1, errno, then zeros), and for READ it is the 8 bytes with no data.
• Unknown ops reply with the 8-byte -1, ENOSYS. A malformed request (wrong length) replies -1, EINVAL.
• Changes from the legacy Python host:
  • Errors used the host's errno numbering; they now use newlib's.
  • READ and WRITE errors returned r = 0; they now return -1.
  • Unknown and unimplemented ops got no reply; they now get ENOSYS.
  • The fid mask was 27 bits; it is now 31 bits.

5.5.5 Who answers, and the sandbox¶

Sandbox. Every path resolves against the host's files directory. / and \ both separate components.

• absolute paths (including C:… and \\?\…), .. that escapes, and symlinks or Windows junctions that resolve outside → EACCES;
• names a Windows host could not store as plain files → EACCES on every host, so a device sees the same result wherever it runs:
  • names containing : (drive letters, alternate data streams);
  • the Windows device names CON, PRN, AUX, NUL, COM0–COM9, LPT0–LPT9 (also with ¹²³), CONIN$ and CONOUT$, in any case and with any extension (nul.txt);
  • names ending in . or a space;
  • names containing control characters (0x00–0x1F) or <>"|?*;
• no files directory configured → EACCES for OPEN and MKDIR (the handler implements file ops, but has no directory to work in).

ENOSYS is only for ops the answering handler doesn't implement: the unsupported ops in §5.5.2, and OPEN and MKDIR where the table above says ENOSYS (a hub's default handler).

5.6 Port 60 — plots¶

CBOR maps from the device. The first key that is present determines the message:

• Data for a plot that does not exist yet is counted and dropped. This happens when the host connected after the plot was created.
• The device library builds the line-plot messages with log_plot.h (log_plot_create, log_plot_point, log_plot_data, log_plot_clear); heatmaps and spectrograms are hand-built CBOR for now.
• yprops/*props are passed to the renderer as hints; unknown props are ignored.

5.7 Port 61 — terminal¶

• Device → host: UTF-8 with VT100/ANSI escape sequences.
• Host → device: keystrokes, or lines ending in \r.

6. Logdata¶

6.1 .logdata file¶

A CBOR map. The ELF form carries the same bytes in the section .logdata_cbor.

Each fmts entry is one of:

• [level, file, line, fmt, kinds], where:
  • level is text "0"–"5": DEBUG, INFO, WARN, ERROR, FATAL, PANIC;
  • file and line are text;
  • fmt is the format string (§6.2);
  • kinds lists the argument kinds: "int32", "uint32", "int64", "uint64", "double", "pointer", "string", "sym", "bytes" (the rest of the frame, so only as the last kind), or ["enum", name]. Any other kind, or an enum entry that is not a two-element array, makes the entry undecodable: the log becomes an error record (§7).
• [level, file, line]: no format string. Such an entry cannot be decoded and is reported as an error.

6.2 Formatting¶

fmt is already rewritten by the logdata generator:

Argument sizes always come from kinds, never from the conversion. For example, %lu may carry a uint64.

Formatting rules:

• Conversions: %[flags][width][.prec][length]conv.
  • flags: #+- 0
  • length modifiers (hh h l ll z j t L) are accepted and ignored
  • conv: d i o u x X c f F e E g G a A s %
• Semantics are C printf, with values taken at the kind's width. %c prints the code point.
• Exact parity with Python % or any particular libc is not required for corner cases. The logs.json vectors MUST match exactly.
• Type 1 (memory):
  • text is "{fmt} {ptr:08x}: " + bytes as lowercase hex pairs separated by spaces;
  • if line == "<zephyr>", text is the bytes as UTF-8 and the line is empty.

6.3 Argument rendering¶

6.4 Selecting logdata (app_hash)¶

Resolution order on each port-62 app_hash:

1. A file chosen manually for this device.
2. Local cache <cache>/<app_hash hex>.logdata.
3. GET {logdata.url}/logdata/{domain}/{app_hash hex}, with an optional Bearer token (README §5.3).
4. A sibling or workspace image whose SHA-512 after its 512-byte header equals app_hash.

Logs for a target digit with no logdata are shown as errors (§7) with the raw bytes.

7. Error records¶

Every problem MUST become a visible record (level ERROR, source <uclog>) with a per-kind counter. Nothing is silently dropped. Suggested texts:

The kind names are the ones used in logs.json.

8. Hub session (client view)¶

The hub is specified in docs/plans/01-hub-server.md. This section covers what a client sees.

• One client at a time. A second client is turned away:

  Link	How
  TCP	closed before any data
  WebSocket	close code 4001 ("busy")
  UDP	ignored
  - Approval. A connection from a non-loopback address (or a browser Origin not on the allow
  list) may be held open with no data while the hub's operator approves it, for up to 60 s.
  Clients show "waiting for hub approval".
  - On accept, the hub sends, in order:
  1. the port-62 frame in effect at the oldest buffered frame (if any);
  2. the buffered frames (backlog), oldest first;
  3. one pulse (00), marking the end of the backlog;
  4. live data.

  Port-62 frames inside the backlog mark resets and image changes at the right place (§5.2): a client decodes the logs that follow each one with that image's logdata, even when a later port-62 frame arrives before that logdata has been resolved. Since every frame carries both delimiters, two frames are separated by exactly two 00s: a run of three or more (two or more at the start of the stream) contains the pulse that ends the backlog. Clients that do not skip the backlog can ignore it like any pulse. - After that pulse, everything the hub sends comes from the device: its frames, and pulses, which the hub sends only while the device is alive (§2.3). A hub that is still opening the device (a BLE connection takes several seconds) sends nothing more. The first byte after the end-of-backlog pulse therefore shows that the device is open and talking; clients that wait for the device wait for that byte, not for the replay. - The hub asks for the device info when it opens the device (§5.2, F7), so clients need not. A client MAY still send the request; it is forwarded to the device. - Port-63 time ops are answered by the hub. The client sees only the other syscalls (§5.5.5). - Discovery: - mDNS _uclog._tcp.local. (and _uclog._udp), instance "{hostname}-{interface}", TXT keys serial_number description interface kind proto=1 ws tls busy; - local hubs also write a registry file (hub plan §7.1).

9. Changes from the legacy protocol (uclog-client v0)¶

• New port 59 (FW update without inner COBS/CRC). Port 3 is deprecated.
• New semihost time ops 0x10 0x11 0x30 0x31. A hub always answers time ops.
• Semihost errors use newlib errno. Every request gets a reply (ENOSYS for unknown or unimplemented ops). The WRITE fid mask is 31 bits.
• The hub carries only the raw stream: no per-port TCP servers and no decoded CBOR stream.
• Every frame starts with 00, and senders respect the ≥ 50 ms wake gap after more than 2 s idle.
• Frames are limited to 1452 bytes on the wire (§2.4; legacy: a 1500-byte port buffer, and hosts accepted anything). UDP datagrams are sent with Don't Fragment.
• The string argument escaping (§6.3) replaces Python repr().