libwebsockets/READMEs/README.cbor-lecp.md

RFC8949 CBOR Stream Parsing and Writing

cmake	LWS_WITH_CBOR, LWS_WITH_CBOR_FLOAT
Header	./include/libwebsockets/lws-lecp.h
api-test	./minimal-examples/api-tests/api-test-lecp/
test app	./test-apps/test-lecp.c -> libwebsockets-test-lecp

LECP is the RFC8949 CBOR stream parsing counterpart to LEJP for JSON.

Features

• Completely immune to input fragmentation, give it any size blocks of CBOR as they become available; 1 byte, or 100K at a time give identical parsing results
• Input chunks discarded as they are parsed, whole CBOR never needed in memory
• Nonrecursive, fixed stack usage of a few dozen bytes
• No heap allocations at all, just requires ~500 byte context usually on caller stack
• Creates callbacks to a user-provided handler as members are parsed out
• No payload size limit, supports huge / endless strings or blobs bigger than system memory
• Collates utf-8 text and blob payloads into a 250-byte chunk buffer for ease of access
• Write apis don't use any heap allocations or recursion either
• Write apis use an explicit context with its own lifecycle, and printf style vaargs including sized blobs, C strings, double, int, unsigned long etc
• Completely immune to output fragmentation, supports huge strings and blobs into small buffers, api returns to indicates unfinished if it needs to be called again to continue; 1 byte or 100K output buffer give same results
• Write apis completely fill available buffer and if unfinished, continues into same or different buffer when called again with same args; no requirement for subsequent calls to be done sequentially or even from same function

Type limits

CBOR allows negative integers of up to 64 bits, these do not fit into a uint64_t. LECP has a union for numbers that includes the types uint64_t and int64_t, but it does not separately handle negative integers. Only -2^63.. 2^64 -1 can be handled by the C types, the oversize negative numbers wrap and should be avoided.

Floating point support

Floats are handled using the IEEE memory format, it means they can be parsed from the CBOR without needing any floating point support in the build. If floating point is available, you can also enable LWS_WITH_CBOR_FLOAT and a float and double types are available in the number item union. Otherwise these are handled as ctx->item.u.u32 and ctx->item.u.u64 union members.

Half-float (16-bit) is defined in CBOR and always handled as a uint16_t number union member ctx->item.u.hf.

Callback reasons

The user callback does not have to handle any callbacks, it only needs to process the data for the ones it is interested in.

Callback reason	CBOR structure	Associated data
LECPCB_CONSTRUCTED	Created the parse context
LECPCB_DESTRUCTED	Destroyed the parse context
LECPCB_COMPLETE	The parsing completed OK
LECPCB_FAILED	The parsing failed
LECPCB_VAL_TRUE	boolean true
LECPCB_VAL_FALSE	boolean false
LECPCB_VAL_NULL	explicit NULL
LECPCB_VAL_NUM_INT	signed integer	ctx->item.u.i64
LECPCB_VAL_STR_START	A UTF-8 string is starting
LECPCB_VAL_STR_CHUNK	The next string chunk	ctx->npos bytes in ctx->buf
LECPCB_VAL_STR_END	The last string chunk	ctx->npos bytes in ctx->buf
LECPCB_ARRAY_START	An array is starting
LECPCB_ARRAY_END	An array has ended
LECPCB_OBJECT_START	A CBOR map is starting
LECPCB_OBJECT_END	A CBOR map has ended
LECPCB_TAG_START	The following data has a tag index	ctx->item.u.u64
LECPCB_TAG_END	The end of the data referenced by the last tag
LECPCB_VAL_NUM_UINT	Unsigned integer	ctx->item.u.u64
LECPCB_VAL_UNDEFINED	CBOR undefined
LECPCB_VAL_FLOAT16	half-float available as host-endian uint16_t	ctx->item.u.hf
LECPCB_VAL_FLOAT32	float (uint32_t if no float support) available	ctx->item.u.f
LECPCB_VAL_FLOAT64	double (uint64_t if no float support) available	ctx->item.u.d
LECPCB_VAL_SIMPLE	CBOR simple	ctx->item.u.u64
LECPCB_VAL_BLOB_START	A binary blob is starting
LECPCB_VAL_BLOB_CHUNK	The next blob chunk	ctx->npos bytes in ctx->buf
LECPCB_VAL_BLOB_END	The last blob chunk	ctx->npos bytes in ctx->buf
LECPCB_ARRAY_ITEM_START	A logical item in an array is starting
LCEPDB_ARRAY_ITEM_END	A logical item in an array has completed

CBOR indeterminite lengths

Indeterminite lengths are supported, but are concealed in the parser as far as possible, the CBOR lengths or its indeterminacy are not exposed in the callback interface at all, just chunks of data that may be the start, the middle, or the end.

Handling CBOR UTF-8 strings and blobs

When a string or blob is parsed, an advisory callback of LECPCB_VAL_STR_START or LECPCB_VAL_BLOB_START occurs first. The _STR_ callbacks indicate the content is a CBOR UTF-8 string, _BLOB_ indicates it is binary data.

Strings or blobs may have indeterminite length, but if so, they are composed of logical chunks which must have known lengths. When the _START callback occurs, the logical length either of the whole string, or of the sub-chunk if indeterminite length, can be found in ctx->item.u.u64.

Payload is collated into ctx->buf[], the valid length is in ctx->npos.

For short strings or blobs where the length is known, the whole payload is delivered in a single LECPCB_VAL_STR_END or LECPCB_VAL_BLOB_END callback.

For payloads larger than the size of ctx->buf[], LECPCB_VAL_STR_CHUNK or LECPCB_VAL_BLOB_CHUNK callbacks occur delivering each sequential bufferload. If the CBOR indicates the total length, the last chunk is delievered in a LECPCB_VAL_STR_END or LECPCB_VAL_BLOB_END.

If the CBOR indicates the string end after the chunk, a zero-length ..._END callback is provided.

Handling CBOR tags

CBOR tags are exposed as LECPCB_TAG_START and LECPCB_TAG_END pairs, at the _START callback the tag index is available in ctx->item.u.u64.

CBOR maps

You can check if you are on the "key" part of a map "key:value" pair using the helper api lecp_parse_map_is_key(ctx).

Parsing paths

LECP maintains a "parsing path" in ctx->path that represents the context of the callback events. As a convenience, at LECP context creation time, you can pass in an array of path strings you want to match on, and have any match checkable in the callback using ctx->path_match, it's 0 if no active match, or the match index from your path array starting from 1 for the first entry.

CBOR element	Representation in path
CBOR Array	[]
CBOR Map	.
CBOR Map entry key string	keystring

Accessing raw CBOR subtrees

Some CBOR usages like COSE require access to selected raw CBOR from the input stream. lecp_parse_report_raw(ctx, on) lets you turn on and off buffering of raw CBOR and reporting it in the parse callback with LECPCB_LITERAL_CBOR callbacks. The callbacks mean the temp buffer ctx->cbor[] has ctx->cbor_pos bytes of raw CBOR available in it. Callbacks are triggered when the buffer fills, or reporting is turned off and the buffer has something in it.

By turning the reporting on and off according to the outer CBOR parsing state, it's possible to get exactly the raw CBOR subtree that's needed.

Capturing and reporting the raw CBOR does not change that the same CBOR is being passed to the parser as usual as well.

Comparison with LEJP (JSON parser)

LECP is based on the same principles as LEJP and shares most of the callbacks. The major differences:

• LEJP value callbacks all appear in ctx->buf[], ie, floating-point is provided to the callback in ascii form like "1.0". CBOR provides a more strict typing system, and the different type values are provided either in ctx->buf[] for blobs or utf-8 text strtings, or the item.u union for converted types, with additional callback reasons specific to each type.

• CBOR "maps" use _OBJECT_START and _END parsing callbacks around the key / value pairs. LEJP has a special callback type PAIR_NAME for the key string / integer, but in LECP these are provided as generic callbacks dependent on type, ie, generic string callbacks or integer ones, and the value part is represented according to whatever comes.

Writing CBOR

CBOR is written into a lws_lec_pctx_t object that has been initialized to point to an output buffer of a specified size, using printf type formatting.

Output is paused if the buffer fills, and the write api may be called again later with the same context object, to resume emitting to the same or different buffer.

This allows bufferloads of encoded CBOR to be produced on demand, it's designed to fit usage in WRITEABLE callbacks and Secure Streams tx() callbacks where the buffer size for one packet is already fixed.

CBOR array and map lengths are deduced from the format string, as is whether to use indeterminite length formatting or not. For indeterminite text or binary strings, a container of < >

Format	Arg(s)	Meaning
123		unsigned literal number
-123		signed literal number
%u	unsigned int	number
%lu	unsigned long int	number
%llu	unsigned long long int	number
%d	signed int	number
%ld	signed long int	number
%lld	signed long long int	number
%f	double	floating point number
123(...)		literal tag and scope
%t(...)	unsigned int	tag and scope
%lt(...)	unsigned long int	tag and scope
%llt(...)	unsigned long long int	tag and scope
[...]		Array (fixed len if ] in same format string)
{...}		Map (fixed len if } in same format string)
<t...>		Container for indeterminite text string frags
<b...>		Container for indeterminite binary string frags
'string'		Literal text of known length
%s	const char *	NUL-terminated string
%.*s	int, const char *	length-specified string
%.*b	int, const uint8_t *	length-specified binary
:		separator between Map items (a:b)
,		separator between Map pairs or array items

Backslash is used as an escape in '...' literal strings, so '\\' represents a string consisting of a single backslash, and '\'' a string consisting of a single single-quote.

For integers, various natural C types are available, but in all cases, the number is represented in CBOR using the smallest valid way based on its value, the long or long-long modifiers just apply to the expected C type in the args.

For floats, the C argument is always expected to be a double type following C type promotion, but again it is represented in CBOR using the smallest valid way based on value, half-floats are used for NaN / Infinity and where possible for values like 0.0 and -1.0.

Examples

Literal ints

	uint8_t buf[128];
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "-1");

Return	LWS_LECPCTX_RET_FINISHED
ctx->used	1
buf[]	20

Dynamic ints

	uint8_t buf[128];
	lws_lec_pctx_t cbw;
	int n = -1; /* could be long */

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%d", n); /* use %ld for long */

Return	LWS_LECPCTX_RET_FINISHED
ctx->used	1
buf[]	20

Maps, arrays and dynamic ints

	...
	int args[3] = { 1, 2, 3 };

	lws_lec_printf(ctx, "{'a':%d,'b':[%d,%d]}", args[0], args[1], args[2]);

Return	LWS_LECPCTX_RET_FINISHED
ctx->used	9
buf[]	A2 61 61 01 61 62 82 02 03

String longer than the buffer

Using %s and the same string as an arg gives same results

	uint8_t buf[16];
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "'A literal string > one buf'");
	/* not required to be in same function context or same buf,
	 * but the string must remain the same */
	lws_lec_setbuf(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "'A literal string > one buf'");

First call

Return	LWS_LECPCTX_RET_AGAIN
ctx->used	16
buf[]	78 1A 41 20 6C 69 74 65 72 61 6C 20 73 74 72 69

Second call

Return	LWS_LECPCTX_RET_FINISHED
ctx->used	12
buf[]	6E 67 20 3E 20 6F 6E 65 20 62 75 66

Binary blob longer than the buffer

	uint8_t buf[16], blob[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 };
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);
	/* not required to be in same function context or same buf,
	 * but the length and blob must remain the same */
	lws_lec_setbuf(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);

First call

Return	LWS_LECPCTX_RET_AGAIN
ctx->used	16
buf[]	52 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

Second call

Return	LWS_LECPCTX_RET_FINISHED
ctx->used	3
buf[]	10 11 12