libwebsockets/READMEs/README.fault-injection.md

lws_fi Fault Injection

Most efforts during development go towards trying to make the system do what it is supposed to do during normal operation.

But to provide reliable quality there's a need to not just test the code paths for normal operation, but also to be able to easily confirm that they act correctly under various fault conditions that may be difficult to arrange at test-time. It's otherwise very easy for error conditions that are low probability to be overlooked and turn out to do the wrong thing, eg, try to clean up things they had not actually initialized, or forget to free things etc.

Code handling the operational failures we want to check may be anywhere, including during early initialization or in user code before lws intialization.

To help with this lws has a LWS_WITH_SYS_FAULT_INJECTION build option that provides a simple but powerful api for targeted fault injection in any lws or user code, and provides a wide range of well-known internal faults inside lws you can trigger from outside.

Fault contexts and faults

The basic idea is objects in the user code can choose to initialize "fault contexts" inside objects, that list named, well-known "faults" that the code supoorts and that the user wants to inject.

Although these "fault contexts" can be embedded in objects directly at object creation time, eg, for lws in the lws_context creation info struct, or the client connection info struct, or Secure Stream info struct, it's usually inconvenient to pass the desired faults directly deep into the code and attach them at creation time. Eg, if you want to cause a fault in a wsi instantiated by a Secure Stream, that is internal lws code one step removed from the Secure Stream object creation making it difficult to arrange.

For that reason, faults have a targeted inheritance scheme using namespace paths, it's usually enough to just list the faults you want at context creation time and they will be filter down to the internal objects you want to target when they are created later.

A fault injection request is made in lws_fi_t objects, specifying the fault name and whether, and how often to inject the fault.

The "fault context" objects lws_fi_ctx_t embedded in the creation info structs are linked-lists of lws_fi_t objects. When Fault Injection is enabled at build-time, the key system objects like the lws_context, lws_vhost, wsi and Secure Stream handles / SSPC handles contain their own lws_fi_ctx_t lists that may have any number of lws_fi_t added to them.

When downstream objects are created, eg, when an lws_context creates a Secure Stream, in addition to using any faults provided directly in the SS info, the lws_context faults are consulted to see if any relate to that streamtype and should be applied.

Although faults can be added to objects at creation, it is far more convenient to just pass a list of faults you want into the lws_context and have the objects later match them using namespacing, described later.

Integrating fault injection conditionals into code in private lws code

A simple query api lws_fi(fi_ctx, "name") is provided that returns 0 if no fault to be injected, or 1 if the fault should be synthesized. If there is no rule matching "name", the answer is always to not inject a fault, ie, returns 0.

Similarly for convenience if FAULT_INJECTION is disabled at build, the lws_fi() call always returns the constant 0.

By default then just enabling Fault Injection at build does not have any impact on code operation since the user must also add the fault injection rules he wants to the objects's Fault Injection context.

Integrating fault injection conditionals into user code with public apis

These public apis query the fault context in a wsi, lws_context, ss handle, or sspc handle (client side of proxy) to find any matching rule, if so they return 1 if the conditions (eg, probability) are met and the fault should be injected.

These allow user code to use the whole Fault Injection system without having to understand anything except the common object like a wsi they want to query and the name of the fault rule they are checking.

FI context owner	Public API
lws_context	int lws_fi_user_context_fi(struct lws_context *ctx, const char *rule)
wsi	int lws_fi_user_wsi_fi(struct lws *wsi, const char *rule)
ss handle	int lws_fi_user_ss_fi(struct lws_ss_handle *h, const char *rule)
sspc handle	int lws_fi_user_sspc_fi(struct lws_sspc_handle *h, const char *rule)

For example, the minimal-http-client user code example contains this in its ESTABLISHED callback

		if (lws_fi_user_wsi_fi(wsi, "user_reject_at_est"))
			return -1;

which can be triggered by running it with

lws-minimal-http-client --fault-injection 'wsi/user_reject_at_est', causing

...
[2021/03/11 13:41:05:2769] U: Connected to 46.105.127.147, http response: 200
[2021/03/11 13:41:05:2776] W: lws_fi: Injecting fault unk->user_reject_at_est
[2021/03/11 13:41:05:2789] E: CLIENT_CONNECTION_ERROR: HS: disallowed at ESTABLISHED
...

When LWS_WITH_SYS_FAULT_INJECTION is disabled, these public apis become preprocessor defines to (0), so the related code is removed by the compiler.

Types of fault injection "when" strategy

The api keeps track of each time the context was asked and uses this information to drive the decision about when to say yes, according to the type of rule

Injection rule type	Description
LWSFI_ALWAYS	Unconditionally inject the fault
LWSFI_DETERMINISTIC	after pre times without the fault, the next count times exhibit the fault`
LWSFI_PROBABILISTIC	exhibit a fault pre percentage of the time
LWSFI_PATTERN	Reference pre bits pointed to by pattern and fault if the bit set, pointing to static array
LWSFI_PATTERN_ALLOC	Reference pre bits pointed to by pattern and fault if the bit set, pointing to allocated array, freed when fault goes out of scope

Probabalistic choices are sourced from a PRNG with a seed set in the context creation info Fault Injection Context. By default the lws helper lws_cmdline_option_handle_builtin() sets this to the time in us, but it can be overridden using --fault-seed <decimal>, and the effective PRNG seed is logged when the commandline options are initially parsed.

Addings Fault Injection Rules to lws_fi_ctx_t

Typically the lws_context is used as the central, toplevel place to define faults. This is done by adding prepared lws_fi_t objects on the stack one by one to the context creation info struct's .fic member, using lws_fi_add(lws_fi_ctx_t *fic, const lws_fi_t *fi);, this will allocate and copy the provided fi into the allocation, and attach it to the lws_fi_ctx_t list.

When the context (or other object using the same scheme) is created, it imports all the faults from the info structure .fic and takes ownership of them, leaving the info .fic empty and ready to go out of scope.

Passing in fault injection rules

A key requirement is that Fault Injection rules must be availble to the code creating an object before the object has been created. This is why the user code prepares a Fault Injection context listing his rules in the creation info struct, rather than waiting for the object to be created and then attach Fault Injection rules... it's too late then to test faults during the creation.

Directly applying fault contexts

You can pass in a Fault Injection context prepared with lws_fi_t added to it when creating the following kinds of objects

Object being created	info struct	Fault injection Context member
lws context	struct lws_context_creation_info	fic
vhost	struct lws_context_creation_info	fic
Secure Stream	struct lws_ss_info	fic
client wsi	struct lws_client_connect_info	fic

However typically the approach is just provide a list of faults at context creation time, and let the objects match and inherit using namespacing, described next.

Using the namespace to target specific instances

Lws objects created by the user can directly have a Fault Injection context attached to them at creation time, so the fault injection objects directly relate to the object.

But in other common scenarios, there is no direct visibility of the object that we want to trigger faults in, it may not exist until some time later. Eg, we want to trigger faults in the listen socket of a vhost. To allow this, the fault names can be structured with a /path/ type namespace so objects created later can inherit faults.

Notice that if you are directly creating the vhost, Secure Stream or wsi, you can directly attach the subrule yourself without the namespacing needed. The namespacing is used when you have access to a higher level object at creation- time, like the lws_context, and it will itself create the object you want to target without your having any direct access to it.

namespace form	effect
**vh=myvhost/**subrule	subrule is inherited by the vhost named "myvhost" when it is created
**vh/**subrule	subrule is inherited by any vhost when it is created
**ss=mystream/**subrule	subrule is inherited by SS of streamtype "mystream" (also covers SSPC / proxy client)
**ss/**subrule	subrule is inherited by all SS of any streamtype (also covers SSPC / proxy client)
**wsi=myname/**subrule	subrule is inherited by client wsi created with info->fi_wsi_name "myname"
**wsi/**subrule	subrule is inherited by any wsi

Namespaces can be combined, for example vh=myvhost/wsi/listenskt will set the listenskt fault on wsi created by the server vhost "myvhost", ie, it will cause the listen socket for the vhost to error out on creation.

In the case of wsi migration when it's the network connection wsi on an h2 connection that is migrated to be SID 1, the attached faults also migrate.

Here is which Fault Injection Contexts each type of object inherits matching Fault Injection rules from:

Object type	Initialized with	Inherit matching faults from
context	struct lws_context_creation_info .fic	-
vhost	struct lws_context_creation_info .fic	context FIC
client wsi	struct lws_client_connect_info .fic	context FIC, vhost FIC
ss / sspc	lws_ss_info_t .fic	context FIC
ss / sspc wsi	-	context FIC, vhost FIC, ss / sspc .fic

Since everything can be reached from the lws_context fault context, directly or by additional inheritence, and that's the most convenient to set from the outside, that's typically the original source of all injected faults.

Integration with minimal examples

All the minimal examples that use the lws_cmdline_option_handle_builtin() api can take an additional --fault-injection "...,..." switch, which automatically parses the comma-separated list in the argument to add faults with the given name to the lws_context. For example,

lws-minimal-http-client --fault-injection "wsi/dnsfail"

will force all wsi dns lookups to fail for that run of the example.

Specifying when to inject the fault

By default, if you just give the name part, if the namespace is absent or matches an object, the fault will be injected every time. It's also possible to make the fault inject itself at a random probability, or in a cyclic pattern, by giving additional information in brackets, eg

Syntax	Used with	Meaning
wsi/thefault	lws_fi()	Inject the fault every time
wsi/thefault(10%)	lws_fi()	Randomly inject the fault at 10% probability
wsi/thefault(.............X.X)	lws_fi()	Inject the fault on the 14th and 16th try, every 16 tries
wsi/thefault2(123..456)	lws_fi_range()	Pick a number between 123 and 456

You must quote the strings containing these symbols, since they may otherwise be interpreted by your shell.

The last example above does not decide whether to inject the fault via lws_fi() like the others. Instead you can use it via lws_fi_range() as part of the fault processing, on a secondary fault injection name. For example you may have a fault myfault you use with lws_fi() to decide when to inject the fault, and then a second, related fault name myfault_delay to allow you to add code to delay the fault action by some random amount of ms within an externally- given range. You can get a pseudo-random number within the externally-given range by calling lws_fi_range() on myfault_delay, and control the whole thing by giving, eg, "myfault(10%),myfault_delay(123..456)"

Well-known fault names in lws

Scope	Namespc	Name	Fault effect
context		ctx_createfail1	Fail context creation immediately at entry
context		ctx_createfail_plugin_init	Fail context creation as if a plugin init failed (if plugins enabled)
context		ctx_createfail_evlib_plugin	Fail context creation due to event lib plugin failed init (if evlib plugins enabled)
context		ctx_createfail_evlib_sel	Fail context creation due to unable to select event lib
context		ctx_createfail_oom_ctx	Fail context creation due to OOM on context object
context		ctx_createfail_privdrop	Fail context creation due to failure dropping privileges
context		ctx_createfail_maxfds	Fail context creation due to unable to determine process fd limit
context		ctx_createfail_oom_fds	Fail context creation due to OOM on fds table
context		ctx_createfail_plat_init	Fail context creation due to platform init failed
context		ctx_createfail_evlib_init	Fail context creation due to event lib init failed
context		ctx_createfail_evlib_pt	Fail context creation due to event lib pt init failed
context		ctx_createfail_sys_vh	Fail context creation due to system vhost creation failed
context		ctx_createfail_sys_vh_init	Fail context creaton due to system vhost init failed
context		ctx_createfail_def_vh	Fail context creation due to default vhost creation failed
context		ctx_createfail_ss_pol1	Fail context creation due to ss policy parse start failed (if policy enabled)
context		ctx_createfail_ss_pol2	Fail context creation due to ss policy parse failed (if policy enabled)
context		ctx_createfail_ss_pol3	Fail context creation due to ss policy set failed (if policy enabled)
context		cache_createfail	Fail lws_cache creation due to OOM
context		cache_lookup_oom	Fail lws_cache lookup due to OOM
vhost	vh	vh_create_oom	Fail vh creation on vh object alloc OOM
vhost	vh	vh_create_oom	Fail vh creation on vh object alloc OOM
vhost	vh	vh_create_pcols_oom	Fail vh creation at protocols alloc OOM
vhost	vh	vh_create_access_log_open_fail	Fail vh creation due to unable to open access log (LWS_WITH_ACCESS_LOG)
vhost	vh	vh_create_ssl_srv	Fail server ssl_ctx init
vhost	vh	vh_create_ssl_cli	Fail client ssl_ctx init
vhost	vh	vh_create_srv_init	Fail server init
vhost	vh	vh_create_protocol_init	Fail late protocol init (for late vhost creation)
srv vhost	vh=xxx/wsi	listenskt	Causes socket() allocation for vhost listen socket to fail
cli wsi	wsi	dnsfail	Sync: getaddrinfo() is not called and a EAI_FAIL return synthesized, Async: request not started and immediate fail synthesized
cli wsi	wsi	sendfail	Attempts to send data on the wsi socket fail
cli wsi	wsi	connfail	Attempts to connect on the wsi socket fail
cli wsi	wsi	createfail	Creating the client wsi itself fails
udp wsi	wsi	udp_rx_loss	Drop UDP RX that was actually received, useful with probabalistic mode
udp wsi	wsi	udp_tx_loss	Drop UDP TX so that it's not actually sent, useful with probabalistic mode
srv ss	ss	ss_srv_vh_fail	Secure Streams Server vhost creation forced to fail
cli ss	ss	ss_no_streamtype_policy	The policy for the streamtype is made to seem as if it is missing
sspc	ss	sspc_fail_on_linkup	Reject the connection to the proxy when we hear it has succeeded, it will provoke endless retries
sspc	ss	sspc_fake_rxparse_disconnect_me	Force client-proxy link parse to seem to ask to be disconnected, it will provoke endless retries
sspc	ss	sspc_fake_rxparse_destroy_me	Force client-proxy link parse to seem to ask to destroy the SS, it will destroy the SS cleanly
sspc	ss	sspc_link_write_fail	Force write on the link to fail, it will provoke endless retries
sspc	ss	sspc_create_oom	Cause the sspc handle allocation to fail as if OOM at creation time
sspc	ss	sspc_fail_metadata_set	Cause the metadata allocation to fail
sspc	ss	sspc_rx_fake_destroy_me	Make it seem that client's user code *rx() returned DESTROY_ME
sspc	ss	sspc_rx_metadata_oom	Cause metadata from proxy allocation to fail
ssproxy	ss	ssproxy_dsh_create_oom	Cause proxy's creation of DSH to fail
ssproxy	ss	ssproxy_dsh_rx_queue_oom	Cause proxy's allocation in the onward SS->P[->C] DSH rx direction to fail as if OOM, this causes the onward connection to disconnect
ssproxy	wsi	ssproxy_client_adopt_oom	Cause proxy to be unable to allocate for new client - proxy link connection object
ssproxy	wsi	ssproxy_client_write_fail	Cause proxy write to client to fail
ssproxy	wsi	sspc_dsh_ss2p_oom	Cause ss->proxy dsh allocation to fail
ssproxy	ss	ssproxy_onward_conn_fail	Act as if proxy onward client connection failed immediately
ssproxy	ss	ssproxy_dsh_c2p_pay_oom	Cause proxy's DSH alloc for C->P payload to fail
ss	ss	ss_create_smd	SMD: ss creation smd registration fail
ss	ss	ss_create_vhost	Server: ss creation acts like no vhost matching typename (only for !vhost)
ss	ss	ss_create_pcol	Server: ss creation acts like no protocol given in policy
ss	ss	ss_srv_vh_fail	Server: ss creation acts like unable to create vhost
ss	ss	ss_create_destroy_me	ss creation acts like CREATING state returned DESTROY_ME
ss	ss	ss_create_no_ts	Static Policy: ss creation acts like no trust store
ss	ss	ss_create_smd_1	SMD: ss creation acts like CONNECTING said DESTROY_ME
ss	ss	ss_create_smd_2	SMD: ss creation acts like CONNECTED said DESTROY_ME
ss	ss	ss_create_conn	Nailed up: ss creation client connection fails with DESTROY_ME
wsi	wsi	timedclose	(see next) Cause wsi to close after some time
wsi	wsi	timedclose_ms	Range of ms for timedclose (eg, "timedclose_ms(10..250)"

Well-known namespace targets

Namespaces can be used to target these more precisely, for example even though we are only passing the faults we want inject at the lws_context, we can use the namespace "paths" to target only the wsis created by other things.

To target wsis from SS-based connections, you can use ss=stream_type_name/, eg for captive portal detection, to have it unable to find its policy entry:

ss=captive_portal_detect/ss_no_streamtype_policy (disables CPD from operating)

...to force it to fail to resolve the server DNS:

ss=captive_portal_detect/wsi/dnsfail (this makes CPD feel there is no internet)

...to target the connection part of the captive portal testing instead:

ss=captive_portal_detect/wsi/connfail (this also makes CPD feel there is no internet)

Well-known internal wsi type names

Wsi created for internal features like Async DNS processing can also be targeted

wsi target	Meaning
wsi=asyncdns/	UDP wsi used by lws Async DNS support to talk to DNS servers
wsi=dhcpc/	UDP wsi used by lws DHCP Client
wsi=ntpclient/	UDP wsi used by lws NTP Client

For example, passing in at lws_context level wsi=asyncdns/udp_tx_loss will force async dns to be unable to resolve anything since its UDP tx is being suppressed.

At client connection creation time, user code can also specify their own names to match on these wsi=xxx/ namespace parts, so the faults only apply to specific wsi they are creating themselves later. This is done by setting the client creation info struct .fi_wsi_name to the string "xxx".