mirror of
https://libwebsockets.org/repo/libwebsockets
synced 2024-12-12 08:47:25 +00:00
346 lines
16 KiB
Markdown
346 lines
16 KiB
Markdown
## Using CTest with lws
|
|
|
|
### Updating ancient cmake
|
|
|
|
You need a recent cmake to have the CTest tests work properly, if you're on an
|
|
older distro you need to update your cmake. Luckily Kitware provide a repo for
|
|
common distros. These instructions work for bionic and xenial.
|
|
|
|
First remove the old distro cmake and install the pieces needed to get the new repo keys
|
|
|
|
```
|
|
# apt purge --auto-remove cmake
|
|
# apt install gnupg wget apt-transport-https ca-certificates
|
|
# wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -
|
|
# apt edit-sources
|
|
```
|
|
|
|
Add the line `deb https://apt.kitware.com/ubuntu/ bionic main` at the end
|
|
replacing `bionic` with `xenial` as needed, and save (:wq). Then
|
|
|
|
```
|
|
# apt update
|
|
# apt install cmake
|
|
```
|
|
|
|
## Tests live in CMakeLists.txt
|
|
|
|
The rules for tests are described in ctest / cmake language inside the minimal
|
|
examples and api tests that are enabled by current build options, so you need
|
|
to build with `-DLWS_WITH_MINIMAL_EXAMPLES=1` to build the examples along with
|
|
the library.
|
|
|
|
The tests are typically running the examples or api tests and regarding the
|
|
process exiting with exit code 0 as success, anything else as failure.
|
|
|
|
## Generating the tests
|
|
|
|
The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`. You can optionally set
|
|
`-DLWS_CTEST_INTERNET_AVAILABLE=0` to indicate you can't run the tests that need
|
|
internet connectivity.
|
|
|
|
## Preparing to run the tests
|
|
|
|
The tests have to be able to run without root and without disturbing any other
|
|
install of lws in the build machine.
|
|
|
|
For that reason you have to do an unprivileged side-install into `../destdir`,
|
|
using `make install DESTDIR=../destdir` from the build directory and perform the
|
|
tests on the pieces in there.
|
|
|
|
## Running the tests
|
|
|
|
We must take care to run the pieces (.so etc) we just built, without having
|
|
root access, and not any of the same pieces from some other lws version that may
|
|
have been installed on the build machine. That includes, eg, plugins that
|
|
we just built, to ensure precedence of those in the search path we can set our
|
|
DESTDIR unprivileged install path in `LD_LIBRARY_PATH`.
|
|
|
|
Then we can run ctest on the unprivileged install. The whole step looks
|
|
something like this:
|
|
|
|
```
|
|
build $ make -j12 && \
|
|
rm -rf ../destdir && \
|
|
make -j12 DESTDIR=../destdir install && \\
|
|
LD_LIBRARY_PATH=../destdir/usr/local/share/libwebsockets-test-server/plugins ctest -j2 --output-on-failure
|
|
```
|
|
|
|
On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build
|
|
type.
|
|
|
|
Good results look something like this (which tests can run depend on your
|
|
build options)
|
|
|
|
```
|
|
Test project /projects/libwebsockets/build
|
|
Start 71: st_wcs_srv
|
|
Start 43: st_hcp_srv
|
|
1/73 Test #71: st_wcs_srv .................................. Passed 5.01 sec
|
|
Start 19: st_hcmp_srv
|
|
2/73 Test #43: st_hcp_srv .................................. Passed 5.01 sec
|
|
Start 17: st_hcm_srv
|
|
3/73 Test #19: st_hcmp_srv ................................. Passed 5.01 sec
|
|
Start 55: st_ssproxyctx
|
|
4/73 Test #17: st_hcm_srv .................................. Passed 5.01 sec
|
|
Start 52: st_ssproxy
|
|
5/73 Test #55: st_ssproxyctx ............................... Passed 1.02 sec
|
|
Start 67: st_sstfproxy
|
|
6/73 Test #52: st_ssproxy .................................. Passed 1.02 sec
|
|
Start 60: st_ssprxsmd_sspc
|
|
7/73 Test #67: st_sstfproxy ................................ Passed 1.01 sec
|
|
Start 63: st_mulssprxsmd_sspc
|
|
8/73 Test #60: st_ssprxsmd_sspc ............................ Passed 1.01 sec
|
|
Start 69: sspc-minimaltf
|
|
9/73 Test #63: st_mulssprxsmd_sspc ......................... Passed 1.02 sec
|
|
Start 73: ws-client-spam
|
|
10/73 Test #73: ws-client-spam .............................. Passed 12.21 sec
|
|
Start 57: sspc-minimaltx
|
|
11/73 Test #57: sspc-minimaltx .............................. Passed 5.90 sec
|
|
Start 65: mulsspcsmd_sspc
|
|
12/73 Test #65: mulsspcsmd_sspc ............................. Passed 3.58 sec
|
|
Start 62: sspcsmd_sspc
|
|
13/73 Test #62: sspcsmd_sspc ................................ Passed 1.73 sec
|
|
Start 22: http-client-multi-h1
|
|
14/73 Test #22: http-client-multi-h1 ........................ Passed 5.04 sec
|
|
Start 25: http-client-multi-stag
|
|
15/73 Test #25: http-client-multi-stag ...................... Passed 4.53 sec
|
|
Start 26: http-client-multi-stag-h1
|
|
16/73 Test #26: http-client-multi-stag-h1 ................... Passed 4.40 sec
|
|
Start 21: http-client-multi
|
|
17/73 Test #21: http-client-multi ........................... Passed 4.37 sec
|
|
Start 36: http-client-multi-post-h1
|
|
18/73 Test #36: http-client-multi-post-h1 ................... Passed 2.73 sec
|
|
Start 54: sspc-minimal
|
|
19/73 Test #54: sspc-minimal ................................ Passed 0.93 sec
|
|
Start 39: http-client-multi-post-stag
|
|
20/73 Test #39: http-client-multi-post-stag ................. Passed 2.29 sec
|
|
Start 40: http-client-multi-post-stag-h1
|
|
21/73 Test #69: sspc-minimaltf .............................. Passed 49.83 sec
|
|
Start 35: http-client-multi-post
|
|
22/73 Test #40: http-client-multi-post-stag-h1 .............. Passed 4.30 sec
|
|
Start 33: http-client-multi-restrict-nopipe-fail
|
|
23/73 Test #35: http-client-multi-post ...................... Passed 3.23 sec
|
|
Start 28: http-client-multi-stag-h1-pipe
|
|
24/73 Test #33: http-client-multi-restrict-nopipe-fail ...... Passed 2.86 sec
|
|
Start 32: http-client-multi-restrict-stag-h1-pipe
|
|
25/73 Test #28: http-client-multi-stag-h1-pipe .............. Passed 2.86 sec
|
|
Start 27: http-client-multi-stag-pipe
|
|
26/73 Test #32: http-client-multi-restrict-stag-h1-pipe ..... Passed 1.51 sec
|
|
Start 31: http-client-multi-restrict-stag-pipe
|
|
27/73 Test #27: http-client-multi-stag-pipe ................. Passed 1.52 sec
|
|
Start 34: http-client-multi-restrict-h1-nopipe-fail
|
|
28/73 Test #34: http-client-multi-restrict-h1-nopipe-fail ... Passed 2.78 sec
|
|
Start 46: http-client-post-m
|
|
29/73 Test #31: http-client-multi-restrict-stag-pipe ........ Passed 2.80 sec
|
|
Start 42: http-client-multi-post-stag-h1-pipe
|
|
30/73 Test #42: http-client-multi-post-stag-h1-pipe ......... Passed 1.51 sec
|
|
Start 41: http-client-multi-post-stag-pipe
|
|
31/73 Test #46: http-client-post-m .......................... Passed 1.59 sec
|
|
Start 48: http-client-post-m-h1
|
|
32/73 Test #48: http-client-post-m-h1 ....................... Passed 1.10 sec
|
|
Start 23: http-client-multi-pipe
|
|
33/73 Test #41: http-client-multi-post-stag-pipe ............ Passed 1.51 sec
|
|
Start 29: http-client-multi-restrict-pipe
|
|
34/73 Test #23: http-client-multi-pipe ...................... Passed 1.09 sec
|
|
Start 24: http-client-multi-h1-pipe
|
|
35/73 Test #29: http-client-multi-restrict-pipe ............. Passed 0.74 sec
|
|
Start 30: http-client-multi-restrict-h1-pipe
|
|
36/73 Test #24: http-client-multi-h1-pipe ................... Passed 1.14 sec
|
|
Start 45: http-client-post
|
|
37/73 Test #30: http-client-multi-restrict-h1-pipe .......... Passed 1.14 sec
|
|
Start 38: http-client-multi-post-h1-pipe
|
|
38/73 Test #45: http-client-post ............................ Passed 0.30 sec
|
|
Start 37: http-client-multi-post-pipe
|
|
39/73 Test #38: http-client-multi-post-h1-pipe .............. Passed 0.49 sec
|
|
Start 47: http-client-post-h1
|
|
40/73 Test #37: http-client-multi-post-pipe ................. Passed 0.31 sec
|
|
Start 50: hs_evlib_foreign_event
|
|
41/73 Test #47: http-client-post-h1 ......................... Passed 0.29 sec
|
|
Start 66: ss-tf
|
|
42/73 Test #50: hs_evlib_foreign_event ...................... Passed 22.02 sec
|
|
Start 49: hs_evlib_foreign_uv
|
|
43/73 Test #49: hs_evlib_foreign_uv ......................... Passed 21.03 sec
|
|
Start 51: ss-warmcat
|
|
44/73 Test #51: ss-warmcat .................................. Passed 2.69 sec
|
|
Start 59: ss-smd
|
|
45/73 Test #59: ss-smd ...................................... Passed 1.78 sec
|
|
Start 10: api-test-secure-streams
|
|
46/73 Test #10: api-test-secure-streams ..................... Passed 1.34 sec
|
|
Start 11: http-client-warmcat
|
|
47/73 Test #11: http-client-warmcat ......................... Passed 0.27 sec
|
|
Start 58: sspost-warmcat
|
|
48/73 Test #58: sspost-warmcat .............................. Passed 0.84 sec
|
|
Start 12: http-client-warmcat-h1
|
|
49/73 Test #12: http-client-warmcat-h1 ...................... Passed 0.25 sec
|
|
Start 2: api-test-jose
|
|
50/73 Test #2: api-test-jose ............................... Passed 0.27 sec
|
|
Start 70: ws-client-rx-warmcat
|
|
51/73 Test #70: ws-client-rx-warmcat ........................ Passed 0.27 sec
|
|
Start 56: ki_ssproxyctx
|
|
52/73 Test #56: ki_ssproxyctx ............................... Passed 0.12 sec
|
|
Start 68: ki_ssproxy
|
|
53/73 Test #68: ki_ssproxy .................................. Passed 0.11 sec
|
|
Start 64: ki_mulssprxsmd_sspc
|
|
54/73 Test #64: ki_mulssprxsmd_sspc ......................... Passed 0.10 sec
|
|
Start 61: ki_ssprxsmd_sspc
|
|
55/73 Test #61: ki_ssprxsmd_sspc ............................ Passed 0.11 sec
|
|
Start 13: http-client-h2-rxflow-warmcat
|
|
56/73 Test #13: http-client-h2-rxflow-warmcat ............... Passed 0.28 sec
|
|
Start 14: http-client-h2-rxflow-warmcat-h1
|
|
57/73 Test #14: http-client-h2-rxflow-warmcat-h1 ............ Passed 0.34 sec
|
|
Start 16: http-client-hugeurl-warmcat-h1
|
|
58/73 Test #16: http-client-hugeurl-warmcat-h1 .............. Passed 0.16 sec
|
|
Start 15: http-client-hugeurl-warmcat
|
|
59/73 Test #15: http-client-hugeurl-warmcat ................. Passed 0.16 sec
|
|
Start 72: ki_wcs_srv
|
|
60/73 Test #72: ki_wcs_srv .................................. Passed 0.12 sec
|
|
Start 44: ki_hcp_srv
|
|
61/73 Test #44: ki_hcp_srv .................................. Passed 0.11 sec
|
|
Start 20: ki_hcmp_srv
|
|
62/73 Test #20: ki_hcmp_srv ................................. Passed 0.11 sec
|
|
Start 18: ki_hcm_srv
|
|
63/73 Test #18: ki_hcm_srv .................................. Passed 0.11 sec
|
|
Start 7: api-test-lws_struct_sqlite
|
|
64/73 Test #7: api-test-lws_struct_sqlite .................. Passed 0.03 sec
|
|
Start 1: api-test-gencrypto
|
|
65/73 Test #1: api-test-gencrypto .......................... Passed 0.02 sec
|
|
Start 6: api-test-lws_struct-json
|
|
66/73 Test #6: api-test-lws_struct-json .................... Passed 0.01 sec
|
|
Start 4: api-test-lws_dsh
|
|
67/73 Test #4: api-test-lws_dsh ............................ Passed 0.01 sec
|
|
Start 8: api-test-lws_tokenize
|
|
68/73 Test #8: api-test-lws_tokenize ....................... Passed 0.01 sec
|
|
Start 9: api-test-lwsac
|
|
69/73 Test #9: api-test-lwsac .............................. Passed 0.00 sec
|
|
Start 3: api-test-lejp
|
|
70/73 Test #3: api-test-lejp ............................... Passed 0.00 sec
|
|
Start 53: ki_ssproxy
|
|
71/73 Test #53: ki_ssproxy .................................. Passed 0.11 sec
|
|
72/73 Test #66: ss-tf ....................................... Passed 55.51 sec
|
|
Start 5: api-test-lws_smd
|
|
73/73 Test #5: api-test-lws_smd ............................ Passed 4.22 sec
|
|
|
|
100% tests passed, 0 tests failed out of 73
|
|
|
|
Total Test time (real) = 137.76 sec
|
|
```
|
|
|
|
## Considerations for creating tests
|
|
|
|
### Timeout
|
|
|
|
The default test timeout is 1500s, for that reason it's good practice to set
|
|
a more suitable `TIMEOUT` property on every test.
|
|
|
|
### Working Directory
|
|
|
|
Server-side test apps usually need to be run from their `./minimal-examples/...`
|
|
directory so they can access their assets like index.html etc.
|
|
|
|
However when building with `-DLWS_WITH_MBEDTLS=1` then even client-side apps
|
|
need to be run from their directory, since they need to get the trusted CA for
|
|
warmcat.com or libwebsockets.org additionally.
|
|
|
|
For that reason it's good practice to set the `WORKING_DIRECTORY` property to
|
|
the home dir of the example app in all cases.
|
|
|
|
### Spawning Buddies
|
|
|
|
Many networking tests need to either spawn a client or a server in order to
|
|
have a "buddy" to talk to during the test for the opposing side. This is a
|
|
bit awkward in cmake since it does not directly support spawning daemons as
|
|
test dependencies.
|
|
|
|
Lws provides helper scripts for unix type targets in `./scripts/ctest-background.sh`
|
|
and `./scripts/ctest-background-kill.sh`, which spawn background processes,
|
|
save the pid in a decorated /tmp file and can later take the process down. This
|
|
also has arrangements to dump the log of any background process that exited
|
|
early.
|
|
|
|
To arrange the buddy to run aligned with the test, you first explain to cmake
|
|
how to start and stop the buddy using phony tests to make a "fixture" in cmake
|
|
terms.
|
|
|
|
In this example, taken from minimal-http-client-multi, we arrange for
|
|
minimal-http-server-tls to be available for our actual test. The starting and
|
|
stopping definition, for "st_hcm_srv" and "ki_hcm_srv":
|
|
|
|
```
|
|
add_test(NAME st_hcm_srv COMMAND
|
|
${CMAKE_SOURCE_DIR}/scripts/ctest-background.sh
|
|
hcm_srv $<TARGET_FILE:lws-minimal-http-server-tls>
|
|
--port ${PORT_HCM_SRV} )
|
|
add_test(NAME ki_hcm_srv COMMAND
|
|
${CMAKE_SOURCE_DIR}/scripts/ctest-background-kill.sh
|
|
hcm_srv $<TARGET_FILE_NAME:lws-minimal-http-server-tls>
|
|
--port ${PORT_HCM_SRV})
|
|
```
|
|
|
|
... and binding those together so cmake knows they start and stop a specific
|
|
named fixture "hcm_srv", itself with an 800s timeout
|
|
|
|
```
|
|
set_tests_properties(st_hcm_srv PROPERTIES
|
|
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples-lowlevel/http-server/minimal-http-server-tls
|
|
FIXTURES_SETUP hcm_srv
|
|
TIMEOUT 800)
|
|
set_tests_properties(ki_hcm_srv PROPERTIES
|
|
FIXTURES_CLEANUP hcm_srv)
|
|
```
|
|
|
|
... and finally, adding the "hcm_srv" fixture as a requirement on the actual
|
|
test (http-client-multi) we are testing
|
|
|
|
```
|
|
set_tests_properties(http-client-multi
|
|
PROPERTIES
|
|
FIXTURES_REQUIRED "hcm_srv"
|
|
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples-lowlevel/http-client/minimal-http-client-multi
|
|
TIMEOUT 50)
|
|
```
|
|
|
|
Once all that explaining is done, ctest itself will take care about starting
|
|
and killing hcm_srv before and after http-client-multi test.
|
|
|
|
### Buddy sockets and test concurrency
|
|
|
|
For tests with local buddies using tcp sockets inside the same VM or systemd-
|
|
nspawn networking context, you cannot just use a well-known port like 7681.
|
|
|
|
ctest itself is usually executed concurrently, and Sai is typically building
|
|
multiple different instances concurrently as well (typically 3), so it may be
|
|
running different ctests inside the same VM simultaneously.
|
|
|
|
Different tests can have their own convention for port ranges, to solve the
|
|
problem about Sai running different tests concurrently inside one ctest.
|
|
|
|
For the case there are multiple ctests running, we can use the env var
|
|
`$ENV{SAI_INSTANCE_IDX}`, which is an ordinal like 0 or 1, to further ensure
|
|
that port selections won't conflict. If not using Sai, you can just set this
|
|
in the evironment yourself to reflect your build instance index.
|
|
|
|
```
|
|
#
|
|
# instantiate the server per sai builder instance, they are running in the same
|
|
# machine context in parallel so they can tread on each other otherwise
|
|
#
|
|
set(PORT_HCM_SRV "7670")
|
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "0")
|
|
set(PORT_HCM_SRV 7671)
|
|
endif()
|
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "1")
|
|
set(PORT_HCM_SRV 7672)
|
|
endif()
|
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "2")
|
|
set(PORT_HCM_SRV 7673)
|
|
endif()
|
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "3")
|
|
set(PORT_HCM_SRV 7674)
|
|
endif()
|
|
```
|
|
|
|
This is complicated enough that the best approach is copy an existing simple
|
|
case like the CMakeLists.txt for minimal-http-client and change the names and
|
|
ports to be unique.
|