9 commits

Author	SHA1	Message	Date
Costa Tsaousis	5f72d4279b	Streaming improvements No 3 (#19168) ... * ML uses synchronous queries * do not call malloc_trim() to free memory, since to locks everything * Reschedule dimensions for training from worker threads. * when we collect or read from the database, it is SAMPLES. When we generate points for a chart is POINTS * keep the receiver send buffer 10x the default * support autoscaling stream circular buffers * nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible * fix last commit * allow removing receiver and senders inline, if the stream thread is not working on them * fix logs * Revert "nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible" This reverts commit 51539a97da. * do not access receiver or sender after it has been removed * open cache hot2clean * open cache hot2clean does not need flushing * use aral for extent pages up to 65k * track aral malloc and mmap allocations separately; add 8192 as a possible value to PGD * do not evict too frequently if not needed * fix aral metrics * fix aral metrics again * accurate accounting of memory for dictionaries, strings, labels and MRG * log during shutdown the progress of dbengine flushing * move metasync shutfown after dbengine * max iterations per I/O events * max iterations per I/O events - break the loop * max iterations per I/O events - break the loop - again * disable inline evictions for all caches * when writing to sockets, send everything that can be sent * cleanup code to trigger evictions * fix calculation of eviction size * fix calculation of eviction size once more * fix calculation of eviction size once more - again * ml and replication stop while backfilling is running * process opcodes while draining the sockets; log with limit when asking to disconnect a node * fix log * ml stops when replication queries are running * report pgd_padding to pulse * aral precise memory accounting * removed all alignas() and fix the 2 issues that resulted in unaligned memory accesses (one in mqtt and another in streaming) * remove the bigger sizes from PGD, but keep multiples of gorilla buffers * exclude judy from sanitizers * use 16 bytes alignment on 32 bit machines * internal check about memory alignment * experiment: do not allow more children to connect while there is backfilling or replication queries running * when the node is initializing, retry in 30 seconds * connector cleanup and isolation of control logic about enabling/disabling various parts * stop also health queries while backfilling is running * tuning * drain the input * improve interactivity when suspending * more interactive stream_control * debug logs to find the connection issue * abstracted everything about stream control * Add ml_host_{start,stop} again. * Do not create/update anomaly-detection charts when ML is not running for a host. * rrdhost flag RECEIVER_DISCONNECTED has been reversed to COLLECTOR_ONLINE and has been used for localhost and virtual hosts too, to have a single point of truth about the availability of collected data or not * ml_host_start() and ml_host_stop() are used by streaming receivers; ml_host_start() is used for localhost and virtual hosts * fixed typo * allow up to 3 backfills at a time * add throttling based on user queries * restore cache line paddings * unify streaming logs to make it easier to grep logs * tuning of stream_control * more logs unification * use mallocz_release_as_much_memory_to_the_system() under extreme conditions * do not rely on the response code of evict_pages() * log the gap of the database every time a node is connected * updated ram requirements --------- Co-authored-by: vkalintiris <vasilis@netdata.cloud>	2024-12-11 18:02:17 +02:00
Costa Tsaousis	9ecf021ec2	Streaming improvements #1 (#19137) ... * prefer tinysleep over yielding the processor * split spinlocks to separate files * rename spinlock initializers * Optimize ML queuing operations. - Allocate 25% of cores for ML. - Split queues by request type. - Accurate stats for queue operations by type. * abstracted circular buffer into a new private structure to enable using it in receiver sending side - no features added yet, only abstracted the existing functionality - not tested yet * completed the abstraction of stream circular buffer * unified list of receivers and senders; opcodes now support both receivers and senders * use strings in pluginsd * stream receivers send data back to the child using the event loop * do not share pgc aral between caches * pgc uses 4 to 256 partitions, by default equal to the number of CPU cores * add forgotten worker job * workers now monitor spinlock contention * stream sender tries to lock the sender, but does not wait for it - it will be handled later * increase the number of web server threads to the number of cpu cores, with a minimum of 6 * use the nowait versions of nd_sock functions * handle EAGAIN properly * add spinlock contention tracing for rw_spinlock * aral lock/unlock contention tracing * allocate the compressed buffer * use 128KiB for aral default page size; limit memory protection to 5GiB * aral uses mmap() for big pages * enrich log messages * renamed telemetry to pulse * unified sender and receiver socket event loops * logging improvements * NETDATA_LOG_STREAM_SENDER logs inbound and outbound traffic * 16k receiver buffer size to improve interactivity * fix NETDATA_LOG_STREAM_SENDER in sender_execute * do not stream ML models for charts and dimensions that have not been exposed * add support for sending QUIT to plugins and waiting for some time for them to quit gracefully * global spinlock contention per function * use an aral per pgc partition; use 8 partitions for PGD * rrdcalc: do not change the frequency of alerts - it uses arbitrary values used during replication, changing permanently the frequency of alerts replication: use 1/3 of the cores or 1 core every 10 nodes (min of the two) pgd: use as many aral partitions as the CPU cores, up to 256 * aral does 1 allocation per page (the structure and the elements together), instead of two * use the evitor thread only when we run out of memory; restore the optimization about prepending or appending clean pages based on their accesses; use the main cache free memory for the other caches, reducing I/O when the main cache has enough room * reduce the number of events per poll() to 10 * aral allocates pages of up to 1MiB; restore processing 100 events per nd_poll() call * drain the sockets while reading * receiver sockets should be non-blocking * add stability detector to aral * increase the receivers send buffer * do not remove the sender or the receiver while we drain the input sockets --------- Co-authored-by: vkalintiris <vasilis@netdata.cloud>	2024-12-09 02:37:44 +02:00
Costa Tsaousis	6b8c6baac2	Balance streaming parents (#18945) ... * recreate the circular buffer from time to time * do not update cloud url if the node id is not updated * remove deadlock and optimize pipe size * removed const * finer control on randomized delays * restore children re-connecting to parents * handle partial pipe reads; sender_commit() now checks if the sender is still connected to avoid bombarding it with data that cannot be sent * added commented code about optimizing the array of pollfds * improve interactivity of sender; code cleanup * do not use the pipe for sending messages, instead use a queue in memory (that can never be full) * fix dictionaries families * do not destroy aral on replication exit - it crashes the senders * support multiple dispatchers and connectors; code cleanup * more cleanup * Add serde support for KMeans models. - Serialization/Deserialization support of KMeans models. - Send/receive ML models between a child/parent. - Fix some rare and old crash reports. - Reduce allocations by a couple thousand per second when training. - Enable ML statistics temporarily which might increase CPU consumption. * fix ml models streaming * up to 10 dispatchers and 2 connectors * experiment: limit the number of receivers to the number of cores - 2 * reworked compression at the receiver to minimize read operations * multi-core receivers * use slot 0 on receivers * use slot 0 on receivers * use half the cores for receivers with a minimum of 4 * cancel receiver threads * use offsets instead of pointers in the compressed buffer; track last reads * fix crash on using freed decompressor; core re-org * fix incorrect job registration * fix send_to_plugin() for SSL * add reason to disconnect message * fix signaling receivers to stop * added --dev option to netdata-installer.sh to prevent it from removing the build directory * Fix serde of double values. NaNs and +/- infinities are encoded as strings. * unused param * reset max cbuffer size when it is recreated * struct receiver_state is now private * 1 dispatcher, 1 connector, 2/3 cores for receivers * all replication requests are served by replication threads - never the dispatcher threads * optimize partitions and cache lines for dbengine cache * fix crash on receiver shutdown * rw spinlock now prioritizes writers * backfill all higher tiers * extent cache to 10% * automatic sizing of replication threads * add more replication threads * configure cache eviction parameters to avoid running in aggressive mode all the time * run evictions and flushes every 100ms * add missing initialization * add missing initialization - again * add evictors for all caches * add dedicated evict thread per cache * destroy the completion * avoid sending too many signals to eviction threads * alternative way to make sure there are data to evict * measure inline cache events * disable inline evictions and flushing for open and extent cache * use a spinlock to avoid sending too many signals * batch evictions are not in steps of pages * fix wanted cache size when there are no clean entries in it * fix wanted cache size when there are no clean entries in it * fix wanted cache size again * adaptive batch evictions; batch evictions first try all partitions * move waste events to waste chart * added evict_traversed * evict is smaller steps * removed obsolete code * disabled inlining of evictions and flushing; added timings for evictions * more detailed timings for evictions * use inline evictors * use aral for gorilla pages of 512 bytes, when they are loaded from disk * use aral for all gorilla page sizes loaded from disk * disable inlining again to test it after the memory optimization * timings for dbengine evictions * added timing names * detailed timings * detailed timings - again * removed timings and restored inline evictions * eviction on release only under critical pressure * cleanup and replication tuning * tune cache size calculation * tune replication threads calculation * make streaming receiver exit * Do not allocate/copy extent data twice. * Build/link mimalloc Just for testing, it will be reverted. * lower memory requirements * Link mimalloc statically * run replication with synchronous queries * added missing worker jobs in sender dispatcher * enable batch evictions in pgc * fix sender-dispatcher workers * set max dispatchers to 2 * increase the default replication threads * log stream_info errors * increase replication threads * log the json text when we fail to parse json response of stream_info * stream info response may come back in multiple steps * print the socket error of stream info * added debug to stream info socket error * loop while content-length is smaller than the payload received * Revert "Link mimalloc statically" This reverts commit c98e482d47. * Revert "Build/link mimalloc" This reverts commit 8aae22a28a. * Remove NEED_PROTOBUF * Use mimalloc * Revert "Use mimalloc" This reverts commit 9a68034786. * Use mimalloc * support 256 bytes gorilla pages, when they are loaded from disk * added os_mem_available() * test memory protection * use protection only on one cache * use the free memory of the main cache in the other caches too * use the free memory of the main cache in the open cache too * Batch gorilla writes by tracking the last written number. In a setup with 200 children, `perf` shows that the worst offender is the gorilla write operation, reporting ~17% overhead. With this change `perf` reports ~4% overhead and netdata's CPU consumption decreased by ~16%. * make buffered_reader_next_line() a couple times faster * flushing open cache * Use re2c for the line splitting pluginsd. Function get's optimized around 3x. We should delete old code and use the re2c for the rest of the functions, but we need to keep the PR size as minimal as possible. Will do in follow up PRs. * use cores - 1 for receivers, use only 1 sender * move sender processing to a separate function * Revert "Batch gorilla writes by tracking the last written number." This reverts commit 2e72a5c56d. * Batch gorilla writes only from writers This reapplies df79be2f01145bd79091a8934d7c80b4b3eb915b and introduces a couple changes to remomove writes from readers. * log information for buffer overflow * fix heap use after free * added comments to the main stream receiver loop * 3 dispatchers * single threaded receiver and sender * code cleanup * de-associate hosts from streaming threads when both the receiver and sender stop, so that each time the threads are re-balanced * fix heap use after free * properly get the slot number of pollfd * fixes * fixes * revert worker changes * reuse streaming threads * backfilling should be synchronous * remove the node last * do not keep a pointer to rellocatable buffer * give to pgc the right page size, not less * restore spreading metrics size across time * use the calculated slots for gorilla pages * accurately track gorilla page size changes * check the sth pointer for validity * code cleanup, files re-org and renames to reflect the new structure of streaming * updated referenced size when the size of a page changes; removed flush spins - fluhses cancelled is a waste event * improve families in netdata statistics * page size histogram per cache * page size histogram per cache queue (hot, dirty, clean) * fix heap after use in pdc.c * rw_spinlocks: when preferring a writer yield so that the writer has the chance to get the lock * do not balloon open and extent caches more than needed (it fragments memory and there is not enough memory for the main cache) * fixed typo * enable trace allocations to work * Skip adding kmeans model when ML dimension has not been created. * PGD is now entirely on ARAL for all types of pages * 2 partitions for PGD * Check for ML queue prior to pushing as well. * merge multiple arals, to avoid wasting memory * significantly less arals; proper calculation of gorilla efficiency * report pgd buffers separately from pgc * aral only for sizes less than 512 bytes * tune aral caches * log the functions using the streaming buffer when concurrent use is detected * aral supporting different pages for collected pages and clean pages - an attempt to minimize fragmentation at high performance * fix misuse of sender thread buffers * select the right buffer, based on the receiver tid * no more rrdpush, renamed to stream * lower aral max page size to 16KiB - in an attempt to lower fragmentation under memory pressure * update opcode handling * automatic sizing of aral limiting its size to 200 items per page or 4 x system pages * tune cache eviction strategy * renamed global statistics to telemetry and split it into multiple files * left over renames of global statistics to telemetry * added heatmap to chart types * note about re-balancing a parents cluster * fix formating * added aral telemetry to find the fragmentation per aral * experiment with a different strategy when making clean pages: always append so that the cache is being constantly rotated; aral telemetry reports utilization instead of fragmentation * aral now takes into account waiting deallocators when it creates new pages * split netdata-conf functions into multiple files; added dbengine use all caches and dbengine out of memory protection settings * tune cache eviction strategy * cache parameters cleanup * rename mem_available to system_memory * Fix variable type. * Add fuzzer for pluginsd line splitter. * use cgroup v1 and v2 to detect memory protection; log on start the detection of memory * fixed typo * added logs about system memory detection * remove debug logs from system memory detection * move the rest of dbengine config to netdata-conf * respect streaming buffer size configured * add workers to pgc eviction threads * renamed worker * fixed flip-flop in size and entries conversions * use aral_by_size when we actually agreegate stats to aral by size * use keyword defintions * move opcode definitions to stream-thread.h * swap struct pollfd slots to make sure all the sockets have an equal chance of being processed * Revert "Add fuzzer for pluginsd line splitter." This reverts commit 454cbcf6e1. * Revert "Use re2c for the line splitting pluginsd." This reverts commit 2b2f9d3887. * stream thread use judy arrays instead of linked lists and pre-allocated arrays * added comment about pfd structure on sender and receiver * fixed logs and made the defaut sender timeout 5 seconds * Spawn ML worker threads based on number of CPUs. * Add statistics for ML allocations/deallocations. * Add host flag to check for pending alert transitions to save Remove precompiled statements Offload processing of alerts in the event loop Queue alert transitions to the metadata event loop to be saved Run metadata checks every 5 seconds * do not block doing socket retries when errno indicates EWOULDBLOCK; insist sending data in send_to_plugin() * Revert "Add host flag to check for pending alert transitions to save" This reverts commit 86ade0e87e. * fix error reasons * Disable ML memory statistics when using mimalloc * add reason when ml cannot acquire the dimension * added ML memory and depending on the DICT_WITH_STATS define, add aral by size too * do not stream ML when the parent does not have ML enabled * nd_poll() to overcome the starvation of poll() and use epoll() under Linux * nd_poll() optimization to minimize the number of system calls * nd_poll() fix * nd_poll() fix again * make glibc release memory to the system when the system is critical in memory * try bigger aral pages, to enable releasing memory back to the system * Queue alert transitions to the metadata event loop (global list not per host) Add host count to check for pending alert transitions to save Remove precompiled statements Offload processing of alerts in the event loop Run metadata checks every 5 seconds * round robin aral allocations * fix aral round robin * ask glibc to release memory when the allocations are aggressive * tinysleep yields the processor instead of waiting * run malloc_trim() more frequently * Add reference count on alarm_entry * selective tinysleep and processor yielding * revert gorilla batch writes * codacy fixes --------- Co-authored-by: vkalintiris <vasilis@netdata.cloud> Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>	2024-12-05 18:17:36 +02:00
Stelios Fragkakis	fc5605da7f	Adjust max possible extent size (#18960) ... * Adjust max possible extent size * Simplify calculation Fix also the have_read_error	2024-11-07 10:52:14 +00:00
thiagoftsm	6bcfc4972b	Fix warnings (#17940)	2024-06-18 12:22:04 +00:00
Costa Tsaousis	fe06e8495f	Windows Support Phase 1 (#17497) ... * abstraction layer for O/S * updates * updates * updates * temp fix for protobuf * emulated waitid() * fix * fix * compatibility layer * fix for idtype * fix for missing includes * fix for missing includes * added missing includes * added missing includes * added missing includes * added missing includes * added missing includes * added missing includes * UUID renamed to ND_UUID to avoid conflict with windows.h * include libnetdata.h always - no conflicts * simplify abstraction headers * fix missing functions * fix missing functions * fix missing functions * fix missing functions * rename MSYS to WINDOWS * moved byteorder.h * structure for an internal windows plugin * 1st windows plugin * working plugin * fix printf * Special case windows for protobuf * remove cygwin, compile both as windows * log windows libraries used * fix cmake * fix protobuf * compilation * updated compilation script * added system.ram * windows uptime * perflib * working perflibdump * minify dump * updates to windows plugins, enable ML * minor compatibility fixes for cygwin and msys * perflib-dump to its own file * perflib now indexes names * improvements to the library; disks module WIP * API for selectively traversing the metrics * first working perflib chart: disk.space * working chart on logical and physical disks * added windows protocols * fix datatypes for loops * tinysleep for native smallest sleep support * remove libuuid dependency on windows * fix uuid functions for macos compilation * fix uuid comparison function * do not overwrite uuid library functions, define them as aliases to our own * fixed uuid_unparse functions * fixed typo * added perflib processor charts * updates for compiling without posix emulation * gather common contexts together * fix includes on linux * perflib-memory * windows mem.available * Update variable names for protobuf * network traffic * add network adapters that have traffic as virtual interfaces * add -pipe to windows compilation * reset or overflow flag is now per dimension * dpc is now counted separately * verified all perflib fields are processed and no text fields are present in the data * more common contexts * fix crash * do not add system.net multiple times * install deps update and shortcut * all threads are now joinable behind the scenes * fix threads cleanup * prepare for abstracting threads API * netdata threads full abstraction from pthreads * more threads abstraction and cleanup * more compatibility changes * fix compiler warnings * add base-devel to packages * removed duplicate base-devel * check for strndup * check headers in quotes * fix linux compilation * fix attribute gnu_printf on macos * fix for threads on macos * mingw64 compatibility * enable compilation on windows clion * added instructions * enable cloud * compatibility fixes * compatibility fixes * compatibility fixes * clion works on windows * support both MSYSTEM=MSYS and MSYSTEM=MINGW64 for configure * cleanup and docs * rename uuid_t to nd_uuid_t to avoid conflict with windows uuid_t * leftovers uuid_t * do not include uuid.h on macos * threads signaled cancellations * do not install v0 dashboard on windows * script to install openssh server on windows * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * update openssh installation script * use cleanup variable instead of pthreads push and pop * replace all calls to netdata_thread_cleanup_push() and netdata_thread_cleanup_pop() with __attribute__((cleanup(...))) * remove left-over freez * make sure there are no locks acquired at thread exit * add missing parameter * stream receivers and senders are now voluntarily cancelled * plugins.d now voluntarily exits its threads * uuid_t may not be aligned to word boundaries - fix the uuid_t functions to work on unaligned objects too. * collectors evloop is now using the new threading cancellation; ml is now not using pthread_cancel; more fixes * eliminate threads cancellability from the code base * fix exit timings and logs; fix uv_threads tags * use SSL_has_pending() only when it is available * do not use SSL_has_pending() * dyncfg files on windows escape collon and pipe characters * fix compilation on older systems * fix compilation on older systems * Create windows installer. The installer will install everything under C:\netdata by default. It will: - Install msys2 at C:\netdata - Install netdata dependencies with pacman - Install the agent itself under C:\netdata\opt You can start the agent by running an MSYS shell with C:\netdata\msys2_shell.cmd and then start the agent normally with: /opt/netdata/usr/sbin/netdata -D There are a more couple things to work on: - Verify publisher. - Install all deps not just libuv & protobuf. - Figure out how we want to auto-start the agent as a service. - Check how to uninstall things. * fixed typo * code cleanup * Create uninstaller --------- Co-authored-by: vkalintiris <vasilis@netdata.cloud>	2024-05-16 13:33:00 +03:00
Costa Tsaousis	f1c26d0e2b	DBENGINE: support ZSTD compression (#17244) ... * extract dbengine compression to separate files * added ZSTD support in dbengine * automatically select best compression * handle decompression errors * eliminate fatals from compression algorithms; fallback to uncompressed pages if compression fails or generates bigger data * have the unit test generate many data files	2024-03-25 12:30:31 +02:00
Costa Tsaousis	00f897a883	Code cleanup (#17237) ... * renames in dbengine * remove leftovers from memory mode save and map * fix docs about 3 tiers by default * split linked-lists, bitmaps and storage-points from libnetdata.h	2024-03-23 22:55:31 +00:00
vkalintiris	115d074a6c	Create a top-level directory to contain source code. (#16896) ... * Move ML under src * Move spwan under src * Move cli/ under src/ * move registry/ under src/ * move streaming/ under src/ * Move claim under src. Update docs * Move database/ under src/ * Move libnetdata/ under src/ * Update references to libnetdata * Fix logsmanagement includes * Update generated script path.	2024-02-01 13:41:44 +02:00

Renamed from database/engine/pdc.c (Browse further)