* protection against extreme cardinality
* do not cleanup everything, but only what is required to reach 50%
* check for collection flags and settings to prevent deletion
* 1000 to 10000
* 1000 old instances
* log extreme cardinality cleanup
* add configuration for extreme cardinality
* add documentation about extreme cardinality protection
* allow retention to be configured in durations less than a day
* min db size 32MiB
* extreme cardinality working
* fixed cleanup
* check for extreme cardinality on rotation, not on startup
* added option to enable/disable extreme cardinality protection
* enable it by default
* make unknown UUID errors and debug log
* extreme cardinality logs more information about the current status
* enable protection conditions
* never delete more than requested
* evict, a page at a time
* 4 replication ahead requests per replication thread
* added per job average timings for workers and dbengine query router
* debug statement to find what is slow
* yield the processor to avoid monopolizing the cache
* test more page sizes in aral
* more polite journal v2 indexing
* pulse macros for atomics
* added profile so that users can control the defaults of the agent
* fix windows warnings; journal v2 generation yields the processor for every page
* replication threads are 1/3 of the cores and they are synchronous
* removed the default from the list of profiles
* turn pgc locks into macros to have tracing on the functions that called them
* log the size of madvise() when failing
* more work on profiles
* restore batch cache evictions, but lower the batch size significantly
* do not spin while searching for pages in the cache - handle currently being deleted pages within the search logic itself
* remove bottleneck in epdl processing while merging extents
* allocate outside the lock
* rw spinlock implemented without spinlocks; both spinlocks and r/w spinlocks now support exponential backoff while waiting
* apply max sleep to spinlocks
* tune replication
* r/w spinlock prefers writers again, but this time recursive readers are bypassing the writer wait
* tuning of rw spinlock
* more tuning of the rw spinlock
* configure glibc arenas based on profile
* moving global variables into nd_profile
* do not accept sockets that have not received any data; once sockets with data have been accepted, check they are not closed already before processing them
* poll_events is now using nd_poll(), resulting in vast simplification of the code; static web files are now served inline resulting in another simplification of the web server logic (was required because epoll does not support normal files)
* startup fixes
* added debug info to poll_fd_close()
* closed sockets are automatically removed from epoll(), by the kernel
* fix for mrg acquired and referenced going negative
* fixed bug in mrg cleanup, not deleting metrics that do not have retention
* monitor strings index size
* strings memory chart is now stacked
* replication: do not lock data collection when running in batches
* explicitly set socket flags for sender and receiver
* normalize the event loop for sending data (receiver and sender)
* normalize the event loop for receiving data (receiver and sender)
* check all sender nodes every half a second
* fix bug on sender, not enabling sending
* first cleanup then destroy
* normalize nd_poll() to handle all possible events
* cleanup
* normalize socket helper functions
* fixed warnings on alpine
* fix for POLLRDHUP missing
* fix cleanup on shutdown
* added detailed replication summary
* moved logs to INFO
* prevent crash when sender is not there
* madvise _dontfork() should not be used with aral, madvise_dontdump() is only used for file backed maps
* fix wording
* fix log wording
* split replication receiver and sender; add logs to find missing replication requests
* fix compilation
* fixed bug in backfilling, having garbage for counters - malloc instead of calloc
* backfilling logs if it misses callbacks
* log replication rcv and replication snd in node info
* remove contention from aral_page_free_lock(), but having 2 free lists per page, one for incoming and another for available items and moving incoming to available when the available is empty - this allows aral_mallocz() and aral_freez() to operate concurrently on the same page
* fix internal checks
* log errors for all replication receiver exceptions
* removed wrong error log
* prevent health crashing
* cleanup logs that are irrelevant with the missing replication events
* replication tracking: added replication tracking to figure out how replication missed requests
* fix compilation and fix bug on spawn server cleanup calling uv_shutdown at exit
* merged receiver initialization
* prevent compilation warnings
* fix race condition in nd_poll() returning events for deleted fds
* for user queries, prepare as many queries as half the processors
* fix log
* add option dont_dump to netdata_mmap and aral_create
* add logging missing receiver and sender charts
* reviewed judy memory accounting; adbstracted flags handling to ensure they all work the same way; introduced atomic_flags_set_and_clear() to set and clear atomic flags with a single atomic operation
* improvement(go.d/nats): add server_id label (#19280)
* Regenerate integrations docs (#19281)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-30-nightly.
* docs: improve on-prem troubleshooting readability (#19279)
* docs: improve on-prem troubleshooting readability
* Apply suggestions from code review
---------
Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>
* improvement(go.d/nats): add leafz metrics (#19282)
* Regenerate integrations docs (#19283)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-34-nightly.
* fix go.d/nats tests (#19284)
* improvement(go.d/nats): add basic jetstream metrics (#19285)
* Regenerate integrations docs (#19286)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-38-nightly.
* bump dag req jinja version (#19287)
* more strict control on replication counters
* do not flush the log files - to cope with the rate
* [ci skip] Update changelog and version for nightly build: v2.1.0-40-nightly.
* fix aral on windows
* add waiting queue to sender commit, to allow the streaming thread go fast and put replication threads in order
* use the receiver tid
* fix(netdata-updater.sh): remove commit_check_file directory (#19288)
* receiver now has periodic checks too (like the senders have)
* fixed logs
* replication periodic checks: resending of chart definitions
* strict checking on rrdhost state id
* replication periodic checks: added for receivers
* shorter replication status messages
* do not log about ieee754
* receiver logs replication traffic without RSET
* object state: rrdhost_state_id has become object_state in libnetdata so that it can be reused
* fixed metadata; added journal message id for netdata fatal messages
* replication: undo bypassing the pipeline
* receiver cleanup: free all structures at the end, to ensure there are not crashes while cleaning up
* replication periodic checks: do not run it on receivers, when there is replication in progress
* nd_log: prevent fatal statements from recursing
* replication tracking: disabled (compile time)
* fix priority and log
* disconnect on stale replication - detected on both sender and receiver
* update our tagline
* when sending data from within opcode handling do not remove the receiver/sender
* improve interactivity of streaming sockets
* log the replication cmd counters on disconnect and reset them on reconnect
* rrdhost object state activate/deactivate should happen in set/clear receiver
* remove writer preference from rw spinlocks
* show the value in health logs
* move counter to the right place to avoid double counting replication commands
* do not run opcodes when running inline
* fix replication log messages
* make IoT harmless for the moment
---------
Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
Co-authored-by: Netdata bot <43409846+netdatabot@users.noreply.github.com>
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
Co-authored-by: netdatabot <bot@netdata.cloud>
Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>
* updated copyright notices everywhere (I hope)
* Update makeself.lsm
* Update coverity-scan.sh
* make all newlines be linux, not windows
* remove copyright from all files (the take it from the repo), unless it is printed to users
* ML uses synchronous queries
* do not call malloc_trim() to free memory, since to locks everything
* Reschedule dimensions for training from worker threads.
* when we collect or read from the database, it is SAMPLES. When we generate points for a chart is POINTS
* keep the receiver send buffer 10x the default
* support autoscaling stream circular buffers
* nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible
* fix last commit
* allow removing receiver and senders inline, if the stream thread is not working on them
* fix logs
* Revert "nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible"
This reverts commit 51539a97da.
* do not access receiver or sender after it has been removed
* open cache hot2clean
* open cache hot2clean does not need flushing
* use aral for extent pages up to 65k
* track aral malloc and mmap allocations separately; add 8192 as a possible value to PGD
* do not evict too frequently if not needed
* fix aral metrics
* fix aral metrics again
* accurate accounting of memory for dictionaries, strings, labels and MRG
* log during shutdown the progress of dbengine flushing
* move metasync shutfown after dbengine
* max iterations per I/O events
* max iterations per I/O events - break the loop
* max iterations per I/O events - break the loop - again
* disable inline evictions for all caches
* when writing to sockets, send everything that can be sent
* cleanup code to trigger evictions
* fix calculation of eviction size
* fix calculation of eviction size once more
* fix calculation of eviction size once more - again
* ml and replication stop while backfilling is running
* process opcodes while draining the sockets; log with limit when asking to disconnect a node
* fix log
* ml stops when replication queries are running
* report pgd_padding to pulse
* aral precise memory accounting
* removed all alignas() and fix the 2 issues that resulted in unaligned memory accesses (one in mqtt and another in streaming)
* remove the bigger sizes from PGD, but keep multiples of gorilla buffers
* exclude judy from sanitizers
* use 16 bytes alignment on 32 bit machines
* internal check about memory alignment
* experiment: do not allow more children to connect while there is backfilling or replication queries running
* when the node is initializing, retry in 30 seconds
* connector cleanup and isolation of control logic about enabling/disabling various parts
* stop also health queries while backfilling is running
* tuning
* drain the input
* improve interactivity when suspending
* more interactive stream_control
* debug logs to find the connection issue
* abstracted everything about stream control
* Add ml_host_{start,stop} again.
* Do not create/update anomaly-detection charts when ML is not running for a host.
* rrdhost flag RECEIVER_DISCONNECTED has been reversed to COLLECTOR_ONLINE and has been used for localhost and virtual hosts too, to have a single point of truth about the availability of collected data or not
* ml_host_start() and ml_host_stop() are used by streaming receivers; ml_host_start() is used for localhost and virtual hosts
* fixed typo
* allow up to 3 backfills at a time
* add throttling based on user queries
* restore cache line paddings
* unify streaming logs to make it easier to grep logs
* tuning of stream_control
* more logs unification
* use mallocz_release_as_much_memory_to_the_system() under extreme conditions
* do not rely on the response code of evict_pages()
* log the gap of the database every time a node is connected
* updated ram requirements
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
* recreate the circular buffer from time to time
* do not update cloud url if the node id is not updated
* remove deadlock and optimize pipe size
* removed const
* finer control on randomized delays
* restore children re-connecting to parents
* handle partial pipe reads; sender_commit() now checks if the sender is still connected to avoid bombarding it with data that cannot be sent
* added commented code about optimizing the array of pollfds
* improve interactivity of sender; code cleanup
* do not use the pipe for sending messages, instead use a queue in memory (that can never be full)
* fix dictionaries families
* do not destroy aral on replication exit - it crashes the senders
* support multiple dispatchers and connectors; code cleanup
* more cleanup
* Add serde support for KMeans models.
- Serialization/Deserialization support of KMeans models.
- Send/receive ML models between a child/parent.
- Fix some rare and old crash reports.
- Reduce allocations by a couple thousand per second when training.
- Enable ML statistics temporarily which might increase CPU consumption.
* fix ml models streaming
* up to 10 dispatchers and 2 connectors
* experiment: limit the number of receivers to the number of cores - 2
* reworked compression at the receiver to minimize read operations
* multi-core receivers
* use slot 0 on receivers
* use slot 0 on receivers
* use half the cores for receivers with a minimum of 4
* cancel receiver threads
* use offsets instead of pointers in the compressed buffer; track last reads
* fix crash on using freed decompressor; core re-org
* fix incorrect job registration
* fix send_to_plugin() for SSL
* add reason to disconnect message
* fix signaling receivers to stop
* added --dev option to netdata-installer.sh to prevent it from removing the build directory
* Fix serde of double values.
NaNs and +/- infinities are encoded as strings.
* unused param
* reset max cbuffer size when it is recreated
* struct receiver_state is now private
* 1 dispatcher, 1 connector, 2/3 cores for receivers
* all replication requests are served by replication threads - never the dispatcher threads
* optimize partitions and cache lines for dbengine cache
* fix crash on receiver shutdown
* rw spinlock now prioritizes writers
* backfill all higher tiers
* extent cache to 10%
* automatic sizing of replication threads
* add more replication threads
* configure cache eviction parameters to avoid running in aggressive mode all the time
* run evictions and flushes every 100ms
* add missing initialization
* add missing initialization - again
* add evictors for all caches
* add dedicated evict thread per cache
* destroy the completion
* avoid sending too many signals to eviction threads
* alternative way to make sure there are data to evict
* measure inline cache events
* disable inline evictions and flushing for open and extent cache
* use a spinlock to avoid sending too many signals
* batch evictions are not in steps of pages
* fix wanted cache size when there are no clean entries in it
* fix wanted cache size when there are no clean entries in it
* fix wanted cache size again
* adaptive batch evictions; batch evictions first try all partitions
* move waste events to waste chart
* added evict_traversed
* evict is smaller steps
* removed obsolete code
* disabled inlining of evictions and flushing; added timings for evictions
* more detailed timings for evictions
* use inline evictors
* use aral for gorilla pages of 512 bytes, when they are loaded from disk
* use aral for all gorilla page sizes loaded from disk
* disable inlining again to test it after the memory optimization
* timings for dbengine evictions
* added timing names
* detailed timings
* detailed timings - again
* removed timings and restored inline evictions
* eviction on release only under critical pressure
* cleanup and replication tuning
* tune cache size calculation
* tune replication threads calculation
* make streaming receiver exit
* Do not allocate/copy extent data twice.
* Build/link mimalloc
Just for testing, it will be reverted.
* lower memory requirements
* Link mimalloc statically
* run replication with synchronous queries
* added missing worker jobs in sender dispatcher
* enable batch evictions in pgc
* fix sender-dispatcher workers
* set max dispatchers to 2
* increase the default replication threads
* log stream_info errors
* increase replication threads
* log the json text when we fail to parse json response of stream_info
* stream info response may come back in multiple steps
* print the socket error of stream info
* added debug to stream info socket error
* loop while content-length is smaller than the payload received
* Revert "Link mimalloc statically"
This reverts commit c98e482d47.
* Revert "Build/link mimalloc"
This reverts commit 8aae22a28a.
* Remove NEED_PROTOBUF
* Use mimalloc
* Revert "Use mimalloc"
This reverts commit 9a68034786.
* Use mimalloc
* support 256 bytes gorilla pages, when they are loaded from disk
* added os_mem_available()
* test memory protection
* use protection only on one cache
* use the free memory of the main cache in the other caches too
* use the free memory of the main cache in the open cache too
* Batch gorilla writes by tracking the last written number.
In a setup with 200 children, `perf` shows that
the worst offender is the gorilla write operation,
reporting ~17% overhead.
With this change `perf` reports ~4% overhead and
netdata's CPU consumption decreased by ~16%.
* make buffered_reader_next_line() a couple times faster
* flushing open cache
* Use re2c for the line splitting pluginsd.
Function get's optimized around 3x.
We should delete old code and use the re2c for
the rest of the functions, but we need to keep
the PR size as minimal as possible. Will do in
follow up PRs.
* use cores - 1 for receivers, use only 1 sender
* move sender processing to a separate function
* Revert "Batch gorilla writes by tracking the last written number."
This reverts commit 2e72a5c56d.
* Batch gorilla writes only from writers
This reapplies df79be2f01145bd79091a8934d7c80b4b3eb915b
and introduces a couple changes to remomove writes
from readers.
* log information for buffer overflow
* fix heap use after free
* added comments to the main stream receiver loop
* 3 dispatchers
* single threaded receiver and sender
* code cleanup
* de-associate hosts from streaming threads when both the receiver and sender stop, so that each time the threads are re-balanced
* fix heap use after free
* properly get the slot number of pollfd
* fixes
* fixes
* revert worker changes
* reuse streaming threads
* backfilling should be synchronous
* remove the node last
* do not keep a pointer to rellocatable buffer
* give to pgc the right page size, not less
* restore spreading metrics size across time
* use the calculated slots for gorilla pages
* accurately track gorilla page size changes
* check the sth pointer for validity
* code cleanup, files re-org and renames to reflect the new structure of streaming
* updated referenced size when the size of a page changes; removed flush spins - fluhses cancelled is a waste event
* improve families in netdata statistics
* page size histogram per cache
* page size histogram per cache queue (hot, dirty, clean)
* fix heap after use in pdc.c
* rw_spinlocks: when preferring a writer yield so that the writer has the chance to get the lock
* do not balloon open and extent caches more than needed (it fragments memory and there is not enough memory for the main cache)
* fixed typo
* enable trace allocations to work
* Skip adding kmeans model when ML dimension has not been created.
* PGD is now entirely on ARAL for all types of pages
* 2 partitions for PGD
* Check for ML queue prior to pushing as well.
* merge multiple arals, to avoid wasting memory
* significantly less arals; proper calculation of gorilla efficiency
* report pgd buffers separately from pgc
* aral only for sizes less than 512 bytes
* tune aral caches
* log the functions using the streaming buffer when concurrent use is detected
* aral supporting different pages for collected pages and clean pages - an attempt to minimize fragmentation at high performance
* fix misuse of sender thread buffers
* select the right buffer, based on the receiver tid
* no more rrdpush, renamed to stream
* lower aral max page size to 16KiB - in an attempt to lower fragmentation under memory pressure
* update opcode handling
* automatic sizing of aral limiting its size to 200 items per page or 4 x system pages
* tune cache eviction strategy
* renamed global statistics to telemetry and split it into multiple files
* left over renames of global statistics to telemetry
* added heatmap to chart types
* note about re-balancing a parents cluster
* fix formating
* added aral telemetry to find the fragmentation per aral
* experiment with a different strategy when making clean pages: always append so that the cache is being constantly rotated; aral telemetry reports utilization instead of fragmentation
* aral now takes into account waiting deallocators when it creates new pages
* split netdata-conf functions into multiple files; added dbengine use all caches and dbengine out of memory protection settings
* tune cache eviction strategy
* cache parameters cleanup
* rename mem_available to system_memory
* Fix variable type.
* Add fuzzer for pluginsd line splitter.
* use cgroup v1 and v2 to detect memory protection; log on start the detection of memory
* fixed typo
* added logs about system memory detection
* remove debug logs from system memory detection
* move the rest of dbengine config to netdata-conf
* respect streaming buffer size configured
* add workers to pgc eviction threads
* renamed worker
* fixed flip-flop in size and entries conversions
* use aral_by_size when we actually agreegate stats to aral by size
* use keyword defintions
* move opcode definitions to stream-thread.h
* swap struct pollfd slots to make sure all the sockets have an equal chance of being processed
* Revert "Add fuzzer for pluginsd line splitter."
This reverts commit 454cbcf6e1.
* Revert "Use re2c for the line splitting pluginsd."
This reverts commit 2b2f9d3887.
* stream thread use judy arrays instead of linked lists and pre-allocated arrays
* added comment about pfd structure on sender and receiver
* fixed logs and made the defaut sender timeout 5 seconds
* Spawn ML worker threads based on number of CPUs.
* Add statistics for ML allocations/deallocations.
* Add host flag to check for pending alert transitions to save
Remove precompiled statements
Offload processing of alerts in the event loop
Queue alert transitions to the metadata event loop to be saved
Run metadata checks every 5 seconds
* do not block doing socket retries when errno indicates EWOULDBLOCK; insist sending data in send_to_plugin()
* Revert "Add host flag to check for pending alert transitions to save"
This reverts commit 86ade0e87e.
* fix error reasons
* Disable ML memory statistics when using mimalloc
* add reason when ml cannot acquire the dimension
* added ML memory and depending on the DICT_WITH_STATS define, add aral by size too
* do not stream ML when the parent does not have ML enabled
* nd_poll() to overcome the starvation of poll() and use epoll() under Linux
* nd_poll() optimization to minimize the number of system calls
* nd_poll() fix
* nd_poll() fix again
* make glibc release memory to the system when the system is critical in memory
* try bigger aral pages, to enable releasing memory back to the system
* Queue alert transitions to the metadata event loop (global list not per host)
Add host count to check for pending alert transitions to save
Remove precompiled statements
Offload processing of alerts in the event loop
Run metadata checks every 5 seconds
* round robin aral allocations
* fix aral round robin
* ask glibc to release memory when the allocations are aggressive
* tinysleep yields the processor instead of waiting
* run malloc_trim() more frequently
* Add reference count on alarm_entry
* selective tinysleep and processor yielding
* revert gorilla batch writes
* codacy fixes
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
