* extended dbegnine stats should be enabled
* creash reports are enabled when the agent is claimed (directly or indirectly)
* make mmap() report out of memory
* for open cache, use the current hot size, not the max ever used
* remove the RRDHOST hostname index
* do not use flags for indexing hosts
* add dictionary_garbage_collect() to dictionary_flush()
* just destroy remaining hot/dirty pages on shutdown
* fix dictionaries not adding dictionaries for garbage collect
* detect netdata exit reasons
* log exit initiated
* commented debug logs
* commented debug logs again
* fix windows system shutdown detection
* commented debug logs again
* added exit reason msgid
* test shutdown detection by writing to exit.reason
* implement status file loading/saving
* accept also the shutdown event
* fix windows logs
* run as service from the script - not working yet
* save the first fatal message into the status file
* save memory information in the status file
* load machine guid early enough
* fix loading sequence
* simplify function run once logic; add dependencies on netdata.conf loading when required
* accept service parameter
* build for packaging is required for services
* log last exit status with a proper message; log node id and claim id in the status file
* added /var/cache disk space; fixed bug in rfc3339 parsing
* change log priority based on condition
* SIGINT is normal exit under windows
* wait to wevt provider to be initialized before logging
* Revert "fix windows logs (#19632)"
This reverts commit d8c3dc087c.
* fix windows logs - the right way
* set default event log sizes
* added detection of netdata update
* added systemd dbus watcher for systemd shutdown/suspend events
* log system shutdown
* detect system reboot in a better way
* cleanup static thread
* on fatal, call _exit(); linunwind should not skip top calls on the stack
* make the sd bus watcher exit on netdata shutdown
* make the netdata agent version log also print the last exit status
* start watcher when shutdown is initiated; prevent double logging of shutdown initiation
* prepare for sending reports
* a single read per receiver
* track memory calls per worker
* use 4 malloc arenas on parents
* spread higher tiers flushing over time
* pgc and replication tuning
* on child disconnect, get retention from the rrdcontexts worker
* BUFFER: the default size is now 1024 bytes
* use dedicated jemalloc arena for judy allocations
* ARAL: do not double the page size unconditionally; cleanup old members
* double pgc partitions
* fix compiler warning
* make the default replication commit buffer big enough to avoid constant realloc
* post crash reports
* revert log2journal changes
* log2journal minor
* disable the crash report when there was no status file
* increase buffer sizes
* added os_boottime() and os_boot_id(), which are now used in the status file
* log2journal: convert \u000A to \n
* fix headers includes
* fix compilation on non-linux
* for host prefix when getting boot_id and boottime
* write status file to /run/netdata too
* fix /run/netdata on startup
* move the IPC pipe inside the run directory
* exclusive file lock to avoid running concurrently
* allow netdatacli to run from any user and still find the run dir of netdata
* fix pipe failure message
* fix nested loop sharing same variable in ADCS
* fix run_dir and netdatacli on windows
* fix status files on windows
* initialize nd_threads early enough to allow creating threads during initialization
* fix compiler warnings
* on shutdown ignore points with delayed flushing
* fix macos compilation
* added os_type to daemon status
* make daemon status schema ecs compatible
* save daemon status file on every signal
* fix external plugins log to journal
* use special allocators for judy, only on netdata - not the external plugins
* systemd-cat-native: default newline string is \n
* when generating json, prefer special 2 character sequences for common control characters
* fix daemon-status filenames
* log errors when the status file cannot be opened/saved/parsed
* make status file world readable
* do not write status file in /run/netdata; add fall back locations when the file cannot be saved in the cache dir
* move ram and disk into host
* simplified inline subobject parsing for jsonc
* ensure path is an array of at least 128 bytes
* fix non-linux compilation
* inline dbengine query critical path
* more dbengine inlining
* optimize pgc_page_add to optimistically allocate the new entry
* backfill related inlining
* more inlining
* simple re-organization of members of sender_state
* descriptive SSL error logs
* nd-poll() wanted mismatch is now a debug statement
* improve SSL error handling
* restore ssl error handling code
* strict error checking on enable_streaming replication flag
* selected functions flattening
* size replication queue
* fix replication step inicfg parsing; fix local variable exposure
* replication sender code cleanup
* process opcodes every 100ms and reorganize so that the latency per processing can be computed
* more inlining at critical sections
* make replication step 20m
* configure replication prefetch
* use the inicfg range function for libuv threads
* move libuv initialize closer to the beginning
* replication fixes
* replication defaults tuning
* aral to use 2MiB page sizes and nd_mmap() to madvise HUGEPAGE when size matches
* use dedicated aral for replication sender
* tune replication sender to max 6 threads
* aral incoming lock now tries all partitions
* health should be running together with replication
* refcount added refcount_release_and_acquire_for_deletion_advanced()
* minimize aral_lock() use
* aral having page_lock
* aral_page refcount with simple atomics
* cleanup aral
* inline functions related to metrics ingestion
* move pulse outside the lock
* cleanup senders in connector thread - untested
* receiver and sender cleanup
* buffer functions inlining
* more inlining
* do not disconnect receivers when the sender fails
* heavy parts of sender connect/disconnect run with connector unlocked
* added uuidmap registry of uuids, giving a uint32_t
* writer code in mrg is now used only in NETDATA_INTERNAL_CHECKS
* working mrg with uuidmap
* rrd contexts now uses uuidmap; refcount has been abstracted everywhere
* uuidmap uses get before write locking; uuidmap unittest
* uuidmap supports 256 partitions
* added uuidmap to memory accounting; simplified mrg memory accounting
* uuidmap memory accounting
* rrdddim now uses uuidmap; cleanup rrdmetric and rrdinstance; rrddim, rrdmetric, rrdinstance now free uuidmap entries on cleanup; mrg left-overs of JudyHS are removed; mrg uses the same number of partitions as uuidmap and actually the same partition per uuid
* fix bad merge
* db ram/alloc now uses judyL instead of judyHS
* storage engine function using UUIDMAP_ID
* use waitq in sender
* fix warning
* hot pages may become dirty prematurely during shutdown
* set uuidmap partitions to 8
* reuse old uuidmap metric ids
* do not get before add in mrg
* undo resuing host sender buffer
* fix compilation with FSANITIZE_ADDRESS
* aral should not unmark allocations when FSANITIZE_ADDRESS is set
* enable reusing sender buffers
* do not load instances and dimensions for unknown contexts - instead of creating contexts and adding instances and dimensions
* node updates to NC
* unify all liveness calls
* prevent wanted cache size from getting negative
* do not log when a socket is closed on accept()
* waiting queues use futex under linux; use waiting-queues for pgc-queue-locks
* do not use waiting queue for pgc-queue-lock
* multiplex releasing to ensure better ordering; disable futex implementation until further testing
* initialize pulse counters to zero
* fixed web connected clients count in pulse
* coverity fixes
* evict, a page at a time
* 4 replication ahead requests per replication thread
* added per job average timings for workers and dbengine query router
* debug statement to find what is slow
* yield the processor to avoid monopolizing the cache
* test more page sizes in aral
* more polite journal v2 indexing
* pulse macros for atomics
* added profile so that users can control the defaults of the agent
* fix windows warnings; journal v2 generation yields the processor for every page
* replication threads are 1/3 of the cores and they are synchronous
* removed the default from the list of profiles
* turn pgc locks into macros to have tracing on the functions that called them
* log the size of madvise() when failing
* more work on profiles
* restore batch cache evictions, but lower the batch size significantly
* do not spin while searching for pages in the cache - handle currently being deleted pages within the search logic itself
* remove bottleneck in epdl processing while merging extents
* allocate outside the lock
* rw spinlock implemented without spinlocks; both spinlocks and r/w spinlocks now support exponential backoff while waiting
* apply max sleep to spinlocks
* tune replication
* r/w spinlock prefers writers again, but this time recursive readers are bypassing the writer wait
* tuning of rw spinlock
* more tuning of the rw spinlock
* configure glibc arenas based on profile
* moving global variables into nd_profile
* do not accept sockets that have not received any data; once sockets with data have been accepted, check they are not closed already before processing them
* poll_events is now using nd_poll(), resulting in vast simplification of the code; static web files are now served inline resulting in another simplification of the web server logic (was required because epoll does not support normal files)
* startup fixes
* added debug info to poll_fd_close()
* closed sockets are automatically removed from epoll(), by the kernel
* fix for mrg acquired and referenced going negative
* fixed bug in mrg cleanup, not deleting metrics that do not have retention
* monitor strings index size
* strings memory chart is now stacked
* replication: do not lock data collection when running in batches
* explicitly set socket flags for sender and receiver
* normalize the event loop for sending data (receiver and sender)
* normalize the event loop for receiving data (receiver and sender)
* check all sender nodes every half a second
* fix bug on sender, not enabling sending
* first cleanup then destroy
* normalize nd_poll() to handle all possible events
* cleanup
* normalize socket helper functions
* fixed warnings on alpine
* fix for POLLRDHUP missing
* fix cleanup on shutdown
* added detailed replication summary
* moved logs to INFO
* prevent crash when sender is not there
* madvise _dontfork() should not be used with aral, madvise_dontdump() is only used for file backed maps
* fix wording
* fix log wording
* split replication receiver and sender; add logs to find missing replication requests
* fix compilation
* fixed bug in backfilling, having garbage for counters - malloc instead of calloc
* backfilling logs if it misses callbacks
* log replication rcv and replication snd in node info
* remove contention from aral_page_free_lock(), but having 2 free lists per page, one for incoming and another for available items and moving incoming to available when the available is empty - this allows aral_mallocz() and aral_freez() to operate concurrently on the same page
* fix internal checks
* log errors for all replication receiver exceptions
* removed wrong error log
* prevent health crashing
* cleanup logs that are irrelevant with the missing replication events
* replication tracking: added replication tracking to figure out how replication missed requests
* fix compilation and fix bug on spawn server cleanup calling uv_shutdown at exit
* merged receiver initialization
* prevent compilation warnings
* fix race condition in nd_poll() returning events for deleted fds
* for user queries, prepare as many queries as half the processors
* fix log
* add option dont_dump to netdata_mmap and aral_create
* add logging missing receiver and sender charts
* reviewed judy memory accounting; adbstracted flags handling to ensure they all work the same way; introduced atomic_flags_set_and_clear() to set and clear atomic flags with a single atomic operation
* improvement(go.d/nats): add server_id label (#19280)
* Regenerate integrations docs (#19281)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-30-nightly.
* docs: improve on-prem troubleshooting readability (#19279)
* docs: improve on-prem troubleshooting readability
* Apply suggestions from code review
---------
Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>
* improvement(go.d/nats): add leafz metrics (#19282)
* Regenerate integrations docs (#19283)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-34-nightly.
* fix go.d/nats tests (#19284)
* improvement(go.d/nats): add basic jetstream metrics (#19285)
* Regenerate integrations docs (#19286)
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
* [ci skip] Update changelog and version for nightly build: v2.1.0-38-nightly.
* bump dag req jinja version (#19287)
* more strict control on replication counters
* do not flush the log files - to cope with the rate
* [ci skip] Update changelog and version for nightly build: v2.1.0-40-nightly.
* fix aral on windows
* add waiting queue to sender commit, to allow the streaming thread go fast and put replication threads in order
* use the receiver tid
* fix(netdata-updater.sh): remove commit_check_file directory (#19288)
* receiver now has periodic checks too (like the senders have)
* fixed logs
* replication periodic checks: resending of chart definitions
* strict checking on rrdhost state id
* replication periodic checks: added for receivers
* shorter replication status messages
* do not log about ieee754
* receiver logs replication traffic without RSET
* object state: rrdhost_state_id has become object_state in libnetdata so that it can be reused
* fixed metadata; added journal message id for netdata fatal messages
* replication: undo bypassing the pipeline
* receiver cleanup: free all structures at the end, to ensure there are not crashes while cleaning up
* replication periodic checks: do not run it on receivers, when there is replication in progress
* nd_log: prevent fatal statements from recursing
* replication tracking: disabled (compile time)
* fix priority and log
* disconnect on stale replication - detected on both sender and receiver
* update our tagline
* when sending data from within opcode handling do not remove the receiver/sender
* improve interactivity of streaming sockets
* log the replication cmd counters on disconnect and reset them on reconnect
* rrdhost object state activate/deactivate should happen in set/clear receiver
* remove writer preference from rw spinlocks
* show the value in health logs
* move counter to the right place to avoid double counting replication commands
* do not run opcodes when running inline
* fix replication log messages
* make IoT harmless for the moment
---------
Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
Co-authored-by: Netdata bot <43409846+netdatabot@users.noreply.github.com>
Co-authored-by: ilyam8 <22274335+ilyam8@users.noreply.github.com>
Co-authored-by: netdatabot <bot@netdata.cloud>
Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>
* add sqlite3 memory to total netdata memory
* recheck stream_conf_is_parent() when stream.conf is loaded
* cleanup about getting the number of cores; we now also use the official posix way
* fix the log message when a node connects for the first time
* better alternative for freebsd
* allow empty flags
* do not expect a filename on non-linux machines
* enable deleting dyncfg alerts, even if they are on disk
* pulse aral stats fixes
* double is not needed there
* control the number of pages evicted at once, to avoid gaps on data collection
* lower the aggresiveness of evictions to prevent evicting signifanctly more pages than really needed
* add aral judy statistics to pulse aral
* metadata sync shutsdown in the background
* renumber
* when just 1 host is still pending, log also its hostname
* call mallocz_release_as_much_memory_to_the_system() at most once per second
* improve ML logs and move metadata_sync_shutdown() at the proper place
* stack size of external plugins; moved all function evloop data off the stack
* fix for incorrect replication accounting
* metadata cmds in aral
* fix function payload parsing for external plugins
* split main service control and netdata shutdown to separate files - no code changes
* add more information to shutdown watcher
* fix exit log messages
* parallel dbengine exits for all tiers
* log with limit
* alerts not alarms
* minor fixes in health
* added logs to trace the root cause of delays when closing data files
* added STORAGE_PRIORITY_SYNCHRONOUS_FIRST
* Revert "added logs to trace the root cause of delays when closing data files"
This reverts commit 75515c1484.
* log datafile lock wait
* print the number of writers
* print the number of writers again
* print the number of writers again again
* fix watcher messages
* single node agents use 1 replication and 1 ML thread; max threads for both is 256
* log the progress of flushers
* spawn intense flushing on quiesce
* cleanup logs added
* print the status of the cache on exit
* dbengine shutdown cleanup
* cleanup logs
* cleanup logs
* more cleanup on logs
* proper percentage calculation
* sentry include added
* ML uses synchronous queries
* do not call malloc_trim() to free memory, since to locks everything
* Reschedule dimensions for training from worker threads.
* when we collect or read from the database, it is SAMPLES. When we generate points for a chart is POINTS
* keep the receiver send buffer 10x the default
* support autoscaling stream circular buffers
* nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible
* fix last commit
* allow removing receiver and senders inline, if the stream thread is not working on them
* fix logs
* Revert "nd_poll() prefers sending data vs receiving data - in an attempt to dequeue as soon as possible"
This reverts commit 51539a97da.
* do not access receiver or sender after it has been removed
* open cache hot2clean
* open cache hot2clean does not need flushing
* use aral for extent pages up to 65k
* track aral malloc and mmap allocations separately; add 8192 as a possible value to PGD
* do not evict too frequently if not needed
* fix aral metrics
* fix aral metrics again
* accurate accounting of memory for dictionaries, strings, labels and MRG
* log during shutdown the progress of dbengine flushing
* move metasync shutfown after dbengine
* max iterations per I/O events
* max iterations per I/O events - break the loop
* max iterations per I/O events - break the loop - again
* disable inline evictions for all caches
* when writing to sockets, send everything that can be sent
* cleanup code to trigger evictions
* fix calculation of eviction size
* fix calculation of eviction size once more
* fix calculation of eviction size once more - again
* ml and replication stop while backfilling is running
* process opcodes while draining the sockets; log with limit when asking to disconnect a node
* fix log
* ml stops when replication queries are running
* report pgd_padding to pulse
* aral precise memory accounting
* removed all alignas() and fix the 2 issues that resulted in unaligned memory accesses (one in mqtt and another in streaming)
* remove the bigger sizes from PGD, but keep multiples of gorilla buffers
* exclude judy from sanitizers
* use 16 bytes alignment on 32 bit machines
* internal check about memory alignment
* experiment: do not allow more children to connect while there is backfilling or replication queries running
* when the node is initializing, retry in 30 seconds
* connector cleanup and isolation of control logic about enabling/disabling various parts
* stop also health queries while backfilling is running
* tuning
* drain the input
* improve interactivity when suspending
* more interactive stream_control
* debug logs to find the connection issue
* abstracted everything about stream control
* Add ml_host_{start,stop} again.
* Do not create/update anomaly-detection charts when ML is not running for a host.
* rrdhost flag RECEIVER_DISCONNECTED has been reversed to COLLECTOR_ONLINE and has been used for localhost and virtual hosts too, to have a single point of truth about the availability of collected data or not
* ml_host_start() and ml_host_stop() are used by streaming receivers; ml_host_start() is used for localhost and virtual hosts
* fixed typo
* allow up to 3 backfills at a time
* add throttling based on user queries
* restore cache line paddings
* unify streaming logs to make it easier to grep logs
* tuning of stream_control
* more logs unification
* use mallocz_release_as_much_memory_to_the_system() under extreme conditions
* do not rely on the response code of evict_pages()
* log the gap of the database every time a node is connected
* updated ram requirements
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
* prefer tinysleep over yielding the processor
* split spinlocks to separate files
* rename spinlock initializers
* Optimize ML queuing operations.
- Allocate 25% of cores for ML.
- Split queues by request type.
- Accurate stats for queue operations by type.
* abstracted circular buffer into a new private structure to enable using it in receiver sending side - no features added yet, only abstracted the existing functionality - not tested yet
* completed the abstraction of stream circular buffer
* unified list of receivers and senders; opcodes now support both receivers and senders
* use strings in pluginsd
* stream receivers send data back to the child using the event loop
* do not share pgc aral between caches
* pgc uses 4 to 256 partitions, by default equal to the number of CPU cores
* add forgotten worker job
* workers now monitor spinlock contention
* stream sender tries to lock the sender, but does not wait for it - it will be handled later
* increase the number of web server threads to the number of cpu cores, with a minimum of 6
* use the nowait versions of nd_sock functions
* handle EAGAIN properly
* add spinlock contention tracing for rw_spinlock
* aral lock/unlock contention tracing
* allocate the compressed buffer
* use 128KiB for aral default page size; limit memory protection to 5GiB
* aral uses mmap() for big pages
* enrich log messages
* renamed telemetry to pulse
* unified sender and receiver socket event loops
* logging improvements
* NETDATA_LOG_STREAM_SENDER logs inbound and outbound traffic
* 16k receiver buffer size to improve interactivity
* fix NETDATA_LOG_STREAM_SENDER in sender_execute
* do not stream ML models for charts and dimensions that have not been exposed
* add support for sending QUIT to plugins and waiting for some time for them to quit gracefully
* global spinlock contention per function
* use an aral per pgc partition; use 8 partitions for PGD
* rrdcalc: do not change the frequency of alerts - it uses arbitrary values used during replication, changing permanently the frequency of alerts
replication: use 1/3 of the cores or 1 core every 10 nodes (min of the two)
pgd: use as many aral partitions as the CPU cores, up to 256
* aral does 1 allocation per page (the structure and the elements together), instead of two
* use the evitor thread only when we run out of memory; restore the optimization about prepending or appending clean pages based on their accesses; use the main cache free memory for the other caches, reducing I/O when the main cache has enough room
* reduce the number of events per poll() to 10
* aral allocates pages of up to 1MiB; restore processing 100 events per nd_poll() call
* drain the sockets while reading
* receiver sockets should be non-blocking
* add stability detector to aral
* increase the receivers send buffer
* do not remove the sender or the receiver while we drain the input sockets
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
* recreate the circular buffer from time to time
* do not update cloud url if the node id is not updated
* remove deadlock and optimize pipe size
* removed const
* finer control on randomized delays
* restore children re-connecting to parents
* handle partial pipe reads; sender_commit() now checks if the sender is still connected to avoid bombarding it with data that cannot be sent
* added commented code about optimizing the array of pollfds
* improve interactivity of sender; code cleanup
* do not use the pipe for sending messages, instead use a queue in memory (that can never be full)
* fix dictionaries families
* do not destroy aral on replication exit - it crashes the senders
* support multiple dispatchers and connectors; code cleanup
* more cleanup
* Add serde support for KMeans models.
- Serialization/Deserialization support of KMeans models.
- Send/receive ML models between a child/parent.
- Fix some rare and old crash reports.
- Reduce allocations by a couple thousand per second when training.
- Enable ML statistics temporarily which might increase CPU consumption.
* fix ml models streaming
* up to 10 dispatchers and 2 connectors
* experiment: limit the number of receivers to the number of cores - 2
* reworked compression at the receiver to minimize read operations
* multi-core receivers
* use slot 0 on receivers
* use slot 0 on receivers
* use half the cores for receivers with a minimum of 4
* cancel receiver threads
* use offsets instead of pointers in the compressed buffer; track last reads
* fix crash on using freed decompressor; core re-org
* fix incorrect job registration
* fix send_to_plugin() for SSL
* add reason to disconnect message
* fix signaling receivers to stop
* added --dev option to netdata-installer.sh to prevent it from removing the build directory
* Fix serde of double values.
NaNs and +/- infinities are encoded as strings.
* unused param
* reset max cbuffer size when it is recreated
* struct receiver_state is now private
* 1 dispatcher, 1 connector, 2/3 cores for receivers
* all replication requests are served by replication threads - never the dispatcher threads
* optimize partitions and cache lines for dbengine cache
* fix crash on receiver shutdown
* rw spinlock now prioritizes writers
* backfill all higher tiers
* extent cache to 10%
* automatic sizing of replication threads
* add more replication threads
* configure cache eviction parameters to avoid running in aggressive mode all the time
* run evictions and flushes every 100ms
* add missing initialization
* add missing initialization - again
* add evictors for all caches
* add dedicated evict thread per cache
* destroy the completion
* avoid sending too many signals to eviction threads
* alternative way to make sure there are data to evict
* measure inline cache events
* disable inline evictions and flushing for open and extent cache
* use a spinlock to avoid sending too many signals
* batch evictions are not in steps of pages
* fix wanted cache size when there are no clean entries in it
* fix wanted cache size when there are no clean entries in it
* fix wanted cache size again
* adaptive batch evictions; batch evictions first try all partitions
* move waste events to waste chart
* added evict_traversed
* evict is smaller steps
* removed obsolete code
* disabled inlining of evictions and flushing; added timings for evictions
* more detailed timings for evictions
* use inline evictors
* use aral for gorilla pages of 512 bytes, when they are loaded from disk
* use aral for all gorilla page sizes loaded from disk
* disable inlining again to test it after the memory optimization
* timings for dbengine evictions
* added timing names
* detailed timings
* detailed timings - again
* removed timings and restored inline evictions
* eviction on release only under critical pressure
* cleanup and replication tuning
* tune cache size calculation
* tune replication threads calculation
* make streaming receiver exit
* Do not allocate/copy extent data twice.
* Build/link mimalloc
Just for testing, it will be reverted.
* lower memory requirements
* Link mimalloc statically
* run replication with synchronous queries
* added missing worker jobs in sender dispatcher
* enable batch evictions in pgc
* fix sender-dispatcher workers
* set max dispatchers to 2
* increase the default replication threads
* log stream_info errors
* increase replication threads
* log the json text when we fail to parse json response of stream_info
* stream info response may come back in multiple steps
* print the socket error of stream info
* added debug to stream info socket error
* loop while content-length is smaller than the payload received
* Revert "Link mimalloc statically"
This reverts commit c98e482d47.
* Revert "Build/link mimalloc"
This reverts commit 8aae22a28a.
* Remove NEED_PROTOBUF
* Use mimalloc
* Revert "Use mimalloc"
This reverts commit 9a68034786.
* Use mimalloc
* support 256 bytes gorilla pages, when they are loaded from disk
* added os_mem_available()
* test memory protection
* use protection only on one cache
* use the free memory of the main cache in the other caches too
* use the free memory of the main cache in the open cache too
* Batch gorilla writes by tracking the last written number.
In a setup with 200 children, `perf` shows that
the worst offender is the gorilla write operation,
reporting ~17% overhead.
With this change `perf` reports ~4% overhead and
netdata's CPU consumption decreased by ~16%.
* make buffered_reader_next_line() a couple times faster
* flushing open cache
* Use re2c for the line splitting pluginsd.
Function get's optimized around 3x.
We should delete old code and use the re2c for
the rest of the functions, but we need to keep
the PR size as minimal as possible. Will do in
follow up PRs.
* use cores - 1 for receivers, use only 1 sender
* move sender processing to a separate function
* Revert "Batch gorilla writes by tracking the last written number."
This reverts commit 2e72a5c56d.
* Batch gorilla writes only from writers
This reapplies df79be2f01145bd79091a8934d7c80b4b3eb915b
and introduces a couple changes to remomove writes
from readers.
* log information for buffer overflow
* fix heap use after free
* added comments to the main stream receiver loop
* 3 dispatchers
* single threaded receiver and sender
* code cleanup
* de-associate hosts from streaming threads when both the receiver and sender stop, so that each time the threads are re-balanced
* fix heap use after free
* properly get the slot number of pollfd
* fixes
* fixes
* revert worker changes
* reuse streaming threads
* backfilling should be synchronous
* remove the node last
* do not keep a pointer to rellocatable buffer
* give to pgc the right page size, not less
* restore spreading metrics size across time
* use the calculated slots for gorilla pages
* accurately track gorilla page size changes
* check the sth pointer for validity
* code cleanup, files re-org and renames to reflect the new structure of streaming
* updated referenced size when the size of a page changes; removed flush spins - fluhses cancelled is a waste event
* improve families in netdata statistics
* page size histogram per cache
* page size histogram per cache queue (hot, dirty, clean)
* fix heap after use in pdc.c
* rw_spinlocks: when preferring a writer yield so that the writer has the chance to get the lock
* do not balloon open and extent caches more than needed (it fragments memory and there is not enough memory for the main cache)
* fixed typo
* enable trace allocations to work
* Skip adding kmeans model when ML dimension has not been created.
* PGD is now entirely on ARAL for all types of pages
* 2 partitions for PGD
* Check for ML queue prior to pushing as well.
* merge multiple arals, to avoid wasting memory
* significantly less arals; proper calculation of gorilla efficiency
* report pgd buffers separately from pgc
* aral only for sizes less than 512 bytes
* tune aral caches
* log the functions using the streaming buffer when concurrent use is detected
* aral supporting different pages for collected pages and clean pages - an attempt to minimize fragmentation at high performance
* fix misuse of sender thread buffers
* select the right buffer, based on the receiver tid
* no more rrdpush, renamed to stream
* lower aral max page size to 16KiB - in an attempt to lower fragmentation under memory pressure
* update opcode handling
* automatic sizing of aral limiting its size to 200 items per page or 4 x system pages
* tune cache eviction strategy
* renamed global statistics to telemetry and split it into multiple files
* left over renames of global statistics to telemetry
* added heatmap to chart types
* note about re-balancing a parents cluster
* fix formating
* added aral telemetry to find the fragmentation per aral
* experiment with a different strategy when making clean pages: always append so that the cache is being constantly rotated; aral telemetry reports utilization instead of fragmentation
* aral now takes into account waiting deallocators when it creates new pages
* split netdata-conf functions into multiple files; added dbengine use all caches and dbengine out of memory protection settings
* tune cache eviction strategy
* cache parameters cleanup
* rename mem_available to system_memory
* Fix variable type.
* Add fuzzer for pluginsd line splitter.
* use cgroup v1 and v2 to detect memory protection; log on start the detection of memory
* fixed typo
* added logs about system memory detection
* remove debug logs from system memory detection
* move the rest of dbengine config to netdata-conf
* respect streaming buffer size configured
* add workers to pgc eviction threads
* renamed worker
* fixed flip-flop in size and entries conversions
* use aral_by_size when we actually agreegate stats to aral by size
* use keyword defintions
* move opcode definitions to stream-thread.h
* swap struct pollfd slots to make sure all the sockets have an equal chance of being processed
* Revert "Add fuzzer for pluginsd line splitter."
This reverts commit 454cbcf6e1.
* Revert "Use re2c for the line splitting pluginsd."
This reverts commit 2b2f9d3887.
* stream thread use judy arrays instead of linked lists and pre-allocated arrays
* added comment about pfd structure on sender and receiver
* fixed logs and made the defaut sender timeout 5 seconds
* Spawn ML worker threads based on number of CPUs.
* Add statistics for ML allocations/deallocations.
* Add host flag to check for pending alert transitions to save
Remove precompiled statements
Offload processing of alerts in the event loop
Queue alert transitions to the metadata event loop to be saved
Run metadata checks every 5 seconds
* do not block doing socket retries when errno indicates EWOULDBLOCK; insist sending data in send_to_plugin()
* Revert "Add host flag to check for pending alert transitions to save"
This reverts commit 86ade0e87e.
* fix error reasons
* Disable ML memory statistics when using mimalloc
* add reason when ml cannot acquire the dimension
* added ML memory and depending on the DICT_WITH_STATS define, add aral by size too
* do not stream ML when the parent does not have ML enabled
* nd_poll() to overcome the starvation of poll() and use epoll() under Linux
* nd_poll() optimization to minimize the number of system calls
* nd_poll() fix
* nd_poll() fix again
* make glibc release memory to the system when the system is critical in memory
* try bigger aral pages, to enable releasing memory back to the system
* Queue alert transitions to the metadata event loop (global list not per host)
Add host count to check for pending alert transitions to save
Remove precompiled statements
Offload processing of alerts in the event loop
Run metadata checks every 5 seconds
* round robin aral allocations
* fix aral round robin
* ask glibc to release memory when the allocations are aggressive
* tinysleep yields the processor instead of waiting
* run malloc_trim() more frequently
* Add reference count on alarm_entry
* selective tinysleep and processor yielding
* revert gorilla batch writes
* codacy fixes
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
* add randomness per thread to heartbeat
* tune randomness
* use the thread tag name in the randomness hash
* use os_gettid()
* fix xenstat
* fix all occurences of heartbeat_init and heartbeat_next
* initialize clocks with a constructor; on windows enable high resolution timer globally
* better distribution of heartbeats; prevent heartbeats to be aligned with system HZ
* move randomness away from scheduler ticks
* make sure randomness is not the same across processes
* make randomness spread even across processes
* eliminate compiler warning on windows
* when a label does not exist in queries, set it to [unset]; fix windows network interface charts to have labels
* abstraction layer for O/S
* updates
* updates
* updates
* temp fix for protobuf
* emulated waitid()
* fix
* fix
* compatibility layer
* fix for idtype
* fix for missing includes
* fix for missing includes
* added missing includes
* added missing includes
* added missing includes
* added missing includes
* added missing includes
* added missing includes
* UUID renamed to ND_UUID to avoid conflict with windows.h
* include libnetdata.h always - no conflicts
* simplify abstraction headers
* fix missing functions
* fix missing functions
* fix missing functions
* fix missing functions
* rename MSYS to WINDOWS
* moved byteorder.h
* structure for an internal windows plugin
* 1st windows plugin
* working plugin
* fix printf
* Special case windows for protobuf
* remove cygwin, compile both as windows
* log windows libraries used
* fix cmake
* fix protobuf
* compilation
* updated compilation script
* added system.ram
* windows uptime
* perflib
* working perflibdump
* minify dump
* updates to windows plugins, enable ML
* minor compatibility fixes for cygwin and msys
* perflib-dump to its own file
* perflib now indexes names
* improvements to the library; disks module WIP
* API for selectively traversing the metrics
* first working perflib chart: disk.space
* working chart on logical and physical disks
* added windows protocols
* fix datatypes for loops
* tinysleep for native smallest sleep support
* remove libuuid dependency on windows
* fix uuid functions for macos compilation
* fix uuid comparison function
* do not overwrite uuid library functions, define them as aliases to our own
* fixed uuid_unparse functions
* fixed typo
* added perflib processor charts
* updates for compiling without posix emulation
* gather common contexts together
* fix includes on linux
* perflib-memory
* windows mem.available
* Update variable names for protobuf
* network traffic
* add network adapters that have traffic as virtual interfaces
* add -pipe to windows compilation
* reset or overflow flag is now per dimension
* dpc is now counted separately
* verified all perflib fields are processed and no text fields are present in the data
* more common contexts
* fix crash
* do not add system.net multiple times
* install deps update and shortcut
* all threads are now joinable behind the scenes
* fix threads cleanup
* prepare for abstracting threads API
* netdata threads full abstraction from pthreads
* more threads abstraction and cleanup
* more compatibility changes
* fix compiler warnings
* add base-devel to packages
* removed duplicate base-devel
* check for strndup
* check headers in quotes
* fix linux compilation
* fix attribute gnu_printf on macos
* fix for threads on macos
* mingw64 compatibility
* enable compilation on windows clion
* added instructions
* enable cloud
* compatibility fixes
* compatibility fixes
* compatibility fixes
* clion works on windows
* support both MSYSTEM=MSYS and MSYSTEM=MINGW64 for configure
* cleanup and docs
* rename uuid_t to nd_uuid_t to avoid conflict with windows uuid_t
* leftovers uuid_t
* do not include uuid.h on macos
* threads signaled cancellations
* do not install v0 dashboard on windows
* script to install openssh server on windows
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* update openssh installation script
* use cleanup variable instead of pthreads push and pop
* replace all calls to netdata_thread_cleanup_push() and netdata_thread_cleanup_pop() with __attribute__((cleanup(...)))
* remove left-over freez
* make sure there are no locks acquired at thread exit
* add missing parameter
* stream receivers and senders are now voluntarily cancelled
* plugins.d now voluntarily exits its threads
* uuid_t may not be aligned to word boundaries - fix the uuid_t functions to work on unaligned objects too.
* collectors evloop is now using the new threading cancellation; ml is now not using pthread_cancel; more fixes
* eliminate threads cancellability from the code base
* fix exit timings and logs; fix uv_threads tags
* use SSL_has_pending() only when it is available
* do not use SSL_has_pending()
* dyncfg files on windows escape collon and pipe characters
* fix compilation on older systems
* fix compilation on older systems
* Create windows installer.
The installer will install everything
under C:\netdata by default.
It will:
- Install msys2 at C:\netdata
- Install netdata dependencies with pacman
- Install the agent itself under C:\netdata\opt
You can start the agent by running an MSYS
shell with C:\netdata\msys2_shell.cmd and
then start the agent normally with:
/opt/netdata/usr/sbin/netdata -D
There are a more couple things to work on:
- Verify publisher.
- Install all deps not just libuv & protobuf.
- Figure out how we want to auto-start the agent as a service.
- Check how to uninstall things.
* fixed typo
* code cleanup
* Create uninstaller
---------
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
