* rrdset - in progress
* rrdset optimal constructor; rrdset conflict
* rrdset final touches
* re-organization of rrdset object members
* prevent use-after-free
* dictionary dfe supports also counting of iterations
* rrddim managed by dictionary
* rrd.h cleanup
* DICTIONARY_ITEM now is referencing actual dictionary items in the code
* removed rrdset linked list
* Revert "removed rrdset linked list"
This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.
* removed rrdset linked list
* added comments
* Switch chart uuid to static allocation in rrdset
Remove unused functions
* rrdset_archive() and friends...
* always create rrdfamily
* enable ml_free_dimension
* rrddim_foreach done with dfe
* most custom rrddim loops replaced with rrddim_foreach
* removed accesses to rrddim->dimensions
* removed locks that are no longer needed
* rrdsetvar is now managed by the dictionary
* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853
* conflict callback of rrdsetvar now properly checks if it has to reset the variable
* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM
* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe
* dictionary walkthrough callbacks get dictionary acquired items
* dictionary reference counters that can be dupped from zero
* added advanced functions for get and del
* rrdvar managed by dictionaries
* thread safety for rrdsetvar
* faster rrdvar initialization
* rrdvar string lengths should match in all add, del, get functions
* rrdvar internals hidden from the rest of the world
* rrdvar is now acquired throughout netdata
* hide the internal structures of rrdsetvar
* rrdsetvar is now acquired through out netdata
* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata
* better error handling
* dont create variables if not initialized for health
* dont create variables if not initialized for health again
* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items
* type checking on acquired objects
* rrdcalc renaming of functions
* type checking for rrdfamily_acquired
* rrdcalc managed by dictionaries
* rrdcalc double free fix
* host rrdvars is always needed
* attempt to fix deadlock 1
* attempt to fix deadlock 2
* Remove unused variable
* attempt to fix deadlock 3
* snprintfz
* rrdcalc index in rrdset fix
* Stop storing active charts and computing chart hashes
* Remove store active chart function
* Remove compute chart hash function
* Remove sql_store_chart_hash function
* Remove store_active_dimension function
* dictionary delayed destruction
* formatting and cleanup
* zero dictionary base on rrdsetvar
* added internal error to log delayed destructions of dictionaries
* typo in rrddimvar
* added debugging info to dictionary
* debug info
* fix for rrdcalc keys being empty
* remove forgotten unlock
* remove deadlock
* Switch to metadata version 5 and drop
chart_hash
chart_hash_map
chart_active
dimension_active
v_chart_hash
* SQL cosmetic changes
* do not busy wait while destroying a referenced dictionary
* remove deadlock
* code cleanup; re-organization;
* fast cleanup and flushing of dictionaries
* number formatting fixes
* do not delete configured alerts when archiving a chart
* rrddim obsolete linked list management outside dictionaries
* removed duplicate contexts call
* fix crash when rrdfamily is not initialized
* dont keep rrddimvar referenced
* properly cleanup rrdvar
* removed some locks
* Do not attempt to cleanup chart_hash / chart_hash_map
* rrdcalctemplate managed by dictionary
* register callbacks on the right dictionary
* removed some more locks
* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread
* when looking up for an alarm look using both chart id and chart name
* host initialization a bit more modular
* init rrdlabels on host update
* preparation for dictionary views
* improved comment
* unused variables without internal checks
* service threads isolation and worker info
* more worker info in service thread
* thread cancelability debugging with internal checks
* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647
* dictionary modularization
* Remove unused SQL statement definition
* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated
* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops
* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/proc.plugin/proc_net_dev.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* fix memory leak in rrdset cache_dir
* minor dictionary changes
* dont use index locks in single threaded
* obsolete dict option
* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;
* fix jump on uninitialized value in dictionary; remove double free of cache_dir
* addressed codacy findings
* removed debugging code
* use the private refcount on dictionaries
* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;
* more dictionary statistics
* global statistics about dictionary operations, memory, items, callbacks
* dictionary support for views - missing the public API
* removed warning about unused parameter
* chart and context name for cloud
* chart and context name for cloud, again
* dictionary statistics fixed; first implementation of dictionary views - not currently used
* only the master can globally delete an item
* context needs netdata prefix
* fix context and chart it of spins
* fix for host variables when health is not enabled
* run garbage collector on item insert too
* Fix info message; remove extra "using"
* update dict unittest for new placement of garbage collector
* we need RRDHOST->rrdvars for maintaining custom host variables
* Health initialization needs the host->host_uuid
* split STRING to its own files; no code changes other than that
* initialize health unconditionally
* unit tests do not pollute the global scope with their variables
* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* moved rrdcontexts processing to worker thread
* added loggings
* check for aclk deeper in the code
* removed unessesary logs
* code re-organization; cleanup; more comments; better error handling; rrdcontext locks optimization; more clarity
* updated 2 comments
* make instances walkthrough reentrant; move context lock to the place is really needed
* created macro for reentrant dictionary walkthrough
* incremental updates on instances and metrics
* renamed family of rrdcontext workers
* prevent crash in case RRDINSTANCE or RRDMETRIC is freed during shutdown
* prevent crash during rrddim save, on out of memory fatal()
* always post-process contexts
* added tracing for tracking the caller that trigger updates
* more details on tracing info
* fix for charts that are collected without metrics
* rrdfamily
* rrddim
* rrdset plugin and module names
* rrdset units
* rrdset type
* rrdset family
* rrdset title
* rrdset title more
* rrdset context
* rrdcalctemplate context and removal of context hash from rrdset
* strings statistics
* rrdset name
* rearranged members of rrdset
* eliminate rrdset name hash; rrdcalc chart converted to STRING
* rrdset id, eliminated rrdset hash
* rrdcalc, alarm_entry, alert_config and some of rrdcalctemplate
* rrdcalctemplate
* rrdvar
* eval_variable
* rrddimvar and rrdsetvar
* rrdhost hostname, os and tags
* fix master commits
* added thread cache; implemented string_dup without locks
* faster thread cache
* rrdset and rrddim now use dictionaries for indexing
* rrdhost now uses dictionary
* rrdfamily now uses DICTIONARY
* rrdvar using dictionary instead of AVL
* allocate the right size to rrdvar flag members
* rrdhost remaining char * members to STRING *
* better error handling on indexing
* strings now use a read/write lock to allow parallel searches to the index
* removed AVL support from dictionaries; implemented STRING with native Judy calls
* string releases should be negative
* only 31 bits are allowed for enum flags
* proper locking on strings
* string threading unittest and fixes
* fix lgtm finding
* fixed naming
* stream chart/dimension definitions at the beginning of a streaming session
* thread stack variable is undefined on thread cancel
* rrdcontext garbage collect per host on startup
* worker control in garbage collection
* relaxed deletion of rrdmetrics
* type checking on dictfe
* netdata chart to monitor rrdcontext triggers
* Group chart label updates
* rrdcontext better handling of collected rrdsets
* rrdpush incremental transmition of definitions should use as much buffer as possible
* require 1MB per chart
* empty the sender buffer before enabling metrics streaming
* fill up to 50% of buffer
* reset signaling metrics sending
* use the shared variable for status
* use separate host flag for enabling streaming of metrics
* make sure the flag is clear
* add logging for streaming
* add logging for streaming on buffer overflow
* circular_buffer proper sizing
* removed obsolete logs
* do not execute worker jobs if not necessary
* better messages about compression disabling
* proper use of flags and updating rrdset last access time every time the obsoletion flag is flipped
* monitor stream sender used buffer ratio
* Update exporting unit tests
* no need to compare label value with strcmp
* streaming send workers now monitor bandwidth
* workers now use strings
* streaming receiver monitors incoming bandwidth
* parser shift of worker ids
* minor fixes
* Group chart label updates
* Populate context with dimensions that have data
* Fix chart id
* better shift of parser worker ids
* fix for streaming compression
* properly count received bytes
* ensure LZ4 compression ring buffer does not wrap prematurely
* do not stream empty charts; do not process empty instances in rrdcontext
* need_to_send_chart_definition() does not need an rrdset lock any more
* rrdcontext objects are collected, after data have been written to the db
* better logging of RRDCONTEXT transitions
* always set all variables needed by the worker utilization charts
* implemented double linked list for most objects; eliminated alarm indexes from rrdhost; and many more fixes
* lockless strings design - string_dup() and string_freez() are totally lockless when they dont need to touch Judy - only Judy is protected with a read/write lock
* STRING code re-organization for clarity
* thread_cache improvements; double numbers precision on worker threads
* STRING_ENTRY now shadown STRING, so no duplicate definition is required; string_length() renamed to string_strlen() to follow the paradigm of all other functions, STRING internal statistics are now only compiled with NETDATA_INTERNAL_CHECKS
* rrdhost index by hostname now cleans up; aclk queries of archieved hosts do not index hosts
* Add index to speed up database context searches
* Removed last_updated optimization (was also buggy after latest merge with master)
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* netdata doubles
* fix cmocka test
* fix cmocka test again
* fix left-overs of long double to NETDATA_DOUBLE
* RRDDIM detached from disk representation; db settings in [db] section of netdata.conf
* update the memory before saving
* rrdset is now detached from file structures too
* on memory mode map, update the memory mapped structures on every iteration
* allow RRD_ID_LENGTH_MAX to be changed
* granularity secs, back to update every
* fix formatting
* more formatting
Before:
```
struct foobar {
avl avl;
...
}
```
After:
```
struct foobar {
avl_t avl;
...
};
```
Which makes figuring out the type from field name easier.
Initial work on host labels from the dedicated branch. Includes work for issues #7096, #7400, #7411, #7369, #7410, #7458, #7459, #7412 and #7408 by @vlvkobal, @thiagoftsm, @cakrit and @amoss.
* configure.ac: Add support for GCC's __attribute__((unused))
When compiling under GCC with -Wextra (along with -Wall) there are a lot
of compiler warnings such as
collectors/cgroups.plugin/cgroup-network.c:89:45: warning: unused parameter ‘scope’ [-Wunused-parameter]
89 | struct iface *read_proc_net_dev(const char *scope, const char *prefix) {
| ~~~~~~~~~~~~^~~~~
Some arguments may be able to be got rid off, others won't.
GCC (and at least clang[0]) provide an __attribute__((unused))
annotation that can be used on function parameters (also on variables,
functions, labels, enums, structs etc) to inform the compiler of such
and will squash warnings of the above nature.
A check is added to configure.ac for the use of GCC (I believe $GCC will
be set to 'yes' for clang also) and if found it creates __always_unused
& __maybe_unused #define's set to __attribute__((unused)) otherwise it
just sets them empty.
If other compilers have a similar feature this check can be adjusted to
accommodate them.
The reason for the two defines is that some variables may always be
unused in a function, others may or may not depending on #ifdef's for
example. So we are able to document both cases.
Subsequent commits will start making use of these to squash such
compiler warnings.
[0]: https://clang.llvm.org/docs/AttributeReference.html#maybe-unused-unused
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* collectors/statsd.plugin: Mark a function argument as __maybe_unused
In collectors/statsd.plugin/statsd.c the app function argument to
STATSD_APP_CHART_DIM() might be unused if NETDATA_INTERNAL_CHECKS is not
defined, then the debug() macro that it's used in from
libnetdata/log/log.h will be defined to a dummy function where none of
the arguments are used.
This fixes a compiler warning [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* collectors/apps.plugin: Mark a function argument as __maybe_unused
In collectors/apps.plugin/apps_plugin.c the function
debug_print_process_tree() takes an argument 'msg' that might be unused
if NETDATA_INTERNAL_CHECKS is not defined, then debug_log() will be set
to a dummy function that takes no arguments.
This fixes a compiler warning [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* libnetdata/locks/locks: Mark function arguments as __maybe_unused
In libnetdata/locks/locks.c there a number of functions that take
arguments 'file', 'function' & 'line' that might be unused if
NETDATA_INTERNAL_CHECKS is not defined, then the debug() macro that it's
used in from libnetdata/log/log.h will be defined to a dummy function
where none of the arguments are used.
This fixes compiler warnings [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* libnetdata/socket/security: Mark a function argument as __maybe_unused
In libnetdata/socket/security.c the function security_info_callback()
takes an argument 'ret' that might be unused if NETDATA_INTERNAL_CHECKS
is not defined, then the debug() macro that it's used in from
libnetdata/log/log.h will be defined to a dummy function where none of
the arguments are used.
This fixes a compiler warning [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* collectors/cgroups.plugin: Mark a function argument as __maybe_unused
In collectors/cgroups.plugin/cgroup-network.c the function
read_proc_net_dev() takes an argument 'scope' that might be unused if
the NETDATA_INTERNAL_CHECKS is not defined.
This fixes a compiler warning [-Wunused-parameter] when compiling with
-Wextra *and* -Wall".
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* database/rrddim: Mark function arguments as __maybe_unused
In database/rrddim.c there a couple of functions that take a 'st'
argument that might be unused if NETDATA_INTERNAL_CHECKS is not defined,
then the debug() macro that it's used in from libnetdata/log/log.h will
be defined to a dummy function where none of the arguments are used.
This fixes compiler warnings [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
* database/rrdvar: Mark a function argument as __maybe_unused
In database/rrdvar.c the function rrdvar_create_and_index() take an
argument 'scope' that might be unused if NETDATA_INTERNAL_CHECKS is not
defined, then the debug() macro that it's used in from
libnetdata/log/log.h will be defined to a dummy function where none of
the arguments are used.
This fixes a compiler warning [-Wunused-parameter] when compiling with
-Wextra *and* -Wall.
Signed-off-by: Andrew Clayton <andrew@zeta.digital-domain.net>
##### Summary
fixes#2673fixes#2149fixes#5017fixes#3830fixes#3187fixes#5154
Implements a command API for health which will accept commands via a socket to selectively suppress health checks.
Allows different ports to accept different request types (streaming, dashboard, api, registry, netdata.conf, badges, management)
Removes support for multi-threaded and single-threaded web servers.
##### Component Name
health, daemon
