archive/netdata_netdata
0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-03 21:15:36 +00:00
Code Issues Projects Releases Wiki Activity

RRD structures managed by dictionaries ()

Browse source 
* rrdset - in progress

* rrdset optimal constructor; rrdset conflict

* rrdset final touches

* re-organization of rrdset object members

* prevent use-after-free

* dictionary dfe supports also counting of iterations

* rrddim managed by dictionary

* rrd.h cleanup

* DICTIONARY_ITEM now is referencing actual dictionary items in the code

* removed rrdset linked list

* Revert "removed rrdset linked list"

This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.

* removed rrdset linked list

* added comments

* Switch chart uuid to static allocation in rrdset
Remove unused functions

* rrdset_archive() and friends...

* always create rrdfamily

* enable ml_free_dimension

* rrddim_foreach done with dfe

* most custom rrddim loops replaced with rrddim_foreach

* removed accesses to rrddim->dimensions

* removed locks that are no longer needed

* rrdsetvar is now managed by the dictionary

* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853

* conflict callback of rrdsetvar now properly checks if it has to reset the variable

* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM

* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe

* dictionary walkthrough callbacks get dictionary acquired items

* dictionary reference counters that can be dupped from zero

* added advanced functions for get and del

* rrdvar managed by dictionaries

* thread safety for rrdsetvar

* faster rrdvar initialization

* rrdvar string lengths should match in all add, del, get functions

* rrdvar internals hidden from the rest of the world

* rrdvar is now acquired throughout netdata

* hide the internal structures of rrdsetvar

* rrdsetvar is now acquired through out netdata

* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata

* better error handling

* dont create variables if not initialized for health

* dont create variables if not initialized for health again

* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items

* type checking on acquired objects

* rrdcalc renaming of functions

* type checking for rrdfamily_acquired

* rrdcalc managed by dictionaries

* rrdcalc double free fix

* host rrdvars is always needed

* attempt to fix deadlock 1

* attempt to fix deadlock 2

* Remove unused variable

* attempt to fix deadlock 3

* snprintfz

* rrdcalc index in rrdset fix

* Stop storing active charts and computing chart hashes

* Remove store active chart function

* Remove compute chart hash function

* Remove sql_store_chart_hash function

* Remove store_active_dimension function

* dictionary delayed destruction

* formatting and cleanup

* zero dictionary base on rrdsetvar

* added internal error to log delayed destructions of dictionaries

* typo in rrddimvar

* added debugging info to dictionary

* debug info

* fix for rrdcalc keys being empty

* remove forgotten unlock

* remove deadlock

* Switch to metadata version 5 and drop
  chart_hash
  chart_hash_map
  chart_active
  dimension_active
  v_chart_hash

* SQL cosmetic changes

* do not busy wait while destroying a referenced dictionary

* remove deadlock

* code cleanup; re-organization;

* fast cleanup and flushing of dictionaries

* number formatting fixes

* do not delete configured alerts when archiving a chart

* rrddim obsolete linked list management outside dictionaries

* removed duplicate contexts call

* fix crash when rrdfamily is not initialized

* dont keep rrddimvar referenced

* properly cleanup rrdvar

* removed some locks

* Do not attempt to cleanup chart_hash / chart_hash_map

* rrdcalctemplate managed by dictionary

* register callbacks on the right dictionary

* removed some more locks

* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread

* when looking up for an alarm look using both chart id and chart name

* host initialization a bit more modular

* init rrdlabels on host update

* preparation for dictionary views

* improved comment

* unused variables without internal checks

* service threads isolation and worker info

* more worker info in service thread

* thread cancelability debugging with internal checks

* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647

* dictionary modularization

* Remove unused SQL statement definition

* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated

* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops

* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/proc.plugin/proc_net_dev.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* fix memory leak in rrdset cache_dir

* minor dictionary changes

* dont use index locks in single threaded

* obsolete dict option

* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;

* fix jump on uninitialized value in dictionary; remove double free of cache_dir

* addressed codacy findings

* removed debugging code

* use the private refcount on dictionaries

* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;

* more dictionary statistics

* global statistics about dictionary operations, memory, items, callbacks

* dictionary support for views - missing the public API

* removed warning about unused parameter

* chart and context name for cloud

* chart and context name for cloud, again

* dictionary statistics fixed; first implementation of dictionary views - not currently used

* only the master can globally delete an item

* context needs netdata prefix

* fix context and chart it of spins

* fix for host variables when health is not enabled

* run garbage collector on item insert too

* Fix info message; remove extra "using"

* update dict unittest for new placement of garbage collector

* we need RRDHOST->rrdvars for maintaining custom host variables

* Health initialization needs the host->host_uuid

* split STRING to its own files; no code changes other than that

* initialize health unconditionally

* unit tests do not pollute the global scope with their variables

* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
This commit is contained in:
Costa Tsaousis 2022-09-19 23:46:13 +03:00 • committed by GitHub
parent 6224602916
commit cb7af25c09
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
106 changed files with 7106 additions and 5280 deletions
CMakeLists.txtMakefile.am
collectors
cgroups.plugin
sys_fs_cgroup.c
tests
test_doubles.c
cups.plugin
cups_plugin.c
diskspace.plugin
plugin_diskspace.c
plugins.d
pluginsd_parser.c
proc.plugin
ipc.cproc_diskstats.cproc_interrupts.cproc_loadavg.cproc_mdstat.cproc_net_dev.cproc_net_snmp.cproc_net_sockstat.cproc_net_stat_conntrack.cproc_net_wireless.cproc_self_mountinfo.cproc_softirqs.cproc_spl_kstat_zfs.cproc_stat.csys_block_zram.csys_class_infiniband.csys_class_power_supply.csys_fs_btrfs.c
statsd.plugin
statsd.c
tc.plugin
plugin_tc.c
configure.ac
daemon
analytics.cglobal_statistics.cmain.cservice.cunit_test.c
database
engine
journalfile.crrdengineapi.c
ram
rrddim_mem.c
rrd.hrrdcalc.crrdcalc.hrrdcalctemplate.crrdcalctemplate.hrrdcontext.crrdcontext.hrrddim.crrddimvar.crrddimvar.hrrdfamily.crrdhost.crrdlabels.crrdset.crrdsetvar.crrdsetvar.hrrdvar.crrdvar.h
sqlite
sqlite_aclk.csqlite_aclk_chart.csqlite_aclk_node.csqlite_db_migration.csqlite_functions.csqlite_functions.hsqlite_health.c
exporting
json
json.c
process_data.c
prometheus
prometheus.c
remote_write
remote_write.c
tests
netdata_doubles.c
health
health.chealth.hhealth_config.chealth_json.chealth_log.c
libnetdata
Makefile.am
dictionary
README.mddictionary.cdictionary.h
libnetdata.h
locks
README.md
log
log.c
required_dummies.h
string
Makefile.amREADME.mdstring.cstring.h
threads
threads.cthreads.h
ml
Dimension.h
registry
registry.cregistry_db.cregistry_init.cregistry_internals.hregistry_machine.c
streaming
rrdpush.crrdpush.hsender.c
tests/profile
benchmark-dictionary.c
web/api
badges
web_buffer_svg.c
exporters/shell
allmetrics_shell.c
formatters
charts2json.cjson_wrapper.crrd2json.c
Show more

31
CMakeLists.txt
View file

@ -220,7 +220,11 @@ pkg_check_modules(CURL libcurl)
# -----------------------------------------------------------------------------
# Detect libcups
pkg_check_modules(CURL libcups)
pkg_check_modules(CUPS libcups)
IF(NOT CUPS_LIBRARIES)
pkg_check_modules(CUPS cups)
ENDIF()
# later we use:
# ${CUPS_LIBRARIES}
# ${CUPS_CFLAGS_OTHER}
@ -446,6 +450,8 @@ set(LIBNETDATA_FILES
libnetdata/avl/avl.h
libnetdata/buffer/buffer.c
libnetdata/buffer/buffer.h
libnetdata/circular_buffer/circular_buffer.c
libnetdata/circular_buffer/circular_buffer.h
libnetdata/clocks/clocks.c
libnetdata/clocks/clocks.h
libnetdata/completion/completion.c
@ -454,10 +460,15 @@ set(LIBNETDATA_FILES
libnetdata/dictionary/dictionary.h
libnetdata/eval/eval.c
libnetdata/eval/eval.h
libnetdata/health/health.c
libnetdata/health/health.h
libnetdata/inlined.h
libnetdata/json/json.c
libnetdata/json/json.h
libnetdata/json/jsmn.c
libnetdata/json/jsmn.h
libnetdata/libnetdata.c
libnetdata/libnetdata.h
libnetdata/required_dummies.h
libnetdata/locks/locks.c
libnetdata/locks/locks.h
libnetdata/log/log.c
@ -470,6 +481,9 @@ set(LIBNETDATA_FILES
libnetdata/popen/popen.h
libnetdata/procfile/procfile.c
libnetdata/procfile/procfile.h
libnetdata/required_dummies.h
libnetdata/socket/security.c
libnetdata/socket/security.h
libnetdata/simple_pattern/simple_pattern.c
libnetdata/simple_pattern/simple_pattern.h
libnetdata/socket/socket.c
@ -478,23 +492,16 @@ set(LIBNETDATA_FILES
libnetdata/statistical/statistical.h
libnetdata/storage_number/storage_number.c
libnetdata/storage_number/storage_number.h
libnetdata/string/string.c
libnetdata/string/string.h
libnetdata/threads/threads.c
libnetdata/threads/threads.h
libnetdata/url/url.c
libnetdata/url/url.h
libnetdata/json/json.c
libnetdata/json/json.h
libnetdata/json/jsmn.c
libnetdata/json/jsmn.h
libnetdata/health/health.c
libnetdata/health/health.h
libnetdata/string/utf8.h
libnetdata/socket/security.c
libnetdata/socket/security.h
libnetdata/worker_utilization/worker_utilization.c
libnetdata/worker_utilization/worker_utilization.h
libnetdata/circular_buffer/circular_buffer.c
libnetdata/circular_buffer/circular_buffer.h)
)
IF(ENABLE_PLUGIN_EBPF)
list(APPEND LIBNETDATA_FILES

2
Makefile.am
View file

@ -173,6 +173,8 @@ LIBNETDATA_FILES = \
libnetdata/socket/security.h \
libnetdata/statistical/statistical.c \
libnetdata/statistical/statistical.h \
libnetdata/string/string.c \
libnetdata/string/string.h \
libnetdata/storage_number/storage_number.c \
libnetdata/storage_number/storage_number.h \
libnetdata/threads/threads.c \

23
collectors/cgroups.plugin/sys_fs_cgroup.c
View file

@ -857,16 +857,16 @@ struct cgroup {
char *filename_cpu_cfs_quota;
unsigned long long cpu_cfs_quota;
RRDSETVAR *chart_var_cpu_limit;
const RRDSETVAR_ACQUIRED *chart_var_cpu_limit;
NETDATA_DOUBLE prev_cpu_usage;
char *filename_memory_limit;
unsigned long long memory_limit;
RRDSETVAR *chart_var_memory_limit;
const RRDSETVAR_ACQUIRED *chart_var_memory_limit;
char *filename_memoryswap_limit;
unsigned long long memoryswap_limit;
RRDSETVAR *chart_var_memoryswap_limit;
const RRDSETVAR_ACQUIRED *chart_var_memoryswap_limit;
// services
RRDDIM *rd_cpu;
@ -3708,10 +3708,10 @@ cpu_limits2_err:
}
}
static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, unsigned long long *value, const char *chart_var_name, struct cgroup *cg) {
static inline int update_memory_limits(char **filename, const RRDSETVAR_ACQUIRED **chart_var, unsigned long long *value, const char *chart_var_name, struct cgroup *cg) {
if(*filename) {
if(unlikely(!*chart_var)) {
*chart_var = rrdsetvar_custom_chart_variable_create(cg->st_mem_usage, chart_var_name);
*chart_var = rrdsetvar_custom_chart_variable_add_and_acquire(cg->st_mem_usage, chart_var_name);
if(!*chart_var) {
error("Cannot create cgroup %s chart variable '%s'. Will not update its limit anymore.", cg->id, chart_var_name);
freez(*filename);
@ -3727,7 +3727,7 @@ static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, u
*filename = NULL;
}
else {
rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
return 1;
}
} else {
@ -3742,11 +3742,11 @@ static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, u
char *s = "max\n\0";
if(strcmp(s, buffer) == 0){
*value = UINT64_MAX;
rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
return 1;
}
*value = str2ull(buffer);
rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
return 1;
}
}
@ -3843,7 +3843,7 @@ void update_cgroup_charts(int update_every) {
}
if(unlikely(!cg->chart_var_cpu_limit)) {
cg->chart_var_cpu_limit = rrdsetvar_custom_chart_variable_create(cg->st_cpu, "cpu_limit");
cg->chart_var_cpu_limit = rrdsetvar_custom_chart_variable_add_and_acquire(cg->st_cpu, "cpu_limit");
if(!cg->chart_var_cpu_limit) {
error("Cannot create cgroup %s chart variable 'cpu_limit'. Will not update its limit anymore.", cg->id);
if(cg->filename_cpuset_cpus) freez(cg->filename_cpuset_cpus);
@ -3868,8 +3868,6 @@ void update_cgroup_charts(int update_every) {
value = (NETDATA_DOUBLE)cg->cpuset_cpus * 100;
}
if(likely(value)) {
rrdsetvar_custom_chart_variable_set(cg->chart_var_cpu_limit, value);
if(unlikely(!cg->st_cpu_limit)) {
snprintfz(title, CHART_TITLE_MAX, "CPU Usage within the limits");
@ -3909,14 +3907,15 @@ void update_cgroup_charts(int update_every) {
cg->prev_cpu_usage = cpu_usage;
rrdsetvar_custom_chart_variable_set(cg->st_cpu, cg->chart_var_cpu_limit, value);
rrdset_done(cg->st_cpu_limit);
}
else {
rrdsetvar_custom_chart_variable_set(cg->chart_var_cpu_limit, NAN);
if(unlikely(cg->st_cpu_limit)) {
rrdset_is_obsolete(cg->st_cpu_limit);
cg->st_cpu_limit = NULL;
}
rrdsetvar_custom_chart_variable_set(cg->st_cpu, cg->chart_var_cpu_limit, NAN);
}
}
}

8
collectors/cgroups.plugin/tests/test_doubles.c
View file

@ -146,14 +146,6 @@ void netdev_rename_device_del(const char *host_device)
UNUSED(host_device);
}
void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
{
UNUSED(chart_uuid);
UNUSED(source_type);
UNUSED(label);
UNUSED(value);
}
void rrdcalc_update_rrdlabels(RRDSET *st) {
(void)st;
}

13
collectors/cups.plugin/cups_plugin.c
View file

@ -137,10 +137,7 @@ getIntegerOption(
return ((int)intvalue);
}
static int reset_job_metrics(const char *name, void *entry, void *data) {
(void)name;
(void)data;
static int reset_job_metrics(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data __maybe_unused) {
struct job_metrics *jm = (struct job_metrics *)entry;
jm->is_collected = 0;
@ -175,8 +172,8 @@ struct job_metrics *get_job_metrics(char *dest) {
return jm;
}
int collect_job_metrics(const char *name, void *entry, void *data) {
(void)data;
int collect_job_metrics(const DICTIONARY_ITEM *item, void *entry, void *data __maybe_unused) {
const char *name = dictionary_acquired_item_name(item);
struct job_metrics *jm = (struct job_metrics *)entry;
@ -205,7 +202,7 @@ int collect_job_metrics(const char *name, void *entry, void *data) {
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
dictionary_del_having_write_lock(dict_dest_job_metrics, name);
dictionary_del(dict_dest_job_metrics, name);
}
return 0;
@ -243,7 +240,7 @@ int main(int argc, char **argv) {
errno = 0;
dict_dest_job_metrics = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
dict_dest_job_metrics = dictionary_create(DICT_OPTION_SINGLE_THREADED);
// ------------------------------------------------------------------------
// the main loop

6
collectors/diskspace.plugin/plugin_diskspace.c
View file

@ -95,8 +95,8 @@ int mount_point_cleanup(const char *name, void *entry, int slow) {
return 0;
}
int mount_point_cleanup_cb(const char *name, void *entry, void *data) {
UNUSED(data);
int mount_point_cleanup_cb(const DICTIONARY_ITEM *item, void *entry, void *data __maybe_unused) {
const char *name = dictionary_acquired_item_name(item);
return mount_point_cleanup(name, (struct mount_point_metadata *)entry, 0);
}
@ -330,7 +330,7 @@ static inline void do_disk_space_stats(struct mountinfo *mi, int update_every) {
, SIMPLE_PATTERN_EXACT
);
dict_mountpoints = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
dict_mountpoints = dictionary_create(DICT_OPTION_SINGLE_THREADED);
}
struct mount_point_metadata *m = dictionary_get(dict_mountpoints, mi->mount_point);

26
collectors/plugins.d/pluginsd_parser.c
View file

@ -101,15 +101,19 @@ PARSER_RC pluginsd_variable_action(void *user, RRDHOST *host, RRDSET *st, char *
UNUSED(user);
if (global) {
RRDVAR *rv = rrdvar_custom_host_variable_create(host, name);
if (rv)
rrdvar_custom_host_variable_set(host, rv, value);
const RRDVAR_ACQUIRED *rva = rrdvar_custom_host_variable_add_and_acquire(host, name);
if (rva) {
rrdvar_custom_host_variable_set(host, rva, value);
rrdvar_custom_host_variable_release(host, rva);
}
else
error("cannot find/create HOST VARIABLE '%s' on host '%s'", name, rrdhost_hostname(host));
} else {
RRDSETVAR *rs = rrdsetvar_custom_chart_variable_create(st, name);
if (rs)
rrdsetvar_custom_chart_variable_set(rs, value);
const RRDSETVAR_ACQUIRED *rsa = rrdsetvar_custom_chart_variable_add_and_acquire(st, name);
if (rsa) {
rrdsetvar_custom_chart_variable_set(st, rsa, value);
rrdsetvar_custom_chart_variable_release(st, rsa);
}
else
error("cannot find/create CHART VARIABLE '%s' on host '%s', chart '%s'", name, rrdhost_hostname(host), rrdset_id(st));
}
@ -127,7 +131,7 @@ PARSER_RC pluginsd_dimension_action(void *user, RRDSET *st, char *id, char *name
RRDDIM *rd = rrddim_add(st, id, name, multiplier, divisor, algorithm_type);
int unhide_dimension = 1;
rrddim_flag_clear(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
rrddim_option_clear(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
if (options && *options) {
if (strstr(options, "obsolete") != NULL)
rrddim_is_obsolete(st, rd);
@ -137,20 +141,20 @@ PARSER_RC pluginsd_dimension_action(void *user, RRDSET *st, char *id, char *name
unhide_dimension = !strstr(options, "hidden");
if (strstr(options, "noreset") != NULL)
rrddim_flag_set(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
rrddim_option_set(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
if (strstr(options, "nooverflow") != NULL)
rrddim_flag_set(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
rrddim_option_set(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
} else
rrddim_isnot_obsolete(st, rd);
if (likely(unhide_dimension)) {
rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
(void)sql_set_dimension_option(&rd->metric_uuid, NULL);
rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
}
} else {
rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
(void)sql_set_dimension_option(&rd->metric_uuid, "hidden");
rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);

12
collectors/proc.plugin/ipc.c
View file

@ -281,7 +281,7 @@ int do_ipc(int update_every, usec_t dt) {
static int read_limits_next = -1;
static struct ipc_limits limits;
static struct ipc_status status;
static RRDVAR *arrays_max = NULL, *semaphores_max = NULL;
static const RRDVAR_ACQUIRED *arrays_max = NULL, *semaphores_max = NULL;
static RRDSET *st_semaphores = NULL, *st_arrays = NULL;
static RRDDIM *rd_semaphores = NULL, *rd_arrays = NULL;
static char *msg_filename = NULL;
@ -352,8 +352,8 @@ int do_ipc(int update_every, usec_t dt) {
}
// variables
semaphores_max = rrdvar_custom_host_variable_create(localhost, "ipc_semaphores_max");
arrays_max = rrdvar_custom_host_variable_create(localhost, "ipc_semaphores_arrays_max");
semaphores_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "ipc_semaphores_max");
arrays_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "ipc_semaphores_arrays_max");
}
struct stat stbuf;
@ -483,11 +483,7 @@ int do_ipc(int update_every, usec_t dt) {
rrdset_done(st_msq_messages);
rrdset_done(st_msq_bytes);
long long dimensions_num = 0;
RRDDIM *rd;
rrdset_rdlock(st_msq_messages);
rrddim_foreach_read(rd, st_msq_messages) dimensions_num++;
rrdset_unlock(st_msq_messages);
long long dimensions_num = rrdset_number_of_dimensions(st_msq_messages);
if(unlikely(dimensions_num > dimensions_limit)) {
info("Message queue statistics has been disabled");

2
collectors/proc.plugin/proc_diskstats.c
View file

@ -874,8 +874,6 @@ static void add_labels_to_disk(struct disk *d, RRDSET *st) {
rrdlabels_add(st->rrdlabels, "device_type", "virtual", RRDLABEL_SRC_AUTO);
break;
}
rrdcalc_update_rrdlabels(st);
}
int do_proc_diskstats(int update_every, usec_t dt) {

4
collectors/proc.plugin/proc_interrupts.c
View file

@ -175,7 +175,7 @@ int do_proc_interrupts(int update_every, usec_t dt) {
// calls of this function.
if(unlikely(!irr->rd || strncmp(rrddim_name(irr->rd), irr->name, MAX_INTERRUPT_NAME) != 0)) {
irr->rd = rrddim_add(st_system_interrupts, irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrddim_set_name(st_system_interrupts, irr->rd, irr->name);
rrddim_reset_name(st_system_interrupts, irr->rd, irr->name);
// also reset per cpu RRDDIMs to avoid repeating strncmp() in the per core loop
if(likely(do_per_core != CONFIG_BOOLEAN_NO)) {
@ -237,7 +237,7 @@ int do_proc_interrupts(int update_every, usec_t dt) {
if(irr->used && (do_per_core == CONFIG_BOOLEAN_YES || irr->cpu[c].value)) {
if(unlikely(!irr->cpu[c].rd)) {
irr->cpu[c].rd = rrddim_add(core_st[c], irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrddim_set_name(core_st[c], irr->cpu[c].rd, irr->name);
rrddim_reset_name(core_st[c], irr->cpu[c].rd, irr->name);
}
rrddim_set_by_pointer(core_st[c], irr->cpu[c].rd, irr->cpu[c].value);

6
collectors/proc.plugin/proc_loadavg.c
View file

@ -99,7 +99,7 @@ int do_proc_loadavg(int update_every, usec_t dt) {
if(likely(do_all_processes)) {
static RRDSET *processes_chart = NULL;
static RRDDIM *rd_active = NULL;
static RRDSETVAR *rd_pidmax;
static const RRDSETVAR_ACQUIRED *rd_pidmax;
if(unlikely(!processes_chart)) {
processes_chart = rrdset_create_localhost(
@ -118,12 +118,12 @@ int do_proc_loadavg(int update_every, usec_t dt) {
);
rd_active = rrddim_add(processes_chart, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
rd_pidmax = rrdsetvar_custom_chart_variable_create(processes_chart, "pidmax");
rd_pidmax = rrdsetvar_custom_chart_variable_add_and_acquire(processes_chart, "pidmax");
}
else rrdset_next(processes_chart);
rrddim_set_by_pointer(processes_chart, rd_active, active_processes);
rrdsetvar_custom_chart_variable_set(rd_pidmax, max_processes);
rrdsetvar_custom_chart_variable_set(processes_chart, rd_pidmax, max_processes);
rrdset_done(processes_chart);
}

1
collectors/proc.plugin/proc_mdstat.c
View file

@ -80,7 +80,6 @@ static inline void make_chart_obsolete(char *name, const char *id_modifier)
static void add_labels_to_mdstat(struct raid *raid, RRDSET *st) {
rrdlabels_add(st->rrdlabels, "device", raid->name, RRDLABEL_SRC_AUTO);
rrdlabels_add(st->rrdlabels, "raid_level", raid->level, RRDLABEL_SRC_AUTO);
rrdcalc_update_rrdlabels(st);
}
int do_proc_mdstat(int update_every, usec_t dt)

9
collectors/proc.plugin/proc_net_dev.c
View file

@ -188,7 +188,7 @@ static struct netdev {
RRDDIM *rd_mtu;
char *filename_speed;
RRDSETVAR *chart_var_speed;
const RRDSETVAR_ACQUIRED *chart_var_speed;
char *filename_duplex;
char *filename_operstate;
@ -967,7 +967,8 @@ int do_proc_net_dev(int update_every, usec_t dt) {
// update the interface speed
if(d->filename_speed) {
if(unlikely(!d->chart_var_speed)) {
d->chart_var_speed = rrdsetvar_custom_chart_variable_create(d->st_bandwidth, "nic_speed_max");
d->chart_var_speed =
rrdsetvar_custom_chart_variable_add_and_acquire(d->st_bandwidth, "nic_speed_max");
if(!d->chart_var_speed) {
error("Cannot create interface %s chart variable 'nic_speed_max'. Will not update its speed anymore.", d->name);
freez(d->filename_speed);
@ -990,8 +991,6 @@ int do_proc_net_dev(int update_every, usec_t dt) {
d->filename_speed = NULL;
}
else {
rrdsetvar_custom_chart_variable_set(d->chart_var_speed, (NETDATA_DOUBLE) d->speed * KILOBITS_IN_A_MEGABIT);
if(d->do_speed != CONFIG_BOOLEAN_NO) {
if(unlikely(!d->st_speed)) {
d->st_speed = rrdset_create_localhost(
@ -1020,6 +1019,8 @@ int do_proc_net_dev(int update_every, usec_t dt) {
rrddim_set_by_pointer(d->st_speed, d->rd_speed, (collected_number)d->speed * KILOBITS_IN_A_MEGABIT);
rrdset_done(d->st_speed);
}
rrdsetvar_custom_chart_variable_set(d->st_bandwidth, d->chart_var_speed, (NETDATA_DOUBLE) d->speed * KILOBITS_IN_A_MEGABIT);
}
}
}

4
collectors/proc.plugin/proc_net_snmp.c
View file

@ -104,7 +104,7 @@ int do_proc_net_snmp(int update_every, usec_t dt) {
*arl_udp = NULL,
*arl_udplite = NULL;
static RRDVAR *tcp_max_connections_var = NULL;
static const RRDVAR_ACQUIRED *tcp_max_connections_var = NULL;
if(unlikely(!arl_ip)) {
do_ip_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp", "ipv4 packets", CONFIG_BOOLEAN_AUTO);
@ -216,7 +216,7 @@ int do_proc_net_snmp(int update_every, usec_t dt) {
arl_expect(arl_udplite, "InCsumErrors", &snmp_root.udplite_InCsumErrors);
arl_expect(arl_udplite, "IgnoredMulti", &snmp_root.udplite_IgnoredMulti);
tcp_max_connections_var = rrdvar_custom_host_variable_create(localhost, "tcp_max_connections");
tcp_max_connections_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_max_connections");
}
if(unlikely(!ff)) {

12
collectors/proc.plugin/proc_net_sockstat.c
View file

@ -27,14 +27,14 @@ static struct proc_net_sockstat {
static int read_tcp_mem(void) {
static char *filename = NULL;
static RRDVAR *tcp_mem_low_threshold = NULL,
static const RRDVAR_ACQUIRED *tcp_mem_low_threshold = NULL,
*tcp_mem_pressure_threshold = NULL,
*tcp_mem_high_threshold = NULL;
if(unlikely(!tcp_mem_low_threshold)) {
tcp_mem_low_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_low");
tcp_mem_pressure_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_pressure");
tcp_mem_high_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_high");
tcp_mem_low_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_low");
tcp_mem_pressure_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_pressure");
tcp_mem_high_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_high");
}
if(unlikely(!filename)) {
@ -69,7 +69,7 @@ static int read_tcp_mem(void) {
static kernel_uint_t read_tcp_max_orphans(void) {
static char *filename = NULL;
static RRDVAR *tcp_max_orphans_var = NULL;
static const RRDVAR_ACQUIRED *tcp_max_orphans_var = NULL;
if(unlikely(!filename)) {
char buffer[FILENAME_MAX + 1];
@ -81,7 +81,7 @@ static kernel_uint_t read_tcp_max_orphans(void) {
if(read_single_number_file(filename, &tcp_max_orphans) == 0) {
if(unlikely(!tcp_max_orphans_var))
tcp_max_orphans_var = rrdvar_custom_host_variable_create(localhost, "tcp_max_orphans");
tcp_max_orphans_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_max_orphans");
rrdvar_custom_host_variable_set(localhost, tcp_max_orphans_var, tcp_max_orphans);
return tcp_max_orphans;

4
collectors/proc.plugin/proc_net_stat_conntrack.c
View file

@ -12,7 +12,7 @@ int do_proc_net_stat_conntrack(int update_every, usec_t dt) {
static usec_t get_max_every = 10 * USEC_PER_SEC, usec_since_last_max = 0;
static int read_full = 1;
static char *nf_conntrack_filename, *nf_conntrack_count_filename, *nf_conntrack_max_filename;
static RRDVAR *rrdvar_max = NULL;
static const RRDVAR_ACQUIRED *rrdvar_max = NULL;
unsigned long long aentries = 0, asearched = 0, afound = 0, anew = 0, ainvalid = 0, aignore = 0, adelete = 0, adelete_list = 0,
ainsert = 0, ainsert_failed = 0, adrop = 0, aearly_drop = 0, aicmp_error = 0, aexpect_new = 0, aexpect_create = 0, aexpect_delete = 0, asearch_restart = 0;
@ -50,7 +50,7 @@ int do_proc_net_stat_conntrack(int update_every, usec_t dt) {
if(!do_sockets && !read_full)
return 1;
rrdvar_max = rrdvar_custom_host_variable_create(localhost, "netfilter_conntrack_max");
rrdvar_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "netfilter_conntrack_max");
}
if(likely(read_full)) {

1
collectors/proc.plugin/proc_net_wireless.c
View file

@ -200,7 +200,6 @@ static void configure_device(int do_status, int do_quality, int do_discarded_pac
static void add_labels_to_wireless(struct netwireless *w, RRDSET *st) {
rrdlabels_add(st->rrdlabels, "device", w->name, RRDLABEL_SRC_AUTO);
rrdcalc_update_rrdlabels(st);
}
int do_proc_net_wireless(int update_every, usec_t dt)

3
collectors/proc.plugin/proc_self_mountinfo.c
View file

@ -227,7 +227,8 @@ struct mountinfo *mountinfo_read(int do_statvfs) {
struct mountinfo *root = NULL, *last = NULL, *mi = NULL;
// create a dictionary to track uniqueness
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE);
DICTIONARY *dict = dictionary_create(
DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_NAME_LINK_DONT_CLONE);
unsigned long l, lines = procfile_lines(ff);
for(l = 0; l < lines ;l++) {

4
collectors/proc.plugin/proc_softirqs.c
View file

@ -155,7 +155,7 @@ int do_proc_softirqs(int update_every, usec_t dt) {
// calls of this function.
if(unlikely(!irr->rd || strncmp(irr->name, rrddim_name(irr->rd), MAX_INTERRUPT_NAME) != 0)) {
irr->rd = rrddim_add(st_system_softirqs, irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrddim_set_name(st_system_softirqs, irr->rd, irr->name);
rrddim_reset_name(st_system_softirqs, irr->rd, irr->name);
// also reset per cpu RRDDIMs to avoid repeating strncmp() in the per core loop
if(likely(do_per_core != CONFIG_BOOLEAN_NO)) {
@ -231,7 +231,7 @@ int do_proc_softirqs(int update_every, usec_t dt) {
if(irr->used && (do_per_core == CONFIG_BOOLEAN_YES || irr->cpu[c].value)) {
if(unlikely(!irr->cpu[c].rd)) {
irr->cpu[c].rd = rrddim_add(core_st[c], irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrddim_set_name(core_st[c], irr->cpu[c].rd, irr->name);
rrddim_reset_name(core_st[c], irr->cpu[c].rd, irr->name);
}
rrddim_set_by_pointer(core_st[c], irr->cpu[c].rd, irr->cpu[c].value);

6
collectors/proc.plugin/proc_spl_kstat_zfs.c
View file

@ -252,8 +252,8 @@ void disable_zfs_pool_state(struct zfs_pool *pool)
pool->disabled = 1;
}
int update_zfs_pool_state_chart(const char *name, void *pool_p, void *update_every_p)
{
int update_zfs_pool_state_chart(const DICTIONARY_ITEM *item, void *pool_p, void *update_every_p) {
const char *name = dictionary_acquired_item_name(item);
struct zfs_pool *pool = (struct zfs_pool *)pool_p;
int update_every = *(int *)update_every_p;
@ -321,7 +321,7 @@ int do_proc_spl_kstat_zfs_pool_state(int update_every, usec_t dt)
snprintfz(filename, FILENAME_MAX, "%s%s", netdata_configured_host_prefix, "/proc/spl/kstat/zfs");
dirname = config_get("plugin:proc:" ZFS_PROC_POOLS, "directory to monitor", filename);
zfs_pools = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
zfs_pools = dictionary_create(DICT_OPTION_SINGLE_THREADED);
do_zfs_pool_state = 1;
}

4
collectors/proc.plugin/proc_stat.c
View file

@ -481,7 +481,7 @@ int do_proc_stat(int update_every, usec_t dt) {
static uint32_t hash_intr, hash_ctxt, hash_processes, hash_procs_running, hash_procs_blocked;
static char *core_throttle_count_filename = NULL, *package_throttle_count_filename = NULL, *scaling_cur_freq_filename = NULL,
*time_in_state_filename = NULL, *schedstat_filename = NULL, *cpuidle_name_filename = NULL, *cpuidle_time_filename = NULL;
static RRDVAR *cpus_var = NULL;
static const RRDVAR_ACQUIRED *cpus_var = NULL;
static int accurate_freq_avail = 0, accurate_freq_is_used = 0;
size_t cores_found = (size_t)processors;
@ -713,7 +713,7 @@ int do_proc_stat(int update_every, usec_t dt) {
rrddim_hide(cpu_chart->st, "idle");
if(unlikely(core == 0 && cpus_var == NULL))
cpus_var = rrdvar_custom_host_variable_create(localhost, "active_processors");
cpus_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "active_processors");
}
else rrdset_next(cpu_chart->st);

13
collectors/proc.plugin/sys_block_zram.c
View file

@ -177,7 +177,7 @@ static void free_device(DICTIONARY *dict, const char *name)
rrdset_obsolete_and_pointer_null(d->st_savings);
rrdset_obsolete_and_pointer_null(d->st_alloc_efficiency);
rrdset_obsolete_and_pointer_null(d->st_comp_ratio);
dictionary_del_having_write_lock(dict, name);
dictionary_del(dict, name);
}
// --------------------------------------------------------------------
@ -239,13 +239,16 @@ static inline int _collect_zram_metrics(const char* name, ZRAM_DEVICE *d, int ad
return 0;
}
static int collect_first_zram_metrics(const char *name, void *entry, void *data) {
static int collect_first_zram_metrics(const DICTIONARY_ITEM *item, void *entry, void *data) {
const char *name = dictionary_acquired_item_name(item);
// collect without calling rrdset_next (init only)
return _collect_zram_metrics(name, (ZRAM_DEVICE *)entry, 0, (DICTIONARY *)data);
}
static int collect_zram_metrics(const char *name, void *entry, void *data) {
(void)name;
static int collect_zram_metrics(const DICTIONARY_ITEM *item, void *entry, void *data) {
const char *name = dictionary_acquired_item_name(item);
// collect with calling rrdset_next
return _collect_zram_metrics(name, (ZRAM_DEVICE *)entry, 1, (DICTIONARY *)data);
}
@ -280,7 +283,7 @@ int do_sys_block_zram(int update_every, usec_t dt) {
}
procfile_close(ff);
devices = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
devices = dictionary_create(DICT_OPTION_SINGLE_THREADED);
device_count = init_devices(devices, (unsigned int)zram_id, update_every);
if (device_count < 1)
return 1;

6
collectors/proc.plugin/sys_class_infiniband.c
View file

@ -184,7 +184,7 @@ static struct ibport {
RRDSET *st_hwpackets;
RRDSET *st_hwerrors;
RRDSETVAR *stv_speed;
const RRDSETVAR_ACQUIRED *stv_speed;
usec_t speed_last_collected_usec;
@ -543,7 +543,7 @@ int do_sys_class_infiniband(int update_every, usec_t dt)
// x4 lanes multiplier as per Documentation/ABI/stable/sysfs-class-infiniband
FOREACH_COUNTER_BYTES(GEN_RRD_DIM_ADD_CUSTOM, port, 4 * 8 * port->width, 1024, RRD_ALGORITHM_INCREMENTAL)
port->stv_speed = rrdsetvar_custom_chart_variable_create(port->st_bytes, "link_speed");
port->stv_speed = rrdsetvar_custom_chart_variable_add_and_acquire(port->st_bytes, "link_speed");
} else
rrdset_next(port->st_bytes);
@ -551,7 +551,7 @@ int do_sys_class_infiniband(int update_every, usec_t dt)
FOREACH_COUNTER_BYTES(GEN_RRD_DIM_SETP, port)
// For link speed set only variable
rrdsetvar_custom_chart_variable_set(port->stv_speed, port->speed);
rrdsetvar_custom_chart_variable_set(port->st_bytes, port->stv_speed, port->speed);
rrdset_done(port->st_bytes);
}

1
collectors/proc.plugin/sys_class_power_supply.c
View file

@ -114,7 +114,6 @@ void power_supply_free(struct power_supply *ps) {
static void add_labels_to_power_supply(struct power_supply *ps, RRDSET *st) {
rrdlabels_add(st->rrdlabels, "device", ps->name, RRDLABEL_SRC_AUTO);
rrdcalc_update_rrdlabels(st);
}
int do_sys_class_power_supply(int update_every, usec_t dt) {

1
collectors/proc.plugin/sys_fs_btrfs.c
View file

@ -451,7 +451,6 @@ static inline int find_all_btrfs_pools(const char *path) {
static void add_labels_to_btrfs(BTRFS_NODE *n, RRDSET *st) {
rrdlabels_add(st->rrdlabels, "device", n->id, RRDLABEL_SRC_AUTO);
rrdlabels_add(st->rrdlabels, "device_label", n->label, RRDLABEL_SRC_AUTO);
rrdcalc_update_rrdlabels(st);
}
int do_sys_fs_btrfs(int update_every, usec_t dt) {

53
collectors/statsd.plugin/statsd.c
View file

@ -24,9 +24,9 @@
#ifdef STATSD_MULTITHREADED
// DO NOT ENABLE MULTITHREADING - IT IS NOT WELL TESTED
#define STATSD_DICTIONARY_OPTIONS DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_ADD_IN_FRONT
#define STATSD_DICTIONARY_OPTIONS (DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_ADD_IN_FRONT)
#else
#define STATSD_DICTIONARY_OPTIONS DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_ADD_IN_FRONT|DICTIONARY_FLAG_SINGLE_THREADED
#define STATSD_DICTIONARY_OPTIONS (DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_ADD_IN_FRONT | DICT_OPTION_SINGLE_THREADED)
#endif
#define STATSD_DECIMAL_DETAIL 1000 // floating point values get multiplied by this, with the same divisor
@ -192,6 +192,7 @@ typedef struct statsd_app_chart_dimension {
collected_number multiplier; // the multiplier of the dimension
collected_number divisor; // the divisor of the dimension
RRDDIM_FLAGS flags; // the RRDDIM flags for this dimension
RRDDIM_OPTIONS options; // the RRDDIM options for this dimension
STATSD_APP_CHART_DIM_VALUE_TYPE value_type; // which value to use of the source metric
@ -371,9 +372,10 @@ static struct statsd {
// --------------------------------------------------------------------------------------------------------------------
// statsd index management - add/find metrics
static void dictionary_metric_insert_callback(const char *name, void *value, void *data) {
static void dictionary_metric_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
STATSD_INDEX *index = (STATSD_INDEX *)data;
STATSD_METRIC *m = (STATSD_METRIC *)value;
const char *name = dictionary_acquired_item_name(item);
debug(D_STATSD, "Creating new %s metric '%s'", index->name, name);
@ -390,9 +392,9 @@ static void dictionary_metric_insert_callback(const char *name, void *value, voi
__atomic_fetch_add(&index->metrics, 1, __ATOMIC_RELAXED);
}
static void dictionary_metric_delete_callback(const char *name, void *value, void *data) {
static void dictionary_metric_delete_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
(void)data; // STATSD_INDEX *index = (STATSD_INDEX *)data;
(void)name;
(void)item;
STATSD_METRIC *m = (STATSD_METRIC *)value;
if(m->type == STATSD_METRIC_TYPE_HISTOGRAM || m->type == STATSD_METRIC_TYPE_TIMER) {
@ -416,7 +418,7 @@ static inline STATSD_METRIC *statsd_find_or_add_metric(STATSD_INDEX *index, cons
// no locks here, go faster
// this will call the dictionary_metric_insert_callback() if an item
// is inserted, otherwise it will return the existing one.
// We used the flag DICTIONARY_FLAG_DONT_OVERWRITE_VALUE to support this.
// We used the flag DICT_OPTION_DONT_OVERWRITE_VALUE to support this.
STATSD_METRIC *m = dictionary_set(index->dict, name, NULL, sizeof(STATSD_METRIC));
#endif
@ -572,8 +574,8 @@ static inline void statsd_process_histogram_or_timer(STATSD_METRIC *m, const cha
#define statsd_process_timer(m, value, sampling) statsd_process_histogram_or_timer(m, value, sampling, "timer")
#define statsd_process_histogram(m, value, sampling) statsd_process_histogram_or_timer(m, value, sampling, "histogram")
static void dictionary_metric_set_value_insert_callback(const char *name, void *value, void *data) {
(void)name;
static void dictionary_metric_set_value_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
(void)item;
(void)value;
STATSD_METRIC *m = (STATSD_METRIC *)data;
m->set.unique++;
@ -617,8 +619,8 @@ static inline void statsd_process_set(STATSD_METRIC *m, const char *value) {
}
}
static void dictionary_metric_dict_value_insert_callback(const char *name, void *value, void *data) {
(void)name;
static void dictionary_metric_dict_value_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
(void)item;
(void)value;
STATSD_METRIC *m = (STATSD_METRIC *)data;
m->dictionary.unique++;
@ -1209,6 +1211,7 @@ static STATSD_APP_CHART_DIM *add_dimension_to_app_chart(
, collected_number multiplier
, collected_number divisor
, RRDDIM_FLAGS flags
, RRDDIM_OPTIONS options
, STATSD_APP_CHART_DIM_VALUE_TYPE value_type
) {
STATSD_APP_CHART_DIM *dim = callocz(sizeof(STATSD_APP_CHART_DIM), 1);
@ -1221,6 +1224,7 @@ static STATSD_APP_CHART_DIM *add_dimension_to_app_chart(
dim->divisor = divisor;
dim->value_type = value_type;
dim->flags = flags;
dim->options = options;
if(!dim->multiplier)
dim->multiplier = 1;
@ -1323,7 +1327,7 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
else if(app) {
if(!strcmp(s, "dictionary")) {
if(!app->dict)
app->dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
app->dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
dict = app->dict;
}
@ -1474,17 +1478,18 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
pattern = 1;
}
char *dim_name = words[i++];
char *type = words[i++];
char *multiplier = words[i++];
char *divisor = words[i++];
char *options = words[i++];
char *dim_name = words[i++];
char *type = words[i++];
char *multiplier = words[i++];
char *divisor = words[i++];
char *opts = words[i++];
RRDDIM_FLAGS flags = RRDDIM_FLAG_NONE;
if(options && *options) {
if(strstr(options, "hidden") != NULL) flags |= RRDDIM_FLAG_HIDDEN;
if(strstr(options, "noreset") != NULL) flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS;
if(strstr(options, "nooverflow") != NULL) flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS;
RRDDIM_OPTIONS options = RRDDIM_OPTION_NONE;
if(opts && *opts) {
if(strstr(opts, "hidden") != NULL) options |= RRDDIM_OPTION_HIDDEN;
if(strstr(opts, "noreset") != NULL) options |= RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS;
if(strstr(opts, "nooverflow") != NULL) options |= RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS;
}
if(!pattern) {
@ -1510,7 +1515,8 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
, (multiplier && *multiplier)?str2l(multiplier):1
, (divisor && *divisor)?str2l(divisor):1
, flags
, string2valuetype(type, line, filename)
,
options, string2valuetype(type, line, filename)
);
if(pattern)
@ -1775,7 +1781,7 @@ static inline void statsd_private_chart_dictionary(STATSD_METRIC *m) {
STATSD_METRIC_DICTIONARY_ITEM *t;
dfe_start_read(m->dictionary.dict, t) {
if (!t->rd) t->rd = rrddim_add(m->st, t_name, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
if (!t->rd) t->rd = rrddim_add(m->st, t_dfe.name, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrddim_set_by_pointer(m->st, t->rd, (collected_number)t->count);
}
dfe_done(t);
@ -2130,6 +2136,7 @@ static inline void check_if_metric_is_for_app(STATSD_INDEX *index, STATSD_METRIC
, dim->multiplier
, dim->divisor
, dim->flags
, dim->options
, dim->value_type
);
@ -2186,11 +2193,13 @@ static inline RRDDIM *statsd_add_dim_to_app_chart(STATSD_APP *app, STATSD_APP_CH
dim->rd = rrddim_add(chart->st, metric, dim->name, dim->multiplier, dim->divisor, dim->algorithm);
if(dim->flags != RRDDIM_FLAG_NONE) dim->rd->flags |= dim->flags;
if(dim->options != RRDDIM_OPTION_NONE) dim->rd->options |= dim->options;
return dim->rd;
}
dim->rd = rrddim_add(chart->st, dim->metric, dim->name, dim->multiplier, dim->divisor, dim->algorithm);
if(dim->flags != RRDDIM_FLAG_NONE) dim->rd->flags |= dim->flags;
if(dim->options != RRDDIM_OPTION_NONE) dim->rd->options |= dim->options;
return dim->rd;
}

75
collectors/tc.plugin/plugin_tc.c
View file

@ -74,7 +74,7 @@ struct tc_device {
// ----------------------------------------------------------------------------
// tc_class index
static void tc_class_free_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
static void tc_class_free_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
// struct tc_device *d = data;
struct tc_class *c = value;
@ -84,22 +84,21 @@ static void tc_class_free_callback(const char *name __maybe_unused, void *value,
string_freez(c->parentid);
}
static void tc_class_conflict_callback(const char *name __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
static bool tc_class_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
struct tc_device *d = data; (void)d;
struct tc_class *c = old_value; (void)c;
struct tc_class *new_c = new_value; (void)new_c;
error("TC: class '%s' is already in device '%s'. Ignoring duplicate.", name, string2str(d->id));
error("TC: class '%s' is already in device '%s'. Ignoring duplicate.", dictionary_acquired_item_name(item), string2str(d->id));
tc_class_free_callback(name, new_value, data);
tc_class_free_callback(item, new_value, data);
return true;
}
static void tc_class_index_init(struct tc_device *d) {
if(!d->classes) {
d->classes = dictionary_create(
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
|DICTIONARY_FLAG_SINGLE_THREADED
);
d->classes = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_SINGLE_THREADED);
dictionary_register_delete_callback(d->classes, tc_class_free_callback, d);
dictionary_register_conflict_callback(d->classes, tc_class_conflict_callback, d);
@ -128,12 +127,12 @@ static inline struct tc_class *tc_class_index_find(struct tc_device *d, const ch
static DICTIONARY *tc_device_root_index = NULL;
static void tc_device_add_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
static void tc_device_add_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
struct tc_device *d = value;
tc_class_index_init(d);
}
static void tc_device_free_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
static void tc_device_free_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
struct tc_device *d = value;
tc_class_index_destroy(d);
@ -146,10 +145,7 @@ static void tc_device_free_callback(const char *name __maybe_unused, void *value
static void tc_device_index_init() {
if(!tc_device_root_index) {
tc_device_root_index = dictionary_create(
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
|DICTIONARY_FLAG_SINGLE_THREADED
|DICTIONARY_FLAG_ADD_IN_FRONT
);
DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_SINGLE_THREADED | DICT_OPTION_ADD_IN_FRONT);
dictionary_register_insert_callback(tc_device_root_index, tc_device_add_callback, NULL);
dictionary_register_delete_callback(tc_device_root_index, tc_device_free_callback, NULL);
@ -196,7 +192,7 @@ static inline void tc_device_classes_cleanup(struct tc_device *d) {
d->family_updated = false;
struct tc_class *c;
dfe_start_unsafe(d->classes, c) {
dfe_start_write(d->classes, c) {
if(unlikely(cleanup_every && c->unupdated >= cleanup_every))
tc_class_free(d, c);
@ -254,7 +250,7 @@ static inline void tc_device_commit(struct tc_device *d) {
// prepare all classes
// we set reasonable defaults for the rest of the code below
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
c->render = false; // do not render this class
c->isleaf = true; // this is a leaf class
c->hasparent = false; // without a parent
@ -283,7 +279,7 @@ static inline void tc_device_commit(struct tc_device *d) {
error("TC: device '%s' has active both classes (%d) and qdiscs (%d). Will render only qdiscs.", string2str(d->id), updated_classes, updated_qdiscs);
// set all classes to !updated
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if (unlikely(!c->isqdisc && c->updated))
c->updated = false;
}
@ -307,7 +303,7 @@ static inline void tc_device_commit(struct tc_device *d) {
// so, here we remove the isleaf flag from nodes in the middle
// and we add the hasparent flag to leaf nodes we found their parent
if(likely(!d->enabled_all_classes_qdiscs)) {
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->updated))
continue;
@ -319,7 +315,7 @@ static inline void tc_device_commit(struct tc_device *d) {
// c->parentid?c->parentid:"NULL");
// find if c is leaf or not
dfe_start_unsafe(d->classes, x) {
dfe_start_read(d->classes, x) {
if(unlikely(!x->updated || c == x || !x->parentid))
continue;
@ -339,7 +335,7 @@ static inline void tc_device_commit(struct tc_device *d) {
dfe_done(c);
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->updated))
continue;
@ -367,7 +363,7 @@ static inline void tc_device_commit(struct tc_device *d) {
// dump all the list to see what we know
if(unlikely(debug_flags & D_TC_LOOP)) {
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(c->render) debug(D_TC_LOOP, "TC: final nodes dump for '%s': class %s, OK", string2str(d->name), string2str(c->id));
else debug(D_TC_LOOP, "TC: final nodes dump for '%s': class '%s', IGNORE (updated: %d, isleaf: %d, hasparent: %d, parent: '%s')",
string2str(d->name?d->name:d->id), string2str(c->id), c->updated, c->isleaf, c->hasparent, string2str(c->parentid));
@ -426,7 +422,8 @@ static inline void tc_device_commit(struct tc_device *d) {
}
else {
rrdset_next(d->st_bytes);
if(unlikely(d->name_updated)) rrdset_set_name(d->st_bytes, string2str(d->name));
if(unlikely(d->name_updated))
rrdset_reset_name(d->st_bytes, string2str(d->name));
if(d->name && d->name_updated)
rrdlabels_add(d->st_bytes->rrdlabels, "name", string2str(d->name), RRDLABEL_SRC_AUTO);
@ -438,13 +435,13 @@ static inline void tc_device_commit(struct tc_device *d) {
// update the family
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->render)) continue;
if(unlikely(!c->rd_bytes))
c->rd_bytes = rrddim_add(d->st_bytes, string2str(c->id), string2str(c->name?c->name:c->id), 8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
else if(unlikely(c->name_updated))
rrddim_set_name(d->st_bytes, c->rd_bytes, string2str(c->name));
rrddim_reset_name(d->st_bytes, c->rd_bytes, string2str(c->name));
rrddim_set_by_pointer(d->st_bytes, c->rd_bytes, c->bytes);
}
@ -491,7 +488,7 @@ static inline void tc_device_commit(struct tc_device *d) {
if(unlikely(d->name_updated)) {
char name[RRD_ID_LENGTH_MAX + 1];
snprintfz(name, RRD_ID_LENGTH_MAX, "%s_packets", string2str(d->name?d->name:d->id));
rrdset_set_name(d->st_packets, name);
rrdset_reset_name(d->st_packets, name);
}
if(d->name && d->name_updated)
@ -504,13 +501,13 @@ static inline void tc_device_commit(struct tc_device *d) {
// update the family
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->render)) continue;
if(unlikely(!c->rd_packets))
c->rd_packets = rrddim_add(d->st_packets, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_INCREMENTAL);
else if(unlikely(c->name_updated))
rrddim_set_name(d->st_packets, c->rd_packets, string2str(c->name));
rrddim_reset_name(d->st_packets, c->rd_packets, string2str(c->name));
rrddim_set_by_pointer(d->st_packets, c->rd_packets, c->packets);
}
@ -557,7 +554,7 @@ static inline void tc_device_commit(struct tc_device *d) {
if(unlikely(d->name_updated)) {
char name[RRD_ID_LENGTH_MAX + 1];
snprintfz(name, RRD_ID_LENGTH_MAX, "%s_dropped", string2str(d->name?d->name:d->id));
rrdset_set_name(d->st_dropped, name);
rrdset_reset_name(d->st_dropped, name);
}
if(d->name && d->name_updated)
@ -570,13 +567,13 @@ static inline void tc_device_commit(struct tc_device *d) {
// update the family
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->render)) continue;
if(unlikely(!c->rd_dropped))
c->rd_dropped = rrddim_add(d->st_dropped, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_INCREMENTAL);
else if(unlikely(c->name_updated))
rrddim_set_name(d->st_dropped, c->rd_dropped, string2str(c->name));
rrddim_reset_name(d->st_dropped, c->rd_dropped, string2str(c->name));
rrddim_set_by_pointer(d->st_dropped, c->rd_dropped, c->dropped);
}
@ -623,7 +620,7 @@ static inline void tc_device_commit(struct tc_device *d) {
if(unlikely(d->name_updated)) {
char name[RRD_ID_LENGTH_MAX + 1];
snprintfz(name, RRD_ID_LENGTH_MAX, "%s_tokens", string2str(d->name?d->name:d->id));
rrdset_set_name(d->st_tokens, name);
rrdset_reset_name(d->st_tokens, name);
}
if(d->name && d->name_updated)
@ -636,14 +633,14 @@ static inline void tc_device_commit(struct tc_device *d) {
// update the family
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->render)) continue;
if(unlikely(!c->rd_tokens)) {
c->rd_tokens = rrddim_add(d->st_tokens, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_ABSOLUTE);
}
else if(unlikely(c->name_updated))
rrddim_set_name(d->st_tokens, c->rd_tokens, string2str(c->name));
rrddim_reset_name(d->st_tokens, c->rd_tokens, string2str(c->name));
rrddim_set_by_pointer(d->st_tokens, c->rd_tokens, c->tokens);
}
@ -691,7 +688,7 @@ static inline void tc_device_commit(struct tc_device *d) {
if(unlikely(d->name_updated)) {
char name[RRD_ID_LENGTH_MAX + 1];
snprintfz(name, RRD_ID_LENGTH_MAX, "%s_ctokens", string2str(d->name?d->name:d->id));
rrdset_set_name(d->st_ctokens, name);
rrdset_reset_name(d->st_ctokens, name);
}
if(d->name && d->name_updated)
@ -704,13 +701,13 @@ static inline void tc_device_commit(struct tc_device *d) {
// update the family
}
dfe_start_unsafe(d->classes, c) {
dfe_start_read(d->classes, c) {
if(unlikely(!c->render)) continue;
if(unlikely(!c->rd_ctokens))
c->rd_ctokens = rrddim_add(d->st_ctokens, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_ABSOLUTE);
else if(unlikely(c->name_updated))
rrddim_set_name(d->st_ctokens, c->rd_ctokens, string2str(c->name));
rrddim_reset_name(d->st_ctokens, c->rd_ctokens, string2str(c->name));
rrddim_set_by_pointer(d->st_ctokens, c->rd_ctokens, c->ctokens);
}
@ -1146,12 +1143,12 @@ void *tc_main(void *ptr) {
worker_is_busy(WORKER_TC_WORKTIME);
worker_set_metric(WORKER_TC_PLUGIN_TIME, str2ll(words[1], NULL));
size_t number_of_devices = dictionary_stats_entries(tc_device_root_index);
size_t number_of_devices = dictionary_entries(tc_device_root_index);
size_t number_of_classes = 0;
struct tc_device *d;
dfe_start_unsafe(tc_device_root_index, d) {
number_of_classes += dictionary_stats_entries(d->classes);
dfe_start_read(tc_device_root_index, d) {
number_of_classes += dictionary_entries(d->classes);
}
dfe_done(d);

1
configure.ac
View file

@ -1718,6 +1718,7 @@ AC_CONFIG_FILES([
libnetdata/simple_pattern/Makefile
libnetdata/socket/Makefile
libnetdata/statistical/Makefile
libnetdata/string/Makefile
libnetdata/storage_number/Makefile
libnetdata/storage_number/tests/Makefile
libnetdata/threads/Makefile

60
daemon/analytics.c
View file

@ -249,8 +249,7 @@ void analytics_exporters(void)
buffer_free(bi);
}
int collector_counter_callb(const char *name, void *entry, void *data) {
(void)name;
int collector_counter_callb(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
struct array_printer *ap = (struct array_printer *)data;
struct collector *col = (struct collector *)entry;
@ -279,21 +278,22 @@ int collector_counter_callb(const char *name, void *entry, void *data) {
void analytics_collectors(void)
{
RRDSET *st;
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
char name[500];
BUFFER *bt = buffer_create(1000);
rrdset_foreach_read(st, localhost)
{
if (rrdset_is_available_for_viewers(st)) {
struct collector col = {
.plugin = rrdset_plugin_name(st),
.module = rrdset_module_name(st)
};
snprintfz(name, 499, "%s:%s", col.plugin, col.module);
dictionary_set(dict, name, &col, sizeof(struct collector));
}
rrdset_foreach_read(st, localhost) {
if(!rrdset_is_available_for_viewers(st))
continue;
struct collector col = {
.plugin = rrdset_plugin_name(st),
.module = rrdset_module_name(st)
};
snprintfz(name, 499, "%s:%s", col.plugin, col.module);
dictionary_set(dict, name, &col, sizeof(struct collector));
}
rrdset_foreach_done(st);
struct array_printer ap;
ap.c = 0;
@ -398,12 +398,11 @@ void analytics_charts(void)
{
RRDSET *st;
int c = 0;
rrdset_foreach_read(st, localhost)
{
if (rrdset_is_available_for_viewers(st)) {
c++;
}
}
if(rrdset_is_available_for_viewers(st)) c++;
rrdset_foreach_done(st);
{
char b[7];
snprintfz(b, 6, "%d", c);
@ -415,22 +414,19 @@ void analytics_metrics(void)
{
RRDSET *st;
long int dimensions = 0;
RRDDIM *rd;
rrdset_foreach_read(st, localhost)
{
rrdset_rdlock(st);
rrdset_foreach_read(st, localhost) {
if (rrdset_is_available_for_viewers(st)) {
rrddim_foreach_read(rd, st)
{
if (rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
RRDDIM *rd;
rrddim_foreach_read(rd, st) {
if (rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
continue;
dimensions++;
}
rrddim_foreach_done(rd);
}
rrdset_unlock(st);
}
rrdset_foreach_done(st);
{
char b[7];
snprintfz(b, 6, "%ld", dimensions);
@ -443,7 +439,7 @@ void analytics_alarms(void)
int alarm_warn = 0, alarm_crit = 0, alarm_normal = 0;
char b[10];
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(localhost, rc) {
foreach_rrdcalc_in_rrdhost_read(localhost, rc) {
if (unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
continue;
@ -458,6 +454,7 @@ void analytics_alarms(void)
alarm_normal++;
}
}
foreach_rrdcalc_in_rrdhost_done(rc);
snprintfz(b, 9, "%d", alarm_normal);
analytics_set_data(&analytics_data.netdata_alarms_normal, b);
@ -527,16 +524,11 @@ void analytics_gather_immutable_meta_data(void)
*/
void analytics_gather_mutable_meta_data(void)
{
rrdhost_rdlock(localhost);
analytics_collectors();
analytics_alarms();
analytics_charts();
analytics_metrics();
analytics_aclk();
rrdhost_unlock(localhost);
analytics_mirrored_hosts();
analytics_alarms_notifications();

369
daemon/global_statistics.c
View file

@ -10,8 +10,9 @@
#define WORKER_JOB_DBENGINE 3
#define WORKER_JOB_HEARTBEAT 4
#define WORKER_JOB_STRINGS 5
#define WORKER_JOB_DICTIONARIES 6
#if WORKER_UTILIZATION_MAX_JOB_TYPES < 6
#if WORKER_UTILIZATION_MAX_JOB_TYPES < 7
#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 5
#endif
@ -1215,6 +1216,367 @@ static void update_heartbeat_charts() {
rrdset_done(st_heartbeat);
}
// ---------------------------------------------------------------------------------------------------------------------
// dictionary statistics
struct dictionary_categories {
struct dictionary_stats *stats;
const char *family;
const char *context_prefix;
int priority;
RRDSET *st_dicts;
RRDDIM *rd_dicts_active;
RRDDIM *rd_dicts_deleted;
RRDSET *st_items;
RRDDIM *rd_items_entries;
RRDDIM *rd_items_referenced;
RRDDIM *rd_items_pending_deletion;
RRDSET *st_ops;
RRDDIM *rd_ops_creations;
RRDDIM *rd_ops_destructions;
RRDDIM *rd_ops_flushes;
RRDDIM *rd_ops_traversals;
RRDDIM *rd_ops_walkthroughs;
RRDDIM *rd_ops_garbage_collections;
RRDDIM *rd_ops_searches;
RRDDIM *rd_ops_inserts;
RRDDIM *rd_ops_resets;
RRDDIM *rd_ops_deletes;
RRDSET *st_callbacks;
RRDDIM *rd_callbacks_inserts;
RRDDIM *rd_callbacks_conflicts;
RRDDIM *rd_callbacks_reacts;
RRDDIM *rd_callbacks_deletes;
RRDSET *st_memory;
RRDDIM *rd_memory_indexed;
RRDDIM *rd_memory_values;
RRDDIM *rd_memory_dict;
RRDSET *st_spins;
RRDDIM *rd_spins_use;
RRDDIM *rd_spins_search;
RRDDIM *rd_spins_insert;
} dictionary_categories[] = {
{ .stats = &dictionary_stats_category_other, "dictionaries", "dictionaries", 900000 },
// terminator
{ .stats = NULL, NULL, NULL, 0 },
};
#define load_dictionary_stats_entry(x) total += (size_t)(stats.x = __atomic_load_n(&c->stats->x, __ATOMIC_RELAXED))
static void update_dictionary_category_charts(struct dictionary_categories *c) {
struct dictionary_stats stats;
stats.name = c->stats->name;
// ------------------------------------------------------------------------
size_t total = 0;
load_dictionary_stats_entry(dictionaries.active);
load_dictionary_stats_entry(dictionaries.deleted);
if(c->st_dicts || total != 0) {
if (unlikely(!c->st_dicts)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.dictionaries", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.dictionaries", c->context_prefix);
c->st_dicts = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionaries"
, "dictionaries"
, "netdata"
, "stats"
, c->priority + 0
, localhost->rrd_update_every
, RRDSET_TYPE_LINE
);
c->rd_dicts_active = rrddim_add(c->st_dicts, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
c->rd_dicts_deleted = rrddim_add(c->st_dicts, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(c->st_dicts->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_dicts);
rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_active, (collected_number)stats.dictionaries.active);
rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_deleted, (collected_number)stats.dictionaries.deleted);
rrdset_done(c->st_dicts);
}
// ------------------------------------------------------------------------
total = 0;
load_dictionary_stats_entry(items.entries);
load_dictionary_stats_entry(items.referenced);
load_dictionary_stats_entry(items.pending_deletion);
if(c->st_items || total != 0) {
if (unlikely(!c->st_items)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.items", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.items", c->context_prefix);
c->st_items = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionary Items"
, "items"
, "netdata"
, "stats"
, c->priority + 1
, localhost->rrd_update_every
, RRDSET_TYPE_LINE
);
c->rd_items_entries = rrddim_add(c->st_items, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
c->rd_items_pending_deletion = rrddim_add(c->st_items, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
c->rd_items_referenced = rrddim_add(c->st_items, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(c->st_items->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_items);
rrddim_set_by_pointer(c->st_items, c->rd_items_entries, stats.items.entries);
rrddim_set_by_pointer(c->st_items, c->rd_items_pending_deletion, stats.items.pending_deletion);
rrddim_set_by_pointer(c->st_items, c->rd_items_referenced, stats.items.referenced);
rrdset_done(c->st_items);
}
// ------------------------------------------------------------------------
total = 0;
load_dictionary_stats_entry(ops.creations);
load_dictionary_stats_entry(ops.destructions);
load_dictionary_stats_entry(ops.flushes);
load_dictionary_stats_entry(ops.traversals);
load_dictionary_stats_entry(ops.walkthroughs);
load_dictionary_stats_entry(ops.garbage_collections);
load_dictionary_stats_entry(ops.searches);
load_dictionary_stats_entry(ops.inserts);
load_dictionary_stats_entry(ops.resets);
load_dictionary_stats_entry(ops.deletes);
if(c->st_ops || total != 0) {
if (unlikely(!c->st_ops)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.ops", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.ops", c->context_prefix);
c->st_ops = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionary Operations"
, "ops/s"
, "netdata"
, "stats"
, c->priority + 2
, localhost->rrd_update_every
, RRDSET_TYPE_LINE
);
c->rd_ops_creations = rrddim_add(c->st_ops, "creations", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_destructions = rrddim_add(c->st_ops, "destructions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_flushes = rrddim_add(c->st_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_traversals = rrddim_add(c->st_ops, "traversals", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_walkthroughs = rrddim_add(c->st_ops, "walkthroughs", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_garbage_collections = rrddim_add(c->st_ops, "garbage_collections", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_searches = rrddim_add(c->st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_inserts = rrddim_add(c->st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_resets = rrddim_add(c->st_ops, "resets", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_ops_deletes = rrddim_add(c->st_ops, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrdlabels_add(c->st_ops->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_ops);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_creations, (collected_number)stats.ops.creations);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_destructions, (collected_number)stats.ops.destructions);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_flushes, (collected_number)stats.ops.flushes);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_traversals, (collected_number)stats.ops.traversals);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_walkthroughs, (collected_number)stats.ops.walkthroughs);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_garbage_collections, (collected_number)stats.ops.garbage_collections);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_searches, (collected_number)stats.ops.searches);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_inserts, (collected_number)stats.ops.inserts);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_resets, (collected_number)stats.ops.resets);
rrddim_set_by_pointer(c->st_ops, c->rd_ops_deletes, (collected_number)stats.ops.deletes);
rrdset_done(c->st_ops);
}
// ------------------------------------------------------------------------
total = 0;
load_dictionary_stats_entry(callbacks.inserts);
load_dictionary_stats_entry(callbacks.conflicts);
load_dictionary_stats_entry(callbacks.reacts);
load_dictionary_stats_entry(callbacks.deletes);
if(c->st_callbacks || total != 0) {
if (unlikely(!c->st_callbacks)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.callbacks", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.callbacks", c->context_prefix);
c->st_callbacks = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionary Callbacks"
, "callbacks/s"
, "netdata"
, "stats"
, c->priority + 3
, localhost->rrd_update_every
, RRDSET_TYPE_LINE
);
c->rd_callbacks_inserts = rrddim_add(c->st_callbacks, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_callbacks_deletes = rrddim_add(c->st_callbacks, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_callbacks_conflicts = rrddim_add(c->st_callbacks, "conflicts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_callbacks_reacts = rrddim_add(c->st_callbacks, "reacts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrdlabels_add(c->st_callbacks->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_callbacks);
rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_inserts, (collected_number)stats.callbacks.inserts);
rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_conflicts, (collected_number)stats.callbacks.conflicts);
rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_reacts, (collected_number)stats.callbacks.reacts);
rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_deletes, (collected_number)stats.callbacks.deletes);
rrdset_done(c->st_callbacks);
}
// ------------------------------------------------------------------------
total = 0;
load_dictionary_stats_entry(memory.indexed);
load_dictionary_stats_entry(memory.values);
load_dictionary_stats_entry(memory.dict);
if(c->st_memory || total != 0) {
if (unlikely(!c->st_memory)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.memory", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.memory", c->context_prefix);
c->st_memory = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionary Memory"
, "bytes"
, "netdata"
, "stats"
, c->priority + 4
, localhost->rrd_update_every
, RRDSET_TYPE_STACKED
);
c->rd_memory_indexed = rrddim_add(c->st_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
c->rd_memory_values = rrddim_add(c->st_memory, "data", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
c->rd_memory_dict = rrddim_add(c->st_memory, "structures", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(c->st_memory->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_memory);
rrddim_set_by_pointer(c->st_memory, c->rd_memory_indexed, (collected_number)stats.memory.indexed);
rrddim_set_by_pointer(c->st_memory, c->rd_memory_values, (collected_number)stats.memory.values);
rrddim_set_by_pointer(c->st_memory, c->rd_memory_dict, (collected_number)stats.memory.dict);
rrdset_done(c->st_memory);
}
// ------------------------------------------------------------------------
total = 0;
load_dictionary_stats_entry(spin_locks.use);
load_dictionary_stats_entry(spin_locks.search);
load_dictionary_stats_entry(spin_locks.insert);
if(c->st_spins || total != 0) {
if (unlikely(!c->st_spins)) {
char id[RRD_ID_LENGTH_MAX + 1];
snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.spins", c->context_prefix, stats.name);
char context[RRD_ID_LENGTH_MAX + 1];
snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.spins", c->context_prefix);
c->st_spins = rrdset_create_localhost(
"netdata"
, id
, NULL
, c->family
, context
, "Dictionary Spins"
, "count"
, "netdata"
, "stats"
, c->priority + 5
, localhost->rrd_update_every
, RRDSET_TYPE_LINE
);
c->rd_spins_use = rrddim_add(c->st_spins, "use", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_spins_search = rrddim_add(c->st_spins, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
c->rd_spins_insert = rrddim_add(c->st_spins, "insert", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
rrdlabels_add(c->st_spins->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(c->st_spins);
rrddim_set_by_pointer(c->st_spins, c->rd_spins_use, (collected_number)stats.spin_locks.use);
rrddim_set_by_pointer(c->st_spins, c->rd_spins_search, (collected_number)stats.spin_locks.search);
rrddim_set_by_pointer(c->st_spins, c->rd_spins_insert, (collected_number)stats.spin_locks.insert);
rrdset_done(c->st_spins);
}
}
static void dictionary_statistics(void) {
for(int i = 0; dictionary_categories[i].stats ;i++) {
update_dictionary_category_charts(&dictionary_categories[i]);
}
}
// ---------------------------------------------------------------------------------------------------------------------
// worker utilization
@ -1334,6 +1696,7 @@ static struct worker_utilization all_workers_utilization[] = {
{ .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
{ .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
{ .name = "RRDCONTEXT", .family = "workers contexts", .priority = 1000000 },
{ .name = "SERVICE", .family = "workers service", .priority = 1000000 },
// has to be terminated with a NULL
{ .name = NULL, .family = NULL }
@ -2011,6 +2374,7 @@ void *global_statistics_main(void *ptr)
worker_register_job_name(WORKER_JOB_WORKERS, "workers");
worker_register_job_name(WORKER_JOB_DBENGINE, "dbengine");
worker_register_job_name(WORKER_JOB_STRINGS, "strings");
worker_register_job_name(WORKER_JOB_DICTIONARIES, "dictionaries");
netdata_thread_cleanup_push(global_statistics_cleanup, ptr);
@ -2048,6 +2412,9 @@ void *global_statistics_main(void *ptr)
worker_is_busy(WORKER_JOB_STRINGS);
update_strings_charts();
worker_is_busy(WORKER_JOB_DICTIONARIES);
dictionary_statistics();
}
netdata_thread_cleanup_pop(1);

4
daemon/main.c
View file

@ -1004,6 +1004,7 @@ int main(int argc, char **argv) {
if(test_dbengine()) return 1;
#endif
if(test_sqlite()) return 1;
if(string_unittest(10000)) return 1;
if (dictionary_unittest(10000))
return 1;
if (rrdlabels_unittest())
@ -1028,6 +1029,9 @@ int main(int argc, char **argv) {
else if(strcmp(optarg, "dicttest") == 0) {
return dictionary_unittest(10000);
}
else if(strcmp(optarg, "stringtest") == 0) {
return string_unittest(10000);
}
else if(strcmp(optarg, "rrdlabelstest") == 0) {
return rrdlabels_unittest();
}

251
daemon/service.c
View file

@ -5,12 +5,238 @@
/* Run service jobs every X seconds */
#define SERVICE_HEARTBEAT 10
#define WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK 1
#define WORKER_JOB_CLEANUP_OBSOLETE_CHARTS 2
#define WORKER_JOB_ARCHIVE_CHART 3
#define WORKER_JOB_ARCHIVE_CHART_DIMENSIONS 4
#define WORKER_JOB_ARCHIVE_DIMENSION 5
#define WORKER_JOB_CLEANUP_ORPHAN_HOSTS 6
#define WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS 7
#define WORKER_JOB_FREE_HOST 9
#define WORKER_JOB_SAVE_HOST_CHARTS 10
#define WORKER_JOB_DELETE_HOST_CHARTS 11
#define WORKER_JOB_FREE_CHART 12
#define WORKER_JOB_SAVE_CHART 13
#define WORKER_JOB_DELETE_CHART 14
#define WORKER_JOB_FREE_DIMENSION 15
static void svc_rrddim_obsolete_to_archive(RRDDIM *rd) {
RRDSET *st = rd->rrdset;
if(rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED | RRDDIM_FLAG_ACLK) || !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
return;
worker_is_busy(WORKER_JOB_ARCHIVE_DIMENSION);
rrddim_flag_set(rd, RRDDIM_FLAG_ARCHIVED);
rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
const char *cache_filename = rrddim_cache_filename(rd);
if(cache_filename) {
info("Deleting dimension file '%s'.", cache_filename);
if (unlikely(unlink(cache_filename) == -1))
error("Cannot delete dimension file '%s'", cache_filename);
}
if (rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
rrddimvar_delete_all(rd);
/* only a collector can mark a chart as obsolete, so we must remove the reference */
size_t tiers_available = 0, tiers_said_yes = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(rd->tiers[tier]) {
tiers_available++;
if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
tiers_said_yes++;
rd->tiers[tier]->db_collection_handle = NULL;
}
}
if (tiers_available == tiers_said_yes && tiers_said_yes) {
/* This metric has no data and no references */
delete_dimension_uuid(&rd->metric_uuid);
}
else {
/* Do not delete this dimension */
#ifdef ENABLE_ACLK
queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
#endif
return;
}
}
worker_is_busy(WORKER_JOB_FREE_DIMENSION);
rrddim_free(st, rd);
}
static void svc_rrdset_archive_obsolete_dimensions(RRDSET *st, bool all_dimensions) {
worker_is_busy(WORKER_JOB_ARCHIVE_CHART_DIMENSIONS);
RRDDIM *rd;
time_t now = now_realtime_sec();
dfe_start_reentrant(st->rrddim_root_index, rd) {
if(unlikely(
all_dimensions ||
(rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE) && (rd->last_collected_time.tv_sec + rrdset_free_obsolete_time < now))
)) {
info("Removing obsolete dimension '%s' (%s) of '%s' (%s).", rrddim_name(rd), rrddim_id(rd), rrdset_name(st), rrdset_id(st));
svc_rrddim_obsolete_to_archive(rd);
}
}
dfe_done(rd);
}
static void svc_rrdset_obsolete_to_archive(RRDSET *st) {
worker_is_busy(WORKER_JOB_ARCHIVE_CHART);
rrdset_flag_set(st, RRDSET_FLAG_ARCHIVED);
rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
rrdcalc_unlink_all_rrdset_alerts(st);
svc_rrdset_archive_obsolete_dimensions(st, true);
rrdsetvar_release_and_delete_all(st);
// has to be run after all dimensions are archived - or use-after-free will occur
rrdvar_delete_all(st->rrdvars);
if(st->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) {
if(rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS)) {
worker_is_busy(WORKER_JOB_DELETE_CHART);
rrdset_delete_files(st);
}
else {
worker_is_busy(WORKER_JOB_SAVE_CHART);
rrdset_save(st);
}
worker_is_busy(WORKER_JOB_FREE_CHART);
rrdset_free(st);
}
}
static void svc_rrdhost_cleanup_obsolete_charts(RRDHOST *host) {
worker_is_busy(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS);
time_t now = now_realtime_sec();
RRDSET *st;
rrdset_foreach_reentrant(st, host) {
if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)
&& st->last_accessed_time + rrdset_free_obsolete_time < now
&& st->last_updated.tv_sec + rrdset_free_obsolete_time < now
&& st->last_collected_time.tv_sec + rrdset_free_obsolete_time < now
)) {
svc_rrdset_obsolete_to_archive(st);
}
else if(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS)) {
rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
svc_rrdset_archive_obsolete_dimensions(st, false);
}
#ifdef ENABLE_ACLK
else
sql_check_chart_liveness(st);
#endif
}
rrdset_foreach_done(st);
}
static void svc_rrdset_check_obsoletion(RRDHOST *host) {
worker_is_busy(WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK);
time_t last_entry_t;
RRDSET *st;
rrdset_foreach_read(st, host) {
last_entry_t = rrdset_last_entry_t(st);
if(last_entry_t && last_entry_t < host->senders_connect_time)
rrdset_is_obsolete(st);
}
rrdset_foreach_done(st);
}
static void svc_rrd_cleanup_obsolete_charts_from_all_hosts() {
worker_is_busy(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS);
rrd_rdlock();
RRDHOST *host;
rrdhost_foreach_read(host) {
if(rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS|RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS)) {
rrdhost_flag_clear(host, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS|RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS);
svc_rrdhost_cleanup_obsolete_charts(host);
}
if(host != localhost
&& host->trigger_chart_obsoletion_check
&& (
(
host->senders_last_chart_command
&& host->senders_last_chart_command + host->health_delay_up_to < now_realtime_sec()
)
|| (host->senders_connect_time + 300 < now_realtime_sec())
)
) {
svc_rrdset_check_obsoletion(host);
host->trigger_chart_obsoletion_check = 0;
}
}
rrd_unlock();
}
static void svc_rrdhost_cleanup_orphan_hosts(RRDHOST *protected_host) {
worker_is_busy(WORKER_JOB_CLEANUP_ORPHAN_HOSTS);
rrd_wrlock();
time_t now = now_realtime_sec();
RRDHOST *host;
restart_after_removal:
rrdhost_foreach_write(host) {
if(rrdhost_should_be_removed(host, protected_host, now)) {
info("Host '%s' with machine guid '%s' is obsolete - cleaning up.", rrdhost_hostname(host), host->machine_guid);
if (rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST)
#ifdef ENABLE_DBENGINE
/* don't delete multi-host DB host files */
&& !(host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && is_storage_engine_shared(host->storage_instance[0]))
#endif
) {
worker_is_busy(WORKER_JOB_DELETE_HOST_CHARTS);
rrdhost_delete_charts(host);
}
else {
worker_is_busy(WORKER_JOB_SAVE_HOST_CHARTS);
rrdhost_save_charts(host);
}
worker_is_busy(WORKER_JOB_FREE_HOST);
rrdhost_free(host, 0);
goto restart_after_removal;
}
}
rrd_unlock();
}
static void service_main_cleanup(void *ptr)
{
struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
debug(D_SYSTEM, "Cleaning up...");
worker_unregister();
static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
}
@ -20,6 +246,22 @@ static void service_main_cleanup(void *ptr)
*/
void *service_main(void *ptr)
{
worker_register("SERVICE");
worker_register_job_name(WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK, "child chart obsoletion check");
worker_register_job_name(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS, "cleanup obsolete charts");
worker_register_job_name(WORKER_JOB_ARCHIVE_CHART, "archive chart");
worker_register_job_name(WORKER_JOB_ARCHIVE_CHART_DIMENSIONS, "archive chart dimensions");
worker_register_job_name(WORKER_JOB_ARCHIVE_DIMENSION, "archive dimension");
worker_register_job_name(WORKER_JOB_CLEANUP_ORPHAN_HOSTS, "cleanup orphan hosts");
worker_register_job_name(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS, "cleanup obsolete charts on all hosts");
worker_register_job_name(WORKER_JOB_FREE_HOST, "free host");
worker_register_job_name(WORKER_JOB_SAVE_HOST_CHARTS, "save host charts");
worker_register_job_name(WORKER_JOB_DELETE_HOST_CHARTS, "delete host charts");
worker_register_job_name(WORKER_JOB_FREE_CHART, "free chart");
worker_register_job_name(WORKER_JOB_SAVE_CHART, "save chart");
worker_register_job_name(WORKER_JOB_DELETE_CHART, "delete chart");
worker_register_job_name(WORKER_JOB_FREE_DIMENSION, "free dimension");
netdata_thread_cleanup_push(service_main_cleanup, ptr);
heartbeat_t hb;
heartbeat_init(&hb);
@ -28,14 +270,11 @@ void *service_main(void *ptr)
debug(D_SYSTEM, "Service thread starts");
while (!netdata_exit) {
worker_is_idle();
heartbeat_next(&hb, step);
rrd_cleanup_obsolete_charts();
rrd_wrlock();
rrdhost_cleanup_orphan_hosts_nolock(localhost);
rrd_unlock();
svc_rrd_cleanup_obsolete_charts_from_all_hosts();
svc_rrdhost_cleanup_orphan_hosts(localhost);
}
netdata_thread_cleanup_pop(1);

36
daemon/unit_test.c
View file

@ -1269,27 +1269,27 @@ static int test_variable_renames(void) {
fprintf(stderr, "Created dimension with id '%s', name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
fprintf(stderr, "Renaming chart to CHARTNAME1\n");
rrdset_set_name(st, "CHARTNAME1");
rrdset_reset_name(st, "CHARTNAME1");
fprintf(stderr, "Renamed chart with id '%s' to name '%s'\n", rrdset_id(st), rrdset_name(st));
fprintf(stderr, "Renaming chart to CHARTNAME2\n");
rrdset_set_name(st, "CHARTNAME2");
rrdset_reset_name(st, "CHARTNAME2");
fprintf(stderr, "Renamed chart with id '%s' to name '%s'\n", rrdset_id(st), rrdset_name(st));
fprintf(stderr, "Renaming dimension DIM1 to DIM1NAME1\n");
rrddim_set_name(st, rd1, "DIM1NAME1");
rrddim_reset_name(st, rd1, "DIM1NAME1");
fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd1), rrddim_name(rd1));
fprintf(stderr, "Renaming dimension DIM1 to DIM1NAME2\n");
rrddim_set_name(st, rd1, "DIM1NAME2");
rrddim_reset_name(st, rd1, "DIM1NAME2");
fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd1), rrddim_name(rd1));
fprintf(stderr, "Renaming dimension DIM2 to DIM2NAME1\n");
rrddim_set_name(st, rd2, "DIM2NAME1");
rrddim_reset_name(st, rd2, "DIM2NAME1");
fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
fprintf(stderr, "Renaming dimension DIM2 to DIM2NAME2\n");
rrddim_set_name(st, rd2, "DIM2NAME2");
rrddim_reset_name(st, rd2, "DIM2NAME2");
fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
BUFFER *buf = buffer_create(1);
@ -1447,9 +1447,8 @@ int unit_test(long delay, long shift)
long increment = 1000;
collected_number i = 0;
unsigned long c, dimensions = 0;
unsigned long c, dimensions = rrdset_number_of_dimensions(st);
RRDDIM *rd;
for(rd = st->dimensions; rd ; rd = rd->next) dimensions++;
for(c = 0; c < 20 ;c++) {
i += increment;
@ -1470,8 +1469,10 @@ int unit_test(long delay, long shift)
}
// prevent it from deleting the dimensions
for(rd = st->dimensions; rd ; rd = rd->next)
rrddim_foreach_read(rd, st) {
rd->last_collected_time.tv_sec = st->last_collected_time.tv_sec;
}
rrddim_foreach_done(rd);
rrdset_done(st);
}
@ -1486,7 +1487,7 @@ int unit_test(long delay, long shift)
for(c = 0 ; c < st->counter ; c++) {
fprintf(stderr, "\nPOSITION: c = %lu, EXPECTED VALUE %lu\n", c, (oincrement + c * increment + increment * (1000000 - shift) / 1000000 )* 10);
for(rd = st->dimensions; rd ; rd = rd->next) {
rrddim_foreach_read(rd, st) {
sn = rd->db[c];
cn = unpack_storage_number(sn);
fprintf(stderr, "\t %s " NETDATA_DOUBLE_FORMAT " (PACKED AS " STORAGE_NUMBER_FORMAT ") -> ", rrddim_id(rd), cn, sn);
@ -1508,6 +1509,7 @@ int unit_test(long delay, long shift)
ret = 1;
}
}
rrddim_foreach_done(rd);
}
if(ret)
@ -1967,7 +1969,11 @@ static int test_dbengine_check_rrdr(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DIMS]
time_retrieved = r->t[c];
// for each dimension
for (j = 0, d = r->st->dimensions; d && j < r->d ; ++j, d = d->next) {
rrddim_foreach_read(d, r->st) {
if(unlikely((int)d_dfe.counter >= r->d)) break; // d_counter is provided by the dictionary dfe
j = (int)d_dfe.counter;
NETDATA_DOUBLE *cn = &r->v[ c * r->d ];
value = cn[j];
assert(rd[i][j] == d);
@ -1990,6 +1996,7 @@ static int test_dbengine_check_rrdr(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DIMS]
time_errors++;
}
}
rrddim_foreach_done(d);
}
rrdr_free(owa, r);
}
@ -2103,7 +2110,11 @@ int test_dbengine(void)
time_t time_retrieved = r->t[c];
// for each dimension
for(j = 0, d = r->st->dimensions; d && j < r->d ; ++j, d = d->next) {
rrddim_foreach_read(d, r->st) {
if(unlikely((int)d_dfe.counter >= r->d)) break; // d_counter is provided by the dictionary dfe
j = (int)d_dfe.counter;
NETDATA_DOUBLE *cn = &r->v[ c * r->d ];
NETDATA_DOUBLE value = cn[j];
assert(rd[i][j] == d);
@ -2126,6 +2137,7 @@ int test_dbengine(void)
time_errors++;
}
}
rrddim_foreach_done(d);
}
rrdr_free(owa, r);
}

2
database/engine/journalfile.c
View file

@ -519,7 +519,7 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
journalfile->file = file;
journalfile->pos = file_size;
journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0);
info("Loading journal file \"%s\" using %s.", path, journalfile->data?"using MMAP":"using uv_fs_read");
info("Loading journal file \"%s\" using %s.", path, journalfile->data?"MMAP":"uv_fs_read");
max_id = iterate_transactions(ctx, journalfile);

2
database/engine/rrdengineapi.c
View file

@ -138,7 +138,7 @@ STORAGE_METRIC_HANDLE *rrdeng_metric_init(RRDDIM *rd, STORAGE_INSTANCE *db_insta
uuid_copy(rd->metric_uuid, multihost_legacy_uuid);
if (unlikely(need_to_store && !ctx->tier))
(void)sql_store_dimension(&rd->metric_uuid, rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
(void)sql_store_dimension(&rd->metric_uuid, &rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
}
struct rrdeng_metric_handle *mh = mallocz(sizeof(struct rrdeng_metric_handle));

2
database/ram/rrddim_mem.c
View file

@ -42,7 +42,7 @@ void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec
void rrddim_store_metric_flush(STORAGE_COLLECT_HANDLE *collection_handle) {
struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
RRDDIM *rd = ch->rd;
memset(rd->db, 0, rd->entries * sizeof(storage_number));
memset(rd->db, 0, rd->rrdset->entries * sizeof(storage_number));
}
int rrddim_collect_finalize(STORAGE_COLLECT_HANDLE *collection_handle) {

463
database/rrd.h
View file

@ -16,20 +16,21 @@ typedef struct storage_metric_handle STORAGE_METRIC_HANDLE;
typedef struct rrdhost RRDHOST;
typedef struct rrddim RRDDIM;
typedef struct rrdset RRDSET;
typedef struct rrdvar RRDVAR;
typedef struct rrdsetvar RRDSETVAR;
typedef struct rrddimvar RRDDIMVAR;
typedef struct rrdcalc RRDCALC;
typedef struct rrdcalctemplate RRDCALCTEMPLATE;
typedef struct alarm_entry ALARM_ENTRY;
typedef struct context_param CONTEXT_PARAM;
typedef struct rrdfamily_acquired RRDFAMILY_ACQUIRED;
typedef struct rrdvar_acquired RRDVAR_ACQUIRED;
typedef struct rrdsetvar_acquired RRDSETVAR_ACQUIRED;
typedef struct rrdcalc_acquired RRDCALC_ACQUIRED;
typedef void *ml_host_t;
typedef void *ml_dimension_t;
// forward declarations
struct rrddim_tier;
struct rrdset_volatile;
struct context_param;
#ifdef ENABLE_DBENGINE
@ -75,11 +76,6 @@ struct context_param {
uint8_t flags;
};
#define META_CHART_UPDATED 1
#define META_PLUGIN_UPDATED 2
#define META_MODULE_UPDATED 4
#define META_CHART_ACTIVATED 8
#define UPDATE_EVERY 1
#define UPDATE_EVERY_MAX 3600
@ -122,7 +118,9 @@ typedef enum rrd_memory_mode {
RRD_MEMORY_MODE_MAP = 2,
RRD_MEMORY_MODE_SAVE = 3,
RRD_MEMORY_MODE_ALLOC = 4,
RRD_MEMORY_MODE_DBENGINE = 5
RRD_MEMORY_MODE_DBENGINE = 5,
// this is 8-bit
} RRD_MEMORY_MODE;
#define RRD_MEMORY_MODE_NONE_NAME "none"
@ -145,7 +143,9 @@ typedef enum rrd_algorithm {
RRD_ALGORITHM_ABSOLUTE = 0,
RRD_ALGORITHM_INCREMENTAL = 1,
RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL = 2,
RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL = 3
RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL = 3,
// this is 8-bit
} RRD_ALGORITHM;
#define RRD_ALGORITHM_ABSOLUTE_NAME "absolute"
@ -159,34 +159,42 @@ extern const char *rrd_algorithm_name(RRD_ALGORITHM algorithm);
// ----------------------------------------------------------------------------
// RRD FAMILY
struct rrdfamily {
STRING *family;
DICTIONARY *rrdvar_root_index;
size_t use_count;
};
typedef struct rrdfamily RRDFAMILY;
extern const RRDFAMILY_ACQUIRED *rrdfamily_add_and_acquire(RRDHOST *host, const char *id);
extern void rrdfamily_release(RRDHOST *host, const RRDFAMILY_ACQUIRED *rfa);
extern void rrdfamily_index_init(RRDHOST *host);
extern void rrdfamily_index_destroy(RRDHOST *host);
extern DICTIONARY *rrdfamily_rrdvars_dict(const RRDFAMILY_ACQUIRED *rf);
// ----------------------------------------------------------------------------
// flags
// use this for configuration flags, not for state control
// flags are set/unset in a manner that is not thread safe
// and may lead to missing information.
// flags & options
// options are permanent configuration options (no atomics to alter/access them)
typedef enum rrddim_options {
RRDDIM_OPTION_NONE = 0,
RRDDIM_OPTION_HIDDEN = (1 << 0), // this dimension will not be offered to callers
RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS = (1 << 1), // do not offer RESET or OVERFLOW info to callers
// this is 8-bit
} RRDDIM_OPTIONS;
#define rrddim_option_check(rd, flag) ((rd)->flags & (flag))
#define rrddim_option_set(rd, flag) (rd)->flags |= (flag)
#define rrddim_option_clear(rd, flag) (rd)->flags &= ~(flag)
// flags are runtime changing status flags (atomics are required to alter/access them)
typedef enum rrddim_flags {
RRDDIM_FLAG_NONE = 0,
RRDDIM_FLAG_HIDDEN = (1 << 0), // this dimension will not be offered to callers
RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS = (1 << 1), // do not offer RESET or OVERFLOW info to callers
RRDDIM_FLAG_OBSOLETE = (1 << 2), // this is marked by the collector/module as obsolete
// No new values have been collected for this dimension since agent start or it was marked RRDDIM_FLAG_OBSOLETE at
// No new values have been collected for this dimension since agent start, or it was marked RRDDIM_FLAG_OBSOLETE at
// least rrdset_free_obsolete_time seconds ago.
RRDDIM_FLAG_ARCHIVED = (1 << 3),
RRDDIM_FLAG_ACLK = (1 << 4),
RRDDIM_FLAG_PENDING_FOREACH_ALARM = (1 << 5), // set when foreach alarm has not been initialized yet
RRDDIM_FLAG_PENDING_FOREACH_ALARMS = (1 << 5), // set when foreach alarm has not been initialized yet
RRDDIM_FLAG_META_HIDDEN = (1 << 6), // Status of hidden option in the metadata database
RRDDIM_FLAG_INDEXED_ID = (1 << 7),
// this is 8 bit
} RRDDIM_FLAGS;
#define rrddim_flag_check(rd, flag) (__atomic_load_n(&((rd)->flags), __ATOMIC_SEQ_CST) & (flag))
@ -231,6 +239,7 @@ extern void rrdlabels_copy(DICTIONARY *dst, DICTIONARY *src);
void reload_host_labels(void);
extern void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels);
extern void rrdset_save_rrdlabels_to_sql(RRDSET *st);
extern int rrdlabels_unittest(void);
@ -244,13 +253,15 @@ struct rrddim {
uuid_t metric_uuid; // global UUID for this metric (unique_across hosts)
// ------------------------------------------------------------------------
// the dimension definition
// dimension definition
STRING *id; // the id of this dimension (for internal identification)
STRING *name; // the name of this dimension (as presented to user)
RRD_ALGORITHM algorithm; // the algorithm that is applied to add new collected values
RRD_MEMORY_MODE rrd_memory_mode; // the memory mode for this dimension
RRDDIM_FLAGS flags; // configuration flags for the dimension
RRD_ALGORITHM algorithm:8; // the algorithm that is applied to add new collected values
RRDDIM_OPTIONS options:8; // permanent configuration options
RRD_MEMORY_MODE rrd_memory_mode:8; // the memory mode for this dimension
/*RRDDIM_FLAGS*/ uint8_t flags; // run time changing status flags
bool updated; // 1 when the dimension has been updated since the last processing
bool exposed; // 1 when set what have sent this dimension to the central netdata
@ -258,20 +269,38 @@ struct rrddim {
collected_number multiplier; // the multiplier of the collected values
collected_number divisor; // the divider of the collected values
// ------------------------------------------------------------------------
// members for temporary data we need for calculations
int update_every; // every how many seconds is this updated
// TODO - remove update_every from rrddim
// it is always the same in rrdset
struct timeval last_collected_time; // when was this dimension last updated
// this is actual date time we updated the last_collected_value
// THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET
// ------------------------------------------------------------------------
// operational state members
#ifdef ENABLE_ACLK
int aclk_live_status;
#endif
ml_dimension_t ml_dimension;
ml_dimension_t ml_dimension; // machine learning data about this dimension
// ------------------------------------------------------------------------
// linking to siblings and parents
struct rrddim *next; // linking of dimensions within the same data set
struct rrddim *prev; // linking of dimensions within the same data set
struct rrdset *rrdset;
RRDMETRIC_ACQUIRED *rrdmetric; // the rrdmetric of this dimension
// ------------------------------------------------------------------------
// data collection members
struct rrddim_tier *tiers[RRD_STORAGE_TIERS]; // our tiers of databases
struct timeval last_collected_time; // when was this dimension last updated
// this is actual date time we updated the last_collected_value
// THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET
size_t collections_counter; // the number of times we added values to this rrddim
collected_number collected_value_max; // the absolute maximum of the collected value
@ -282,33 +311,12 @@ struct rrddim {
collected_number collected_value; // the current value, as collected - resets to 0 after being used
collected_number last_collected_value; // the last value that was collected, after being processed
// the *_volume members are used to calculate the accuracy of the rounding done by the
// storage number - they are printed to debug.log when debug is enabled for a set.
NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
struct rrddim *next; // linking of dimensions within the same data set
struct rrddim *prev; // linking of dimensions within the same data set
struct rrdset *rrdset;
RRDMETRIC_ACQUIRED *rrdmetric; // the rrdmetric of this dimension
// ------------------------------------------------------------------------
// members for checking the data when loading from disk
long entries; // how many entries this dimension has in ram
// this is the same to the entries of the data set
// we set it here, to check the data when we load it from disk.
int update_every; // every how many seconds is this updated
// db mode RAM, SAVE, MAP, ALLOC, NONE specifics
// TODO - they should be managed by storage engine
// (RRDDIM_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
size_t memsize; // the memory allocated for this dimension (without RRDDIM)
struct rrddimvar *variables;
// ------------------------------------------------------------------------
// the values stored in this dimension, using our floating point numbers
void *rd_on_file; // pointer to the header written on disk
storage_number *db; // the array of values
};
@ -346,7 +354,6 @@ struct rrddim_query_handle {
RRDDIM *rd;
time_t start_time;
time_t end_time;
TIER_QUERY_FETCH tier_query_fetch_type;
STORAGE_QUERY_HANDLE* handle;
};
@ -399,7 +406,7 @@ struct rrddim_collect_ops {
// run this to flush / reset the current data collection sequence
void (*flush)(STORAGE_COLLECT_HANDLE *collection_handle);
// an finalization function to run after collection is over
// a finalization function to run after collection is over
// returns 1 if it's safe to delete the dimension
int (*finalize)(STORAGE_COLLECT_HANDLE *collection_handle);
};
@ -432,7 +439,6 @@ struct rrddim_query_ops {
struct rrddim_tier {
int tier_grouping;
RRD_MEMORY_MODE mode; // the memory mode of this tier
RRD_BACKFILL backfill; // backfilling configuration
STORAGE_METRIC_HANDLE *db_metric_handle; // the metric handle inside the database
STORAGE_COLLECT_HANDLE *db_collection_handle; // the data collection handle
STORAGE_POINT virtual_point;
@ -448,11 +454,16 @@ extern void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, int tier, time_t n
// these loop macros make sure the linked list is accessed with the right lock
#define rrddim_foreach_read(rd, st) \
for((rd) = (st)->dimensions, rrdset_check_rdlock(st); (rd) ; (rd) = (rd)->next)
dfe_start_read((st)->rrddim_root_index, rd)
#define rrddim_foreach_write(rd, st) \
for((rd) = (st)->dimensions, rrdset_check_wrlock(st); (rd) ; (rd) = (rd)->next)
dfe_start_write((st)->rrddim_root_index, rd)
#define rrddim_foreach_reentrant(rd, st) \
dfe_start_reentrant((st)->rrddim_root_index, rd)
#define rrddim_foreach_done(rd) \
dfe_done(rd)
// ----------------------------------------------------------------------------
// RRDSET - this is a chart
@ -477,7 +488,7 @@ typedef enum rrdset_flags {
RRDSET_FLAG_HIDDEN = (1 << 12), // if set, do not show this chart on the dashboard, but use it for exporting
RRDSET_FLAG_SYNC_CLOCK = (1 << 13), // if set, microseconds on next data collection will be ignored (the chart will be synced to now)
RRDSET_FLAG_OBSOLETE_DIMENSIONS = (1 << 14), // this is marked by the collector/module when a chart has obsolete dimensions
// No new values have been collected for this chart since agent start or it was marked RRDSET_FLAG_OBSOLETE at
// No new values have been collected for this chart since agent start, or it was marked RRDSET_FLAG_OBSOLETE at
// least rrdset_free_obsolete_time seconds ago.
RRDSET_FLAG_ARCHIVED = (1 << 15),
RRDSET_FLAG_ACLK = (1 << 16),
@ -496,14 +507,19 @@ typedef enum rrdset_flags {
#define rrdset_is_ar_chart(st) rrdset_flag_check(st, RRDSET_FLAG_ANOMALY_RATE_CHART)
struct rrdset {
uuid_t uuid;
uuid_t chart_uuid; // the global UUID for this chart
// ------------------------------------------------------------------------
// the set configuration
// chart configuration
STRING *id; // the ID of the data set
STRING *name; // the name of this dimension (as presented to user)
STRING *type; // the type of graph RRD_TYPE_* (a category, for determining graphing options)
struct {
STRING *type; // the type of {type}.{id}
STRING *id; // the id of {type}.{id}
STRING *name; // the name of {type}.{name}
} parts;
STRING *id; // the unique ID of the rrdset as {type}.{id}
STRING *name; // the unique name of the rrdset as {type}.{name}
STRING *family; // grouping sets under the same family
STRING *title; // title shown to user
STRING *units; // units of measurement
@ -511,44 +527,51 @@ struct rrdset {
STRING *plugin_name; // the name of the plugin that generated this
STRING *module_name; // the name of the plugin module that generated this
RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
RRD_MEMORY_MODE rrd_memory_mode; // the db mode of this rrdset
RRDSET_TYPE chart_type; // line, area, stacked
RRDSET_FLAGS flags; // configuration flags
RRDSET_FLAGS *exporting_flags; // array of flags for exporting connector instances
int update_every; // every how many seconds is this updated?
int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored
// netdata will interpolate values for gaps lower than this
long entries; // total number of entries in the data set
long current_entry; // the entry that is currently being updated
// it goes around in a round-robin fashion
long priority; // the sorting priority of this chart
int update_every; // data collection frequency
DICTIONARY *rrdlabels; // chart labels
DICTIONARY *rrdsetvar_root_index; // chart variables
DICTIONARY *rrddimvar_root_index; // dimension variables
// we use this dictionary to manage their allocation
// TODO - dimensions linked list and lock to be removed
netdata_rwlock_t rrdset_rwlock; // protects the dimensions linked list
RRDDIM *dimensions; // chart metrics
// ------------------------------------------------------------------------
// members for temporary data we need for calculations
// operational state members
char *cache_dir; // the directory to store dimensions
RRDSET_FLAGS flags; // flags
RRD_MEMORY_MODE rrd_memory_mode; // the db mode of this rrdset
uuid_t hash_uuid; // hash_id for syncing with cloud
// TODO - obsolete now - cleanup
DICTIONARY *rrddim_root_index; // dimensions index
int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored
// netdata will interpolate values for gaps lower than this
// TODO - use the global - all charts have the same value
// ------------------------------------------------------------------------
// linking to siblings and parents
RRDHOST *rrdhost; // pointer to RRDHOST this chart belongs to
RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
// ------------------------------------------------------------------------
// data collection members
size_t counter; // the number of times we added values to this database
size_t counter_done; // the number of times rrdset_done() has been called
union {
time_t last_accessed_time; // the last time this RRDSET has been accessed
time_t last_entry_t; // the last_entry_t computed for transient RRDSET
};
time_t upstream_resync_time; // the timestamp up to which we should resync clock upstream
uuid_t *chart_uuid; // Store the global GUID for this chart
// this object.
size_t rrddim_page_alignment; // keeps metric pages in alignment when using dbengine
time_t last_accessed_time; // the last time this RRDSET has been accessed
usec_t usec_since_last_update; // the time in microseconds since the last collection of data
@ -558,46 +581,73 @@ struct rrdset {
total_number collected_total; // used internally to calculate percentages
total_number last_collected_total; // used internally to calculate percentages
RRDFAMILY *rrdfamily; // pointer to RRDFAMILY this chart belongs to
RRDHOST *rrdhost; // pointer to RRDHOST this chart belongs to
struct rrdset *next; // linking of rrdsets
struct rrdset *prev; // linking of rrdsets
size_t rrdlabels_last_saved_version;
// ------------------------------------------------------------------------
// local variables
// data collection - streaming to parents, temp variables
NETDATA_DOUBLE green; // green threshold for this chart
NETDATA_DOUBLE red; // red threshold for this chart
DICTIONARY *rrdvar_root_index; // RRDVAR index for this chart
RRDSETVAR *variables; // RRDSETVAR linked list for this chart (one RRDSETVAR, many RRDVARs)
RRDCALC *alarms; // RRDCALC linked list for this chart
time_t upstream_resync_time; // the timestamp up to which we should resync clock upstream
// ------------------------------------------------------------------------
// members for checking the data when loading from disk
// context queries temp variables
// TODO - eliminate these
time_t last_entry_t; // the last_entry_t computed for transient RRDSET
// ------------------------------------------------------------------------
// dbengine specifics
// TODO - they should be managed by storage engine
// (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
size_t rrddim_page_alignment; // keeps metric pages in alignment when using dbengine
// ------------------------------------------------------------------------
// db mode SAVE, MAP specifics
// TODO - they should be managed by storage engine
// (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
char *cache_dir; // the directory to store dimensions
unsigned long memsize; // how much mem we have allocated for this (without dimensions)
void *st_on_file; // compatibility with V019 RRDSET files
// ------------------------------------------------------------------------
// chart labels
// db mode RAM, SAVE, MAP, ALLOC, NONE specifics
// TODO - they should be managed by storage engine
// (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
DICTIONARY *rrdlabels;
long entries; // total number of entries in the data set
long current_entry; // the entry that is currently being updated
// it goes around in a round-robin fashion
// ------------------------------------------------------------------------
// the dimensions
// exporting to 3rd party time-series members
// TODO - they should be managed by exporting engine
// (RRDSET_EXPORTING_STATE ptr to an undefined structure, and a call to clean this up during destruction)
DICTIONARY *rrddim_root_index; // the root of the dimensions index
RRDSET_FLAGS *exporting_flags; // array of flags for exporting connector instances
netdata_rwlock_t rrdset_rwlock; // protects dimensions linked list
RRDDIM *dimensions; // the actual data for every dimension
// ------------------------------------------------------------------------
// health monitoring members
// TODO - they should be managed by health
// (RRDSET_HEALTH_STATE ptr to an undefined structure, and a call to clean this up during destruction)
NETDATA_DOUBLE green; // green threshold for this chart
NETDATA_DOUBLE red; // red threshold for this chart
DICTIONARY *rrdvars; // RRDVAR index for this chart
const RRDFAMILY_ACQUIRED *rrdfamily; // pointer to RRDFAMILY dictionary item, this chart belongs to
struct {
netdata_rwlock_t rwlock; // protection for RRDCALC *base
RRDCALC *base; // double linked list of RRDCALC related to this RRDSET
} alerts;
};
#define rrdset_plugin_name(st) string2str((st)->plugin_name)
#define rrdset_module_name(st) string2str((st)->module_name)
#define rrdset_units(st) string2str((st)->units)
#define rrdset_type(st) string2str((st)->type)
#define rrdset_parts_type(st) string2str((st)->parts.type)
#define rrdset_family(st) string2str((st)->family)
#define rrdset_title(st) string2str((st)->title)
#define rrdset_context(st) string2str((st)->context)
@ -614,11 +664,19 @@ extern STRING *rrd_string_strdupz(const char *s);
// these loop macros make sure the linked list is accessed with the right lock
#define rrdset_foreach_read(st, host) \
for((st) = (host)->rrdset_root, rrdhost_check_rdlock(host); st ; (st) = (st)->next)
dfe_start_read((host)->rrdset_root_index, st)
#define rrdset_foreach_write(st, host) \
for((st) = (host)->rrdset_root, rrdhost_check_wrlock(host); st ; (st) = (st)->next)
dfe_start_write((host)->rrdset_root_index, st)
#define rrdset_foreach_reentrant(st, host) \
dfe_start_reentrant((host)->rrdset_root_index, st)
#define rrdset_foreach_done(st) \
dfe_done(st)
#define rrdset_number_of_dimensions(st) \
dictionary_entries((st)->rrddim_root_index)
extern void rrdset_memory_file_save(RRDSET *st);
extern void rrdset_memory_file_free(RRDSET *st);
@ -633,19 +691,23 @@ extern bool rrdset_memory_load_or_create_map_save(RRDSET *st_on_file, RRD_MEMORY
// and may lead to missing information.
typedef enum rrdhost_flags {
RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
RRDHOST_FLAG_INDEXED_MACHINE_GUID = (1 << 11), // when set, we have indexed its machine guid
RRDHOST_FLAG_INDEXED_HOSTNAME = (1 << 12), // when set, we have indexed its hostname
RRDHOST_FLAG_STREAM_COLLECTED_METRICS = (1 << 13), // when set, rrdset_done() should push metrics to parent
RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
RRDHOST_FLAG_INDEXED_MACHINE_GUID = (1 << 11), // when set, we have indexed its machine guid
RRDHOST_FLAG_INDEXED_HOSTNAME = (1 << 12), // when set, we have indexed its hostname
RRDHOST_FLAG_STREAM_COLLECTED_METRICS = (1 << 13), // when set, rrdset_done() should push metrics to parent
RRDHOST_FLAG_INITIALIZED_HEALTH = (1 << 14), // the host has initialized health structures
RRDHOST_FLAG_INITIALIZED_RRDPUSH = (1 << 15), // the host has initialized rrdpush structures
RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS = (1 << 16), // the host has pending chart obsoletions
RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS = (1 << 17), // the host has pending dimension obsoletions
} RRDHOST_FLAGS;
#define rrdhost_flag_check(host, flag) (__atomic_load_n(&((host)->flags), __ATOMIC_SEQ_CST) & (flag))
@ -868,27 +930,18 @@ struct rrdhost {
// all RRDCALCs are primarily allocated and linked here
// RRDCALCs may be linked to charts at any point
// (charts may or may not exist when these are loaded)
RRDCALC *host_alarms;
RRDCALC *alarms_with_foreach;
DICTIONARY *rrdcalc_root_index;
// templates of alarms
// these are used to create alarms when charts
// are created or renamed, that match them
DICTIONARY *rrdcalctemplate_root_index;
ALARM_LOG health_log; // alarms historical events (event log)
uint32_t health_last_processed_id; // the last processed health id from the log
uint32_t health_max_unique_id; // the max alarm log unique id given for the host
uint32_t health_max_alarm_id; // the max alarm id given for the host
// templates of alarms
// these are used to create alarms when charts
// are created or renamed, that match them
RRDCALCTEMPLATE *alarms_templates;
// ------------------------------------------------------------------------
// the charts of the host
RRDSET *rrdset_root; // the host charts
unsigned int obsolete_charts_count;
// ------------------------------------------------------------------------
// locks
@ -909,7 +962,8 @@ struct rrdhost {
DICTIONARY *rrdset_root_index_name; // the host's charts index (by name)
DICTIONARY *rrdfamily_root_index; // the host's chart families index
DICTIONARY *rrdvar_root_index; // the host's chart variables index
DICTIONARY *rrdvars; // the host's chart variables index
// this includes custom host variables
STORAGE_INSTANCE *storage_instance[RRD_STORAGE_TIERS]; // the database instances of the storage tiers
@ -973,6 +1027,11 @@ extern netdata_rwlock_t rrd_rwlock;
// ----------------------------------------------------------------------------
extern bool is_storage_engine_shared(STORAGE_INSTANCE *engine);
extern void rrdset_index_init(RRDHOST *host);
extern void rrdset_index_destroy(RRDHOST *host);
extern void rrddim_index_init(RRDSET *st);
extern void rrddim_index_destroy(RRDSET *st);
// ----------------------------------------------------------------------------
@ -1058,7 +1117,7 @@ extern void __rrd_check_wrlock(const char *file, const char *function, const uns
// ----------------------------------------------------------------------------
// RRDSET functions
extern int rrdset_set_name(RRDSET *st, const char *name);
extern int rrdset_reset_name(RRDSET *st, const char *name);
extern RRDSET *rrdset_create_custom(RRDHOST *host
, const char *type
@ -1086,12 +1145,10 @@ extern void rrdhost_free_all(void);
extern void rrdhost_save_all(void);
extern void rrdhost_cleanup_all(void);
extern void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host);
extern void rrdhost_system_info_free(struct rrdhost_system_info *system_info);
extern void rrdhost_free(RRDHOST *host, bool force);
extern void rrdhost_save_charts(RRDHOST *host);
extern void rrdhost_delete_charts(RRDHOST *host);
extern void rrd_cleanup_obsolete_charts();
extern int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now);
@ -1140,95 +1197,19 @@ extern void rrdset_is_obsolete(RRDSET *st);
extern void rrdset_isnot_obsolete(RRDSET *st);
// checks if the RRDSET should be offered to viewers
#define rrdset_is_available_for_viewers(st) (!rrdset_flag_check(st, RRDSET_FLAG_HIDDEN) && !rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions && (st)->rrd_memory_mode != RRD_MEMORY_MODE_NONE)
#define rrdset_is_available_for_viewers(st) (!rrdset_flag_check(st, RRDSET_FLAG_HIDDEN) && !rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && rrdset_number_of_dimensions(st) && (st)->rrd_memory_mode != RRD_MEMORY_MODE_NONE)
#define rrdset_is_available_for_exporting_and_alarms(st) (!rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions)
#define rrdset_is_archived(st) (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions)
// get the timestamp of the last entry in the round robin database
static inline time_t rrddim_last_entry_t(RRDDIM *rd) {
time_t latest = rd->tiers[0]->query_ops.latest_time(rd->tiers[0]->db_metric_handle);
for(int tier = 1; tier < storage_tiers ;tier++) {
if(unlikely(!rd->tiers[tier])) continue;
time_t t = rd->tiers[tier]->query_ops.latest_time(rd->tiers[tier]->db_metric_handle);
if(t > latest)
latest = t;
}
return latest;
}
static inline time_t rrddim_first_entry_t(RRDDIM *rd) {
time_t oldest = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(unlikely(!rd->tiers[tier])) continue;
time_t t = rd->tiers[tier]->query_ops.oldest_time(rd->tiers[tier]->db_metric_handle);
if(t != 0 && (oldest == 0 || t < oldest))
oldest = t;
}
return oldest;
}
// get the timestamp of the last entry in the round robin database
static inline time_t rrdset_last_entry_t_nolock(RRDSET *st) {
RRDDIM *rd;
time_t last_entry_t = 0;
rrddim_foreach_read(rd, st) {
time_t t = rrddim_last_entry_t(rd);
if(t > last_entry_t) last_entry_t = t;
}
return last_entry_t;
}
static inline time_t rrdset_last_entry_t(RRDSET *st) {
time_t last_entry_t;
netdata_rwlock_rdlock(&st->rrdset_rwlock);
last_entry_t = rrdset_last_entry_t_nolock(st);
netdata_rwlock_unlock(&st->rrdset_rwlock);
return last_entry_t;
}
// get the timestamp of first entry in the round robin database
static inline time_t rrdset_first_entry_t_nolock(RRDSET *st) {
RRDDIM *rd;
time_t first_entry_t = LONG_MAX;
rrddim_foreach_read(rd, st) {
time_t t = rrddim_first_entry_t(rd);
if(t < first_entry_t)
first_entry_t = t;
}
if (unlikely(LONG_MAX == first_entry_t)) return 0;
return first_entry_t;
}
static inline time_t rrdset_first_entry_t(RRDSET *st)
{
time_t first_entry_t;
netdata_rwlock_rdlock(&st->rrdset_rwlock);
first_entry_t = rrdset_first_entry_t_nolock(st);
netdata_rwlock_unlock(&st->rrdset_rwlock);
return first_entry_t;
}
time_t rrdhost_last_entry_t(RRDHOST *h);
extern time_t rrddim_first_entry_t(RRDDIM *rd);
extern time_t rrddim_last_entry_t(RRDDIM *rd);
extern time_t rrdset_last_entry_t(RRDSET *st);
extern time_t rrdset_first_entry_t(RRDSET *st);
extern time_t rrdhost_last_entry_t(RRDHOST *h);
// ----------------------------------------------------------------------------
// RRD DIMENSION functions
extern void rrdcalc_link_to_rrddim(RRDDIM *rd, RRDSET *st, RRDHOST *host);
extern RRDDIM *rrddim_add_custom(RRDSET *st
, const char *id
, const char *name
@ -1241,7 +1222,7 @@ extern RRDDIM *rrddim_add_custom(RRDSET *st
#define rrddim_add(st, id, name, multiplier, divisor, algorithm) \
rrddim_add_custom(st, id, name, multiplier, divisor, algorithm, (st)->rrd_memory_mode)
extern int rrddim_set_name(RRDSET *st, RRDDIM *rd, const char *name);
extern int rrddim_reset_name(RRDSET *st, RRDDIM *rd, const char *name);
extern int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm);
extern int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multiplier);
extern int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor);
@ -1270,19 +1251,17 @@ extern char *rrdset_strncpyz_name(char *to, const char *from, size_t length);
// ----------------------------------------------------------------------------
// RRD internal functions
extern void rrdset_delete_files(RRDSET *st);
extern void rrdset_save(RRDSET *st);
extern void rrdset_free(RRDSET *st);
#ifdef NETDATA_RRD_INTERNALS
extern char *rrdset_cache_dir(RRDHOST *host, const char *id);
extern void rrddim_free(RRDSET *st, RRDDIM *rd);
extern RRDFAMILY *rrdfamily_create(RRDHOST *host, const char *id);
extern void rrdfamily_free(RRDHOST *host, RRDFAMILY *rc);
extern void rrdset_free(RRDSET *st);
extern void rrdset_reset(RRDSET *st);
extern void rrdset_save(RRDSET *st);
extern void rrdset_delete_files(RRDSET *st);
extern void rrdset_delete_obsolete_dimensions(RRDSET *st);
extern RRDHOST *rrdhost_create(

875
database/rrdcalc.c
View file

File diff suppressed because it is too large Load diff

111
database/rrdcalc.h
View file

@ -10,28 +10,35 @@
// (defined in their update_every member below)
// They increase the overhead of netdata.
//
// These calculations are allocated and linked (->next)
// under RRDHOST.
// Then are also linked to RRDSET (of course only when the
// chart is found, via ->rrdset_next and ->rrdset_prev).
// This double-linked list is maintained sorted at all times
// having as RRDSET.calculations the RRDCALC to be processed
// next.
// These calculations are stored under RRDHOST.
// Then are also linked to RRDSET (of course only when a
// matching chart is found).
#define RRDCALC_FLAG_DB_ERROR 0x00000001
#define RRDCALC_FLAG_DB_NAN 0x00000002
/* #define RRDCALC_FLAG_DB_STALE 0x00000004 */
#define RRDCALC_FLAG_CALC_ERROR 0x00000008
#define RRDCALC_FLAG_WARN_ERROR 0x00000010
#define RRDCALC_FLAG_CRIT_ERROR 0x00000020
#define RRDCALC_FLAG_RUNNABLE 0x00000040
#define RRDCALC_FLAG_DISABLED 0x00000080
#define RRDCALC_FLAG_SILENCED 0x00000100
#define RRDCALC_FLAG_RUN_ONCE 0x00000200
#define RRDCALC_FLAG_NO_CLEAR_NOTIFICATION 0x80000000
typedef enum {
RRDCALC_FLAG_DB_ERROR = (1 << 0),
RRDCALC_FLAG_DB_NAN = (1 << 1),
// RRDCALC_FLAG_DB_STALE = (1 << 2),
RRDCALC_FLAG_CALC_ERROR = (1 << 3),
RRDCALC_FLAG_WARN_ERROR = (1 << 4),
RRDCALC_FLAG_CRIT_ERROR = (1 << 5),
RRDCALC_FLAG_RUNNABLE = (1 << 6),
RRDCALC_FLAG_DISABLED = (1 << 7),
RRDCALC_FLAG_SILENCED = (1 << 8),
RRDCALC_FLAG_RUN_ONCE = (1 << 9),
RRDCALC_FLAG_FROM_TEMPLATE = (1 << 10), // the rrdcalc has been created from a template
} RRDCALC_FLAGS;
typedef enum {
// This list uses several other options from RRDR_OPTIONS for db lookups.
// To add an item here, you need to reserve a bit in RRDR_OPTIONS.
RRDCALC_OPTION_NO_CLEAR_NOTIFICATION = 0x80000000,
} RRDCALC_OPTIONS;
#define RRDCALC_ALL_OPTIONS_EXCLUDING_THE_RRDR_ONES (RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)
struct rrdcalc {
STRING *key; // the unique key in the host's rrdcalc_root_index
uint32_t id; // the unique id of this alarm
uint32_t next_event_id; // the next event id that will be used for this alarm
@ -68,14 +75,12 @@ struct rrdcalc {
// database lookup settings
STRING *dimensions; // the chart dimensions
STRING *foreachdim; // the group of dimensions that the `foreach` will be applied.
SIMPLE_PATTERN *spdim; // used if and only if there is a simple pattern for the chart.
int foreachcounter; // the number of alarms created with foreachdim, this also works as an id of the
// children
STRING *foreach_dimension; // the group of dimensions that the `foreach` will be applied.
SIMPLE_PATTERN *foreach_dimension_pattern; // used if and only if there is a simple pattern for the chart.
RRDR_GROUPING group; // grouping method: average, max, etc.
int before; // ending point in time-series
int after; // starting point in time-series
uint32_t options; // calculation options
RRDCALC_OPTIONS options; // configuration options
// ------------------------------------------------------------------------
// expressions related to the alarm
@ -113,7 +118,7 @@ struct rrdcalc {
NETDATA_DOUBLE value; // the current value of the alarm
NETDATA_DOUBLE old_value; // the previous value of the alarm
uint32_t rrdcalc_flags; // check RRDCALC_FLAG_*
RRDCALC_FLAGS run_flags; // check RRDCALC_FLAG_*
time_t last_updated; // the last update timestamp of the alarm
time_t next_update; // the next update timestamp of the alarm
@ -132,20 +137,17 @@ struct rrdcalc {
// ------------------------------------------------------------------------
// variables this alarm exposes to the rest of the alarms
RRDVAR *local;
RRDVAR *family;
RRDVAR *hostid;
RRDVAR *hostname;
const RRDVAR_ACQUIRED *rrdvar_local;
const RRDVAR_ACQUIRED *rrdvar_family;
const RRDVAR_ACQUIRED *rrdvar_host_chart_id;
const RRDVAR_ACQUIRED *rrdvar_host_chart_name;
// ------------------------------------------------------------------------
// the chart this alarm it is linked to
size_t labels_version;
struct rrdset *rrdset;
// linking of this alarm on its chart
struct rrdcalc *rrdset_next;
struct rrdcalc *rrdset_prev;
struct rrdcalc *next;
struct rrdcalc *prev;
};
@ -164,14 +166,17 @@ struct rrdcalc {
#define rrdcalc_original_info(rc) string2str((rc)->original_info)
#define rrdcalc_info(rc) string2str((rc)->info)
#define rrdcalc_dimensions(rc) string2str((rc)->dimensions)
#define rrdcalc_foreachdim(rc) string2str((rc)->foreachdim)
#define rrdcalc_foreachdim(rc) string2str((rc)->foreach_dimension)
#define rrdcalc_host_labels(rc) string2str((rc)->host_labels)
#define foreach_rrdcalc_in_rrdset(st, rc) \
DOUBLE_LINKED_LIST_FOREACH_FORWARD((st)->alarms, rc, rrdset_prev, rrdset_next)
#define foreach_rrdcalc_in_rrdhost_read(host, rc) \
dfe_start_read((host)->rrdcalc_root_index, rc) \
#define foreach_rrdcalc_in_rrdhost(host, rc) \
DOUBLE_LINKED_LIST_FOREACH_FORWARD((host)->host_alarms, rc, prev, next)
#define foreach_rrdcalc_in_rrdhost_reentrant(host, rc) \
dfe_start_reentrant((host)->rrdcalc_root_index, rc)
#define foreach_rrdcalc_in_rrdhost_done(rc) \
dfe_done(rc)
struct alert_config {
STRING *alarm;
@ -213,26 +218,24 @@ struct alert_config {
#define RRDCALC_HAS_DB_LOOKUP(rc) ((rc)->after)
extern void rrdsetcalc_link_matching(RRDSET *st);
extern void rrdsetcalc_unlink(RRDCALC *rc);
extern RRDCALC *rrdcalc_find(RRDSET *st, const char *name);
extern void rrdcalc_update_info_using_rrdset_labels(RRDCALC *rc);
extern void rrdcalc_link_matching_alerts_to_rrdset(RRDSET *st);
extern const RRDCALC_ACQUIRED *rrdcalc_from_rrdset_get(RRDSET *st, const char *alert_name);
extern void rrdcalc_from_rrdset_release(RRDSET *st, const RRDCALC_ACQUIRED *rca);
extern RRDCALC *rrdcalc_acquired_to_rrdcalc(const RRDCALC_ACQUIRED *rca);
extern const char *rrdcalc_status2string(RRDCALC_STATUS status);
extern void rrdcalc_free(RRDCALC *rc);
extern void rrdcalc_unlink_and_free(RRDHOST *host, RRDCALC *rc);
extern void rrdcalc_free_unused_rrdcalc_loaded_from_config(RRDCALC *rc);
extern int rrdcalc_exists(RRDHOST *host, const char *chart, const char *name);
extern uint32_t rrdcalc_get_unique_id(RRDHOST *host, STRING *chart, STRING *name, uint32_t *next_event_id);
extern RRDCALC *rrdcalc_create_from_template(RRDHOST *host, RRDCALCTEMPLATE *rt, const char *chart);
extern RRDCALC *rrdcalc_create_from_rrdcalc(RRDCALC *rc, RRDHOST *host, const char *name, const char *dimension);
extern void rrdcalc_add_to_host(RRDHOST *host, RRDCALC *rc);
extern void dimension_remove_pipe_comma(char *str);
extern char *alarm_name_with_dim(const char *name, size_t namelen, const char *dim, size_t dimlen);
extern void rrdcalc_update_rrdlabels(RRDSET *st);
extern void rrdcalc_add_from_rrdcalctemplate(RRDHOST *host, RRDCALCTEMPLATE *rt, RRDSET *st, const char *overwrite_alert_name, const char *overwrite_dimensions);
extern int rrdcalc_add_from_config(RRDHOST *host, RRDCALC *rc);
extern void rrdcalc_labels_unlink();
extern void rrdcalc_labels_unlink_alarm_from_host(RRDHOST *host);
extern void rrdcalc_delete_alerts_not_matching_host_labels_from_all_hosts();
extern void rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(RRDHOST *host);
static inline int rrdcalc_isrepeating(RRDCALC *rc) {
if (unlikely(rc->warn_repeat_every > 0 || rc->crit_repeat_every > 0)) {
@ -241,6 +244,12 @@ static inline int rrdcalc_isrepeating(RRDCALC *rc) {
return 0;
}
extern void rrdcalc_unlink_all_rrdset_alerts(RRDSET *st);
extern void rrdcalc_delete_all(RRDHOST *host);
extern void rrdcalc_rrdhost_index_init(RRDHOST *host);
extern void rrdcalc_rrdhost_index_destroy(RRDHOST *host);
#define RRDCALC_VAR_MAX 100
#define RRDCALC_VAR_FAMILY "$family"
#define RRDCALC_VAR_LABEL "$label:"

208
database/rrdcalctemplate.c
View file

@ -1,6 +1,5 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#define NETDATA_HEALTH_INTERNALS
#include "rrd.h"
// ----------------------------------------------------------------------------
@ -11,45 +10,84 @@
* @param rt is the template used to create the chart.
* @param st is the chart where the alarm will be attached.
*/
void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
if(rt->context != st->context)
return;
if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st)))
return;
static char *rrdcalc_alert_name_with_dimension(const char *name, size_t namelen, const char *dim, size_t dimlen) {
char *newname,*move;
newname = mallocz(namelen + dimlen + 2);
move = newname;
memcpy(move, name, namelen);
move += namelen;
*move++ = '_';
memcpy(move, dim, dimlen);
move += dimlen;
*move = '\0';
return newname;
}
bool rrdcalctemplate_check_rrdset_conditions(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
if(rt->context != st->context)
return false;
if(rt->foreach_dimension_pattern && !rrdset_number_of_dimensions(st))
return false;
if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st)) && !simple_pattern_matches(rt->charts_pattern, rrdset_id(st)))
return false;
if (rt->family_pattern && !simple_pattern_matches(rt->family_pattern, rrdset_family(st)))
return;
return false;
if (rt->module_pattern && !simple_pattern_matches(rt->module_pattern, rrdset_module_name(st)))
return;
return false;
if (rt->plugin_pattern && !simple_pattern_matches(rt->plugin_pattern, rrdset_plugin_name(st)))
return;
return false;
if(host->rrdlabels && rt->host_labels_pattern && !rrdlabels_match_simple_pattern_parsed(host->rrdlabels, rt->host_labels_pattern, '='))
return false;
return true;
}
void rrdcalctemplate_check_rrddim_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDDIM *rd, RRDHOST *host) {
if (simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_id(rd)) || simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_name(rd))) {
char *overwrite_alert_name = rrdcalc_alert_name_with_dimension(
rrdcalctemplate_name(rt), string_strlen(rt->name), rrddim_name(rd), string_strlen(rd->name));
rrdcalc_add_from_rrdcalctemplate(host, rt, st, overwrite_alert_name, rrddim_name(rd));
freez(overwrite_alert_name);
}
}
void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
if(!rrdcalctemplate_check_rrdset_conditions(rt, st, host))
return;
RRDCALC *rc = rrdcalc_create_from_template(host, rt, rrdset_id(st));
if (unlikely(!rc))
info("Health tried to create alarm from template '%s' on chart '%s' of host '%s', but it failed", rrdcalctemplate_name(rt), rrdset_id(st), rrdhost_hostname(host));
#ifdef NETDATA_INTERNAL_CHECKS
else if (rc->rrdset != st && !rc->foreachdim) //When we have a template with foreachdim, the child will be added to the index late
error("Health alarm '%s.%s' should be linked to chart '%s', but it is not", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdset_id(st));
#endif
if(!rt->foreach_dimension_pattern) {
rrdcalc_add_from_rrdcalctemplate(host, rt, st, NULL, NULL);
return;
}
RRDDIM *rd;
rrddim_foreach_read(rd, st) {
rrdcalctemplate_check_rrddim_conditions_and_link(rt, st, rd, host);
}
rrddim_foreach_done(rd);
}
void rrdcalctemplate_link_matching(RRDSET *st) {
void rrdcalctemplate_link_matching_templates_to_rrdset(RRDSET *st) {
RRDHOST *host = st->rrdhost;
RRDCALCTEMPLATE *rt;
foreach_rrdcalctemplate_in_rrdhost(host, rt)
RRDCALCTEMPLATE *rt;
foreach_rrdcalctemplate_read(host, rt) {
rrdcalctemplate_check_conditions_and_link(rt, st, host);
}
foreach_rrdcalctemplate_done(rt);
}
inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) {
if(unlikely(!rt)) return;
static void rrdcalctemplate_free_internals(RRDCALCTEMPLATE *rt) {
expression_free(rt->calculation);
expression_free(rt->warning);
expression_free(rt->critical);
@ -77,19 +115,127 @@ inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) {
string_freez(rt->units);
string_freez(rt->info);
string_freez(rt->dimensions);
string_freez(rt->foreachdim);
string_freez(rt->foreach_dimension);
string_freez(rt->host_labels);
simple_pattern_free(rt->spdim);
simple_pattern_free(rt->foreach_dimension_pattern);
simple_pattern_free(rt->host_labels_pattern);
freez(rt);
}
inline void rrdcalctemplate_unlink_and_free(RRDHOST *host, RRDCALCTEMPLATE *rt) {
void rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(RRDCALCTEMPLATE *rt) {
if(unlikely(!rt)) return;
debug(D_HEALTH, "Health removing template '%s' of host '%s'", rrdcalctemplate_name(rt), rrdhost_hostname(host));
DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->alarms_templates, rt, prev, next);
rrdcalctemplate_free(rt);
rrdcalctemplate_free_internals(rt);
freez(rt);
}
static void rrdcalctemplate_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *added_bool) {
RRDCALCTEMPLATE *rt = rrdcalctemplate; (void)rt;
bool *added = added_bool;
*added = true;
debug(D_HEALTH, "Health configuration adding template '%s'"
": context '%s'"
", exec '%s'"
", recipient '%s'"
", green " NETDATA_DOUBLE_FORMAT_AUTO
", red " NETDATA_DOUBLE_FORMAT_AUTO
", lookup: group %d"
", after %d"
", before %d"
", options %u"
", dimensions '%s'"
", for each dimension '%s'"
", update every %d"
", calculation '%s'"
", warning '%s'"
", critical '%s'"
", source '%s'"
", delay up %d"
", delay down %d"
", delay max %d"
", delay_multiplier %f"
", warn_repeat_every %u"
", crit_repeat_every %u",
rrdcalctemplate_name(rt),
(rt->context)?string2str(rt->context):"NONE",
(rt->exec)?rrdcalctemplate_exec(rt):"DEFAULT",
(rt->recipient)?rrdcalctemplate_recipient(rt):"DEFAULT",
rt->green,
rt->red,
(int)rt->group,
rt->after,
rt->before,
rt->options,
(rt->dimensions)?rrdcalctemplate_dimensions(rt):"NONE",
(rt->foreach_dimension)?rrdcalctemplate_foreachdim(rt):"NONE",
rt->update_every,
(rt->calculation)?rt->calculation->parsed_as:"NONE",
(rt->warning)?rt->warning->parsed_as:"NONE",
(rt->critical)?rt->critical->parsed_as:"NONE",
rrdcalctemplate_source(rt),
rt->delay_up_duration,
rt->delay_down_duration,
rt->delay_max_duration,
rt->delay_multiplier,
rt->warn_repeat_every,
rt->crit_repeat_every
);
}
static void rrdcalctemplate_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *rrdhost __maybe_unused) {
RRDCALCTEMPLATE *rt = rrdcalctemplate;
rrdcalctemplate_free_internals(rt);
}
void rrdcalctemplate_index_init(RRDHOST *host) {
if(!host->rrdcalctemplate_root_index) {
host->rrdcalctemplate_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_insert_callback, NULL);
dictionary_register_delete_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_delete_callback, host);
}
}
void rrdcalctemplate_index_destroy(RRDHOST *host) {
dictionary_destroy(host->rrdcalctemplate_root_index);
host->rrdcalctemplate_root_index = NULL;
}
inline void rrdcalctemplate_delete_all(RRDHOST *host) {
dictionary_flush(host->rrdcalctemplate_root_index);
}
#define RRDCALCTEMPLATE_MAX_KEY_SIZE 1024
static size_t rrdcalctemplate_key(char *dst, size_t dst_len, const char *name, const char *family_match) {
return snprintfz(dst, dst_len, "%s/%s", name, (family_match && *family_match)?family_match:"*");
}
void rrdcalctemplate_add_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) {
if(unlikely(!rt->context)) {
error("Health configuration for template '%s' does not have a context", rrdcalctemplate_name(rt));
return;
}
if(unlikely(!rt->update_every)) {
error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalctemplate_name(rt));
return;
}
if(unlikely(!RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) && !rt->calculation && !rt->warning && !rt->critical)) {
error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rrdcalctemplate_name(rt));
return;
}
char key[RRDCALCTEMPLATE_MAX_KEY_SIZE + 1];
size_t key_len = rrdcalctemplate_key(key, RRDCALCTEMPLATE_MAX_KEY_SIZE, rrdcalctemplate_name(rt), rrdcalctemplate_family_match(rt));
bool added = false;
dictionary_set_advanced(host->rrdcalctemplate_root_index, key, (ssize_t)(key_len + 1), rt, sizeof(*rt), &added);
if(added)
freez(rt);
else {
info("Health configuration template '%s' already exists for host '%s'.", rrdcalctemplate_name(rt), rrdhost_hostname(host));
rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
}
}

35
database/rrdcalctemplate.h
View file

@ -48,14 +48,12 @@ struct rrdcalctemplate {
// database lookup settings
STRING *dimensions; // the chart dimensions
STRING *foreachdim; // the group of dimensions that the lookup will be applied.
SIMPLE_PATTERN *spdim; // used if and only if there is a simple pattern for the chart.
int foreachcounter; // the number of alarms created with foreachdim, this also works as an id of the
// children
STRING *foreach_dimension; // the group of dimensions that the lookup will be applied.
SIMPLE_PATTERN *foreach_dimension_pattern; // used if and only if there is a simple pattern for the chart.
RRDR_GROUPING group; // grouping method: average, max, etc.
int before; // ending point in time-series
int after; // starting point in time-series
uint32_t options; // calculation options
RRDCALC_OPTIONS options; // configuration options
// ------------------------------------------------------------------------
// notification delay settings
@ -87,8 +85,11 @@ struct rrdcalctemplate {
struct rrdcalctemplate *prev;
};
#define foreach_rrdcalctemplate_in_rrdhost(host, rt) \
DOUBLE_LINKED_LIST_FOREACH_FORWARD((host)->alarms_templates, rt, prev, next)
#define foreach_rrdcalctemplate_read(host, rt) \
dfe_start_read((host)->rrdcalctemplate_root_index, rt)
#define foreach_rrdcalctemplate_done(rt) \
dfe_done(rt)
#define rrdcalctemplate_name(rt) string2str((rt)->name)
#define rrdcalctemplate_exec(rt) string2str((rt)->exec)
@ -104,14 +105,24 @@ struct rrdcalctemplate {
#define rrdcalctemplate_info(rt) string2str((rt)->info)
#define rrdcalctemplate_source(rt) string2str((rt)->source)
#define rrdcalctemplate_dimensions(rt) string2str((rt)->dimensions)
#define rrdcalctemplate_foreachdim(rt) string2str((rt)->foreachdim)
#define rrdcalctemplate_foreachdim(rt) string2str((rt)->foreach_dimension)
#define rrdcalctemplate_host_labels(rt) string2str((rt)->host_labels)
#define RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) ((rt)->after)
extern void rrdcalctemplate_link_matching(RRDSET *st);
extern void rrdcalctemplate_link_matching_templates_to_rrdset(RRDSET *st);
extern void rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(RRDCALCTEMPLATE *rt);
extern void rrdcalctemplate_delete_all(RRDHOST *host);
extern void rrdcalctemplate_add_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt);
extern void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host);
extern bool rrdcalctemplate_check_rrdset_conditions(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host);
extern void rrdcalctemplate_check_rrddim_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDDIM *rd, RRDHOST *host);
extern void rrdcalctemplate_index_init(RRDHOST *host);
extern void rrdcalctemplate_index_destroy(RRDHOST *host);
extern void rrdcalctemplate_free(RRDCALCTEMPLATE *rt);
extern void rrdcalctemplate_unlink_and_free(RRDHOST *host, RRDCALCTEMPLATE *rt);
extern void rrdcalctemplate_create_alarms(RRDHOST *host, RRDCALCTEMPLATE *rt, RRDSET *st);
#endif //NETDATA_RRDCALCTEMPLATE_H

188
database/rrdcontext.c
View file

@ -8,6 +8,8 @@
int rrdcontext_enabled = CONFIG_BOOLEAN_YES;
// #define LOG_POST_PROCESSING_QUEUE_INSERTIONS 1
#define MESSAGES_PER_BUNDLE_TO_SEND_TO_HUB_PER_HOST 5000
#define FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS 120
#define RRDCONTEXT_WORKER_THREAD_HEARTBEAT_USEC (1000 * USEC_PER_MS)
@ -432,11 +434,11 @@ static void rrdmetric_free(RRDMETRIC *rm) {
// called when this rrdmetric is inserted to the rrdmetrics dictionary of a rrdinstance
// the constructor of the rrdmetric object
static void rrdmetric_insert_callback(const char *id __maybe_unused, void *value, void *data) {
static void rrdmetric_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance) {
RRDMETRIC *rm = value;
// link it to its parent
rm->ri = data;
rm->ri = rrdinstance;
// remove flags that we need to figure out at runtime
rm->flags = rm->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS; // no need for atomics
@ -447,7 +449,7 @@ static void rrdmetric_insert_callback(const char *id __maybe_unused, void *value
// called when this rrdmetric is deleted from the rrdmetrics dictionary of a rrdinstance
// the destructor of the rrdmetric object
static void rrdmetric_delete_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
static void rrdmetric_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance __maybe_unused) {
RRDMETRIC *rm = value;
internal_error(rm->rrddim, "RRDMETRIC: '%s' is freed but there is a RRDDIM linked to it.", string2str(rm->id));
@ -458,7 +460,7 @@ static void rrdmetric_delete_callback(const char *id __maybe_unused, void *value
// called when the same rrdmetric is inserted again to the rrdmetrics dictionary of a rrdinstance
// while this is called, the dictionary is write locked, but there may be other users of the object
static void rrdmetric_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdinstance __maybe_unused) {
RRDMETRIC *rm = oldv;
RRDMETRIC *rm_new = newv;
@ -518,11 +520,12 @@ static void rrdmetric_conflict_callback(const char *id __maybe_unused, void *old
rrdmetric_free(rm_new);
// the react callback will continue from here
return rrd_flag_is_updated(rm);
}
// this is called after the insert or the conflict callbacks,
// but the dictionary is now unlocked
static void rrdmetric_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
static void rrdmetric_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance __maybe_unused) {
RRDMETRIC *rm = value;
rrdmetric_trigger_updates(rm, __FUNCTION__ );
}
@ -531,11 +534,11 @@ static void rrdmetrics_create_in_rrdinstance(RRDINSTANCE *ri) {
if(unlikely(!ri)) return;
if(likely(ri->rrdmetrics)) return;
ri->rrdmetrics = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(ri->rrdmetrics, rrdmetric_insert_callback, (void *)ri);
dictionary_register_delete_callback(ri->rrdmetrics, rrdmetric_delete_callback, (void *)ri);
dictionary_register_conflict_callback(ri->rrdmetrics, rrdmetric_conflict_callback, (void *)ri);
dictionary_register_react_callback(ri->rrdmetrics, rrdmetric_react_callback, (void *)ri);
ri->rrdmetrics = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(ri->rrdmetrics, rrdmetric_insert_callback, ri);
dictionary_register_delete_callback(ri->rrdmetrics, rrdmetric_delete_callback, ri);
dictionary_register_conflict_callback(ri->rrdmetrics, rrdmetric_conflict_callback, ri);
dictionary_register_react_callback(ri->rrdmetrics, rrdmetric_react_callback, ri);
}
static void rrdmetrics_destroy_from_rrdinstance(RRDINSTANCE *ri) {
@ -669,7 +672,7 @@ static void rrdinstance_free(RRDINSTANCE *ri) {
ri->rrdset = NULL;
}
static void rrdinstance_insert_callback(const char *id __maybe_unused, void *value, void *data) {
static void rrdinstance_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext) {
static STRING *ml_anomaly_rates_id = NULL;
if(unlikely(!ml_anomaly_rates_id))
@ -678,7 +681,7 @@ static void rrdinstance_insert_callback(const char *id __maybe_unused, void *val
RRDINSTANCE *ri = value;
// link it to its parent
ri->rc = data;
ri->rc = rrdcontext;
ri->flags = ri->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS; // no need for atomics
@ -711,9 +714,7 @@ static void rrdinstance_insert_callback(const char *id __maybe_unused, void *val
rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
}
static void rrdinstance_delete_callback(const char *id, void *value, void *data) {
(void)id;
RRDCONTEXT *rc = data; (void)rc;
static void rrdinstance_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext __maybe_unused) {
RRDINSTANCE *ri = (RRDINSTANCE *)value;
internal_error(ri->rrdset, "RRDINSTANCE: '%s' is freed but there is a RRDSET linked to it.", string2str(ri->id));
@ -721,7 +722,7 @@ static void rrdinstance_delete_callback(const char *id, void *value, void *data)
rrdinstance_free(ri);
}
static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
static bool rrdinstance_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdcontext __maybe_unused) {
RRDINSTANCE *ri = (RRDINSTANCE *)oldv;
RRDINSTANCE *ri_new = (RRDINSTANCE *)newv;
@ -739,10 +740,10 @@ static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *o
rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LINKING);
}
if(ri->rrdset && ri->rrdset->chart_uuid && uuid_compare(ri->uuid, *ri->rrdset->chart_uuid) != 0) {
if(ri->rrdset && uuid_compare(ri->uuid, ri->rrdset->chart_uuid) != 0) {
char uuid1[UUID_STR_LEN], uuid2[UUID_STR_LEN];
uuid_unparse(ri->uuid, uuid1);
uuid_unparse(*ri->rrdset->chart_uuid, uuid2);
uuid_unparse(ri->rrdset->chart_uuid, uuid2);
internal_error(true, "RRDINSTANCE: '%s' is linked to RRDSET '%s' but they have different UUIDs. RRDINSTANCE has '%s', RRDSET has '%s'", string2str(ri->id), rrdset_id(ri->rrdset), uuid1, uuid2);
}
@ -823,9 +824,10 @@ static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *o
rrdinstance_free(ri_new);
// the react callback will continue from here
return rrd_flag_is_updated(ri);
}
static void rrdinstance_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
static void rrdinstance_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext __maybe_unused) {
RRDINSTANCE *ri = value;
rrdinstance_trigger_updates(ri, __FUNCTION__ );
@ -837,11 +839,11 @@ void rrdinstances_create_in_rrdcontext(RRDCONTEXT *rc) {
if(unlikely(!rc || rc->rrdinstances)) return;
rc->rrdinstances = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(rc->rrdinstances, rrdinstance_insert_callback, (void *)rc);
dictionary_register_delete_callback(rc->rrdinstances, rrdinstance_delete_callback, (void *)rc);
dictionary_register_conflict_callback(rc->rrdinstances, rrdinstance_conflict_callback, (void *)rc);
dictionary_register_react_callback(rc->rrdinstances, rrdinstance_react_callback, (void *)rc);
rc->rrdinstances = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(rc->rrdinstances, rrdinstance_insert_callback, rc);
dictionary_register_delete_callback(rc->rrdinstances, rrdinstance_delete_callback, rc);
dictionary_register_conflict_callback(rc->rrdinstances, rrdinstance_conflict_callback, rc);
dictionary_register_react_callback(rc->rrdinstances, rrdinstance_react_callback, rc);
}
void rrdinstances_destroy_from_rrdcontext(RRDCONTEXT *rc) {
@ -907,7 +909,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
.flags = RRD_FLAG_NONE, // no need for atomics
.rrdset = st,
};
uuid_copy(tri.uuid, *st->chart_uuid);
uuid_copy(tri.uuid, st->chart_uuid);
RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *)dictionary_set_and_acquire_item(rc->rrdinstances, string2str(tri.id), &tri, sizeof(tri));
@ -934,7 +936,6 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
RRDINSTANCE *ri_old = rrdinstance_acquired_value(ria_old);
// migrate all dimensions to the new metrics
rrdset_rdlock(st);
RRDDIM *rd;
rrddim_foreach_read(rd, st) {
if (!rd->rrdmetric) continue;
@ -950,7 +951,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
rrdmetric_from_rrddim(rd);
}
rrdset_unlock(st);
rrddim_foreach_done(rd);
// mark the old instance, ready to be deleted
if(!rrd_flag_check(ri_old, RRD_FLAG_OWN_LABELS))
@ -966,7 +967,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
/*
// trigger updates on the old context
if(!dictionary_stats_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
if(!dictionary_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
rrdcontext_lock(rc_old);
rc_old->flags = ((rc_old->flags & RRD_FLAG_QUEUED)?RRD_FLAG_QUEUED:RRD_FLAG_NONE)|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
rc_old->first_time_t = 0;
@ -1102,9 +1103,8 @@ static void rrdcontext_freez(RRDCONTEXT *rc) {
string_freez(rc->family);
}
static void rrdcontext_insert_callback(const char *id, void *value, void *data) {
(void)id;
RRDHOST *host = (RRDHOST *)data;
static void rrdcontext_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost) {
RRDHOST *host = (RRDHOST *)rrdhost;
RRDCONTEXT *rc = (RRDCONTEXT *)value;
rc->rrdhost = host;
@ -1167,10 +1167,7 @@ static void rrdcontext_insert_callback(const char *id, void *value, void *data)
rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
}
static void rrdcontext_delete_callback(const char *id, void *value, void *data) {
(void)id;
RRDHOST *host = (RRDHOST *)data;
(void)host;
static void rrdcontext_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost __maybe_unused) {
RRDCONTEXT *rc = (RRDCONTEXT *)value;
@ -1179,18 +1176,14 @@ static void rrdcontext_delete_callback(const char *id, void *value, void *data)
rrdcontext_freez(rc);
}
static void rrdcontext_conflict_callback(const char *id, void *oldv, void *newv, void *data) {
(void)id;
RRDHOST *host = (RRDHOST *)data;
(void)host;
static bool rrdcontext_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdhost __maybe_unused) {
RRDCONTEXT *rc = (RRDCONTEXT *)oldv;
RRDCONTEXT *rc_new = (RRDCONTEXT *)newv;
//current rc is not archived, new_rc is archived, dont merge
if (!rrd_flag_is_archived(rc) && rrd_flag_is_archived(rc_new)) {
rrdcontext_freez(rc_new);
return;
return false;
}
rrdcontext_lock(rc);
@ -1246,11 +1239,11 @@ static void rrdcontext_conflict_callback(const char *id, void *oldv, void *newv,
rrdcontext_freez(rc_new);
// the react callback will continue from here
return rrd_flag_is_updated(rc);
}
static void rrdcontext_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
static void rrdcontext_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost __maybe_unused) {
RRDCONTEXT *rc = (RRDCONTEXT *)value;
rrdcontext_trigger_updates(rc, __FUNCTION__ );
}
@ -1259,33 +1252,35 @@ static void rrdcontext_trigger_updates(RRDCONTEXT *rc, const char *function) {
rrdcontext_queue_for_post_processing(rc, function, rc->flags);
}
static void rrdcontext_hub_queue_insert_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
static void rrdcontext_hub_queue_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
RRDCONTEXT *rc = context;
rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_HUB);
rc->queue.queued_ut = now_realtime_usec();
rc->queue.queued_flags = rrd_flags_get(rc);
}
static void rrdcontext_hub_queue_delete_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
static void rrdcontext_hub_queue_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
RRDCONTEXT *rc = context;
rrd_flag_clear(rc, RRD_FLAG_QUEUED_FOR_HUB);
}
static void rrdcontext_hub_queue_conflict_callback(const char *name __maybe_unused, void *context, void *new_context __maybe_unused, void *data __maybe_unused) {
static bool rrdcontext_hub_queue_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *new_context __maybe_unused, void *nothing __maybe_unused) {
// context and new_context are the same
// we just need to update the timings
RRDCONTEXT *rc = context;
rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_HUB);
rc->queue.queued_ut = now_realtime_usec();
rc->queue.queued_flags |= rrd_flags_get(rc);
return true;
}
static void rrdcontext_post_processing_queue_insert_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
static void rrdcontext_post_processing_queue_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
RRDCONTEXT *rc = context;
rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_POST_PROCESSING);
}
static void rrdcontext_post_processing_queue_delete_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
static void rrdcontext_post_processing_queue_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
RRDCONTEXT *rc = context;
rrd_flag_clear(rc, RRD_FLAG_QUEUED_FOR_POST_PROCESSING);
}
@ -1297,23 +1292,19 @@ void rrdhost_create_rrdcontexts(RRDHOST *host) {
if(unlikely(!host)) return;
if(likely(host->rrdctx)) return;
host->rrdctx = (RRDCONTEXTS *)dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback((DICTIONARY *)host->rrdctx, rrdcontext_insert_callback, (void *)host);
dictionary_register_delete_callback((DICTIONARY *)host->rrdctx, rrdcontext_delete_callback, (void *)host);
dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx, rrdcontext_conflict_callback, (void *)host);
dictionary_register_react_callback((DICTIONARY *)host->rrdctx, rrdcontext_react_callback, (void *)host);
host->rrdctx = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback((DICTIONARY *)host->rrdctx, rrdcontext_insert_callback, host);
dictionary_register_delete_callback((DICTIONARY *)host->rrdctx, rrdcontext_delete_callback, host);
dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx, rrdcontext_conflict_callback, host);
dictionary_register_react_callback((DICTIONARY *)host->rrdctx, rrdcontext_react_callback, host);
host->rrdctx_hub_queue = (RRDCONTEXTS *)dictionary_create(
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
host->rrdctx_hub_queue = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_VALUE_LINK_DONT_CLONE);
dictionary_register_insert_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_insert_callback, NULL);
dictionary_register_delete_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_delete_callback, NULL);
dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_conflict_callback, NULL);
host->rrdctx_post_processing_queue = (RRDCONTEXTS *)dictionary_create(
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
host->rrdctx_post_processing_queue = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_VALUE_LINK_DONT_CLONE);
dictionary_register_insert_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_post_processing_queue_insert_callback, NULL);
dictionary_register_delete_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_post_processing_queue_delete_callback, NULL);
@ -1331,7 +1322,7 @@ void rrdhost_destroy_rrdcontexts(RRDHOST *host) {
RRDCONTEXT *rc;
dfe_start_write(old, rc) {
dictionary_del_having_write_lock(old, string2str(rc->id));
dictionary_del(old, string2str(rc->id));
}
dfe_done(rc);
dictionary_destroy(old);
@ -1343,7 +1334,7 @@ void rrdhost_destroy_rrdcontexts(RRDHOST *host) {
RRDCONTEXT *rc;
dfe_start_write(old, rc) {
dictionary_del_having_write_lock(old, string2str(rc->id));
dictionary_del(old, string2str(rc->id));
}
dfe_done(rc);
dictionary_destroy(old);
@ -1464,6 +1455,35 @@ void rrdcontext_db_rotation(void) {
rrdcontext_next_db_rotation_ut = now_realtime_usec() + FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS * USEC_PER_SEC;
}
int rrdcontext_foreach_instance_with_rrdset_in_context(RRDHOST *host, const char *context, int (*callback)(RRDSET *st, void *data), void *data) {
if(unlikely(!host || !context || !*context || !callback))
return -1;
RRDCONTEXT_ACQUIRED *rca = (RRDCONTEXT_ACQUIRED *)dictionary_get_and_acquire_item((DICTIONARY *)host->rrdctx, context);
if(unlikely(!rca)) return -1;
RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
if(unlikely(!rc)) return -1;
int ret = 0;
RRDINSTANCE *ri;
dfe_start_read(rc->rrdinstances, ri) {
if(ri->rrdset) {
int r = callback(ri->rrdset, data);
if(r >= 0) ret += r;
else {
ret = r;
break;
}
}
}
dfe_done(ri);
rrdcontext_release(rca);
return ret;
}
// ----------------------------------------------------------------------------
// ACLK interface
@ -1601,7 +1621,8 @@ struct rrdcontext_to_json {
RRD_FLAGS combined_flags;
};
static inline int rrdmetric_to_json_callback(const char *id, void *value, void *data) {
static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *id = dictionary_acquired_item_name(item);
struct rrdcontext_to_json * t = data;
RRDMETRIC *rm = value;
BUFFER *wb = t->wb;
@ -1673,7 +1694,9 @@ static inline int rrdmetric_to_json_callback(const char *id, void *value, void *
return 1;
}
static inline int rrdinstance_to_json_callback(const char *id, void *value, void *data) {
static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *id = dictionary_acquired_item_name(item);
struct rrdcontext_to_json *t_parent = data;
RRDINSTANCE *ri = value;
BUFFER *wb = t_parent->wb;
@ -1788,7 +1811,7 @@ static inline int rrdinstance_to_json_callback(const char *id, void *value, void
buffer_strcat(wb, "\"");
}
if(options & RRDCONTEXT_OPTION_SHOW_LABELS && ri->rrdlabels && dictionary_stats_entries(ri->rrdlabels)) {
if(options & RRDCONTEXT_OPTION_SHOW_LABELS && ri->rrdlabels && dictionary_entries(ri->rrdlabels)) {
buffer_sprintf(wb, ",\n\t\t\t\t\t\"labels\": {\n");
rrdlabels_to_buffer(ri->rrdlabels, wb, "\t\t\t\t\t\t", ":", "\"", ",\n", NULL, NULL, NULL, NULL);
buffer_strcat(wb, "\n\t\t\t\t\t}");
@ -1807,7 +1830,8 @@ static inline int rrdinstance_to_json_callback(const char *id, void *value, void
return 1;
}
static inline int rrdcontext_to_json_callback(const char *id, void *value, void *data) {
static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *id = dictionary_acquired_item_name(item);
struct rrdcontext_to_json *t_parent = data;
RRDCONTEXT *rc = value;
BUFFER *wb = t_parent->wb;
@ -1974,7 +1998,7 @@ int rrdcontext_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, R
.written = 0,
.now = now_realtime_sec(),
};
rrdcontext_to_json_callback(context, rc, &t_contexts);
rrdcontext_to_json_callback((DICTIONARY_ITEM *)rca, rc, &t_contexts);
rrdcontext_release(rca);
@ -2300,10 +2324,10 @@ static inline bool rrdinstance_should_be_deleted(RRDINSTANCE *ri) {
if(likely(ri->rrdset))
return false;
if(unlikely(dictionary_stats_referenced_items(ri->rrdmetrics) != 0))
if(unlikely(dictionary_referenced_items(ri->rrdmetrics) != 0))
return false;
if(unlikely(dictionary_stats_entries(ri->rrdmetrics) != 0))
if(unlikely(dictionary_entries(ri->rrdmetrics) != 0))
return false;
if(ri->first_time_t || ri->last_time_t)
@ -2319,10 +2343,10 @@ static inline bool rrdcontext_should_be_deleted(RRDCONTEXT *rc) {
if(likely(rrd_flag_check(rc, RRD_FLAGS_PREVENTING_DELETIONS)))
return false;
if(unlikely(dictionary_stats_referenced_items(rc->rrdinstances) != 0))
if(unlikely(dictionary_referenced_items(rc->rrdinstances) != 0))
return false;
if(unlikely(dictionary_stats_entries(rc->rrdinstances) != 0))
if(unlikely(dictionary_entries(rc->rrdinstances) != 0))
return false;
if(unlikely(rc->first_time_t || rc->last_time_t))
@ -2364,7 +2388,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
dfe_start_write(ri->rrdmetrics, rm) {
if(rrdmetric_should_be_deleted(rm)) {
if(worker_jobs) worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
if(dictionary_del_having_write_lock(ri->rrdmetrics, string2str(rm->id)) != 0)
if(!dictionary_del(ri->rrdmetrics, string2str(rm->id)))
error("RRDCONTEXT: metric '%s' of instance '%s' of context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
string2str(rm->id),
string2str(ri->id),
@ -2384,7 +2408,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
if(rrdinstance_should_be_deleted(ri)) {
if(worker_jobs) worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
if(dictionary_del(rc->rrdinstances, string2str(ri->id)) != 0)
if(!dictionary_del(rc->rrdinstances, string2str(ri->id)))
error("RRDCONTEXT: instance '%s' of context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
string2str(ri->id),
string2str(rc->id),
@ -2405,7 +2429,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
rrdcontext_dequeue_from_post_processing(rc);
rrdcontext_delete_from_sql_unsafe(rc);
if(dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
if(!dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)))
error("RRDCONTEXT: context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
string2str(rc->id),
rrdhost_hostname(host));
@ -2471,7 +2495,7 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
size_t metrics_active = 0, metrics_deleted = 0;
bool live_retention = true, currently_collected = false;
if(dictionary_stats_entries(ri->rrdmetrics) > 0) {
if(dictionary_entries(ri->rrdmetrics) > 0) {
RRDMETRIC *rm;
dfe_start_read((DICTIONARY *)ri->rrdmetrics, rm) {
if(unlikely(netdata_exit)) break;
@ -2574,7 +2598,7 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
size_t instances_active = 0, instances_deleted = 0;
bool live_retention = true, currently_collected = false, hidden = true;
if(dictionary_stats_entries(rc->rrdinstances) > 0) {
if(dictionary_entries(rc->rrdinstances) > 0) {
RRDINSTANCE *ri;
dfe_start_reentrant(rc->rrdinstances, ri) {
if(unlikely(netdata_exit)) break;
@ -2714,7 +2738,7 @@ static void rrdcontext_queue_for_post_processing(RRDCONTEXT *rc, const char *fun
rc,
sizeof(*rc));
#ifdef NETDATA_INTERNAL_CHECKS
#if(defined(NETDATA_INTERNAL_CHECKS) && defined(LOG_POST_PROCESSING_QUEUE_INSERTIONS))
{
BUFFER *wb_flags = buffer_create(1000);
rrd_flags_to_buffer(flags, wb_flags);
@ -2912,7 +2936,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
return;
// check if there are queued items to send
if(!dictionary_stats_entries((DICTIONARY *)host->rrdctx_hub_queue))
if(!dictionary_entries((DICTIONARY *)host->rrdctx_hub_queue))
return;
if(!host->node_id)
@ -2975,7 +2999,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
rrdcontext_unlock(rc);
// delete it from the master dictionary
if(dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
if(!dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)))
error("RRDCONTEXT: '%s' of host '%s' failed to be deleted from rrdcontext dictionary.",
string2str(id), rrdhost_hostname(host));
@ -3010,7 +3034,10 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
static void rrdcontext_main_cleanup(void *ptr) {
struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
// custom code
worker_unregister();
static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
}
@ -3065,12 +3092,13 @@ void *rrdcontext_main(void *ptr) {
worker_is_busy(WORKER_JOB_HOSTS);
if(host->rrdctx_post_processing_queue) {
pp_queued_contexts_for_all_hosts += dictionary_stats_entries((DICTIONARY *)host->rrdctx_post_processing_queue);
pp_queued_contexts_for_all_hosts +=
dictionary_entries((DICTIONARY *)host->rrdctx_post_processing_queue);
rrdcontext_post_process_queued_contexts(host);
}
if(host->rrdctx_hub_queue) {
hub_queued_contexts_for_all_hosts += dictionary_stats_entries((DICTIONARY *)host->rrdctx_hub_queue);
hub_queued_contexts_for_all_hosts += dictionary_entries((DICTIONARY *)host->rrdctx_hub_queue);
rrdcontext_dispatch_queued_contexts_to_hub(host, now_ut);
}
}

2
database/rrdcontext.h
View file

@ -34,6 +34,8 @@ extern void rrdhost_destroy_rrdcontexts(RRDHOST *host);
extern void rrdcontext_host_child_connected(RRDHOST *host);
extern void rrdcontext_host_child_disconnected(RRDHOST *host);
extern int rrdcontext_foreach_instance_with_rrdset_in_context(RRDHOST *host, const char *context, int (*callback)(RRDSET *st, void *data), void *data);
typedef enum {
RRDCONTEXT_OPTION_NONE = 0,
RRDCONTEXT_OPTION_SHOW_METRICS = (1 << 0),

635
database/rrddim.c
View file

@ -10,23 +10,303 @@
// ----------------------------------------------------------------------------
// RRDDIM index
static inline void rrddim_index_add(RRDSET *st, RRDDIM *rd) {
if(likely(dictionary_set(st->rrddim_root_index, string2str(rd->id), rd, sizeof(RRDDIM)) == rd)) {
rrddim_flag_set(rd, RRDDIM_FLAG_INDEXED_ID);
}
else {
rrddim_flag_clear(rd, RRDDIM_FLAG_INDEXED_ID);
error("RRDDIM: %s() attempted to index duplicate dimension with key '%s' of chart '%s' of host '%s'", __FUNCTION__, rrddim_id(rd), rrdset_id(st), rrdhost_hostname(st->rrdhost));
struct rrddim_constructor {
RRDSET *st;
const char *id;
const char *name;
collected_number multiplier;
collected_number divisor;
RRD_ALGORITHM algorithm;
RRD_MEMORY_MODE memory_mode;
enum {
RRDDIM_REACT_NONE = 0,
RRDDIM_REACT_NEW = (1 << 0),
RRDDIM_REACT_UPDATED = (1 << 2),
} react_action;
};
static void rrddim_update_rrddimvars_unsafe(RRDDIM *rd) {
RRDSET *st = rd->rrdset;
RRDHOST *host = st->rrdhost;
if(host->health_enabled && !rrdset_is_ar_chart(st)) {
rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_FLAG_NONE);
rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_FLAG_NONE);
rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_FLAG_NONE);
rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS);
rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
}
}
static inline void rrddim_index_del(RRDSET *st, RRDDIM *rd) {
if(rrddim_flag_check(rd, RRDDIM_FLAG_INDEXED_ID)) {
if (likely(dictionary_del(st->rrddim_root_index, string2str(rd->id)) == 0))
rrddim_flag_clear(rd, RRDDIM_FLAG_INDEXED_ID);
else
error("RRDDIM: %s() attempted to delete non-indexed dimension with key '%s' of chart '%s' of host '%s'", __FUNCTION__, rrddim_id(rd), rrdset_id(st), rrdhost_hostname(st->rrdhost));
static void rrddim_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *constructor_data) {
struct rrddim_constructor *ctr = constructor_data;
RRDDIM *rd = rrddim;
RRDSET *st = ctr->st;
RRDHOST *host = st->rrdhost;
rd->flags = RRDDIM_FLAG_NONE;
rd->id = string_strdupz(ctr->id);
rd->name = (ctr->name && *ctr->name)?rrd_string_strdupz(ctr->name):string_dup(rd->id);
rd->algorithm = ctr->algorithm;
rd->multiplier = ctr->multiplier;
rd->divisor = ctr->divisor;
if(!rd->divisor) rd->divisor = 1;
rd->update_every = st->update_every;
rd->rrdset = st;
if(rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST))
rd->collections_counter = 1;
if(ctr->memory_mode == RRD_MEMORY_MODE_MAP || ctr->memory_mode == RRD_MEMORY_MODE_SAVE) {
if(!rrddim_memory_load_or_create_map_save(st, rd, ctr->memory_mode)) {
info("Failed to use memory mode %s for chart '%s', dimension '%s', falling back to ram", (ctr->memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st), rrddim_name(rd));
ctr->memory_mode = RRD_MEMORY_MODE_RAM;
}
}
if(ctr->memory_mode == RRD_MEMORY_MODE_RAM) {
size_t entries = st->entries;
if(!entries) entries = 5;
rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1);
if(!rd->db) {
info("Failed to use memory mode ram for chart '%s', dimension '%s', falling back to alloc", rrdset_name(st), rrddim_name(rd));
ctr->memory_mode = RRD_MEMORY_MODE_ALLOC;
}
else rd->memsize = entries * sizeof(storage_number);
}
if(ctr->memory_mode == RRD_MEMORY_MODE_ALLOC || ctr->memory_mode == RRD_MEMORY_MODE_NONE) {
size_t entries = st->entries;
if(entries < 5) entries = 5;
rd->db = callocz(entries, sizeof(storage_number));
rd->memsize = entries * sizeof(storage_number);
}
rd->rrd_memory_mode = ctr->memory_mode;
#ifdef ENABLE_ACLK
rd->aclk_live_status = -1;
#endif
(void) find_dimension_uuid(st, rd, &(rd->metric_uuid));
// initialize the db tiers
{
size_t initialized = 0;
RRD_MEMORY_MODE wanted_mode = ctr->memory_mode;
for(int tier = 0; tier < storage_tiers ; tier++, wanted_mode = RRD_MEMORY_MODE_DBENGINE) {
STORAGE_ENGINE *eng = storage_engine_get(wanted_mode);
if(!eng) continue;
rd->tiers[tier] = callocz(1, sizeof(struct rrddim_tier));
rd->tiers[tier]->tier_grouping = get_tier_grouping(tier);
rd->tiers[tier]->mode = eng->id;
rd->tiers[tier]->collect_ops = eng->api.collect_ops;
rd->tiers[tier]->query_ops = eng->api.query_ops;
rd->tiers[tier]->db_metric_handle = eng->api.init(rd, host->storage_instance[tier]);
storage_point_unset(rd->tiers[tier]->virtual_point);
initialized++;
// internal_error(true, "TIER GROUPING of chart '%s', dimension '%s' for tier %d is set to %d", rd->rrdset->name, rd->name, tier, rd->tiers[tier]->tier_grouping);
}
if(!initialized)
error("Failed to initialize all db tiers for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
if(!rd->tiers[0])
error("Failed to initialize the first db tier for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
}
// initialize data collection for all tiers
{
size_t initialized = 0;
for (int tier = 0; tier < storage_tiers; tier++) {
if (rd->tiers[tier]) {
rd->tiers[tier]->db_collection_handle = rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
initialized++;
}
}
if(!initialized)
error("Failed to initialize data collection for all db tiers for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
}
if(st->dimensions) {
RRDDIM *td = st->dimensions;
if(td->algorithm != rd->algorithm || ABS(td->multiplier) != ABS(rd->multiplier) || ABS(td->divisor) != ABS(rd->divisor)) {
if(!rrdset_flag_check(st, RRDSET_FLAG_HETEROGENEOUS)) {
#ifdef NETDATA_INTERNAL_CHECKS
info("Dimension '%s' added on chart '%s' of host '%s' is not homogeneous to other dimensions already present (algorithm is '%s' vs '%s', multiplier is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ", divisor is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ").",
rrddim_name(rd),
rrdset_name(st),
rrdhost_hostname(host),
rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(td->algorithm),
rd->multiplier, td->multiplier,
rd->divisor, td->divisor
);
#endif
rrdset_flag_set(st, RRDSET_FLAG_HETEROGENEOUS);
}
}
}
rrddim_update_rrddimvars_unsafe(rd);
// let the chart resync
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
ml_new_dimension(rd);
ctr->react_action = RRDDIM_REACT_NEW;
internal_error(false, "RRDDIM: inserted dimension '%s' of chart '%s' of host '%s'",
rrddim_name(rd), rrdset_name(st), rrdhost_hostname(st->rrdhost));
}
static void rrddim_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *rrdset) {
RRDDIM *rd = rrddim;
RRDSET *st = rrdset; (void)st;
internal_error(false, "RRDDIM: deleting dimension '%s' of chart '%s' of host '%s'",
rrddim_name(rd), rrdset_name(st), rrdhost_hostname(st->rrdhost));
rrdcontext_removed_rrddim(rd);
ml_delete_dimension(rd);
debug(D_RRD_CALLS, "rrddim_free() %s.%s", rrdset_name(st), rrddim_name(rd));
if (!rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
size_t tiers_available = 0, tiers_said_yes = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(rd->tiers[tier]) {
tiers_available++;
if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
tiers_said_yes++;
rd->tiers[tier]->db_collection_handle = NULL;
}
}
if (tiers_available == tiers_said_yes && tiers_said_yes && rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
/* This metric has no data and no references */
delete_dimension_uuid(&rd->metric_uuid);
}
}
rrddimvar_delete_all(rd);
// free(rd->annotations);
//#ifdef ENABLE_ACLK
// if (!netdata_exit)
// aclk_send_dimension_update(rd);
//#endif
// this will free MEMORY_MODE_SAVE and MEMORY_MODE_MAP structures
rrddim_memory_file_free(rd);
for(int tier = 0; tier < storage_tiers ;tier++) {
if(!rd->tiers[tier]) continue;
STORAGE_ENGINE* eng = storage_engine_get(rd->tiers[tier]->mode);
if(eng)
eng->api.free(rd->tiers[tier]->db_metric_handle);
freez(rd->tiers[tier]);
rd->tiers[tier] = NULL;
}
if(rd->db) {
if(rd->rrd_memory_mode == RRD_MEMORY_MODE_RAM)
munmap(rd->db, rd->memsize);
else
freez(rd->db);
}
string_freez(rd->id);
string_freez(rd->name);
}
static bool rrddim_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *new_rrddim, void *constructor_data) {
(void)new_rrddim; // it is NULL
struct rrddim_constructor *ctr = constructor_data;
RRDDIM *rd = rrddim;
RRDSET *st = ctr->st;
ctr->react_action = RRDDIM_REACT_NONE;
int rc = rrddim_reset_name(st, rd, ctr->name);
rc += rrddim_set_algorithm(st, rd, ctr->algorithm);
rc += rrddim_set_multiplier(st, rd, ctr->multiplier);
rc += rrddim_set_divisor(st, rd, ctr->divisor);
if(rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
for(int tier = 0; tier < storage_tiers ;tier++) {
if (rd->tiers[tier])
rd->tiers[tier]->db_collection_handle =
rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
}
rrddim_flag_clear(rd, RRDDIM_FLAG_ARCHIVED);
rrddim_update_rrddimvars_unsafe(rd);
}
if(unlikely(rc))
ctr->react_action = RRDDIM_REACT_UPDATED;
return ctr->react_action == RRDDIM_REACT_UPDATED;
}
static void rrddim_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *constructor_data) {
struct rrddim_constructor *ctr = constructor_data;
RRDDIM *rd = rrddim;
RRDSET *st = ctr->st;
if(ctr->react_action == RRDDIM_REACT_UPDATED) {
debug(D_METADATALOG, "DIMENSION [%s] metadata updated", rrddim_id(rd));
(void)sql_store_dimension(&rd->metric_uuid, &rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
#ifdef ENABLE_ACLK
queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
#endif
// the chart needs to be updated to the parent
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
}
rrdcontext_updated_rrddim(rd);
}
void rrddim_index_init(RRDSET *st) {
if(!st->rrddim_root_index) {
st->rrddim_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(st->rrddim_root_index, rrddim_insert_callback, NULL);
dictionary_register_conflict_callback(st->rrddim_root_index, rrddim_conflict_callback, NULL);
dictionary_register_delete_callback(st->rrddim_root_index, rrddim_delete_callback, st);
dictionary_register_react_callback(st->rrddim_root_index, rrddim_react_callback, st);
}
}
void rrddim_index_destroy(RRDSET *st) {
dictionary_destroy(st->rrddim_root_index);
st->rrddim_root_index = NULL;
}
static inline RRDDIM *rrddim_index_find(RRDSET *st, const char *id) {
@ -55,14 +335,15 @@ RRDDIM *rrddim_find_active(RRDSET *st, const char *id) {
// ----------------------------------------------------------------------------
// RRDDIM rename a dimension
inline int rrddim_set_name(RRDSET *st, RRDDIM *rd, const char *name) {
inline int rrddim_reset_name(RRDSET *st, RRDDIM *rd, const char *name) {
if(unlikely(!name || !*name || !strcmp(rrddim_name(rd), name)))
return 0;
debug(D_RRD_CALLS, "rrddim_set_name() from %s.%s to %s.%s", rrdset_name(st), rrddim_name(rd), rrdset_name(st), name);
debug(D_RRD_CALLS, "rrddim_reset_name() from %s.%s to %s.%s", rrdset_name(st), rrddim_name(rd), rrdset_name(st), name);
string_freez(rd->name);
STRING *old = rd->name;
rd->name = rrd_string_strdupz(name);
string_freez(old);
if (!rrdset_is_ar_chart(st))
rrddimvar_rename_all(rd);
@ -115,34 +396,34 @@ inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor)
}
// ----------------------------------------------------------------------------
// RRDDIM create a dimension
void rrdcalc_link_to_rrddim(RRDDIM *rd, RRDSET *st, RRDHOST *host) {
RRDCALC *rc;
for (rc = host->alarms_with_foreach; rc; rc = rc->next) {
if (simple_pattern_matches(rc->spdim, rrddim_id(rd)) || simple_pattern_matches(rc->spdim, rrddim_name(rd))) {
if (rc->chart == st->name || rc->chart == st->id) {
char *name = alarm_name_with_dim(rrdcalc_name(rc), string_strlen(rc->name), rrddim_name(rd), string_strlen(rd->name));
if(rrdcalc_exists(host, rrdset_name(st), name)) {
freez(name);
continue;
}
// get the timestamp of the last entry in the round-robin database
time_t rrddim_last_entry_t(RRDDIM *rd) {
time_t latest = rd->tiers[0]->query_ops.latest_time(rd->tiers[0]->db_metric_handle);
netdata_rwlock_wrlock(&host->health_log.alarm_log_rwlock);
RRDCALC *child = rrdcalc_create_from_rrdcalc(rc, host, name, rrddim_name(rd));
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
for(int tier = 1; tier < storage_tiers ;tier++) {
if(unlikely(!rd->tiers[tier])) continue;
if (child)
rrdcalc_add_to_host(host, child);
else {
error("Cannot allocate a new alarm.");
rc->foreachcounter--;
}
}
}
time_t t = rd->tiers[tier]->query_ops.latest_time(rd->tiers[tier]->db_metric_handle);
if(t > latest)
latest = t;
}
return latest;
}
time_t rrddim_first_entry_t(RRDDIM *rd) {
time_t oldest = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(unlikely(!rd->tiers[tier])) continue;
time_t t = rd->tiers[tier]->query_ops.oldest_time(rd->tiers[tier]->db_metric_handle);
if(t != 0 && (oldest == 0 || t < oldest))
oldest = t;
}
return oldest;
}
// Return either
@ -154,12 +435,14 @@ time_t calc_dimension_liveness(RRDDIM *rd, time_t now)
{
time_t last_updated = rd->last_collected_time.tv_sec;
int live;
if (rd->aclk_live_status == 1)
live =
((now - last_updated) <
MIN(rrdset_free_obsolete_time, RRDSET_MINIMUM_DIM_OFFLINE_MULTIPLIER * rd->update_every));
if(rd->aclk_live_status == 1) {
int offline_time = RRDSET_MINIMUM_DIM_OFFLINE_MULTIPLIER * rd->update_every;
live = ((now - last_updated) < MIN(rrdset_free_obsolete_time, offline_time));
}
else
live = ((now - last_updated) < RRDSET_MINIMUM_DIM_LIVE_MULTIPLIER * rd->update_every);
return live ? 0 : last_updated;
}
#endif
@ -172,260 +455,37 @@ RRDDIM *rrddim_add_custom(RRDSET *st
, RRD_ALGORITHM algorithm
, RRD_MEMORY_MODE memory_mode
) {
RRDHOST *host = st->rrdhost;
rrdset_wrlock(st);
struct rrddim_constructor tmp = {
.st = st,
.id = id,
.name = name,
.multiplier = multiplier,
.divisor = divisor,
.algorithm = algorithm,
.memory_mode = memory_mode,
};
RRDDIM *rd = rrddim_find(st, id);
if(unlikely(rd)) {
debug(D_RRD_CALLS, "Cannot create rrd dimension '%s/%s', it already exists.", rrdset_id(st), name?name:"<NONAME>");
RRDDIM *rd = dictionary_set_advanced(st->rrddim_root_index, tmp.id, -1, NULL, sizeof(RRDDIM), &tmp);
int rc = rrddim_set_name(st, rd, name);
rc += rrddim_set_algorithm(st, rd, algorithm);
rc += rrddim_set_multiplier(st, rd, multiplier);
rc += rrddim_set_divisor(st, rd, divisor);
if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
store_active_dimension(&rd->metric_uuid);
for(int tier = 0; tier < storage_tiers ;tier++) {
if (rd->tiers[tier])
rd->tiers[tier]->db_collection_handle =
rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
}
rrddim_flag_clear(rd, RRDDIM_FLAG_ARCHIVED);
rrddimvar_create(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_OPTION_DEFAULT);
rrddimvar_create(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_OPTION_DEFAULT);
rrddimvar_create(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
}
if (unlikely(rc)) {
debug(D_METADATALOG, "DIMENSION [%s] metadata updated", rrddim_id(rd));
(void)sql_store_dimension(&rd->metric_uuid, rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor,
rd->algorithm);
#ifdef ENABLE_ACLK
queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
#endif
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
}
if(tmp.react_action == RRDDIM_REACT_NEW) {
// append this dimension
rrdset_wrlock(st);
DOUBLE_LINKED_LIST_APPEND_UNSAFE(st->dimensions, rd, prev, next);
rrdset_unlock(st);
rrdcontext_updated_rrddim(rd);
return rd;
}
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rd = callocz(1, sizeof(RRDDIM));
rd->id = string_strdupz(id);
rd->name = (name && *name)?rrd_string_strdupz(name):string_dup(rd->id);
rd->algorithm = algorithm;
rd->multiplier = multiplier;
rd->divisor = divisor;
if(!rd->divisor) rd->divisor = 1;
rd->entries = st->entries;
rd->update_every = st->update_every;
if(rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST))
rd->collections_counter = 1;
rd->rrdset = st;
if(memory_mode == RRD_MEMORY_MODE_MAP || memory_mode == RRD_MEMORY_MODE_SAVE) {
if(!rrddim_memory_load_or_create_map_save(st, rd, memory_mode)) {
info("Failed to use memory mode %s for chart '%s', dimension '%s', falling back to ram", (memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st), rrddim_name(rd));
memory_mode = RRD_MEMORY_MODE_RAM;
}
}
if(memory_mode == RRD_MEMORY_MODE_RAM) {
size_t entries = st->entries;
if(!entries) entries = 5;
rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1);
if(!rd->db) {
info("Failed to use memory mode ram for chart '%s', dimension '%s', falling back to alloc", rrdset_name(st), rrddim_name(rd));
memory_mode = RRD_MEMORY_MODE_ALLOC;
}
else rd->memsize = entries * sizeof(storage_number);
}
if(memory_mode == RRD_MEMORY_MODE_ALLOC || memory_mode == RRD_MEMORY_MODE_NONE) {
size_t entries = st->entries;
if(entries < 5) entries = 5;
rd->db = callocz(entries, sizeof(storage_number));
rd->memsize = entries * sizeof(storage_number);
}
rd->rrd_memory_mode = memory_mode;
#ifdef ENABLE_ACLK
rd->aclk_live_status = -1;
#endif
(void) find_dimension_uuid(st, rd, &(rd->metric_uuid));
// initialize the db tiers
{
size_t initialized = 0;
RRD_MEMORY_MODE wanted_mode = memory_mode;
for(int tier = 0; tier < storage_tiers ; tier++, wanted_mode = RRD_MEMORY_MODE_DBENGINE) {
STORAGE_ENGINE *eng = storage_engine_get(wanted_mode);
if(!eng) continue;
rd->tiers[tier] = callocz(1, sizeof(struct rrddim_tier));
rd->tiers[tier]->tier_grouping = get_tier_grouping(tier);
rd->tiers[tier]->mode = eng->id;
rd->tiers[tier]->collect_ops = eng->api.collect_ops;
rd->tiers[tier]->query_ops = eng->api.query_ops;
rd->tiers[tier]->db_metric_handle = eng->api.init(rd, host->storage_instance[tier]);
storage_point_unset(rd->tiers[tier]->virtual_point);
initialized++;
// internal_error(true, "TIER GROUPING of chart '%s', dimension '%s' for tier %d is set to %d", rd->rrdset->name, rd->name, tier, rd->tiers[tier]->tier_grouping);
}
if(!initialized)
error("Failed to initialize all db tiers for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
if(!rd->tiers[0])
error("Failed to initialize the first db tier for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
}
store_active_dimension(&rd->metric_uuid);
// initialize data collection for all tiers
{
size_t initialized = 0;
for (int tier = 0; tier < storage_tiers; tier++) {
if (rd->tiers[tier]) {
rd->tiers[tier]->db_collection_handle = rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
initialized++;
}
}
if(!initialized)
error("Failed to initialize data collection for all db tiers for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
}
if(st->dimensions) {
RRDDIM *td = st->dimensions;
if(td->algorithm != rd->algorithm || ABS(td->multiplier) != ABS(rd->multiplier) || ABS(td->divisor) != ABS(rd->divisor)) {
if(!rrdset_flag_check(st, RRDSET_FLAG_HETEROGENEOUS)) {
#ifdef NETDATA_INTERNAL_CHECKS
info("Dimension '%s' added on chart '%s' of host '%s' is not homogeneous to other dimensions already present (algorithm is '%s' vs '%s', multiplier is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ", divisor is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ").",
rrddim_name(rd),
rrdset_name(st),
rrdhost_hostname(host),
rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(td->algorithm),
rd->multiplier, td->multiplier,
rd->divisor, td->divisor
);
#endif
rrdset_flag_set(st, RRDSET_FLAG_HETEROGENEOUS);
}
}
}
// append this dimension
DOUBLE_LINKED_LIST_APPEND_UNSAFE(st->dimensions, rd, prev, next);
if(host->health_enabled && !rrdset_is_ar_chart(st)) {
rrddimvar_create(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_OPTION_DEFAULT);
rrddimvar_create(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_OPTION_DEFAULT);
rrddimvar_create(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
}
rrddim_index_add(st, rd);
rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
ml_new_dimension(rd);
rrdset_unlock(st);
rrdcontext_updated_rrddim(rd);
return(rd);
}
// ----------------------------------------------------------------------------
// RRDDIM remove / free a dimension
void rrddim_free(RRDSET *st, RRDDIM *rd)
{
rrdcontext_removed_rrddim(rd);
ml_delete_dimension(rd);
debug(D_RRD_CALLS, "rrddim_free() %s.%s", rrdset_name(st), rrddim_name(rd));
if (!rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
size_t tiers_available = 0, tiers_said_yes = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(rd->tiers[tier]) {
tiers_available++;
if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
tiers_said_yes++;
rd->tiers[tier]->db_collection_handle = NULL;
}
}
if (tiers_available == tiers_said_yes && tiers_said_yes && rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
/* This metric has no data and no references */
delete_dimension_uuid(&rd->metric_uuid);
}
}
void rrddim_free(RRDSET *st, RRDDIM *rd) {
rrdset_wrlock(st);
DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->dimensions, rd, prev, next);
rrdset_unlock(st);
while(rd->variables)
rrddimvar_free(rd->variables);
rrddim_index_del(st, rd);
// free(rd->annotations);
//#ifdef ENABLE_ACLK
// if (!netdata_exit)
// aclk_send_dimension_update(rd);
//#endif
// this will free MEMORY_MODE_SAVE and MEMORY_MODE_MAP structures
rrddim_memory_file_free(rd);
for(int tier = 0; tier < storage_tiers ;tier++) {
if(!rd->tiers[tier]) continue;
STORAGE_ENGINE* eng = storage_engine_get(rd->tiers[tier]->mode);
if(eng)
eng->api.free(rd->tiers[tier]->db_metric_handle);
freez(rd->tiers[tier]);
rd->tiers[tier] = NULL;
}
if(rd->db) {
if(rd->rrd_memory_mode == RRD_MEMORY_MODE_RAM)
munmap(rd->db, rd->memsize);
else
freez(rd->db);
}
string_freez(rd->id);
string_freez(rd->name);
freez(rd);
dictionary_del(st->rrddim_root_index, string2str(rd->id));
}
@ -445,7 +505,7 @@ int rrddim_hide(RRDSET *st, const char *id) {
if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN))
(void)sql_set_dimension_option(&rd->metric_uuid, "hidden");
rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
rrdcontext_updated_rrddim_flags(rd);
return 0;
@ -463,7 +523,7 @@ int rrddim_unhide(RRDSET *st, const char *id) {
if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN))
(void)sql_set_dimension_option(&rd->metric_uuid, NULL);
rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
rrdcontext_updated_rrddim_flags(rd);
return 0;
@ -478,6 +538,7 @@ inline void rrddim_is_obsolete(RRDSET *st, RRDDIM *rd) {
}
rrddim_flag_set(rd, RRDDIM_FLAG_OBSOLETE);
rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS);
rrdcontext_updated_rrddim_flags(rd);
}

282
database/rrddimvar.c
View file

@ -1,84 +1,87 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#define NETDATA_HEALTH_INTERNALS
#include "rrd.h"
typedef struct rrddimvar {
struct rrddim *rrddim;
STRING *prefix;
STRING *suffix;
void *value;
const RRDVAR_ACQUIRED *rrdvar_local_dim_id;
const RRDVAR_ACQUIRED *rrdvar_local_dim_name;
const RRDVAR_ACQUIRED *rrdvar_family_id;
const RRDVAR_ACQUIRED *rrdvar_family_name;
const RRDVAR_ACQUIRED *rrdvar_family_context_dim_id;
const RRDVAR_ACQUIRED *rrdvar_family_context_dim_name;
const RRDVAR_ACQUIRED *rrdvar_host_chart_id_dim_id;
const RRDVAR_ACQUIRED *rrdvar_host_chart_id_dim_name;
const RRDVAR_ACQUIRED *rrdvar_host_chart_name_dim_id;
const RRDVAR_ACQUIRED *rrdvar_host_chart_name_dim_name;
RRDVAR_FLAGS flags:24;
RRDVAR_TYPE type:8;
} RRDDIMVAR;
// ----------------------------------------------------------------------------
// RRDDIMVAR management
// DIMENSION VARIABLES
#define RRDDIMVAR_ID_MAX 1024
static inline void rrddimvar_free_variables(RRDDIMVAR *rs) {
static inline void rrddimvar_free_variables_unsafe(RRDDIMVAR *rs) {
RRDDIM *rd = rs->rrddim;
RRDSET *st = rd->rrdset;
RRDHOST *host = st->rrdhost;
// CHART VARIABLES FOR THIS DIMENSION
rrdvar_free(host, st->rrdvar_root_index, rs->var_local_id);
rs->var_local_id = NULL;
if(st->rrdvars) {
rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local_dim_id);
rs->rrdvar_local_dim_id = NULL;
rrdvar_free(host, st->rrdvar_root_index, rs->var_local_name);
rs->var_local_name = NULL;
rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local_dim_name);
rs->rrdvar_local_dim_name = NULL;
}
// FAMILY VARIABLES FOR THIS DIMENSION
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_id);
rs->var_family_id = NULL;
if(st->rrdfamily) {
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_id);
rs->rrdvar_family_id = NULL;
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_name);
rs->var_family_name = NULL;
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_name);
rs->rrdvar_family_name = NULL;
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextid);
rs->var_family_contextid = NULL;
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_context_dim_id);
rs->rrdvar_family_context_dim_id = NULL;
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextname);
rs->var_family_contextname = NULL;
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_context_dim_name);
rs->rrdvar_family_context_dim_name = NULL;
}
// HOST VARIABLES FOR THIS DIMENSION
rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidid);
rs->var_host_chartidid = NULL;
if(host->rrdvars && host->health_enabled) {
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id_dim_id);
rs->rrdvar_host_chart_id_dim_id = NULL;
rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidname);
rs->var_host_chartidname = NULL;
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id_dim_name);
rs->rrdvar_host_chart_id_dim_name = NULL;
rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnameid);
rs->var_host_chartnameid = NULL;
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name_dim_id);
rs->rrdvar_host_chart_name_dim_id = NULL;
rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnamename);
rs->var_host_chartnamename = NULL;
// KEYS
string_freez(rs->key_id);
rs->key_id = NULL;
string_freez(rs->key_name);
rs->key_name = NULL;
string_freez(rs->key_fullidid);
rs->key_fullidid = NULL;
string_freez(rs->key_fullidname);
rs->key_fullidname = NULL;
string_freez(rs->key_contextid);
rs->key_contextid = NULL;
string_freez(rs->key_contextname);
rs->key_contextname = NULL;
string_freez(rs->key_fullnameid);
rs->key_fullnameid = NULL;
string_freez(rs->key_fullnamename);
rs->key_fullnamename = NULL;
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name_dim_name);
rs->rrdvar_host_chart_name_dim_name = NULL;
}
}
static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
rrddimvar_free_variables(rs);
static inline void rrddimvar_update_variables_unsafe(RRDDIMVAR *rs) {
rrddimvar_free_variables_unsafe(rs);
RRDDIM *rd = rs->rrddim;
RRDSET *st = rd->rrdset;
@ -89,28 +92,28 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
// KEYS
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_id(rd), string2str(rs->suffix));
rs->key_id = string_strdupz(buffer);
STRING *key_dim_id = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_name(rd), string2str(rs->suffix));
rs->key_name = string_strdupz(buffer);
STRING *key_dim_name = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_id));
rs->key_fullidid = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(key_dim_id));
STRING *key_chart_id_dim_id = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_name));
rs->key_fullidname = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(key_dim_name));
STRING *key_chart_id_dim_name = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_id));
rs->key_contextid = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(key_dim_id));
STRING *key_context_dim_id = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_name));
rs->key_contextname = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(key_dim_name));
STRING *key_context_dim_name = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_id));
rs->key_fullnameid = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(key_dim_id));
STRING *key_chart_name_dim_id = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_name));
rs->key_fullnamename = string_strdupz(buffer);
snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(key_dim_name));
STRING *key_chart_name_dim_name = string_strdupz(buffer);
// CHART VARIABLES FOR THIS DIMENSION
// -----------------------------------
@ -119,8 +122,10 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
// - $id
// - $name
rs->var_local_id = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_local_name = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
if(st->rrdvars) {
rs->rrdvar_local_dim_id = rrdvar_add_and_acquire("local", st->rrdvars, key_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_local_dim_name = rrdvar_add_and_acquire("local", st->rrdvars, key_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
}
// FAMILY VARIABLES FOR THIS DIMENSION
// -----------------------------------
@ -131,10 +136,12 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
// - $chart-context.id
// - $chart-context.name
rs->var_family_id = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_family_name = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_family_contextid = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_family_contextname = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
if(st->rrdfamily) {
rs->rrdvar_family_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_family_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_family_context_dim_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_context_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_family_context_dim_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_context_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
}
// HOST VARIABLES FOR THIS DIMENSION
// -----------------------------------
@ -145,60 +152,121 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
// - $chart-name.id
// - $chart-name.name
rs->var_host_chartidid = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_host_chartidname = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_host_chartnameid = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnameid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
rs->var_host_chartnamename = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnamename, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
if(host->rrdvars && host->health_enabled) {
rs->rrdvar_host_chart_id_dim_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_host_chart_id_dim_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_host_chart_name_dim_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
rs->rrdvar_host_chart_name_dim_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
}
// free the keys
string_freez(key_dim_id);
string_freez(key_dim_name);
string_freez(key_chart_id_dim_id);
string_freez(key_chart_id_dim_name);
string_freez(key_context_dim_id);
string_freez(key_context_dim_name);
string_freez(key_chart_name_dim_id);
string_freez(key_chart_name_dim_name);
}
RRDDIMVAR *rrddimvar_create(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_OPTIONS options) {
RRDSET *st = rd->rrdset;
(void)st;
struct rrddimvar_constructor {
RRDDIM *rrddim;
const char *prefix;
const char *suffix;
void *value;
RRDVAR_FLAGS flags :16;
RRDVAR_TYPE type:8;
};
debug(D_VARIABLES, "RRDDIMSET create for chart id '%s' name '%s', dimension id '%s', name '%s%s%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), (prefix)?prefix:"", rrddim_name(rd), (suffix)?suffix:"");
static void rrddimvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *constructor_data) {
RRDDIMVAR *rs = rrddimvar;
struct rrddimvar_constructor *ctr = constructor_data;
if(!ctr->prefix) ctr->prefix = "";
if(!ctr->suffix) ctr->suffix = "";
rs->prefix = string_strdupz(ctr->prefix);
rs->suffix = string_strdupz(ctr->suffix);
rs->type = ctr->type;
rs->value = ctr->value;
rs->flags = ctr->flags;
rs->rrddim = ctr->rrddim;
rrddimvar_update_variables_unsafe(rs);
}
static bool rrddimvar_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *new_rrddimvar __maybe_unused, void *constructor_data __maybe_unused) {
RRDDIMVAR *rs = rrddimvar;
rrddimvar_update_variables_unsafe(rs);
return true;
}
static void rrddimvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *rrdset __maybe_unused) {
RRDDIMVAR *rs = rrddimvar;
rrddimvar_free_variables_unsafe(rs);
string_freez(rs->prefix);
string_freez(rs->suffix);
}
void rrddimvar_index_init(RRDSET *st) {
if(!st->rrddimvar_root_index) {
st->rrddimvar_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(st->rrddimvar_root_index, rrddimvar_insert_callback, NULL);
dictionary_register_conflict_callback(st->rrddimvar_root_index, rrddimvar_conflict_callback, NULL);
dictionary_register_delete_callback(st->rrddimvar_root_index, rrddimvar_delete_callback, st);
}
}
void rrddimvar_index_destroy(RRDSET *st) {
dictionary_destroy(st->rrddimvar_root_index);
st->rrddimvar_root_index = NULL;
}
void rrddimvar_add_and_leave_released(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_FLAGS flags) {
if(!prefix) prefix = "";
if(!suffix) suffix = "";
RRDDIMVAR *rs = (RRDDIMVAR *)callocz(1, sizeof(RRDDIMVAR));
char key[RRDDIMVAR_ID_MAX + 1];
size_t key_len = snprintfz(key, RRDDIMVAR_ID_MAX, "%s_%s_%s", prefix, rrddim_id(rd), suffix);
rs->prefix = string_strdupz(prefix);
rs->suffix = string_strdupz(suffix);
rs->type = type;
rs->value = value;
rs->options = options;
rs->rrddim = rd;
DOUBLE_LINKED_LIST_PREPEND_UNSAFE(rd->variables, rs, prev, next);
rrddimvar_create_variables(rs);
return rs;
struct rrddimvar_constructor tmp = {
.suffix = suffix,
.prefix = prefix,
.type = type,
.flags = flags,
.value = value,
.rrddim = rd
};
dictionary_set_advanced(rd->rrdset->rrddimvar_root_index, key, (ssize_t)(key_len + 1), NULL, sizeof(RRDDIMVAR), &tmp);
}
void rrddimvar_rename_all(RRDDIM *rd) {
RRDSET *st = rd->rrdset;
(void)st;
debug(D_VARIABLES, "RRDDIMSET rename for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
debug(D_VARIABLES, "RRDDIMVAR rename for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
RRDDIMVAR *rs, *next = rd->variables;
while((rs = next)) {
next = rs->next;
rrddimvar_create_variables(rs);
RRDDIMVAR *rs;
dfe_start_write(st->rrddimvar_root_index, rs) {
if(unlikely(rs->rrddim == rd))
rrddimvar_update_variables_unsafe(rs);
}
dfe_done(rs);
}
void rrddimvar_free(RRDDIMVAR *rs) {
RRDDIM *rd = rs->rrddim;
debug(D_VARIABLES, "RRDDIMSET free for chart id '%s' name '%s', dimension id '%s', name '%s', prefix='%s', suffix='%s'", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset), rrddim_id(rd), rrddim_name(rd), string2str(rs->prefix), string2str(rs->suffix));
void rrddimvar_delete_all(RRDDIM *rd) {
RRDSET *st = rd->rrdset;
rrddimvar_free_variables(rs);
debug(D_VARIABLES, "RRDDIMVAR delete for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rd->variables, rs, prev, next);
string_freez(rs->prefix);
string_freez(rs->suffix);
freez(rs);
RRDDIMVAR *rs;
dfe_start_write(st->rrddimvar_root_index, rs) {
if(unlikely(rs->rrddim == rd))
dictionary_del(st->rrddimvar_root_index, rs_dfe.name);
}
dfe_done(rs);
}

44
database/rrddimvar.h
View file

@ -10,48 +10,12 @@
// calculated / processed by the normal data collection process
// This means, there will be no speed penalty for using
// these variables
struct rrddimvar {
STRING *prefix;
STRING *suffix;
STRING *key_id; // dimension id
STRING *key_name; // dimension name
STRING *key_contextid; // context + dimension id
STRING *key_contextname; // context + dimension name
STRING *key_fullidid; // chart type.chart id + dimension id
STRING *key_fullidname; // chart type.chart id + dimension name
STRING *key_fullnameid; // chart type.chart name + dimension id
STRING *key_fullnamename; // chart type.chart name + dimension name
RRDVAR_TYPE type;
void *value;
RRDVAR_OPTIONS options;
RRDVAR *var_local_id;
RRDVAR *var_local_name;
RRDVAR *var_family_id;
RRDVAR *var_family_name;
RRDVAR *var_family_contextid;
RRDVAR *var_family_contextname;
RRDVAR *var_host_chartidid;
RRDVAR *var_host_chartidname;
RRDVAR *var_host_chartnameid;
RRDVAR *var_host_chartnamename;
struct rrddim *rrddim;
struct rrddimvar *next;
struct rrddimvar *prev;
};
extern void rrddimvar_rename_all(RRDDIM *rd);
extern RRDDIMVAR *rrddimvar_create(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_OPTIONS options);
extern void rrddimvar_free(RRDDIMVAR *rs);
extern void rrddimvar_add_and_leave_released(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_FLAGS flags);
extern void rrddimvar_delete_all(RRDDIM *rd);
extern void rrddimvar_index_init(RRDSET *st);
extern void rrddimvar_index_destroy(RRDSET *st);
#endif //NETDATA_RRDDIMVAR_H

89
database/rrdfamily.c
View file

@ -3,61 +3,66 @@
#define NETDATA_RRD_INTERNALS
#include "rrd.h"
typedef struct rrdfamily {
STRING *family;
DICTIONARY *rrdvars;
} RRDFAMILY;
// ----------------------------------------------------------------------------
// RRDFAMILY index
static inline RRDFAMILY *rrdfamily_index_add(RRDHOST *host, RRDFAMILY *rc) {
return dictionary_set(host->rrdfamily_root_index, string2str(rc->family), rc, sizeof(RRDFAMILY));
struct rrdfamily_constructor {
const char *family;
};
static void rrdfamily_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdfamily, void *constructor_data) {
RRDFAMILY *rf = rrdfamily;
struct rrdfamily_constructor *ctr = constructor_data;
rf->family = string_strdupz(ctr->family);
rf->rrdvars = rrdvariables_create();
}
static inline RRDFAMILY *rrdfamily_index_del(RRDHOST *host, RRDFAMILY *rc) {
dictionary_del(host->rrdfamily_root_index, string2str(rc->family));
return rc;
static void rrdfamily_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdfamily, void *rrdhost __maybe_unused) {
RRDFAMILY *rf = rrdfamily;
string_freez(rf->family);
rrdvariables_destroy(rf->rrdvars);
rf->family = NULL;
rf->rrdvars = NULL;
}
static inline RRDFAMILY *rrdfamily_index_find(RRDHOST *host, const char *id) {
return dictionary_get(host->rrdfamily_root_index, id);
void rrdfamily_index_init(RRDHOST *host) {
if(!host->rrdfamily_root_index) {
host->rrdfamily_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(host->rrdfamily_root_index, rrdfamily_insert_callback, NULL);
dictionary_register_delete_callback(host->rrdfamily_root_index, rrdfamily_delete_callback, host);
}
}
void rrdfamily_index_destroy(RRDHOST *host) {
dictionary_destroy(host->rrdfamily_root_index);
host->rrdfamily_root_index = NULL;
}
// ----------------------------------------------------------------------------
// RRDFAMILY management
RRDFAMILY *rrdfamily_create(RRDHOST *host, const char *id) {
RRDFAMILY *rc = rrdfamily_index_find(host, id);
if(!rc) {
rc = callocz(1, sizeof(RRDFAMILY));
rc->family = string_strdupz(id);
rc->rrdvar_root_index = dictionary_create(
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
);
RRDFAMILY *ret = rrdfamily_index_add(host, rc);
if(ret != rc)
error("RRDFAMILY: INTERNAL ERROR: Expected to INSERT RRDFAMILY '%s' into index, but inserted '%s'.", string2str(rc->family), (ret)?string2str(ret->family):"NONE");
}
rc->use_count++;
return rc;
const RRDFAMILY_ACQUIRED *rrdfamily_add_and_acquire(RRDHOST *host, const char *id) {
struct rrdfamily_constructor tmp = {
.family = id,
};
return (const RRDFAMILY_ACQUIRED *)dictionary_set_and_acquire_item_advanced(host->rrdfamily_root_index, id, -1, NULL, sizeof(RRDFAMILY), &tmp);
}
void rrdfamily_free(RRDHOST *host, RRDFAMILY *rc) {
rc->use_count--;
if(!rc->use_count) {
RRDFAMILY *ret = rrdfamily_index_del(host, rc);
if(ret != rc)
error("RRDFAMILY: INTERNAL ERROR: Expected to DELETE RRDFAMILY '%s' from index, but deleted '%s'.", string2str(rc->family), (ret)?string2str(ret->family):"NONE");
else {
debug(D_RRD_CALLS, "RRDFAMILY: Cleaning up remaining family variables for host '%s', family '%s'", rrdhost_hostname(host), string2str(rc->family));
rrdvar_free_remaining_variables(host, rc->rrdvar_root_index);
dictionary_destroy(rc->rrdvar_root_index);
string_freez(rc->family);
freez(rc);
}
}
void rrdfamily_release(RRDHOST *host, const RRDFAMILY_ACQUIRED *rfa) {
if(unlikely(!rfa)) return;
dictionary_acquired_item_release(host->rrdfamily_root_index, (const DICTIONARY_ITEM *)rfa);
}
DICTIONARY *rrdfamily_rrdvars_dict(const RRDFAMILY_ACQUIRED *rfa) {
if(unlikely(!rfa)) return NULL;
RRDFAMILY *rf = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rfa);
return(rf->rrdvars);
}

538
database/rrdhost.c
View file

@ -51,18 +51,12 @@ static DICTIONARY *rrdhost_root_index_hostname = NULL;
static inline void rrdhost_init() {
if(unlikely(!rrdhost_root_index)) {
rrdhost_root_index = dictionary_create(
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
| DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
| DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
);
DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
}
if(unlikely(!rrdhost_root_index_hostname)) {
rrdhost_root_index_hostname = dictionary_create(
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
| DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
| DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
);
DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
}
}
@ -70,7 +64,7 @@ static inline void rrdhost_init() {
// RRDHOST index by UUID
inline long rrdhost_hosts_available(void) {
return dictionary_stats_entries(rrdhost_root_index);
return dictionary_entries(rrdhost_root_index);
}
inline RRDHOST *rrdhost_find_by_guid(const char *guid) {
@ -91,7 +85,7 @@ static inline RRDHOST *rrdhost_index_add_by_guid(RRDHOST *host) {
static void rrdhost_index_del_by_guid(RRDHOST *host) {
if(rrdhost_flag_check(host, RRDHOST_FLAG_INDEXED_MACHINE_GUID)) {
if(dictionary_del(rrdhost_root_index, host->machine_guid) != 0)
if(!dictionary_del(rrdhost_root_index, host->machine_guid))
error("RRDHOST: %s() failed to delete machine guid '%s' from index", __FUNCTION__, host->machine_guid);
rrdhost_flag_clear(host, RRDHOST_FLAG_INDEXED_MACHINE_GUID);
@ -126,7 +120,7 @@ static inline void rrdhost_index_del_hostname(RRDHOST *host) {
if(unlikely(!host->hostname)) return;
if(rrdhost_flag_check(host, RRDHOST_FLAG_INDEXED_HOSTNAME)) {
if(dictionary_del(rrdhost_root_index_hostname, rrdhost_hostname(host)) != 0)
if(!dictionary_del(rrdhost_root_index_hostname, rrdhost_hostname(host)))
error("RRDHOST: %s() failed to delete hostname '%s' from index", __FUNCTION__, rrdhost_hostname(host));
rrdhost_flag_clear(host, RRDHOST_FLAG_INDEXED_HOSTNAME);
@ -206,6 +200,114 @@ void set_host_properties(RRDHOST *host, int update_every, RRD_MEMORY_MODE memory
// ----------------------------------------------------------------------------
// RRDHOST - add a host
static void rrdhost_initialize_rrdpush(RRDHOST *host,
unsigned int rrdpush_enabled,
char *rrdpush_destination,
char *rrdpush_api_key,
char *rrdpush_send_charts_matching
) {
if(rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_RRDPUSH)) return;
rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_RRDPUSH);
sender_init(host);
netdata_mutex_init(&host->receiver_lock);
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
if (host->rrdpush_send_destination)
host->destinations = destinations_init(host->rrdpush_send_destination);
host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
host->rrdpush_sender_pipe[0] = -1;
host->rrdpush_sender_pipe[1] = -1;
host->rrdpush_sender_socket = -1;
//host->stream_version = STREAMING_PROTOCOL_CURRENT_VERSION; Unused?
#ifdef ENABLE_HTTPS
host->ssl.conn = NULL;
host->ssl.flags = NETDATA_SSL_START;
host->stream_ssl.conn = NULL;
host->stream_ssl.flags = NETDATA_SSL_START;
#endif
}
static void rrdhost_initialize_health(RRDHOST *host,
int is_localhost
) {
if(!host->health_enabled || rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)) return;
rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_HEALTH);
rrdfamily_index_init(host);
rrdcalctemplate_index_init(host);
rrdcalc_rrdhost_index_init(host);
host->health_default_warn_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat warning", "never");
host->health_default_crit_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat critical", "never");
host->health_log.next_log_id = 1;
host->health_log.next_alarm_id = 1;
host->health_log.max = 1000;
host->health_log.next_log_id = (uint32_t)now_realtime_sec();
host->health_log.next_alarm_id = 0;
long n = config_get_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", host->health_log.max);
if(n < 10) {
error("Host '%s': health configuration has invalid max log entries %ld. Using default %u", rrdhost_hostname(host), n, host->health_log.max);
config_set_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", (long)host->health_log.max);
}
else
host->health_log.max = (unsigned int)n;
netdata_rwlock_init(&host->health_log.alarm_log_rwlock);
char filename[FILENAME_MAX + 1];
if(!is_localhost) {
int r = mkdir(host->varlib_dir, 0775);
if (r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
}
{
snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
int r = mkdir(filename, 0775);
if(r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
}
snprintfz(filename, FILENAME_MAX, "%s/health/health-log.db", host->varlib_dir);
host->health_log_filename = strdupz(filename);
snprintfz(filename, FILENAME_MAX, "%s/alarm-notify.sh", netdata_configured_primary_plugins_dir);
host->health_default_exec = string_strdupz(config_get(CONFIG_SECTION_HEALTH, "script to execute on alarm", filename));
host->health_default_recipient = string_strdupz("root");
// ------------------------------------------------------------------------
// load health configuration
health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
if (!file_is_migrated(host->health_log_filename)) {
int rc = sql_create_health_log_table(host);
if (unlikely(rc)) {
error_report("Failed to create health log table in the database");
health_alarm_log_load(host);
health_alarm_log_open(host);
}
else {
health_alarm_log_load(host);
add_migrated_file(host->health_log_filename, 0);
}
} else {
sql_create_health_log_table(host);
sql_health_alarm_log_load(host);
}
}
RRDHOST *rrdhost_create(const char *hostname,
const char *registry_hostname,
const char *guid,
@ -251,40 +353,18 @@ RRDHOST *rrdhost_create(const char *hostname,
host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
host->health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
sender_init(host);
netdata_mutex_init(&host->receiver_lock);
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
if (host->rrdpush_send_destination)
host->destinations = destinations_init(host->rrdpush_send_destination);
host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
host->rrdpush_sender_pipe[0] = -1;
host->rrdpush_sender_pipe[1] = -1;
host->rrdpush_sender_socket = -1;
//host->stream_version = STREAMING_PROTOCOL_CURRENT_VERSION; Unused?
#ifdef ENABLE_HTTPS
host->ssl.conn = NULL;
host->ssl.flags = NETDATA_SSL_START;
host->stream_ssl.conn = NULL;
host->stream_ssl.flags = NETDATA_SSL_START;
#endif
if (likely(!archived)) {
host->rrdlabels = rrdlabels_create();
rrdhost_initialize_rrdpush(
host, rrdpush_enabled, rrdpush_destination, rrdpush_api_key, rrdpush_send_charts_matching);
}
netdata_rwlock_init(&host->rrdhost_rwlock);
if (likely(!archived))
host->rrdlabels = rrdlabels_create();
netdata_mutex_init(&host->aclk_state_lock);
host->system_info = system_info;
host->rrdset_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
host->rrdset_root_index_name = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
host->rrdfamily_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
host->rrdvar_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
rrdset_index_init(host);
if(config_get_boolean(CONFIG_SECTION_DB, "delete obsolete charts files", 1))
rrdhost_flag_set(host, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS);
@ -292,41 +372,17 @@ RRDHOST *rrdhost_create(const char *hostname,
if(config_get_boolean(CONFIG_SECTION_DB, "delete orphan hosts files", 1) && !is_localhost)
rrdhost_flag_set(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST);
host->health_default_warn_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat warning", "never");
host->health_default_crit_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat critical", "never");
// ------------------------------------------------------------------------
// initialize health variables
host->health_log.next_log_id = 1;
host->health_log.next_alarm_id = 1;
host->health_log.max = 1000;
host->health_log.next_log_id = (uint32_t)now_realtime_sec();
host->health_log.next_alarm_id = 0;
long n = config_get_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", host->health_log.max);
if(n < 10) {
error("Host '%s': health configuration has invalid max log entries %ld. Using default %u", rrdhost_hostname(host), n, host->health_log.max);
config_set_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", (long)host->health_log.max);
}
else
host->health_log.max = (unsigned int)n;
netdata_rwlock_init(&host->health_log.alarm_log_rwlock);
char filename[FILENAME_MAX + 1];
if(is_localhost) {
host->cache_dir = strdupz(netdata_configured_cache_dir);
host->varlib_dir = strdupz(netdata_configured_varlib_dir);
}
else {
// this is not localhost - append our GUID to localhost path
if (is_in_multihost) { // don't append to cache dir in multihost
host->cache_dir = strdupz(netdata_configured_cache_dir);
} else {
}
else {
snprintfz(filename, FILENAME_MAX, "%s/%s", netdata_configured_cache_dir, host->machine_guid);
host->cache_dir = strdupz(filename);
}
@ -340,38 +396,11 @@ RRDHOST *rrdhost_create(const char *hostname,
snprintfz(filename, FILENAME_MAX, "%s/%s", netdata_configured_varlib_dir, host->machine_guid);
host->varlib_dir = strdupz(filename);
if(host->health_enabled) {
int r = mkdir(host->varlib_dir, 0775);
if(r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
}
}
if(host->health_enabled) {
snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
int r = mkdir(filename, 0775);
if(r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
}
snprintfz(filename, FILENAME_MAX, "%s/health/health-log.db", host->varlib_dir);
host->health_log_filename = strdupz(filename);
snprintfz(filename, FILENAME_MAX, "%s/alarm-notify.sh", netdata_configured_primary_plugins_dir);
host->health_default_exec = string_strdupz(config_get(CONFIG_SECTION_HEALTH, "script to execute on alarm", filename));
host->health_default_recipient = string_strdupz("root");
// ------------------------------------------------------------------------
// load health configuration
if(host->health_enabled) {
rrdhost_wrlock(host);
health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
rrdhost_unlock(host);
}
// this is also needed for custom host variables - not only health
if(!host->rrdvars)
host->rrdvars = rrdvariables_create();
RRDHOST *t = rrdhost_index_add_by_guid(host);
if(t != host) {
@ -382,33 +411,20 @@ RRDHOST *rrdhost_create(const char *hostname,
if (likely(!uuid_parse(host->machine_guid, host->host_uuid))) {
int rc;
if (!archived) {
if(!archived) {
rc = sql_store_host_info(host);
if (unlikely(rc))
error_report("Failed to store machine GUID to the database");
}
sql_load_node_id(host);
if (host->health_enabled) {
if (!file_is_migrated(host->health_log_filename)) {
rc = sql_create_health_log_table(host);
if (unlikely(rc)) {
error_report("Failed to create health log table in the database");
health_alarm_log_load(host);
health_alarm_log_open(host);
}
else {
health_alarm_log_load(host);
add_migrated_file(host->health_log_filename, 0);
}
} else {
sql_create_health_log_table(host);
sql_health_alarm_log_load(host);
}
}
}
else
error_report("Host machine GUID %s is not valid", host->machine_guid);
rrdhost_initialize_health(host, is_localhost);
if (host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
#ifdef ENABLE_DBENGINE
char dbenginepath[FILENAME_MAX + 1];
@ -530,6 +546,8 @@ RRDHOST *rrdhost_create(const char *hostname,
ml_new_host(host);
else
rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
return host;
}
@ -605,50 +623,24 @@ void rrdhost_update(RRDHOST *host
// update host tags
rrdhost_init_tags(host, tags);
if(!host->rrdvars)
host->rrdvars = rrdvariables_create();
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
rrdhost_flag_clear(host, RRDHOST_FLAG_ARCHIVED);
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
if (host->rrdpush_send_destination)
host->destinations = destinations_init(host->rrdpush_send_destination);
host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
if(!host->rrdlabels)
host->rrdlabels = rrdlabels_create();
if(host->health_enabled) {
int r;
char filename[FILENAME_MAX + 1];
rrdhost_initialize_rrdpush(host,
rrdpush_enabled,
rrdpush_destination,
rrdpush_api_key,
rrdpush_send_charts_matching);
if (host != localhost) {
r = mkdir(host->varlib_dir, 0775);
if (r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
}
snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
r = mkdir(filename, 0775);
if(r != 0 && errno != EEXIST)
error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
rrdhost_initialize_health(host,
host == localhost);
rrdhost_wrlock(host);
health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
rrdhost_unlock(host);
if (!file_is_migrated(host->health_log_filename)) {
int rc = sql_create_health_log_table(host);
if (unlikely(rc)) {
error_report("Failed to create health log table in the database");
health_alarm_log_load(host);
health_alarm_log_open(host);
} else {
health_alarm_log_load(host);
add_migrated_file(host->health_log_filename, 0);
}
} else {
sql_create_health_log_table(host);
sql_health_alarm_log_load(host);
}
}
rrd_hosts_available++;
ml_new_host(host);
rrdhost_load_rrdcontext_data(host);
@ -762,32 +754,6 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
return 0;
}
void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host) {
time_t now = now_realtime_sec();
RRDHOST *host;
restart_after_removal:
rrdhost_foreach_write(host) {
if(rrdhost_should_be_removed(host, protected_host, now)) {
info("Host '%s' with machine guid '%s' is obsolete - cleaning up.", rrdhost_hostname(host), host->machine_guid);
if (rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST)
#ifdef ENABLE_DBENGINE
/* don't delete multi-host DB host files */
&& !(host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && is_storage_engine_shared(host->storage_instance[0]))
#endif
)
rrdhost_delete_charts(host);
else
rrdhost_save_charts(host);
rrdhost_free(host, 0);
goto restart_after_removal;
}
}
}
// ----------------------------------------------------------------------------
// RRDHOST global / startup initialization
@ -1091,12 +1057,21 @@ void rrdhost_free(RRDHOST *host, bool force) {
// ------------------------------------------------------------------------
// clean up streaming
stop_streaming_sender(host);
if (netdata_exit || force)
stop_streaming_receiver(host);
// ------------------------------------------------------------------------
// clean up alarms
rrdcalc_delete_all(host);
rrdhost_wrlock(host); // lock this RRDHOST
// ------------------------------------------------------------------------
// release its children resources
@ -1109,27 +1084,13 @@ void rrdhost_free(RRDHOST *host, bool force) {
}
#endif
while(host->rrdset_root)
rrdset_free(host->rrdset_root);
// delete all the RRDSETs of the host
rrdset_index_destroy(host);
rrdcalc_rrdhost_index_destroy(host);
rrdcalctemplate_index_destroy(host);
freez(host->exporting_flags);
while(host->host_alarms)
rrdcalc_unlink_and_free(host, host->host_alarms);
RRDCALC *rc,*nc;
for(rc = host->alarms_with_foreach; rc ; rc = nc) {
nc = rc->next;
rrdcalc_free(rc);
}
host->alarms_with_foreach = NULL;
while(host->alarms_templates)
rrdcalctemplate_unlink_and_free(host, host->alarms_templates);
debug(D_RRD_CALLS, "RRDHOST: Cleaning up remaining host variables for host '%s'", rrdhost_hostname(host));
rrdvar_free_remaining_variables(host, host->rrdvar_root_index);
health_alarm_log_free(host);
#ifdef ENABLE_DBENGINE
@ -1208,10 +1169,8 @@ void rrdhost_free(RRDHOST *host, bool force) {
netdata_rwlock_destroy(&host->rrdhost_rwlock);
freez(host->node_id);
dictionary_destroy(host->rrdset_root_index);
dictionary_destroy(host->rrdset_root_index_name);
dictionary_destroy(host->rrdfamily_root_index);
dictionary_destroy(host->rrdvar_root_index);
rrdfamily_index_destroy(host);
rrdvariables_destroy(host->rrdvars);
rrdhost_destroy_rrdcontexts(host);
@ -1249,15 +1208,10 @@ void rrdhost_save_charts(RRDHOST *host) {
// we get a write lock
// to ensure only one thread is saving the database
rrdhost_wrlock(host);
rrdset_foreach_write(st, host) {
rrdset_rdlock(st);
rrdset_save(st);
rrdset_unlock(st);
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
}
static void rrdhost_load_auto_labels(void) {
@ -1411,17 +1365,12 @@ void rrdhost_delete_charts(RRDHOST *host) {
// we get a write lock
// to ensure only one thread is saving the database
rrdhost_wrlock(host);
rrdset_foreach_write(st, host) {
rrdset_rdlock(st);
rrdset_delete_files(st);
rrdset_unlock(st);
rrdset_delete_files(st);
}
rrdset_foreach_done(st);
recursively_delete_dir(host->cache_dir, "left over host");
rrdhost_unlock(host);
}
// ----------------------------------------------------------------------------
@ -1437,22 +1386,19 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
// we get a write lock
// to ensure only one thread is saving the database
rrdhost_wrlock(host);
rrdset_foreach_write(st, host) {
rrdset_rdlock(st);
if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE))
rrdset_delete_files(st);
else if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS))
rrdset_delete_obsolete_dimensions(st);
else
rrdset_save(st);
rrdset_unlock(st);
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
}
@ -1496,157 +1442,6 @@ void rrdhost_cleanup_all(void) {
}
// ----------------------------------------------------------------------------
// RRDHOST - save or delete all the host charts from disk
void rrdhost_cleanup_obsolete_charts(RRDHOST *host) {
time_t now = now_realtime_sec();
RRDSET *st;
uint32_t rrdhost_delete_obsolete_charts = rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS);
restart_after_removal:
rrdset_foreach_write(st, host) {
if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)
&& st->last_accessed_time + rrdset_free_obsolete_time < now
&& st->last_updated.tv_sec + rrdset_free_obsolete_time < now
&& st->last_collected_time.tv_sec + rrdset_free_obsolete_time < now
)) {
st->rrdhost->obsolete_charts_count--;
#ifdef ENABLE_DBENGINE
if(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
RRDDIM *rd, *last;
rrdset_flag_set(st, RRDSET_FLAG_ARCHIVED);
while (st->variables) rrdsetvar_free(st->variables);
while (st->alarms) rrdsetcalc_unlink(st->alarms);
rrdset_wrlock(st);
for (rd = st->dimensions, last = NULL ; likely(rd) ; ) {
if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
last = rd;
rd = rd->next;
continue;
}
if (rrddim_flag_check(rd, RRDDIM_FLAG_ACLK)) {
last = rd;
rd = rd->next;
continue;
}
rrddim_flag_set(rd, RRDDIM_FLAG_ARCHIVED);
while (rd->variables)
rrddimvar_free(rd->variables);
if (rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) {
rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
/* only a collector can mark a chart as obsolete, so we must remove the reference */
size_t tiers_available = 0, tiers_said_yes = 0;
for(int tier = 0; tier < storage_tiers ;tier++) {
if(rd->tiers[tier]) {
tiers_available++;
if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
tiers_said_yes++;
rd->tiers[tier]->db_collection_handle = NULL;
}
}
if (tiers_available == tiers_said_yes && tiers_said_yes) {
/* This metric has no data and no references */
delete_dimension_uuid(&rd->metric_uuid);
rrddim_free(st, rd);
if (unlikely(!last)) {
rd = st->dimensions;
}
else {
rd = last->next;
}
continue;
}
#ifdef ENABLE_ACLK
else
queue_dimension_to_aclk(rd, rd->last_collected_time.tv_sec);
#endif
}
last = rd;
rd = rd->next;
}
rrdset_unlock(st);
debug(D_RRD_CALLS, "RRDSET: Cleaning up remaining chart variables for host '%s', chart '%s'", rrdhost_hostname(host), rrdset_id(st));
rrdvar_free_remaining_variables(host, st->rrdvar_root_index);
rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
if (st->dimensions) {
/* If the chart still has dimensions don't delete it from the metadata log */
continue;
}
}
#endif
rrdset_rdlock(st);
if(rrdhost_delete_obsolete_charts)
rrdset_delete_files(st);
else
rrdset_save(st);
rrdset_unlock(st);
rrdset_free(st);
goto restart_after_removal;
}
#ifdef ENABLE_ACLK
else
sql_check_chart_liveness(st);
#endif
}
}
void rrdset_check_obsoletion(RRDHOST *host)
{
RRDSET *st;
time_t last_entry_t;
rrdset_foreach_read(st, host) {
last_entry_t = rrdset_last_entry_t(st);
if (last_entry_t && last_entry_t < host->senders_connect_time) {
rrdset_is_obsolete(st);
}
}
}
void rrd_cleanup_obsolete_charts()
{
rrd_rdlock();
RRDHOST *host;
rrdhost_foreach_read(host)
{
if (host->obsolete_charts_count) {
rrdhost_wrlock(host);
rrdhost_cleanup_obsolete_charts(host);
rrdhost_unlock(host);
}
if ( host != localhost &&
host->trigger_chart_obsoletion_check &&
((host->senders_last_chart_command &&
host->senders_last_chart_command + host->health_delay_up_to < now_realtime_sec())
|| (host->senders_connect_time + 300 < now_realtime_sec())) ) {
rrdhost_rdlock(host);
rrdset_check_obsoletion(host);
rrdhost_unlock(host);
host->trigger_chart_obsoletion_check = 0;
}
}
rrd_unlock();
}
// ----------------------------------------------------------------------------
// RRDHOST - set system info from environment variables
// system_info fields must be heap allocated or NULL
@ -1786,14 +1581,15 @@ int rrdhost_set_system_info_variable(struct rrdhost_system_info *system_info, ch
// Added for gap-filling, if this proves to be a bottleneck in large-scale systems then we will need to cache
// the last entry times as the metric updates, but let's see if it is a problem first.
time_t rrdhost_last_entry_t(RRDHOST *h) {
rrdhost_rdlock(h);
RRDSET *st;
time_t result = 0;
rrdset_foreach_read(st, h) {
time_t st_last = rrdset_last_entry_t(st);
if (st_last > result)
result = st_last;
}
rrdhost_unlock(h);
rrdset_foreach_done(st);
return result;
}

99
database/rrdlabels.c
View file

@ -478,9 +478,7 @@ typedef struct rrdlabel {
RRDLABEL_SRC label_source;
} RRDLABEL;
static void rrdlabel_insert_callback(const char *name, void *value, void *data) {
(void)name;
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
static void rrdlabel_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
RRDLABEL *lb = (RRDLABEL *)value;
// label_value is already allocated by the STRING
@ -488,42 +486,43 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data)
lb->label_source &= ~RRDLABEL_FLAG_OLD;
}
static void rrdlabel_delete_callback(const char *name, void *value, void *data) {
(void)name;
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
static void rrdlabel_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
RRDLABEL *lb = (RRDLABEL *)value;
string_freez(lb->label_value);
lb->label_value = NULL;
}
static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) {
(void)name;
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
static bool rrdlabel_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldvalue, void *newvalue, void *dict_ptr __maybe_unused) {
RRDLABEL *lbold = (RRDLABEL *)oldvalue;
RRDLABEL *lbnew = (RRDLABEL *)newvalue;
if(lbold->label_value == lbnew->label_value) {
// they are the same
lbold->label_source |= lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_OLD;
lbold->label_source &= ~RRDLABEL_FLAG_NEW;
// free the new one
string_freez(lbnew->label_value);
return false;
}
else {
// they are different
string_freez(lbold->label_value);
lbold->label_value = lbnew->label_value;
lbold->label_source = lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_NEW;
lbold->label_source &= ~RRDLABEL_FLAG_OLD;
}
// they are different
string_freez(lbold->label_value);
lbold->label_value = lbnew->label_value;
lbold->label_source = lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_NEW;
lbold->label_source &= ~RRDLABEL_FLAG_OLD;
return true;
}
DICTIONARY *rrdlabels_create(void) {
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(dict, rrdlabel_insert_callback, dict);
dictionary_register_delete_callback(dict, rrdlabel_delete_callback, dict);
dictionary_register_conflict_callback(dict, rrdlabel_conflict_callback, dict);
@ -623,7 +622,7 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) {
// rrdlabels_get_value_to_buffer_or_null()
void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) {
DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
if(lb && lb->label_value)
@ -638,7 +637,7 @@ void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const
// rrdlabels_get_value_to_char_or_null()
void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const char *key) {
DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
*value = (lb && lb->label_value) ? strdupz(string2str(lb->label_value)) : NULL;
@ -650,10 +649,7 @@ void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const
// rrdlabels_unmark_all()
// remove labels RRDLABEL_FLAG_OLD and RRDLABEL_FLAG_NEW from all dictionary items
static int remove_flags_old_new(const char *name, void *value, void *data) {
(void)name;
(void)data;
static int remove_flags_old_new(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
RRDLABEL *lb = (RRDLABEL *)value;
if(lb->label_source & RRDLABEL_FLAG_OLD) lb->label_source &= ~RRDLABEL_FLAG_OLD;
@ -671,12 +667,13 @@ void rrdlabels_unmark_all(DICTIONARY *labels) {
// rrdlabels_remove_all_unmarked()
// remove dictionary items that are neither old, nor new
static int remove_not_old_not_new_callback(const char *name, void *value, void *data) {
static int remove_not_old_not_new_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
DICTIONARY *dict = (DICTIONARY *)data;
RRDLABEL *lb = (RRDLABEL *)value;
if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) {
dictionary_del_having_write_lock(dict, name);
dictionary_del(dict, name);
return 1;
}
@ -696,7 +693,8 @@ struct labels_walkthrough {
void *data;
};
static int labels_walkthrough_callback(const char *name, void *value, void *data) {
static int labels_walkthrough_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
struct labels_walkthrough *d = (struct labels_walkthrough *)data;
RRDLABEL *lb = (RRDLABEL *)value;
@ -728,7 +726,8 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const
// rrdlabels_migrate_to_these()
// migrate an existing label list to a new list, INPLACE
static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) {
static int copy_label_to_dictionary_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
DICTIONARY *dst = (DICTIONARY *)data;
RRDLABEL *lb = (RRDLABEL *)value;
labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source);
@ -765,7 +764,8 @@ struct simple_pattern_match_name_value {
char equal;
};
static int simple_pattern_match_name_only_callback(const char *name, void *value, void *data) {
static int simple_pattern_match_name_only_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
(void)value;
@ -775,7 +775,8 @@ static int simple_pattern_match_name_only_callback(const char *name, void *value
return 0;
}
static int simple_pattern_match_name_and_value_callback(const char *name, void *value, void *data) {
static int simple_pattern_match_name_and_value_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
RRDLABEL *lb = (RRDLABEL *)value;
@ -841,7 +842,9 @@ bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_patte
// ----------------------------------------------------------------------------
// Log all labels
static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value, void *data) {
static int rrdlabels_log_label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
BUFFER *wb = (BUFFER *)data;
RRDLABEL *lb = (RRDLABEL *)value;
@ -891,7 +894,8 @@ struct labels_to_buffer {
size_t count;
};
static int label_to_buffer_callback(const char *name, void *value, void *data) {
static int label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
const char *name = dictionary_acquired_item_name(item);
struct labels_to_buffer *t = (struct labels_to_buffer *)data;
RRDLABEL *lb = (RRDLABEL *)value;
@ -961,17 +965,30 @@ void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels) {
if (new_rrdlabels)
rrdlabels_migrate_to_these(st->rrdlabels, new_rrdlabels);
rrdcalc_update_rrdlabels(st);
// TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
BUFFER *sql_buf = buffer_create(1024);
struct label_str tmp = {.sql = sql_buf, .count = 0 };
uuid_unparse_lower(*st->chart_uuid, tmp.uuid_str);
rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
db_execute(buffer_tostring(sql_buf));
buffer_free(sql_buf);
rrdset_save_rrdlabels_to_sql(st);
}
void rrdset_save_rrdlabels_to_sql(RRDSET *st) {
if(!st->rrdlabels) return;
size_t old_version = st->rrdlabels_last_saved_version;
size_t new_version = dictionary_version(st->rrdlabels);
if(new_version != old_version) {
// TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
BUFFER *sql_buf = buffer_create(1024);
struct label_str tmp = {.sql = sql_buf, .count = 0};
uuid_unparse_lower(st->chart_uuid, tmp.uuid_str);
rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
db_execute(buffer_tostring(sql_buf));
buffer_free(sql_buf);
st->rrdlabels_last_saved_version = new_version;
}
}
// ----------------------------------------------------------------------------
// rrdlabels unit test

1099
database/rrdset.c
View file

File diff suppressed because it is too large Load diff

319
database/rrdsetvar.c
View file

@ -1,170 +1,268 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#define NETDATA_HEALTH_INTERNALS
#include "rrd.h"
// ----------------------------------------------------------------------------
// RRDSETVAR management
// CHART VARIABLES
typedef struct rrdsetvar {
STRING *name; // variable name
void *value; // we need this to maintain the allocation for custom chart variables
static inline void rrdsetvar_free_variables(RRDSETVAR *rs) {
RRDSET *st = rs->rrdset;
const RRDVAR_ACQUIRED *rrdvar_local;
const RRDVAR_ACQUIRED *rrdvar_family_chart_id;
const RRDVAR_ACQUIRED *rrdvar_family_chart_name;
const RRDVAR_ACQUIRED *rrdvar_host_chart_id;
const RRDVAR_ACQUIRED *rrdvar_host_chart_name;
RRDVAR_FLAGS flags:24;
RRDVAR_TYPE type:8;
} RRDSETVAR;
// should only be called while the rrdsetvar dict is write locked
// otherwise, 2+ threads may be setting the same variables at the same time
static inline void rrdsetvar_free_rrdvars_unsafe(RRDSET *st, RRDSETVAR *rs) {
RRDHOST *host = st->rrdhost;
// ------------------------------------------------------------------------
// CHART
rrdvar_free(host, st->rrdvar_root_index, rs->var_local);
rs->var_local = NULL;
if(st->rrdvars) {
rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local);
rs->rrdvar_local = NULL;
}
// ------------------------------------------------------------------------
// FAMILY
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family);
rs->var_family = NULL;
rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_name);
rs->var_family_name = NULL;
if(st->rrdfamily) {
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_chart_id);
rs->rrdvar_family_chart_id = NULL;
rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_chart_name);
rs->rrdvar_family_chart_name = NULL;
}
// ------------------------------------------------------------------------
// HOST
rrdvar_free(host, host->rrdvar_root_index, rs->var_host);
rs->var_host = NULL;
rrdvar_free(host, host->rrdvar_root_index, rs->var_host_name);
rs->var_host_name = NULL;
if(host->rrdvars && host->health_enabled) {
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id);
rs->rrdvar_host_chart_id = NULL;
// ------------------------------------------------------------------------
// KEYS
string_freez(rs->key_fullid);
rs->key_fullid = NULL;
string_freez(rs->key_fullname);
rs->key_fullname = NULL;
rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name);
rs->rrdvar_host_chart_name = NULL;
}
}
static inline void rrdsetvar_create_variables(RRDSETVAR *rs) {
RRDSET *st = rs->rrdset;
// should only be called while the rrdsetvar dict is write locked
// otherwise, 2+ threads may be setting the same variables at the same time
static inline void rrdsetvar_update_rrdvars_unsafe(RRDSET *st, RRDSETVAR *rs) {
RRDHOST *host = st->rrdhost;
RRDVAR_OPTIONS options = rs->options;
if(rs->options & RRDVAR_OPTION_ALLOCATED)
options &= ~ RRDVAR_OPTION_ALLOCATED;
RRDVAR_FLAGS options = rs->flags;
options &= ~RRDVAR_OPTIONS_REMOVED_WHEN_PROPAGATING_TO_RRDVAR;
// ------------------------------------------------------------------------
// free the old ones (if any)
rrdsetvar_free_variables(rs);
rrdsetvar_free_rrdvars_unsafe(st, rs);
// ------------------------------------------------------------------------
// KEYS
char buffer[RRDVAR_MAX_LENGTH + 1];
snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), string2str(rs->variable));
rs->key_fullid = string_strdupz(buffer);
snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), string2str(rs->name));
STRING *key_chart_id = string_strdupz(buffer);
snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), string2str(rs->variable));
rs->key_fullname = string_strdupz(buffer);
snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), string2str(rs->name));
STRING *key_chart_name = string_strdupz(buffer);
// ------------------------------------------------------------------------
// CHART
rs->var_local = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->variable, rs->type, options, rs->value);
if(st->rrdvars) {
rs->rrdvar_local = rrdvar_add_and_acquire("local", st->rrdvars, rs->name, rs->type, options, rs->value);
}
// ------------------------------------------------------------------------
// FAMILY
rs->var_family = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_fullid, rs->type, options, rs->value);
rs->var_family_name = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_fullname, rs->type, options, rs->value);
if(st->rrdfamily) {
rs->rrdvar_family_chart_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_chart_id, rs->type, options, rs->value);
rs->rrdvar_family_chart_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_chart_name, rs->type, options, rs->value);
}
// ------------------------------------------------------------------------
// HOST
rs->var_host = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullid, rs->type, options, rs->value);
rs->var_host_name = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullname, rs->type, options, rs->value);
if(host->rrdvars && host->health_enabled) {
rs->rrdvar_host_chart_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id, rs->type, options, rs->value);
rs->rrdvar_host_chart_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name, rs->type, options, rs->value);
}
// free the keys
string_freez(key_chart_id);
string_freez(key_chart_name);
}
RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, RRDVAR_TYPE type, void *value, RRDVAR_OPTIONS options) {
debug(D_VARIABLES, "RRDVARSET create for chart id '%s' name '%s' with variable name '%s'", rrdset_id(st), rrdset_name(st), variable);
RRDSETVAR *rs = (RRDSETVAR *)callocz(1, sizeof(RRDSETVAR));
static void rrdsetvar_free_value_unsafe(RRDSETVAR *rs) {
if(rs->flags & RRDVAR_FLAG_ALLOCATED) {
void *old = rs->value;
rs->value = NULL;
rs->flags &= ~RRDVAR_FLAG_ALLOCATED;
freez(old);
}
}
rs->variable = string_strdupz(variable);
rs->type = type;
rs->value = value;
rs->options = options;
rs->rrdset = st;
static void rrdsetvar_set_value_unsafe(RRDSETVAR *rs, void *new_value) {
rrdsetvar_free_value_unsafe(rs);
DOUBLE_LINKED_LIST_PREPEND_UNSAFE(st->variables, rs, prev, next);
if(new_value)
rs->value = new_value;
else {
NETDATA_DOUBLE *n = mallocz(sizeof(NETDATA_DOUBLE));
*n = NAN;
rs->value = n;
rs->flags |= RRDVAR_FLAG_ALLOCATED;
}
}
rrdsetvar_create_variables(rs);
struct rrdsetvar_constructor {
RRDSET *rrdset;
const char *name;
void *value;
RRDVAR_FLAGS flags :16;
RRDVAR_TYPE type:8;
};
return rs;
static void rrdsetvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *constructor_data) {
RRDSETVAR *rs = rrdsetvar;
struct rrdsetvar_constructor *ctr = constructor_data;
ctr->flags &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
rs->name = string_strdupz(ctr->name);
rs->type = ctr->type;
rs->flags = ctr->flags;
rrdsetvar_set_value_unsafe(rs, ctr->value);
// create the rrdvariables while we are having a write lock to the dictionary
rrdsetvar_update_rrdvars_unsafe(ctr->rrdset, rs);
}
static bool rrdsetvar_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *new_rrdsetvar __maybe_unused, void *constructor_data) {
RRDSETVAR *rs = rrdsetvar;
struct rrdsetvar_constructor *ctr = constructor_data;
ctr->flags &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
RRDVAR_FLAGS options = rs->flags;
options &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
if(((ctr->value == NULL && rs->value != NULL && rs->flags & RRDVAR_FLAG_ALLOCATED) || (rs->value == ctr->value))
&& ctr->flags == options && rs->type == ctr->type) {
// don't reset it - everything is the same, or as it should...
return false;
}
internal_error(true, "RRDSETVAR: resetting variable '%s' of chart '%s' of host '%s', options from 0x%x to 0x%x, type from %d to %d",
string2str(rs->name), rrdset_id(ctr->rrdset), rrdhost_hostname(ctr->rrdset->rrdhost),
options, ctr->flags, rs->type, ctr->type);
rrdsetvar_free_value_unsafe(rs); // we are going to change the options, so free it before setting it
rs->flags = ctr->flags;
rs->type = ctr->type;
rrdsetvar_set_value_unsafe(rs, ctr->value);
// recreate the rrdvariables while we are having a write lock to the dictionary
rrdsetvar_update_rrdvars_unsafe(ctr->rrdset, rs);
return true;
}
static void rrdsetvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *rrdset __maybe_unused) {
RRDSET *st = rrdset;
RRDSETVAR *rs = rrdsetvar;
rrdsetvar_free_rrdvars_unsafe(st, rs);
rrdsetvar_free_value_unsafe(rs);
string_freez(rs->name);
rs->name = NULL;
}
void rrdsetvar_index_init(RRDSET *st) {
if(!st->rrdsetvar_root_index) {
st->rrdsetvar_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(st->rrdsetvar_root_index, rrdsetvar_insert_callback, NULL);
dictionary_register_conflict_callback(st->rrdsetvar_root_index, rrdsetvar_conflict_callback, NULL);
dictionary_register_delete_callback(st->rrdsetvar_root_index, rrdsetvar_delete_callback, st);
}
}
void rrdsetvar_index_destroy(RRDSET *st) {
dictionary_destroy(st->rrdsetvar_root_index);
st->rrdsetvar_root_index = NULL;
}
const RRDSETVAR_ACQUIRED *rrdsetvar_add_and_acquire(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags) {
struct rrdsetvar_constructor tmp = {
.name = name,
.type = type,
.value = value,
.flags = flags,
.rrdset = st,
};
const RRDSETVAR_ACQUIRED *rsa = (const RRDSETVAR_ACQUIRED *)dictionary_set_and_acquire_item_advanced(st->rrdsetvar_root_index, name, -1, NULL, sizeof(RRDSETVAR), &tmp);
return rsa;
}
void rrdsetvar_add_and_leave_released(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags) {
const RRDSETVAR_ACQUIRED *rsa = rrdsetvar_add_and_acquire(st, name, type, value, flags);
dictionary_acquired_item_release(st->rrdsetvar_root_index, (const DICTIONARY_ITEM *)rsa);
}
void rrdsetvar_rename_all(RRDSET *st) {
debug(D_VARIABLES, "RRDSETVAR rename for chart id '%s' name '%s'", rrdset_id(st), rrdset_name(st));
RRDSETVAR *rs;
for(rs = st->variables; rs ; rs = rs->next)
rrdsetvar_create_variables(rs);
dfe_start_write(st->rrdsetvar_root_index, rs) {
// should only be called while the rrdsetvar dict is write locked
rrdsetvar_update_rrdvars_unsafe(st, rs);
}
dfe_done(rs);
rrdsetcalc_link_matching(st);
rrdcalc_link_matching_alerts_to_rrdset(st);
}
void rrdsetvar_free(RRDSETVAR *rs) {
RRDSET *st = rs->rrdset;
debug(D_VARIABLES, "RRDSETVAR free for chart id '%s' name '%s', variable '%s'", rrdset_id(st), rrdset_name(st), string2str(rs->variable));
void rrdsetvar_release_and_delete_all(RRDSET *st) {
RRDSETVAR *rs;
dfe_start_write(st->rrdsetvar_root_index, rs) {
dictionary_del_advanced(st->rrdsetvar_root_index, string2str(rs->name), (ssize_t)string_strlen(rs->name) + 1);
}
dfe_done(rs);
}
DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->variables, rs, prev, next);
rrdsetvar_free_variables(rs);
string_freez(rs->variable);
if(rs->options & RRDVAR_OPTION_ALLOCATED)
freez(rs->value);
freez(rs);
void rrdsetvar_release(DICTIONARY *dict, const RRDSETVAR_ACQUIRED *rsa) {
dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rsa);
}
// --------------------------------------------------------------------------------------------------------------------
// custom chart variables
RRDSETVAR *rrdsetvar_custom_chart_variable_create(RRDSET *st, const char *name) {
RRDHOST *host = st->rrdhost;
STRING *n = rrdvar_name_to_string(name);
rrdset_wrlock(st);
// find it
RRDSETVAR *rs;
for(rs = st->variables; rs ; rs = rs->next) {
if(rs->variable == n) {
rrdset_unlock(st);
if(rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR) {
string_freez(n);
return rs;
}
else {
error("RRDSETVAR: custom variable '%s' on chart '%s' of host '%s', conflicts with an internal chart variable", string2str(n), rrdset_id(st), rrdhost_hostname(host));
string_freez(n);
return NULL;
}
}
}
// not found, allocate one
NETDATA_DOUBLE *v = mallocz(sizeof(NETDATA_DOUBLE));
*v = NAN;
rs = rrdsetvar_create(st, string2str(n), RRDVAR_TYPE_CALCULATED, v, RRDVAR_OPTION_ALLOCATED|RRDVAR_OPTION_CUSTOM_CHART_VAR);
rrdset_unlock(st);
string_freez(n);
const RRDSETVAR_ACQUIRED *rrdsetvar_custom_chart_variable_add_and_acquire(RRDSET *st, const char *name) {
STRING *name_string = rrdvar_name_to_string(name);
const RRDSETVAR_ACQUIRED *rs = rrdsetvar_add_and_acquire(st, string2str(name_string), RRDVAR_TYPE_CALCULATED, NULL, RRDVAR_FLAG_CUSTOM_CHART_VAR);
string_freez(name_string);
return rs;
}
void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rs, NETDATA_DOUBLE value) {
if(rs->type != RRDVAR_TYPE_CALCULATED || !(rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR) || !(rs->options & RRDVAR_OPTION_ALLOCATED)) {
void rrdsetvar_custom_chart_variable_set(RRDSET *st, const RRDSETVAR_ACQUIRED *rsa, NETDATA_DOUBLE value) {
if(!rsa) return;
RRDSETVAR *rs = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rsa);
if(rs->type != RRDVAR_TYPE_CALCULATED || !(rs->flags & RRDVAR_FLAG_CUSTOM_CHART_VAR) || !(rs->flags & RRDVAR_FLAG_ALLOCATED)) {
error("RRDSETVAR: requested to set variable '%s' of chart '%s' on host '%s' to value " NETDATA_DOUBLE_FORMAT
" but the variable is not a custom chart one.", string2str(rs->variable), rrdset_id(rs->rrdset), rrdhost_hostname(rs->rrdset->rrdhost), value);
" but the variable is not a custom chart one (it has options 0x%x, value pointer %p). Ignoring request.", string2str(rs->name), rrdset_id(st), rrdhost_hostname(st->rrdhost), value, (uint32_t)rs->flags, rs->value);
}
else {
NETDATA_DOUBLE *v = rs->value;
@ -172,7 +270,24 @@ void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rs, NETDATA_DOUBLE value) {
*v = value;
// mark the chart to be sent upstream
rrdset_flag_clear(rs->rrdset, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
}
}
}
void rrdsetvar_print_to_streaming_custom_chart_variables(RRDSET *st, BUFFER *wb) {
// send the chart local custom variables
RRDSETVAR *rs;
dfe_start_read(st->rrdsetvar_root_index, rs) {
if(unlikely(rs->type == RRDVAR_TYPE_CALCULATED && rs->flags & RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
NETDATA_DOUBLE *value = (NETDATA_DOUBLE *) rs->value;
buffer_sprintf(wb
, "VARIABLE CHART %s = " NETDATA_DOUBLE_FORMAT "\n"
, string2str(rs->name)
, *value
);
}
}
dfe_done(rs);
}

36
database/rrdsetvar.h
View file

@ -11,34 +11,20 @@
// This means, there will be no speed penalty for using
// these variables
struct rrdsetvar {
STRING *variable; // variable name
extern void rrdsetvar_index_init(RRDSET *st);
extern void rrdsetvar_index_destroy(RRDSET *st);
extern void rrdsetvar_release_and_delete_all(RRDSET *st);
STRING *key_fullid; // chart type.chart id.variable
STRING *key_fullname; // chart type.chart name.variable
#define rrdsetvar_custom_chart_variable_release(st, rsa) rrdsetvar_release((st)->rrdsetvar_root_index, rsa)
extern void rrdsetvar_release(DICTIONARY *dict, const RRDSETVAR_ACQUIRED *rsa);
RRDVAR_TYPE type;
void *value;
RRDVAR_OPTIONS options;
RRDVAR *var_local;
RRDVAR *var_family;
RRDVAR *var_host;
RRDVAR *var_family_name;
RRDVAR *var_host_name;
struct rrdset *rrdset;
struct rrdsetvar *next;
struct rrdsetvar *prev;
};
extern RRDSETVAR *rrdsetvar_custom_chart_variable_create(RRDSET *st, const char *name);
extern void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rv, NETDATA_DOUBLE value);
extern const RRDSETVAR_ACQUIRED *rrdsetvar_custom_chart_variable_add_and_acquire(RRDSET *st, const char *name);
extern void rrdsetvar_custom_chart_variable_set(RRDSET *st, const RRDSETVAR_ACQUIRED *rsa, NETDATA_DOUBLE value);
extern void rrdsetvar_rename_all(RRDSET *st);
extern RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, RRDVAR_TYPE type, void *value, RRDVAR_OPTIONS options);
extern void rrdsetvar_free(RRDSETVAR *rs);
extern const RRDSETVAR_ACQUIRED *rrdsetvar_add_and_acquire(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags);
extern void rrdsetvar_add_and_leave_released(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags);
extern void rrdsetvar_print_to_streaming_custom_chart_variables(RRDSET *st, BUFFER *wb);
#endif //NETDATA_RRDSETVAR_H

282
database/rrdvar.c
View file

@ -1,8 +1,19 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#define NETDATA_HEALTH_INTERNALS
#include "rrd.h"
// the variables as stored in the variables indexes
// there are 3 indexes:
// 1. at each chart (RRDSET.rrdvar_root_index)
// 2. at each context (RRDFAMILY.rrdvar_root_index)
// 3. at each host (RRDHOST.rrdvar_root_index)
typedef struct rrdvar {
STRING *name;
void *value;
RRDVAR_FLAGS flags:24;
RRDVAR_TYPE type:8;
} RRDVAR;
// ----------------------------------------------------------------------------
// RRDVAR management
@ -20,41 +31,6 @@ inline int rrdvar_fix_name(char *variable) {
return fixed;
}
static inline RRDVAR *rrdvar_index_add(DICTIONARY *dict, RRDVAR *rv) {
return dictionary_set(dict, rrdvar_name(rv), rv, sizeof(RRDVAR));
}
static inline RRDVAR *rrdvar_index_del(DICTIONARY *dict, RRDVAR *rv) {
if(dictionary_del(dict, rrdvar_name(rv)) != 0) {
error("Request to remove RRDVAR '%s' from index failed. Not Found.", rrdvar_name(rv));
return NULL;
}
return rv;
}
static inline RRDVAR *rrdvar_index_find(DICTIONARY *dict, STRING *name) {
return dictionary_get(dict, string2str(name));
}
inline void rrdvar_free(RRDHOST *host, DICTIONARY *dict, RRDVAR *rv) {
(void)host;
if(!rv) return;
if(dict) {
debug(D_VARIABLES, "Deleting variable '%s'", rrdvar_name(rv));
if(unlikely(!rrdvar_index_del(dict, rv)))
error("RRDVAR: Attempted to delete variable '%s' from host '%s', but it is not found.", rrdvar_name(rv), rrdhost_hostname(host));
}
if(rv->options & RRDVAR_OPTION_ALLOCATED)
freez(rv->value);
string_freez(rv->name);
freez(rv);
}
inline STRING *rrdvar_name_to_string(const char *name) {
char *variable = strdupz(name);
rrdvar_fix_name(variable);
@ -63,99 +39,145 @@ inline STRING *rrdvar_name_to_string(const char *name) {
return name_string;
}
inline RRDVAR *rrdvar_create_and_index(const char *scope __maybe_unused, DICTIONARY *dict, STRING *name,
RRDVAR_TYPE type, RRDVAR_OPTIONS options, void *value) {
struct rrdvar_constructor {
STRING *name;
void *value;
RRDVAR_FLAGS options:16;
RRDVAR_TYPE type:8;
RRDVAR *rv = rrdvar_index_find(dict, name);
if(unlikely(!rv)) {
debug(D_VARIABLES, "Variable '%s' not found in scope '%s'. Creating a new one.", string2str(name), scope);
enum {
RRDVAR_REACT_NONE = 0,
RRDVAR_REACT_NEW = (1 << 0),
} react_action;
};
rv = callocz(1, sizeof(RRDVAR));
rv->name = string_dup(name);
rv->type = type;
rv->options = options;
rv->value = value;
rv->last_updated = now_realtime_sec();
static void rrdvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar, void *constructor_data) {
RRDVAR *rv = rrdvar;
struct rrdvar_constructor *ctr = constructor_data;
RRDVAR *ret = rrdvar_index_add(dict, rv);
if(unlikely(ret != rv)) {
debug(D_VARIABLES, "Variable '%s' in scope '%s' already exists", string2str(name), scope);
freez(rv);
rv = NULL;
}
else
debug(D_VARIABLES, "Variable '%s' created in scope '%s'", string2str(name), scope);
ctr->options &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
rv->name = string_dup(ctr->name);
rv->type = ctr->type;
rv->flags = ctr->options;
if(!ctr->value) {
NETDATA_DOUBLE *v = mallocz(sizeof(NETDATA_DOUBLE));
*v = NAN;
rv->value = v;
rv->flags |= RRDVAR_FLAG_ALLOCATED;
}
else {
debug(D_VARIABLES, "Variable '%s' is already found in scope '%s'.", string2str(name), scope);
else
rv->value = ctr->value;
// this is important
// it must return NULL - not the existing variable - or double-free will happen
rv = NULL;
}
return rv;
ctr->react_action = RRDVAR_REACT_NEW;
}
void rrdvar_free_remaining_variables(RRDHOST *host, DICTIONARY *dict) {
RRDVAR *rv;
dfe_start_reentrant(dict, rv) {
rrdvar_free(host, dict, rv);
}
dfe_done(rv);
static void rrdvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar, void *nothing __maybe_unused) {
RRDVAR *rv = rrdvar;
if(rv->flags & RRDVAR_FLAG_ALLOCATED)
freez(rv->value);
string_freez(rv->name);
rv->name = NULL;
}
DICTIONARY *rrdvariables_create(void) {
DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
dictionary_register_insert_callback(dict, rrdvar_insert_callback, NULL);
dictionary_register_delete_callback(dict, rrdvar_delete_callback, NULL);
return dict;
}
void rrdvariables_destroy(DICTIONARY *dict) {
dictionary_destroy(dict);
}
static inline const RRDVAR_ACQUIRED *rrdvar_get_and_acquire(DICTIONARY *dict, STRING *name) {
return (const RRDVAR_ACQUIRED *)dictionary_get_and_acquire_item_advanced(dict, string2str(name), (ssize_t)string_strlen(name) + 1);
}
inline void rrdvar_release_and_del(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva) {
if(unlikely(!dict || !rva)) return;
RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
dictionary_del_advanced(dict, string2str(rv->name), (ssize_t)string_strlen(rv->name) + 1);
dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rva);
}
inline const RRDVAR_ACQUIRED *rrdvar_add_and_acquire(const char *scope __maybe_unused, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_FLAGS options, void *value) {
if(unlikely(!dict || !name)) return NULL;
struct rrdvar_constructor tmp = {
.name = name,
.value = value,
.type = type,
.options = options,
.react_action = RRDVAR_REACT_NONE,
};
return (const RRDVAR_ACQUIRED *)dictionary_set_and_acquire_item_advanced(dict, string2str(name), (ssize_t)string_strlen(name) + 1, NULL, sizeof(RRDVAR), &tmp);
}
void rrdvar_delete_all(DICTIONARY *dict) {
dictionary_flush(dict);
}
// ----------------------------------------------------------------------------
// CUSTOM HOST VARIABLES
inline int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const char *name, void *rrdvar, void *data), void *data) {
inline int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const DICTIONARY_ITEM *item, void *rrdvar, void *data), void *data) {
if(unlikely(!dict)) return 0; // when health is not enabled
return dictionary_walkthrough_read(dict, callback, data);
}
static RRDVAR *rrdvar_custom_variable_create(const char *scope, DICTIONARY *dict, const char *name) {
NETDATA_DOUBLE *v = callocz(1, sizeof(NETDATA_DOUBLE));
*v = NAN;
const RRDVAR_ACQUIRED *rrdvar_custom_host_variable_add_and_acquire(RRDHOST *host, const char *name) {
DICTIONARY *dict = host->rrdvars;
if(unlikely(!dict)) return NULL; // when health is not enabled
STRING *name_string = rrdvar_name_to_string(name);
RRDVAR *rv = rrdvar_create_and_index(scope, dict, name_string, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_CUSTOM_HOST_VAR|RRDVAR_OPTION_ALLOCATED, v);
if(unlikely(!rv)) {
freez(v);
debug(D_VARIABLES, "Requested variable '%s' already exists - possibly 2 plugins are updating it at the same time.", string2str(name_string));
// find the existing one to return it
rv = rrdvar_index_find(dict, name_string);
}
const RRDVAR_ACQUIRED *rva = rrdvar_add_and_acquire("host", dict, name_string, RRDVAR_TYPE_CALCULATED, RRDVAR_FLAG_CUSTOM_HOST_VAR, NULL);
string_freez(name_string);
return rv;
return rva;
}
RRDVAR *rrdvar_custom_host_variable_create(RRDHOST *host, const char *name) {
return rrdvar_custom_variable_create("host", host->rrdvar_root_index, name);
}
void rrdvar_custom_host_variable_set(RRDHOST *host, const RRDVAR_ACQUIRED *rva, NETDATA_DOUBLE value) {
if(unlikely(!host->rrdvars || !rva)) return; // when health is not enabled
void rrdvar_custom_host_variable_set(RRDHOST *host, RRDVAR *rv, NETDATA_DOUBLE value) {
if(rv->type != RRDVAR_TYPE_CALCULATED || !(rv->options & RRDVAR_OPTION_CUSTOM_HOST_VAR) || !(rv->options & RRDVAR_OPTION_ALLOCATED))
error("requested to set variable '%s' to value " NETDATA_DOUBLE_FORMAT " but the variable is not a custom one.", rrdvar_name(rv), value);
if(rrdvar_type(rva) != RRDVAR_TYPE_CALCULATED || !(rrdvar_flags(rva) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_ALLOCATED)))
error("requested to set variable '%s' to value " NETDATA_DOUBLE_FORMAT " but the variable is not a custom one.", rrdvar_name(rva), value);
else {
RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
NETDATA_DOUBLE *v = rv->value;
if(*v != value) {
*v = value;
rv->last_updated = now_realtime_sec();
// if the host is streaming, send this variable upstream immediately
rrdpush_sender_send_this_host_variable_now(host, rv);
rrdpush_sender_send_this_host_variable_now(host, rva);
}
}
}
void rrdvar_release(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva) {
if(unlikely(!dict || !rva)) return; // when health is not enabled
dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rva);
}
// ----------------------------------------------------------------------------
// RRDVAR lookup
NETDATA_DOUBLE rrdvar2number(RRDVAR *rv) {
NETDATA_DOUBLE rrdvar2number(const RRDVAR_ACQUIRED *rva) {
if(unlikely(!rva)) return NAN;
RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
switch(rv->type) {
case RRDVAR_TYPE_CALCULATED: {
NETDATA_DOUBLE *n = (NETDATA_DOUBLE *)rv->value;
@ -164,17 +186,17 @@ NETDATA_DOUBLE rrdvar2number(RRDVAR *rv) {
case RRDVAR_TYPE_TIME_T: {
time_t *n = (time_t *)rv->value;
return *n;
return (NETDATA_DOUBLE)*n;
}
case RRDVAR_TYPE_COLLECTED: {
collected_number *n = (collected_number *)rv->value;
return *n;
return (NETDATA_DOUBLE)*n;
}
case RRDVAR_TYPE_TOTAL: {
total_number *n = (total_number *)rv->value;
return *n;
return (NETDATA_DOUBLE)*n;
}
case RRDVAR_TYPE_INT: {
@ -193,23 +215,26 @@ int health_variable_lookup(STRING *variable, RRDCALC *rc, NETDATA_DOUBLE *result
if(!st) return 0;
RRDHOST *host = st->rrdhost;
RRDVAR *rv;
const RRDVAR_ACQUIRED *rva;
rv = rrdvar_index_find(st->rrdvar_root_index, variable);
if(rv) {
*result = rrdvar2number(rv);
rva = rrdvar_get_and_acquire(st->rrdvars, variable);
if(rva) {
*result = rrdvar2number(rva);
dictionary_acquired_item_release(st->rrdvars, (const DICTIONARY_ITEM *)rva);
return 1;
}
rv = rrdvar_index_find(st->rrdfamily->rrdvar_root_index, variable);
if(rv) {
*result = rrdvar2number(rv);
rva = rrdvar_get_and_acquire(rrdfamily_rrdvars_dict(st->rrdfamily), variable);
if(rva) {
*result = rrdvar2number(rva);
dictionary_acquired_item_release(rrdfamily_rrdvars_dict(st->rrdfamily), (const DICTIONARY_ITEM *)rva);
return 1;
}
rv = rrdvar_index_find(host->rrdvar_root_index, variable);
if(rv) {
*result = rrdvar2number(rv);
rva = rrdvar_get_and_acquire(host->rrdvars, variable);
if(rva) {
*result = rrdvar2number(rva);
dictionary_acquired_item_release(host->rrdvars, (const DICTIONARY_ITEM *)rva);
return 1;
}
@ -222,19 +247,19 @@ int health_variable_lookup(STRING *variable, RRDCALC *rc, NETDATA_DOUBLE *result
struct variable2json_helper {
BUFFER *buf;
size_t counter;
RRDVAR_OPTIONS options;
RRDVAR_FLAGS options;
};
static int single_variable2json(const char *name __maybe_unused, void *entry, void *data) {
struct variable2json_helper *helper = (struct variable2json_helper *)data;
RRDVAR *rv = (RRDVAR *)entry;
NETDATA_DOUBLE value = rrdvar2number(rv);
static int single_variable2json_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry __maybe_unused, void *helper_data) {
struct variable2json_helper *helper = (struct variable2json_helper *)helper_data;
const RRDVAR_ACQUIRED *rva = (const RRDVAR_ACQUIRED *)item;
NETDATA_DOUBLE value = rrdvar2number(rva);
if (helper->options == RRDVAR_OPTION_DEFAULT || rv->options & helper->options) {
if (helper->options == RRDVAR_FLAG_NONE || rrdvar_flags(rva) & helper->options) {
if(unlikely(isnan(value) || isinf(value)))
buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": null", helper->counter?",":"", rrdvar_name(rv));
buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": null", helper->counter?",":"", rrdvar_name(rva));
else
buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": %0.5" NETDATA_DOUBLE_MODIFIER, helper->counter?",":"", rrdvar_name(rv), (NETDATA_DOUBLE)value);
buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": %0.5" NETDATA_DOUBLE_MODIFIER, helper->counter?",":"", rrdvar_name(rva), (NETDATA_DOUBLE)value);
helper->counter++;
}
@ -246,11 +271,10 @@ void health_api_v1_chart_custom_variables2json(RRDSET *st, BUFFER *buf) {
struct variable2json_helper helper = {
.buf = buf,
.counter = 0,
.options = RRDVAR_OPTION_CUSTOM_CHART_VAR
};
.options = RRDVAR_FLAG_CUSTOM_CHART_VAR};
buffer_sprintf(buf, "{");
rrdvar_walkthrough_read(st->rrdvar_root_index, single_variable2json, &helper);
rrdvar_walkthrough_read(st->rrdvars, single_variable2json_callback, &helper);
buffer_strcat(buf, "\n\t\t\t}");
}
@ -260,20 +284,34 @@ void health_api_v1_chart_variables2json(RRDSET *st, BUFFER *buf) {
struct variable2json_helper helper = {
.buf = buf,
.counter = 0,
.options = RRDVAR_OPTION_DEFAULT
};
.options = RRDVAR_FLAG_NONE};
buffer_sprintf(buf, "{\n\t\"chart\": \"%s\",\n\t\"chart_name\": \"%s\",\n\t\"chart_context\": \"%s\",\n\t\"chart_variables\": {", rrdset_id(st), rrdset_name(st), rrdset_context(st));
rrdvar_walkthrough_read(st->rrdvar_root_index, single_variable2json, &helper);
rrdvar_walkthrough_read(st->rrdvars, single_variable2json_callback, &helper);
buffer_sprintf(buf, "\n\t},\n\t\"family\": \"%s\",\n\t\"family_variables\": {", rrdset_family(st));
helper.counter = 0;
rrdvar_walkthrough_read(st->rrdfamily->rrdvar_root_index, single_variable2json, &helper);
rrdvar_walkthrough_read(rrdfamily_rrdvars_dict(st->rrdfamily), single_variable2json_callback, &helper);
buffer_sprintf(buf, "\n\t},\n\t\"host\": \"%s\",\n\t\"host_variables\": {", rrdhost_hostname(host));
helper.counter = 0;
rrdvar_walkthrough_read(host->rrdvar_root_index, single_variable2json, &helper);
rrdvar_walkthrough_read(host->rrdvars, single_variable2json_callback, &helper);
buffer_strcat(buf, "\n\t}\n}\n");
}
// ----------------------------------------------------------------------------
// RRDVAR private members examination
const char *rrdvar_name(const RRDVAR_ACQUIRED *rva) {
return dictionary_acquired_item_name((const DICTIONARY_ITEM *)rva);
}
RRDVAR_FLAGS rrdvar_flags(const RRDVAR_ACQUIRED *rva) {
RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
return rv->flags;
}
RRDVAR_TYPE rrdvar_type(const RRDVAR_ACQUIRED *rva) {
RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
return rv->type;
}

68
database/rrdvar.h
View file

@ -11,35 +11,32 @@ typedef enum rrdvar_type {
RRDVAR_TYPE_COLLECTED = 3,
RRDVAR_TYPE_TOTAL = 4,
RRDVAR_TYPE_INT = 5
// this is 8 bit
// to increase it you have to set change the bitfield in
// rrdvar, rrdsetvar, rrddimvar
} RRDVAR_TYPE;
typedef enum rrdvar_options {
RRDVAR_OPTION_DEFAULT = 0,
RRDVAR_OPTION_ALLOCATED = (1 << 0), // the value ptr is allocated (not a reference)
RRDVAR_OPTION_CUSTOM_HOST_VAR = (1 << 1), // this is a custom host variable, not associated with a dimension
RRDVAR_OPTION_CUSTOM_CHART_VAR = (1 << 2), // this is a custom chart variable, not associated with a dimension
RRDVAR_OPTION_RRDCALC_LOCAL_VAR = (1 << 3), // this is a an alarm variable, attached to a chart
RRDVAR_OPTION_RRDCALC_FAMILY_VAR = (1 << 4), // this is a an alarm variable, attached to a family
RRDVAR_OPTION_RRDCALC_HOST_CHARTID_VAR = (1 << 5), // this is a an alarm variable, attached to the host, using the chart id
RRDVAR_OPTION_RRDCALC_HOST_CHARTNAME_VAR = (1 << 6), // this is a an alarm variable, attached to the host, using the chart name
} RRDVAR_OPTIONS;
RRDVAR_FLAG_NONE = 0,
RRDVAR_FLAG_ALLOCATED = (1 << 0), // the value ptr is allocated (not a reference)
RRDVAR_FLAG_CUSTOM_HOST_VAR = (1 << 1), // this is a custom host variable, not associated with a dimension
RRDVAR_FLAG_CUSTOM_CHART_VAR = (1 << 2), // this is a custom chart variable, not associated with a dimension
RRDVAR_FLAG_RRDCALC_LOCAL_VAR = (1 << 3), // this is a an alarm variable, attached to a chart
RRDVAR_FLAG_RRDCALC_FAMILY_VAR = (1 << 4), // this is a an alarm variable, attached to a family
RRDVAR_FLAG_RRDCALC_HOST_CHARTID_VAR = (1 << 5), // this is a an alarm variable, attached to the host, using the chart id
RRDVAR_FLAG_RRDCALC_HOST_CHARTNAME_VAR = (1 << 6), // this is a an alarm variable, attached to the host, using the chart name
// the variables as stored in the variables indexes
// there are 3 indexes:
// 1. at each chart (RRDSET.rrdvar_root_index)
// 2. at each context (RRDFAMILY.rrdvar_root_index)
// 3. at each host (RRDHOST.rrdvar_root_index)
struct rrdvar {
STRING *name;
// this is 24 bit
// to increase it you have to set change the bitfield in
// rrdvar, rrdsetvar, rrddimvar
} RRDVAR_FLAGS;
void *value;
time_t last_updated;
#define RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS \
(RRDVAR_FLAG_ALLOCATED)
RRDVAR_OPTIONS options:16;
RRDVAR_TYPE type:8;
};
#define rrdvar_name(rv) string2str((rv)->name)
#define RRDVAR_OPTIONS_REMOVED_WHEN_PROPAGATING_TO_RRDVAR \
(RRDVAR_FLAG_ALLOCATED)
#define RRDVAR_MAX_LENGTH 1024
@ -49,15 +46,26 @@ extern int rrdvar_fix_name(char *variable);
extern STRING *rrdvar_name_to_string(const char *name);
extern RRDVAR *rrdvar_custom_host_variable_create(RRDHOST *host, const char *name);
extern void rrdvar_custom_host_variable_set(RRDHOST *host, RRDVAR *rv, NETDATA_DOUBLE value);
extern void rrdvar_free_remaining_variables(RRDHOST *host, DICTIONARY *dict);
extern const RRDVAR_ACQUIRED *rrdvar_custom_host_variable_add_and_acquire(RRDHOST *host, const char *name);
extern void rrdvar_custom_host_variable_set(RRDHOST *host, const RRDVAR_ACQUIRED *rva, NETDATA_DOUBLE value);
extern int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const char *name, void *rrdvar, void *data), void *data);
extern int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const DICTIONARY_ITEM *item, void *rrdvar, void *data), void *data);
extern NETDATA_DOUBLE rrdvar2number(RRDVAR *rv);
#define rrdvar_custom_host_variable_release(host, rva) rrdvar_release((host)->rrdvars, rva)
extern void rrdvar_release(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva);
extern RRDVAR *rrdvar_create_and_index(const char *scope, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_OPTIONS options, void *value);
extern void rrdvar_free(RRDHOST *host, DICTIONARY *dict, RRDVAR *rv);
extern NETDATA_DOUBLE rrdvar2number(const RRDVAR_ACQUIRED *rva);
extern const RRDVAR_ACQUIRED *rrdvar_add_and_acquire(const char *scope, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_FLAGS options, void *value);
extern void rrdvar_release_and_del(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva);
extern DICTIONARY *rrdvariables_create(void);
extern void rrdvariables_destroy(DICTIONARY *dict);
extern void rrdvar_delete_all(DICTIONARY *dict);
extern const char *rrdvar_name(const RRDVAR_ACQUIRED *rva);
extern RRDVAR_FLAGS rrdvar_flags(const RRDVAR_ACQUIRED *rva);
extern RRDVAR_TYPE rrdvar_type(const RRDVAR_ACQUIRED *rva);
#endif //NETDATA_RRDVAR_H

3
database/sqlite/sqlite_aclk.c
View file

@ -1023,9 +1023,6 @@ void sql_check_aclk_table_list(struct aclk_database_worker_config *wc)
db_execute("DELETE FROM dimension_delete WHERE host_id NOT IN (SELECT host_id FROM host) "
" OR unixepoch() - date_created > 604800;");
db_execute("DELETE FROM chart_hash WHERE CAST(last_used AS INT) < unixepoch() - 604800;");
db_execute("DELETE FROM chart_hash_map WHERE hash_id NOT IN (SELECT hash_id FROM chart_hash);");
return;
}

36
database/sqlite/sqlite_aclk_chart.c
View file

@ -150,7 +150,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
if (likely(claim_id)) {
struct chart_instance_updated chart_payload;
memset(&chart_payload, 0, sizeof(chart_payload));
chart_payload.config_hash = get_str_from_uuid(&st->uuid);
chart_payload.config_hash = get_str_from_uuid(&st->hash_uuid);
chart_payload.update_every = st->update_every;
chart_payload.memory_mode = st->rrd_memory_mode;
chart_payload.name = (char *)rrdset_name(st);
@ -164,7 +164,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
size_t size;
char *payload = generate_chart_instance_updated(&size, &chart_payload);
if (likely(payload))
rc = aclk_add_chart_payload(wc, st->chart_uuid, claim_id, ACLK_PAYLOAD_CHART, (void *) payload, size, NULL, 1);
rc = aclk_add_chart_payload(wc, &st->chart_uuid, claim_id, ACLK_PAYLOAD_CHART, (void *) payload, size, NULL, 1);
freez(payload);
chart_instance_updated_destroy(&chart_payload);
}
@ -588,24 +588,23 @@ void aclk_receive_chart_reset(struct aclk_database_worker_config *wc, struct acl
RRDHOST *host = wc->host;
if (likely(host)) {
rrdhost_rdlock(host);
RRDSET *st;
rrdset_foreach_read(st, host)
{
rrdset_rdlock(st);
rrdset_foreach_read(st, host) {
rrdset_flag_clear(st, RRDSET_FLAG_ACLK);
RRDDIM *rd;
rrddim_foreach_read(rd, st)
{
rrddim_foreach_read(rd, st) {
rrddim_flag_clear(rd, RRDDIM_FLAG_ACLK);
rd->aclk_live_status = (rd->aclk_live_status == 0);
}
rrdset_unlock(st);
rrddim_foreach_done(rd);
}
rrdhost_unlock(host);
} else
rrdset_foreach_done(st);
}
else
error_report("ACLK synchronization thread for %s is not linked to HOST", wc->host_guid);
} else {
}
else {
log_access(
"ACLK STA [%s (%s)]: RESTARTING CHART SYNC FROM SEQUENCE %" PRIu64,
wc->node_id,
@ -1268,29 +1267,26 @@ void sql_check_chart_liveness(RRDSET *st) {
if (unlikely(rrdset_is_ar_chart(st)))
return;
rrdset_rdlock(st);
if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK))) {
rrdset_unlock(st);
if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK)))
return;
}
if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK))) {
if (likely(st->dimensions && st->counter_done && !queue_chart_to_aclk(st))) {
if (likely(rrdset_number_of_dimensions(st) && st->counter_done && !queue_chart_to_aclk(st))) {
debug(D_ACLK_SYNC,"Check chart liveness [%s] submit chart definition", rrdset_name(st));
rrdset_flag_set(st, RRDSET_FLAG_ACLK);
}
}
else
debug(D_ACLK_SYNC,"Check chart liveness [%s] chart definition already submitted", rrdset_name(st));
time_t mark = now_realtime_sec();
debug(D_ACLK_SYNC,"Check chart liveness [%s] scanning dimensions", rrdset_name(st));
rrddim_foreach_read(rd, st) {
if (!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN))
if (!rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN))
queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, mark));
}
rrdset_unlock(st);
rrddim_foreach_done(rd);
}
// ST is read locked

5
database/sqlite/sqlite_aclk_node.c
View file

@ -12,7 +12,6 @@ DICTIONARY *collectors_from_charts(RRDHOST *host, DICTIONARY *dict) {
RRDSET *st;
char name[500];
rrdhost_rdlock(host);
rrdset_foreach_read(st, host) {
if (rrdset_is_available_for_viewers(st)) {
struct collector_info col = {
@ -23,7 +22,7 @@ DICTIONARY *collectors_from_charts(RRDHOST *host, DICTIONARY *dict) {
dictionary_set(dict, name, &col, sizeof(struct collector_info));
}
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
return dict;
}
@ -36,7 +35,7 @@ void sql_build_node_collectors(struct aclk_database_worker_config *wc)
return;
struct update_node_collectors upd_node_collectors;
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
upd_node_collectors.node_id = wc->node_id;
upd_node_collectors.claim_id = get_agent_claimid();

18
database/sqlite/sqlite_db_migration.c
View file

@ -64,6 +64,15 @@ const char *database_migrate_v2_v3[] = {
NULL
};
const char *database_migrate_v4_v5[] = {
"DROP TABLE IF EXISTS chart_active;",
"DROP TABLE IF EXISTS dimension_active;",
"DROP TABLE IF EXISTS chart_hash;",
"DROP TABLE IF EXISTS chart_hash_map;",
"DROP VIEW IF EXISTS v_chart_hash;",
NULL
};
static int do_migration_v1_v2(sqlite3 *database, const char *name)
{
UNUSED(name);
@ -116,6 +125,14 @@ static int do_migration_v3_v4(sqlite3 *database, const char *name)
return 0;
}
static int do_migration_v4_v5(sqlite3 *database, const char *name)
{
UNUSED(name);
info("Running \"%s\" database migration", name);
return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v4_v5[0]);
}
static int do_migration_noop(sqlite3 *database, const char *name)
{
UNUSED(database);
@ -163,6 +180,7 @@ DATABASE_FUNC_MIGRATION_LIST migration_action[] = {
{.name = "v1 to v2", .func = do_migration_v1_v2},
{.name = "v2 to v3", .func = do_migration_v2_v3},
{.name = "v3 to v4", .func = do_migration_v3_v4},
{.name = "v4 to v5", .func = do_migration_v4_v5},
// the terminator of this array
{.name = NULL, .func = NULL}
};

381
database/sqlite/sqlite_functions.c
View file

@ -3,7 +3,7 @@
#include "sqlite_functions.h"
#include "sqlite_db_migration.h"
#define DB_METADATA_VERSION 4
#define DB_METADATA_VERSION 5
const char *database_config[] = {
"CREATE TABLE IF NOT EXISTS host(host_id BLOB PRIMARY KEY, hostname TEXT NOT NULL, "
@ -21,11 +21,6 @@ const char *database_config[] = {
"CREATE TABLE IF NOT EXISTS dimension(dim_id blob PRIMARY KEY, chart_id blob, id text, name text, "
"multiplier int, divisor int , algorithm int, options text);",
"DROP TABLE IF EXISTS chart_active;",
"DROP TABLE IF EXISTS dimension_active;",
"CREATE TABLE IF NOT EXISTS chart_active(chart_id blob PRIMARY KEY, date_created int);",
"CREATE TABLE IF NOT EXISTS dimension_active(dim_id blob primary key, date_created int);",
"CREATE TABLE IF NOT EXISTS metadata_migration(filename text, file_size, date_created int);",
"CREATE INDEX IF NOT EXISTS ind_d1 on dimension (chart_id, id, name);",
"CREATE INDEX IF NOT EXISTS ind_c1 on chart (host_id, id, type, name);",
@ -46,36 +41,16 @@ const char *database_config[] = {
"CREATE TABLE IF NOT EXISTS host_label(host_id blob, source_type int, label_key text NOT NULL, "
"label_value text NOT NULL, date_created INT, PRIMARY KEY (host_id, label_key));",
"CREATE TABLE IF NOT EXISTS chart_hash_map(chart_id blob , hash_id blob, UNIQUE (chart_id, hash_id));",
"CREATE TABLE IF NOT EXISTS chart_hash(hash_id blob PRIMARY KEY,type text, id text, name text, "
"family text, context text, title text, unit text, plugin text, "
"module text, priority integer, chart_type, last_used);",
"CREATE VIEW IF NOT EXISTS v_chart_hash as SELECT ch.*, chm.chart_id FROM chart_hash ch, chart_hash_map chm "
"WHERE ch.hash_id = chm.hash_id;",
"CREATE TRIGGER IF NOT EXISTS ins_host AFTER INSERT ON host BEGIN INSERT INTO node_instance (host_id, date_created)"
" SELECT new.host_id, unixepoch() WHERE new.host_id NOT IN (SELECT host_id FROM node_instance); END;",
"CREATE TRIGGER IF NOT EXISTS tr_v_chart_hash INSTEAD OF INSERT on v_chart_hash BEGIN "
"INSERT INTO chart_hash (hash_id, type, id, name, family, context, title, unit, plugin, "
"module, priority, chart_type, last_used) "
"values (new.hash_id, new.type, new.id, new.name, new.family, new.context, new.title, new.unit, new.plugin, "
"new.module, new.priority, new.chart_type, unixepoch()) "
"ON CONFLICT (hash_id) DO UPDATE SET last_used = unixepoch(); "
"INSERT INTO chart_hash_map (chart_id, hash_id) values (new.chart_id, new.hash_id) "
"on conflict (chart_id, hash_id) do nothing; END; ",
NULL
};
const char *database_cleanup[] = {
"delete from chart where chart_id not in (select chart_id from dimension);",
"delete from host where host_id not in (select host_id from chart);",
"delete from chart_label where chart_id not in (select chart_id from chart);",
"DELETE FROM chart_hash_map WHERE chart_id NOT IN (SELECT chart_id FROM chart);",
"DELETE FROM chart_hash WHERE hash_id NOT IN (SELECT hash_id FROM chart_hash_map);",
"DELETE FROM chart WHERE chart_id NOT IN (SELECT chart_id FROM dimension);",
"DELETE FROM host WHERE host_id NOT IN (SELECT host_id FROM chart);",
"DELETE FROM chart_label WHERE chart_id NOT IN (SELECT chart_id FROM chart);",
"DELETE FROM node_instance WHERE host_id NOT IN (SELECT host_id FROM host);",
"DELETE FROM host_info WHERE host_id NOT IN (SELECT host_id FROM host);",
"DELETE FROM host_label WHERE host_id NOT IN (SELECT host_id FROM host);",
@ -190,88 +165,6 @@ int prepare_statement(sqlite3 *database, char *query, sqlite3_stmt **statement)
return rc;
}
/*
* Store a chart or dimension UUID in chart_active or dimension_active
* The statement that will be prepared determines that
*/
static int store_active_uuid_object(sqlite3_stmt **res, char *statement, uuid_t *uuid)
{
int rc;
// Check if we should need to prepare the statement
if (!*res) {
rc = prepare_statement(db_meta, statement, res);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to store active object, rc = %d", rc);
return rc;
}
}
rc = sqlite3_bind_blob(*res, 1, uuid, sizeof(*uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to bind input parameter to store active object, rc = %d", rc);
else
rc = execute_insert(*res);
return rc;
}
/*
* Marks a chart with UUID as active
* Input: UUID
*/
void store_active_chart(uuid_t *chart_uuid)
{
static __thread sqlite3_stmt *res = NULL;
int rc;
if (unlikely(!db_meta)) {
if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
error_report("Database has not been initialized");
return;
}
if (unlikely(!chart_uuid))
return;
rc = store_active_uuid_object(&res, SQL_STORE_ACTIVE_CHART, chart_uuid);
if (rc != SQLITE_DONE)
error_report("Failed to store active chart, rc = %d", rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to finalize statement in store active chart, rc = %d", rc);
return;
}
/*
* Marks a dimension with UUID as active
* Input: UUID
*/
void store_active_dimension(uuid_t *dimension_uuid)
{
static __thread sqlite3_stmt *res = NULL;
int rc;
if (unlikely(!db_meta)) {
if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
error_report("Database has not been initialized");
return;
}
if (unlikely(!dimension_uuid))
return;
rc = store_active_uuid_object(&res, SQL_STORE_ACTIVE_DIMENSION, dimension_uuid);
if (rc != SQLITE_DONE)
error_report("Failed to store active dimension, rc = %d", rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to finalize statement in store active dimension, rc = %d", rc);
return;
}
static int check_table_integrity_cb(void *data, int argc, char **argv, char **column)
{
int *status = data;
@ -623,7 +516,7 @@ int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid)
}
}
rc = sqlite3_bind_blob(res, 1, st->chart_uuid, sizeof(*st->chart_uuid), SQLITE_STATIC);
rc = sqlite3_bind_blob(res, 1, &st->chart_uuid, sizeof(st->chart_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
@ -642,7 +535,7 @@ int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid)
}
else {
uuid_generate(*store_uuid);
status = sql_store_dimension(store_uuid, st->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
status = sql_store_dimension(store_uuid, &st->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
if (unlikely(status))
error_report("Failed to store dimension metadata in the database");
}
@ -697,20 +590,20 @@ bind_fail:
* Do a database lookup to find the UUID of a chart
*
*/
uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name)
int find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name, uuid_t *store_uuid)
{
static __thread sqlite3_stmt *res = NULL;
uuid_t *uuid = NULL;
int rc;
int status = 1;
if (unlikely(!db_meta) && default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
return NULL;
return 1;
if (unlikely(!res)) {
rc = prepare_statement(db_meta, SQL_FIND_CHART_UUID, &res);
if (rc != SQLITE_OK) {
error_report("Failed to prepare statement to lookup chart UUID in the database");
return NULL;
return 1;
}
}
@ -731,32 +624,23 @@ uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const c
goto bind_fail;
rc = sqlite3_step_monitored(res);
if (likely(rc == SQLITE_ROW)) {
uuid = mallocz(sizeof(uuid_t));
uuid_copy(*uuid, sqlite3_column_blob(res, 0));
if (likely(rc == SQLITE_ROW && sqlite3_column_bytes(res,0) == sizeof(*store_uuid))) {
uuid_copy(*store_uuid, *((uuid_t *) sqlite3_column_blob(res, 0)));
status = 0;
}
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement when searching for a chart UUID, rc = %d", rc);
#ifdef NETDATA_INTERNAL_CHECKS
char uuid_str[GUID_LEN + 1];
if (likely(uuid)) {
uuid_unparse_lower(*uuid, uuid_str);
debug(D_METADATALOG, "Found UUID %s for chart %s.%s", uuid_str, type, name ? name : id);
}
else
debug(D_METADATALOG, "UUID not found for chart %s.%s", type, name ? name : id);
#endif
return uuid;
return status;
bind_fail:
error_report("Failed to bind input parameter to perform chart UUID database lookup, rc = %d", rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement when searching for a chart UUID, rc = %d", rc);
return NULL;
return 1;
}
int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const char *name)
@ -767,7 +651,8 @@ int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const
return 0;
rc = sql_store_chart(
chart_uuid, &st->rrdhost->host_uuid, rrdset_type(st), id, name,
chart_uuid, &st->rrdhost->host_uuid,
rrdset_parts_type(st), id, name,
rrdset_family(st), rrdset_context(st), rrdset_title(st), rrdset_units(st),
rrdset_plugin_name(st), rrdset_module_name(st),
st->priority, st->update_every, st->chart_type,
@ -776,28 +661,6 @@ int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const
return rc;
}
uuid_t *create_chart_uuid(RRDSET *st, const char *id, const char *name)
{
uuid_t *uuid = NULL;
int rc;
uuid = mallocz(sizeof(uuid_t));
uuid_generate(*uuid);
#ifdef NETDATA_INTERNAL_CHECKS
char uuid_str[GUID_LEN + 1];
uuid_unparse_lower(*uuid, uuid_str);
debug(D_METADATALOG,"Generating uuid [%s] for chart %s under host %s", uuid_str, rrdset_id(st), rrdhost_hostname(st->rrdhost));
#endif
rc = update_chart_metadata(uuid, st, id, name);
if (unlikely(rc))
error_report("Failed to store chart metadata in the database");
return uuid;
}
static int exec_statement_with_uuid(const char *sql, uuid_t *uuid)
{
int rc, result = 1;
@ -1695,34 +1558,6 @@ skip_store:
return rc != SQLITE_DONE;
}
#define SQL_INS_CHART_LABEL "insert or replace into chart_label " \
"(chart_id, source_type, label_key, label_value, date_created) " \
"values (@chart, @source, @label, @value, unixepoch());"
void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
{
static __thread sqlite3_stmt *res = NULL;
int rc;
if (unlikely(!db_meta)) {
if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
error_report("Database has not been initialized");
return;
}
if (unlikely(!res)) {
rc = prepare_statement(db_meta, SQL_INS_CHART_LABEL, &res);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement store chart labels");
return;
}
}
sql_store_label(res, chart_uuid, source_type, label, value);
return;
}
#define SQL_INS_HOST_LABEL "INSERT OR REPLACE INTO host_label " \
"(host_id, source_type, label_key, label_value, date_created) " \
"values (@chart, @source, @label, @value, unixepoch());"
@ -1934,7 +1769,7 @@ void sql_build_context_param_list(ONEWAYALLOC *owa, struct context_param **para
if (unlikely(!rd))
continue;
if (sqlite3_column_int(res, 9) == 1)
rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
rd->next = (*param_list)->rd;
(*param_list)->rd = rd;
}
@ -1964,186 +1799,6 @@ failed:
}
/*
* Store a chart hash in the database
*/
#define SQL_STORE_CHART_HASH "insert into v_chart_hash (hash_id, type, id, " \
"name, family, context, title, unit, plugin, module, priority, chart_type, last_used, chart_id) " \
"values (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11, ?12, unixepoch(), ?13);"
int sql_store_chart_hash(
uuid_t *hash_id, uuid_t *chart_id, const char *type, const char *id, const char *name, const char *family,
const char *context, const char *title, const char *units, const char *plugin, const char *module, long priority,
RRDSET_TYPE chart_type)
{
static __thread sqlite3_stmt *res = NULL;
int rc, param = 0;
if (unlikely(!db_meta)) {
if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
return 0;
error_report("Database has not been initialized");
return 1;
}
if (unlikely(!res)) {
rc = prepare_statement(db_meta, SQL_STORE_CHART_HASH, &res);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to store chart, rc = %d", rc);
return 1;
}
}
param++;
rc = sqlite3_bind_blob(res, 1, hash_id, sizeof(*hash_id), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 2, type, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 3, id, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
if (name && *name)
rc = sqlite3_bind_text(res, 4, name, -1, SQLITE_STATIC);
else
rc = sqlite3_bind_null(res, 4);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 5, family, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 6, context, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 7, title, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 8, units, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 9, plugin, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_text(res, 10, module, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_int(res, 11, (int) priority);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_int(res, 12, chart_type);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
param++;
rc = sqlite3_bind_blob(res, 13, chart_id, sizeof(*chart_id), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = execute_insert(res);
if (unlikely(rc != SQLITE_DONE))
error_report("Failed to store chart hash_id, rc = %d", rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement in chart hash_id store function, rc = %d", rc);
return 0;
bind_fail:
error_report("Failed to bind parameter %d to store chart hash_id, rc = %d", param, rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement in chart hash_id store function, rc = %d", rc);
return 1;
}
/*
chart hashes are used for cloud communication.
if cloud is disabled or openssl is not available (which will prevent cloud connectivity)
skip hash calculations
*/
void compute_chart_hash(RRDSET *st)
{
#if !defined DISABLE_CLOUD && defined ENABLE_HTTPS
EVP_MD_CTX *evpctx;
unsigned char hash_value[EVP_MAX_MD_SIZE];
unsigned int hash_len;
char priority_str[32];
if (rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
internal_error(true, "Skipping compute_chart_hash for host %s because context streaming is enabled", rrdhost_hostname(st->rrdhost));
return;
}
sprintf(priority_str, "%ld", st->priority);
evpctx = EVP_MD_CTX_create();
EVP_DigestInit_ex(evpctx, EVP_sha256(), NULL);
//EVP_DigestUpdate(evpctx, st->type, strlen(st->type));
EVP_DigestUpdate(evpctx, rrdset_id(st), string_strlen(st->id));
EVP_DigestUpdate(evpctx, rrdset_name(st), string_strlen(st->name));
EVP_DigestUpdate(evpctx, rrdset_family(st), string_strlen(st->family));
EVP_DigestUpdate(evpctx, rrdset_context(st), string_strlen(st->context));
EVP_DigestUpdate(evpctx, rrdset_title(st), string_strlen(st->title));
EVP_DigestUpdate(evpctx, rrdset_units(st), string_strlen(st->units));
EVP_DigestUpdate(evpctx, rrdset_plugin_name(st), string_strlen(st->plugin_name));
EVP_DigestUpdate(evpctx, rrdset_module_name(st), string_strlen(st->module_name));
// EVP_DigestUpdate(evpctx, priority_str, strlen(priority_str));
EVP_DigestUpdate(evpctx, &st->priority, sizeof(st->priority));
EVP_DigestUpdate(evpctx, &st->chart_type, sizeof(st->chart_type));
EVP_DigestFinal_ex(evpctx, hash_value, &hash_len);
EVP_MD_CTX_destroy(evpctx);
fatal_assert(hash_len > sizeof(uuid_t));
char uuid_str[GUID_LEN + 1];
uuid_unparse_lower(*((uuid_t *) &hash_value), uuid_str);
//info("Calculating HASH %s for chart %s", uuid_str, st->name);
uuid_copy(st->uuid, *((uuid_t *) &hash_value));
(void)sql_store_chart_hash(
(uuid_t *)&hash_value,
st->chart_uuid,
rrdset_type(st),
rrdset_id(st),
rrdset_name(st),
rrdset_family(st),
rrdset_context(st),
rrdset_title(st),
rrdset_units(st),
rrdset_plugin_name(st),
rrdset_module_name(st),
st->priority,
st->chart_type);
#else
UNUSED(st);
#endif
return;
}
#define SQL_STORE_CLAIM_ID "insert into node_instance " \
"(host_id, claim_id, date_created) values (@host_id, @claim_id, unixepoch()) " \
"on conflict(host_id) do update set claim_id = excluded.claim_id;"

13
database/sqlite/sqlite_functions.h
View file

@ -37,18 +37,12 @@ typedef enum db_check_action_type {
#define SQL_FIND_CHART_UUID \
"select chart_id from chart where host_id = @host and type=@type and id=@id and (name is null or name=@name) and chart_id is not null;"
#define SQL_STORE_ACTIVE_CHART \
"insert or replace into chart_active (chart_id, date_created) values (@id, unixepoch());"
#define SQL_STORE_DIMENSION \
"INSERT OR REPLACE into dimension (dim_id, chart_id, id, name, multiplier, divisor , algorithm) values (?0001,?0002,?0003,?0004,?0005,?0006,?0007);"
#define SQL_FIND_DIMENSION_UUID \
"select dim_id from dimension where chart_id=@chart and id=@id and name=@name and length(dim_id)=16;"
#define SQL_STORE_ACTIVE_DIMENSION \
"insert or replace into dimension_active (dim_id, date_created) values (@id, unixepoch());"
#define CHECK_SQLITE_CONNECTION(db_meta) \
if (unlikely(!db_meta)) { \
if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) { \
@ -83,12 +77,9 @@ extern int sql_store_dimension(uuid_t *dim_uuid, uuid_t *chart_uuid, const char
collected_number divisor, int algorithm);
extern int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid);
extern void store_active_dimension(uuid_t *dimension_uuid);
extern uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name);
extern uuid_t *create_chart_uuid(RRDSET *st, const char *id, const char *name);
extern int find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name, uuid_t *store_uuid);
extern int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const char *name);
extern void store_active_chart(uuid_t *dimension_uuid);
extern int find_uuid_type(uuid_t *uuid);
@ -103,7 +94,6 @@ extern void add_migrated_file(char *path, uint64_t file_size);
extern void db_unlock(void);
extern void db_lock(void);
extern void delete_dimension_uuid(uuid_t *dimension_uuid);
extern void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value);
extern void sql_build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_list, RRDHOST *host, char *context, char *chart);
extern void store_claim_id(uuid_t *host_id, uuid_t *claim_id);
extern int update_node_id(uuid_t *host_id, uuid_t *node_id);
@ -112,7 +102,6 @@ extern int get_host_id(uuid_t *node_id, uuid_t *host_id);
extern void invalidate_node_instances(uuid_t *host_id, uuid_t *claim_id);
extern struct node_instance_list *get_node_list(void);
extern void sql_load_node_id(RRDHOST *host);
extern void compute_chart_hash(RRDSET *st);
extern int sql_set_dimension_option(uuid_t *dim_uuid, char *option);
char *get_hostname_by_node_id(char *node_id);
void free_temporary_host(RRDHOST *host);

15
database/sqlite/sqlite_health.c
View file

@ -632,12 +632,15 @@ void sql_health_alarm_log_load(RRDHOST *host) {
return;
}
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
DICTIONARY *all_rrdcalcs = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
DICTIONARY *all_rrdcalcs = dictionary_create(
DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(host, rc)
foreach_rrdcalc_in_rrdhost_read(host, rc) {
dictionary_set(all_rrdcalcs, rrdcalc_name(rc), rc, sizeof(*rc));
}
foreach_rrdcalc_in_rrdhost_done(rc);
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
while (sqlite3_step_monitored(res) == SQLITE_ROW) {
ALARM_ENTRY *ae = NULL;
@ -790,11 +793,11 @@ void sql_health_alarm_log_load(RRDHOST *host) {
loaded++;
}
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
dictionary_destroy(all_rrdcalcs);
all_rrdcalcs = NULL;
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
if(!host->health_max_unique_id) host->health_max_unique_id = (uint32_t)now_realtime_sec();
if(!host->health_max_alarm_id) host->health_max_alarm_id = (uint32_t)now_realtime_sec();

4
exporting/json/json.c
View file

@ -197,7 +197,7 @@ int format_dimension_collected_json_plaintext(struct instance *instance, RRDDIM
rrdset_name(st),
rrdset_family(st),
rrdset_context(st),
rrdset_type(st),
rrdset_parts_type(st),
rrdset_units(st),
rrddim_id(rd),
rrddim_name(rd),
@ -281,7 +281,7 @@ int format_dimension_stored_json_plaintext(struct instance *instance, RRDDIM *rd
rrdset_name(st),
rrdset_family(st),
rrdset_context(st),
rrdset_type(st),
rrdset_parts_type(st),
rrdset_units(st),
rrddim_id(rd),
rrddim_name(rd),

12
exporting/process_data.c
View file

@ -338,26 +338,22 @@ void prepare_buffers(struct engine *engine)
rrd_rdlock();
RRDHOST *host;
rrdhost_foreach_read(host)
{
rrdhost_rdlock(host);
rrdhost_foreach_read(host) {
start_host_formatting(engine, host);
RRDSET *st;
rrdset_foreach_read(st, host)
{
rrdset_rdlock(st);
rrdset_foreach_read(st, host) {
start_chart_formatting(engine, st);
RRDDIM *rd;
rrddim_foreach_read(rd, st)
metric_formatting(engine, rd);
rrddim_foreach_done(rd);
end_chart_formatting(engine, st);
rrdset_unlock(st);
}
rrdset_foreach_done(st);
variables_formatting(engine, host);
end_host_formatting(engine, host);
rrdhost_unlock(host);
}
rrd_unlock();
netdata_thread_enable_cancelability();

28
exporting/prometheus/prometheus.c
View file

@ -349,12 +349,12 @@ struct host_variables_callback_options {
* @param data callback options.
* @return Returns 1 if the chart can be sent, 0 otherwise.
*/
static int print_host_variables(const char *name __maybe_unused, void *rv_ptr, void *data) {
RRDVAR *rv = rv_ptr;
static int print_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
struct host_variables_callback_options *opts = data;
if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
if (!opts->host_header_printed) {
opts->host_header_printed = 1;
@ -519,7 +519,6 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
PROMETHEUS_OUTPUT_OPTIONS output_options)
{
SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
rrdhost_rdlock(host);
char hostname[PROMETHEUS_ELEMENT_MAX + 1];
prometheus_label_copy(hostname, rrdhost_hostname(host), PROMETHEUS_ELEMENT_MAX);
@ -564,17 +563,14 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
.host_header_printed = 0
};
rrdvar_walkthrough_read(host->rrdvar_root_index, print_host_variables, &opts);
rrdvar_walkthrough_read(host->rrdvars, print_host_variables_callback, &opts);
}
// for each chart
RRDSET *st;
rrdset_foreach_read(st, host)
{
rrdset_foreach_read(st, host) {
if (likely(can_send_rrdset(instance, st, filter))) {
rrdset_rdlock(st);
char chart[PROMETHEUS_ELEMENT_MAX + 1];
char context[PROMETHEUS_ELEMENT_MAX + 1];
char family[PROMETHEUS_ELEMENT_MAX + 1];
@ -615,8 +611,7 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
// for each dimension
RRDDIM *rd;
rrddim_foreach_read(rd, st)
{
rrddim_foreach_read(rd, st) {
if (rd->collections_counter && !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) {
char dimension[PROMETHEUS_ELEMENT_MAX + 1];
char *suffix = "";
@ -665,7 +660,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
buffer_sprintf(wb, "# TYPE %s_%s%s %s\n", prefix, context, suffix, p.type);
generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
} else {
}
else {
// the dimensions of the chart, do not have the same algorithm, multiplier or divisor
// we create a metric per dimension
@ -683,7 +679,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
}
} else {
}
else {
// we need average or sum of the data
time_t first_time = instance->after;
@ -750,12 +747,11 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
}
}
}
rrdset_unlock(st);
rrddim_foreach_done(rd);
}
}
rrdset_foreach_done(st);
rrdhost_unlock(host);
simple_pattern_free(filter);
}

8
exporting/prometheus/remote_write/remote_write.c
View file

@ -321,12 +321,12 @@ int format_dimension_prometheus_remote_write(struct instance *instance, RRDDIM *
return 0;
}
static int format_variable_prometheus_remote_write_callback(const char *name_txt __maybe_unused, void *rv_ptr, void *data) {
RRDVAR *rv = rv_ptr;
static int format_variable_prometheus_remote_write_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
struct prometheus_remote_write_variables_callback_options *opts = data;
if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
RRDHOST *host = opts->host;
struct instance *instance = opts->instance;
struct simple_connector_data *simple_connector_data =
@ -363,7 +363,7 @@ int format_variables_prometheus_remote_write(struct instance *instance, RRDHOST
.now = now_realtime_usec(),
};
return rrdvar_walkthrough_read(host->rrdvar_root_index, format_variable_prometheus_remote_write_callback, &opt);
return rrdvar_walkthrough_read(host->rrdvars, format_variable_prometheus_remote_write_callback, &opt);
}
/**

8
exporting/tests/netdata_doubles.c
View file

@ -242,14 +242,6 @@ void __mock_rrddim_query_finalize(struct rrddim_query_handle *handle)
function_called();
}
void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
{
(void)chart_uuid;
(void)source_type;
(void)label;
(void)value;
}
void rrdcalc_update_rrdlabels(RRDSET *st)
{
(void)st;

184
health/health.c
View file

@ -153,63 +153,42 @@ static void health_reload_host(RRDHOST *host) {
char *stock_path = health_stock_config_dir();
// free all running alarms
rrdhost_wrlock(host);
while(host->alarms_templates)
rrdcalctemplate_unlink_and_free(host, host->alarms_templates);
while(host->host_alarms)
rrdcalc_unlink_and_free(host, host->host_alarms);
RRDCALC *rc,*nc;
for(rc = host->alarms_with_foreach; rc ; rc = nc) {
nc = rc->next;
rrdcalc_free(rc);
}
host->alarms_with_foreach = NULL;
rrdhost_unlock(host);
rrdcalc_delete_all(host);
rrdcalctemplate_delete_all(host);
// invalidate all previous entries in the alarm log
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
ALARM_ENTRY *t;
for(t = host->health_log.alarms ; t ; t = t->next) {
if(t->new_status != RRDCALC_STATUS_REMOVED)
t->flags |= HEALTH_ENTRY_FLAG_UPDATED;
}
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
rrdhost_rdlock(host);
// reset all thresholds to all charts
RRDSET *st;
rrdset_foreach_read(st, host) {
st->green = NAN;
st->red = NAN;
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
// load the new alarms
rrdhost_wrlock(host);
health_readdir(host, user_path, stock_path, NULL);
//Discard alarms with labels that do not apply to host
rrdcalc_labels_unlink_alarm_from_host(host);
rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(host);
// link the loaded alarms to their charts
RRDDIM *rd;
rrdset_foreach_write(st, host) {
if (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED))
continue;
rrdsetcalc_link_matching(st);
rrdcalctemplate_link_matching(st);
//This loop must be the last, because ` rrdcalctemplate_link_matching` will create alarms related to it.
rrdset_rdlock(st);
rrddim_foreach_read(rd, st) {
rrdcalc_link_to_rrddim(rd, st, host);
}
rrdset_unlock(st);
rrdcalc_link_matching_alerts_to_rrdset(st);
rrdcalctemplate_link_matching_templates_to_rrdset(st);
}
rrdset_foreach_done(st);
rrdhost_unlock(host);
}
/**
@ -323,7 +302,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
warn_alarms = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE);
crit_alarms = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE);
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
continue;
@ -351,6 +330,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
expr = rc->warning;
}
}
foreach_rrdcalc_in_rrdhost_done(rc);
if (n_warn+n_crit>1)
qsort (active_alerts, n_warn+n_crit, sizeof(active_alerts_t), compare_active_alerts);
@ -477,13 +457,13 @@ static inline void health_alarm_log_process(RRDHOST *host) {
}
}
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
// remember this for the next iteration
host->health_last_processed_id = first_waiting;
bool cleanup_excess_log_entries = host->health_log.count > host->health_log.max;
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
if (!cleanup_excess_log_entries)
return;
@ -554,10 +534,8 @@ static inline int rrdcalc_isrunnable(RRDCALC *rc, time_t now, time_t *next_run)
}
int update_every = rc->rrdset->update_every;
rrdset_rdlock(rc->rrdset);
time_t first = rrdset_first_entry_t_nolock(rc->rrdset);
time_t last = rrdset_last_entry_t_nolock(rc->rrdset);
rrdset_unlock(rc->rrdset);
time_t first = rrdset_first_entry_t(rc->rrdset);
time_t last = rrdset_last_entry_t(rc->rrdset);
if(unlikely(now + update_every < first /* || now - update_every > last */)) {
debug(D_HEALTH
@ -653,29 +631,29 @@ static SILENCE_TYPE check_silenced(RRDCALC *rc, const char *host, SILENCERS *sil
* @return It returns 1 case rrdcalc_flags is DISABLED or 0 otherwise
*/
static int update_disabled_silenced(RRDHOST *host, RRDCALC *rc) {
uint32_t rrdcalc_flags_old = rc->rrdcalc_flags;
uint32_t rrdcalc_flags_old = rc->run_flags;
// Clear the flags
rc->rrdcalc_flags &= ~(RRDCALC_FLAG_DISABLED | RRDCALC_FLAG_SILENCED);
rc->run_flags &= ~(RRDCALC_FLAG_DISABLED | RRDCALC_FLAG_SILENCED);
if (unlikely(silencers->all_alarms)) {
if (silencers->stype == STYPE_DISABLE_ALARMS) rc->rrdcalc_flags |= RRDCALC_FLAG_DISABLED;
else if (silencers->stype == STYPE_SILENCE_NOTIFICATIONS) rc->rrdcalc_flags |= RRDCALC_FLAG_SILENCED;
if (silencers->stype == STYPE_DISABLE_ALARMS) rc->run_flags |= RRDCALC_FLAG_DISABLED;
else if (silencers->stype == STYPE_SILENCE_NOTIFICATIONS) rc->run_flags |= RRDCALC_FLAG_SILENCED;
} else {
SILENCE_TYPE st = check_silenced(rc, rrdhost_hostname(host), silencers);
if (st == STYPE_DISABLE_ALARMS) rc->rrdcalc_flags |= RRDCALC_FLAG_DISABLED;
else if (st == STYPE_SILENCE_NOTIFICATIONS) rc->rrdcalc_flags |= RRDCALC_FLAG_SILENCED;
if (st == STYPE_DISABLE_ALARMS) rc->run_flags |= RRDCALC_FLAG_DISABLED;
else if (st == STYPE_SILENCE_NOTIFICATIONS) rc->run_flags |= RRDCALC_FLAG_SILENCED;
}
if (rrdcalc_flags_old != rc->rrdcalc_flags) {
if (rrdcalc_flags_old != rc->run_flags) {
info("Alarm silencing changed for host '%s' alarm '%s': Disabled %s->%s Silenced %s->%s",
rrdhost_hostname(host),
rrdcalc_name(rc),
(rrdcalc_flags_old & RRDCALC_FLAG_DISABLED)?"true":"false",
(rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)?"true":"false",
(rc->run_flags & RRDCALC_FLAG_DISABLED)?"true":"false",
(rrdcalc_flags_old & RRDCALC_FLAG_SILENCED)?"true":"false",
(rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
(rc->run_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
);
}
if (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)
if (rc->run_flags & RRDCALC_FLAG_DISABLED)
return 1;
else
return 0;
@ -683,36 +661,38 @@ static int update_disabled_silenced(RRDHOST *host, RRDCALC *rc) {
// Create alarms for dimensions that have been added to charts
// since the previous iteration.
static void init_pending_foreach_alarms(RRDHOST *host) {
static void health_execute_pending_updates(RRDHOST *host) {
RRDSET *st;
RRDDIM *rd;
if (!rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS))
return;
rrdhost_wrlock(host);
rrdset_foreach_write(st, host) {
if (!rrdset_flag_check(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS))
rrdset_foreach_reentrant(st, host) {
if(!rrdset_flag_check(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS))
continue;
rrdset_rdlock(st);
RRDDIM *rd;
rrddim_foreach_read(rd, st) {
if (!rrddim_flag_check(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM))
if(!rrddim_flag_check(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS))
continue;
rrdcalc_link_to_rrddim(rd, st, host);
RRDCALCTEMPLATE *rt;
foreach_rrdcalctemplate_read(host, rt) {
if(!rt->foreach_dimension_pattern)
continue;
rrddim_flag_clear(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
if(rrdcalctemplate_check_rrdset_conditions(rt, st, host))
rrdcalctemplate_check_rrddim_conditions_and_link(rt, st, rd, host);
}
foreach_rrdcalctemplate_done(rt);
rrddim_flag_clear(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS);
}
rrddim_foreach_done(rd);
rrdset_flag_clear(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
rrdset_unlock(st);
}
rrdset_foreach_done(st);
rrdhost_flag_clear(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
rrdhost_unlock(host);
}
/**
@ -759,7 +739,7 @@ void *health_main(void *ptr) {
time_t now = now_realtime_sec();
time_t hibernation_delay = config_get_number(CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for seconds", 60);
rrdcalc_labels_unlink();
rrdcalc_delete_alerts_not_matching_host_labels_from_all_hosts();
unsigned int loop = 0;
#ifdef ENABLE_ACLK
@ -829,13 +809,14 @@ void *health_main(void *ptr) {
if(likely(!host->health_log_fp) && (loop == 1 || loop % cleanup_sql_every_loop == 0))
sql_health_alarm_log_cleanup(host);
init_pending_foreach_alarms(host);
health_execute_pending_updates(host);
worker_is_busy(WORKER_HEALTH_JOB_HOST_LOCK);
rrdhost_rdlock(host);
// the first loop is to lookup values from the db
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
rrdcalc_update_info_using_rrdset_labels(rc);
if (update_disabled_silenced(host, rc))
continue;
@ -876,7 +857,7 @@ void *health_main(void *ptr) {
rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0);
if (ae) {
health_alarm_log(host, ae);
health_alarm_log_add_entry(host, ae);
rc->old_status = rc->status;
rc->status = RRDCALC_STATUS_REMOVED;
rc->last_status_change = now;
@ -891,14 +872,14 @@ void *health_main(void *ptr) {
}
if (unlikely(!rrdcalc_isrunnable(rc, now, &next_run))) {
if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_RUNNABLE))
rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUNNABLE;
if (unlikely(rc->run_flags & RRDCALC_FLAG_RUNNABLE))
rc->run_flags &= ~RRDCALC_FLAG_RUNNABLE;
continue;
}
runnable++;
rc->old_value = rc->value;
rc->rrdcalc_flags |= RRDCALC_FLAG_RUNNABLE;
rc->run_flags |= RRDCALC_FLAG_RUNNABLE;
// ------------------------------------------------------------
// if there is database lookup, do it
@ -919,13 +900,13 @@ void *health_main(void *ptr) {
if (unlikely(ret != 200)) {
// database lookup failed
rc->value = NAN;
rc->rrdcalc_flags |= RRDCALC_FLAG_DB_ERROR;
rc->run_flags |= RRDCALC_FLAG_DB_ERROR;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': database lookup returned error %d",
rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc), ret
);
} else
rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_ERROR;
rc->run_flags &= ~RRDCALC_FLAG_DB_ERROR;
/* - RRDCALC_FLAG_DB_STALE not currently used
if (unlikely(old_db_timestamp == rc->db_before)) {
@ -945,14 +926,14 @@ void *health_main(void *ptr) {
if (unlikely(value_is_null)) {
// collected value is null
rc->value = NAN;
rc->rrdcalc_flags |= RRDCALC_FLAG_DB_NAN;
rc->run_flags |= RRDCALC_FLAG_DB_NAN;
debug(D_HEALTH,
"Health on host '%s', alarm '%s.%s': database lookup returned empty value (possibly value is not collected yet)",
rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc)
);
} else
rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_NAN;
rc->run_flags &= ~RRDCALC_FLAG_DB_NAN;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': database lookup gave value " NETDATA_DOUBLE_FORMAT,
rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc), rc->value
@ -968,14 +949,14 @@ void *health_main(void *ptr) {
if (unlikely(!expression_evaluate(rc->calculation))) {
// calculation failed
rc->value = NAN;
rc->rrdcalc_flags |= RRDCALC_FLAG_CALC_ERROR;
rc->run_flags |= RRDCALC_FLAG_CALC_ERROR;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': expression '%s' failed: %s",
rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc),
rc->calculation->parsed_as, buffer_tostring(rc->calculation->error_msg)
);
} else {
rc->rrdcalc_flags &= ~RRDCALC_FLAG_CALC_ERROR;
rc->run_flags &= ~RRDCALC_FLAG_CALC_ERROR;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': expression '%s' gave value "
NETDATA_DOUBLE_FORMAT
@ -985,25 +966,17 @@ void *health_main(void *ptr) {
);
rc->value = rc->calculation->result;
if (rc->local) rc->local->last_updated = now;
if (rc->family) rc->family->last_updated = now;
if (rc->hostid) rc->hostid->last_updated = now;
if (rc->hostname) rc->hostname->last_updated = now;
}
}
}
rrdhost_unlock(host);
foreach_rrdcalc_in_rrdhost_done(rc);
if (unlikely(runnable && !netdata_exit)) {
rrdhost_rdlock(host);
foreach_rrdcalc_in_rrdhost(host, rc) {
if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_RUNNABLE)))
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if (unlikely(!(rc->run_flags & RRDCALC_FLAG_RUNNABLE)))
continue;
if (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED) {
if (rc->run_flags & RRDCALC_FLAG_DISABLED) {
continue;
}
RRDCALC_STATUS warning_status = RRDCALC_STATUS_UNDEFINED;
@ -1017,7 +990,7 @@ void *health_main(void *ptr) {
if (unlikely(!expression_evaluate(rc->warning))) {
// calculation failed
rc->rrdcalc_flags |= RRDCALC_FLAG_WARN_ERROR;
rc->run_flags |= RRDCALC_FLAG_WARN_ERROR;
debug(D_HEALTH,
"Health on host '%s', alarm '%s.%s': warning expression failed with error: %s",
@ -1025,7 +998,7 @@ void *health_main(void *ptr) {
buffer_tostring(rc->warning->error_msg)
);
} else {
rc->rrdcalc_flags &= ~RRDCALC_FLAG_WARN_ERROR;
rc->run_flags &= ~RRDCALC_FLAG_WARN_ERROR;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': warning expression gave value "
NETDATA_DOUBLE_FORMAT
": %s (source: %s)", rrdhost_hostname(host), rrdcalc_chart_name(rc),
@ -1043,7 +1016,7 @@ void *health_main(void *ptr) {
if (unlikely(!expression_evaluate(rc->critical))) {
// calculation failed
rc->rrdcalc_flags |= RRDCALC_FLAG_CRIT_ERROR;
rc->run_flags |= RRDCALC_FLAG_CRIT_ERROR;
debug(D_HEALTH,
"Health on host '%s', alarm '%s.%s': critical expression failed with error: %s",
@ -1051,7 +1024,7 @@ void *health_main(void *ptr) {
buffer_tostring(rc->critical->error_msg)
);
} else {
rc->rrdcalc_flags &= ~RRDCALC_FLAG_CRIT_ERROR;
rc->run_flags &= ~RRDCALC_FLAG_CRIT_ERROR;
debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': critical expression gave value "
NETDATA_DOUBLE_FORMAT
": %s (source: %s)", rrdhost_hostname(host), rrdcalc_chart_name(rc),
@ -1156,13 +1129,13 @@ void *health_main(void *ptr) {
rc->info,
rc->delay_last,
(
((rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
((rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
((rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
((rc->run_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
(rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0)
)
);
health_alarm_log(host, ae);
health_alarm_log_add_entry(host, ae);
rc->last_status_change = now;
rc->old_status = rc->status;
@ -1175,20 +1148,20 @@ void *health_main(void *ptr) {
if (next_run > rc->next_update)
next_run = rc->next_update;
}
foreach_rrdcalc_in_rrdhost_done(rc);
// process repeating alarms
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
int repeat_every = 0;
if(unlikely(rrdcalc_isrepeating(rc) && rc->delay_up_to_timestamp <= now)) {
if(unlikely(rc->status == RRDCALC_STATUS_WARNING)) {
rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUN_ONCE;
rc->run_flags &= ~RRDCALC_FLAG_RUN_ONCE;
repeat_every = rc->warn_repeat_every;
} else if(unlikely(rc->status == RRDCALC_STATUS_CRITICAL)) {
rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUN_ONCE;
rc->run_flags &= ~RRDCALC_FLAG_RUN_ONCE;
repeat_every = rc->crit_repeat_every;
} else if(unlikely(rc->status == RRDCALC_STATUS_CLEAR)) {
if(!(rc->rrdcalc_flags & RRDCALC_FLAG_RUN_ONCE)) {
if(!(rc->run_flags & RRDCALC_FLAG_RUN_ONCE)) {
if(rc->old_status == RRDCALC_STATUS_CRITICAL) {
repeat_every = 1;
} else if (rc->old_status == RRDCALC_STATUS_WARNING) {
@ -1230,25 +1203,24 @@ void *health_main(void *ptr) {
rc->info,
rc->delay_last,
(
((rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
((rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
((rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
((rc->run_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
(rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0)
)
);
ae->last_repeat = rc->last_repeat;
if (!(rc->rrdcalc_flags & RRDCALC_FLAG_RUN_ONCE) && rc->status == RRDCALC_STATUS_CLEAR) {
if (!(rc->run_flags & RRDCALC_FLAG_RUN_ONCE) && rc->status == RRDCALC_STATUS_CLEAR) {
ae->flags |= HEALTH_ENTRY_RUN_ONCE;
}
rc->rrdcalc_flags |= RRDCALC_FLAG_RUN_ONCE;
rc->run_flags |= RRDCALC_FLAG_RUN_ONCE;
health_process_notifications(host, ae);
debug(D_HEALTH, "Notification sent for the repeating alarm %u.", ae->alarm_id);
health_alarm_wait_for_execution(ae);
health_alarm_log_free_one_nochecks_nounlink(ae);
}
}
rrdhost_unlock(host);
foreach_rrdcalc_in_rrdhost_done(rc);
}
if (unlikely(netdata_exit))

4
health/health.h
View file

@ -74,7 +74,7 @@ extern ALARM_ENTRY* health_create_alarm_entry(
int delay,
uint32_t flags);
extern void health_alarm_log(RRDHOST *host, ALARM_ENTRY *ae);
extern void health_alarm_log_add_entry(RRDHOST *host, ALARM_ENTRY *ae);
extern void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path, const char *subpath);
extern char *health_user_config_dir(void);
@ -90,6 +90,4 @@ extern void health_label_log_save(RRDHOST *host);
extern char *health_edit_command_from_source(const char *source);
extern void sql_refresh_hashes(void);
extern SIMPLE_PATTERN *health_pattern_from_foreach(const char *s);
#endif //NETDATA_HEALTH_H

188
health/health_config.c
View file

@ -33,122 +33,6 @@
#define HEALTH_HOST_LABEL_KEY "host labels"
#define HEALTH_FOREACH_KEY "foreach"
static inline int rrdcalc_add_alarm_from_config(RRDHOST *host, RRDCALC *rc) {
if(!rc->chart) {
error("Health configuration for alarm '%s' does not have a chart", rrdcalc_name(rc));
return 0;
}
if(!rc->update_every) {
error("Health configuration for alarm '%s.%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalc_chart_name(rc), rrdcalc_name(rc));
return 0;
}
if(!RRDCALC_HAS_DB_LOOKUP(rc) && !rc->calculation && !rc->warning && !rc->critical) {
error("Health configuration for alarm '%s.%s' is useless (no db lookup, no calculation, no warning and no critical expressions)", rrdcalc_chart_name(rc), rrdcalc_name(rc));
return 0;
}
if (rrdcalc_exists(host, rrdcalc_chart_name(rc), rrdcalc_name(rc)))
return 0;
rc->id = rrdcalc_get_unique_id(host, rc->chart, rc->name, &rc->next_event_id);
debug(D_HEALTH, "Health configuration adding alarm '%s.%s' (%u): exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
", red " NETDATA_DOUBLE_FORMAT_AUTO
", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
rrdcalc_chart_name(rc),
rrdcalc_name(rc),
rc->id,
(rc->exec)?rrdcalc_exec(rc):"DEFAULT",
(rc->recipient)?rrdcalc_recipient(rc):"DEFAULT",
rc->green,
rc->red,
(int)rc->group,
rc->after,
rc->before,
rc->options,
(rc->dimensions)?rrdcalc_dimensions(rc):"NONE",
(rc->foreachdim)?rrdcalc_foreachdim(rc):"NONE",
rc->update_every,
(rc->calculation)?rc->calculation->parsed_as:"NONE",
(rc->warning)?rc->warning->parsed_as:"NONE",
(rc->critical)?rc->critical->parsed_as:"NONE",
rrdcalc_source(rc),
rc->delay_up_duration,
rc->delay_down_duration,
rc->delay_max_duration,
rc->delay_multiplier,
rc->warn_repeat_every,
rc->crit_repeat_every
);
rrdcalc_add_to_host(host, rc);
return 1;
}
static inline int rrdcalctemplate_add_template_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) {
if(unlikely(!rt->context)) {
error("Health configuration for template '%s' does not have a context", rrdcalctemplate_name(rt));
return 0;
}
if(unlikely(!rt->update_every)) {
error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalctemplate_name(rt));
return 0;
}
if(unlikely(!RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) && !rt->calculation && !rt->warning && !rt->critical)) {
error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rrdcalctemplate_name(rt));
return 0;
}
RRDCALCTEMPLATE *t;
foreach_rrdcalctemplate_in_rrdhost(host, t) {
if(unlikely(t->name == rt->name && !strcmp(t->family_match?rrdcalctemplate_family_match(t):"*", rt->family_match?rrdcalctemplate_family_match(rt):"*"))) {
info("Health configuration template '%s' already exists for host '%s'.", rrdcalctemplate_name(rt), rrdhost_hostname(host));
return 0;
}
}
if(rt->foreachdim)
DOUBLE_LINKED_LIST_PREPEND_UNSAFE(host->alarms_templates, rt, prev, next);
else
DOUBLE_LINKED_LIST_APPEND_UNSAFE(host->alarms_templates, rt, prev, next);
debug(D_HEALTH, "Health configuration adding template '%s': context '%s', exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
", red " NETDATA_DOUBLE_FORMAT_AUTO
", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
rrdcalctemplate_name(rt),
(rt->context)?string2str(rt->context):"NONE",
(rt->exec)?rrdcalctemplate_exec(rt):"DEFAULT",
(rt->recipient)?rrdcalctemplate_recipient(rt):"DEFAULT",
rt->green,
rt->red,
(int)rt->group,
rt->after,
rt->before,
rt->options,
(rt->dimensions)?rrdcalctemplate_dimensions(rt):"NONE",
(rt->foreachdim)?rrdcalctemplate_foreachdim(rt):"NONE",
rt->update_every,
(rt->calculation)?rt->calculation->parsed_as:"NONE",
(rt->warning)?rt->warning->parsed_as:"NONE",
(rt->critical)?rt->critical->parsed_as:"NONE",
rrdcalctemplate_source(rt),
rt->delay_up_duration,
rt->delay_down_duration,
rt->delay_max_duration,
rt->delay_multiplier,
rt->warn_repeat_every,
rt->crit_repeat_every
);
return 1;
}
static inline int health_parse_delay(
size_t line, const char *filename, char *string,
int *delay_up_duration,
@ -249,7 +133,7 @@ static inline uint32_t health_parse_options(const char *s) {
buf[count] = '\0';
if(!strcasecmp(buf, "no-clear-notification") || !strcasecmp(buf, "no-clear"))
options |= RRDCALC_FLAG_NO_CLEAR_NOTIFICATION;
options |= RRDCALC_OPTION_NO_CLEAR_NOTIFICATION;
else
error("Ignoring unknown alarm option '%s'", buf);
}
@ -308,13 +192,21 @@ static inline int health_parse_repeat(
*
* @param s the string that will be used to create the simple pattern.
*/
SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
static void dimension_remove_pipe_comma(char *str) {
while(*str) {
if(*str == '|' || *str == ',') *str = ' ';
str++;
}
}
static SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
char *convert= strdupz(s);
SIMPLE_PATTERN *val = NULL;
if(convert) {
dimension_remove_pipe_comma(convert);
val = simple_pattern_create(convert, NULL, SIMPLE_PATTERN_EXACT);
freez(convert);
}
@ -324,7 +216,7 @@ SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
static inline int health_parse_db_lookup(
size_t line, const char *filename, char *string,
RRDR_GROUPING *group_method, int *after, int *before, int *every,
uint32_t *options, STRING **dimensions, STRING **foreachdim
RRDCALC_OPTIONS *options, STRING **dimensions, STRING **foreachdim
) {
debug(D_HEALTH, "Health configuration parsing database lookup %zu@%s: %s", line, filename, string);
@ -335,7 +227,7 @@ static inline int health_parse_db_lookup(
*after = 0;
*before = 0;
*every = 0;
*options = 0;
*options = (*options) & RRDCALC_ALL_OPTIONS_EXCLUDING_THE_RRDR_ONES; // preserve rrdcalc options
char *s = string, *key;
@ -644,16 +536,20 @@ static int health_readfile(const char *filename, void *data) {
if(hash == hash_alarm && !strcasecmp(key, HEALTH_ALARM_KEY)) {
if(rc) {
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc)) {
rrdcalc_free(rc);
}
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
else
rrdcalc_add_from_config(host, rc);
// health_add_alarms_loop(host, rc, ignore_this) ;
}
if(rt) {
if (!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt)) {
rrdcalctemplate_free(rt);
}
if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
else
rrdcalctemplate_add_from_config(host, rt);
rt = NULL;
}
@ -688,15 +584,19 @@ static int health_readfile(const char *filename, void *data) {
else if(hash == hash_template && !strcasecmp(key, HEALTH_TEMPLATE_KEY)) {
if(rc) {
// health_add_alarms_loop(host, rc, ignore_this) ;
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc))
rrdcalc_free(rc);
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
else
rrdcalc_add_from_config(host, rc);
rc = NULL;
}
if(rt) {
if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt))
rrdcalctemplate_free(rt);
if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
else
rrdcalctemplate_add_from_config(host, rt);
}
rt = callocz(1, sizeof(RRDCALCTEMPLATE));
@ -811,10 +711,10 @@ static int health_readfile(const char *filename, void *data) {
else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) {
alert_cfg->lookup = string_strdupz(value);
health_parse_db_lookup(line, filename, value, &rc->group, &rc->after, &rc->before,
&rc->update_every, &rc->options, &rc->dimensions, &rc->foreachdim);
&rc->update_every, &rc->options, &rc->dimensions, &rc->foreach_dimension);
if(rc->foreachdim)
rc->spdim = health_pattern_from_foreach(rrdcalc_foreachdim(rc));
if(rc->foreach_dimension)
rc->foreach_dimension_pattern = health_pattern_from_foreach(rrdcalc_foreachdim(rc));
if (rc->after) {
if (rc->dimensions)
@ -1071,10 +971,10 @@ static int health_readfile(const char *filename, void *data) {
else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) {
alert_cfg->lookup = string_strdupz(value);
health_parse_db_lookup(line, filename, value, &rt->group, &rt->after, &rt->before,
&rt->update_every, &rt->options, &rt->dimensions, &rt->foreachdim);
&rt->update_every, &rt->options, &rt->dimensions, &rt->foreach_dimension);
if(rt->foreachdim)
rt->spdim = health_pattern_from_foreach(rrdcalctemplate_foreachdim(rt));
if(rt->foreach_dimension)
rt->foreach_dimension_pattern = health_pattern_from_foreach(rrdcalctemplate_foreachdim(rt));
if (rt->after) {
if (rt->dimensions)
@ -1237,15 +1137,17 @@ static int health_readfile(const char *filename, void *data) {
if(rc) {
//health_add_alarms_loop(host, rc, ignore_this) ;
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc)) {
rrdcalc_free(rc);
}
if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
else
rrdcalc_add_from_config(host, rc);
}
if(rt) {
if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt)) {
rrdcalctemplate_free(rt);
}
if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
else
rrdcalctemplate_add_from_config(host, rt);
}
if (alert_cfg)

31
health/health_json.c
View file

@ -116,7 +116,6 @@ void health_alarm_entry2json_nolock(BUFFER *wb, ALARM_ENTRY *ae, RRDHOST *host)
}
void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *chart) {
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
buffer_strcat(wb, "[");
@ -125,6 +124,8 @@ void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *char
STRING *chart_string = string_strdupz(chart);
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
ALARM_ENTRY *ae;
for (ae = host->health_log.alarms; ae && count < max; ae = ae->next) {
if ((ae->unique_id > after) && (!chart || chart_string == ae->chart)) {
@ -135,11 +136,11 @@ void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *char
}
}
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
string_freez(chart_string);
buffer_strcat(wb, "\n]\n");
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
}
static inline void health_rrdcalc_values2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC *rc) {
@ -216,8 +217,8 @@ static inline void health_rrdcalc2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC
, rc->component?rrdcalc_component(rc):"Unknown"
, rc->type?rrdcalc_type(rc):"Unknown"
, (rc->rrdset)?"true":"false"
, (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)?"true":"false"
, (rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
, (rc->run_flags & RRDCALC_FLAG_DISABLED)?"true":"false"
, (rc->run_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
, rc->exec?rrdcalc_exec(rc):string2str(host->health_default_exec)
, rc->recipient?rrdcalc_recipient(rc):string2str(host->health_default_recipient)
, rrdcalc_source(rc)
@ -241,7 +242,7 @@ static inline void health_rrdcalc2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC
, (unsigned long)rc->times_repeat
);
if(unlikely(rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)) {
if(unlikely(rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)) {
buffer_strcat(wb, "\t\t\t\"no_clear_notification\": true,\n");
}
@ -306,8 +307,6 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
char *tok = NULL;
char *p = NULL;
rrdhost_rdlock(host);
if (contexts) {
p = (char*)buffer_tostring(contexts);
while(p && *p && (tok = mystrsep(&p, ", |"))) {
@ -315,7 +314,7 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
STRING *tok_string = string_strdupz(tok);
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
continue;
if (unlikely(!rrdset_is_available_for_exporting_and_alarms(rc->rrdset)))
@ -325,12 +324,13 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
&& ((status==RRDCALC_STATUS_RAISED)?(rc->status >= RRDCALC_STATUS_WARNING):rc->status == status)))
numberOfAlarms++;
}
foreach_rrdcalc_in_rrdhost_done(rc);
string_freez(tok_string);
}
}
else {
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
continue;
if (unlikely(!rrdset_is_available_for_exporting_and_alarms(rc->rrdset)))
@ -338,16 +338,16 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
if(unlikely((status==RRDCALC_STATUS_RAISED)?(rc->status >= RRDCALC_STATUS_WARNING):rc->status == status))
numberOfAlarms++;
}
foreach_rrdcalc_in_rrdhost_done(rc);
}
buffer_sprintf(wb, "%d", numberOfAlarms);
rrdhost_unlock(host);
}
static void health_alarms2json_fill_alarms(RRDHOST *host, BUFFER *wb, int all, void (*fp)(RRDHOST *, BUFFER *, RRDCALC *)) {
RRDCALC *rc;
int i = 0;
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
continue;
@ -361,10 +361,10 @@ static void health_alarms2json_fill_alarms(RRDHOST *host, BUFFER *wb, int all, v
fp(host, wb, rc);
i++;
}
foreach_rrdcalc_in_rrdhost_done(rc);
}
void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) {
rrdhost_rdlock(host);
buffer_sprintf(wb, "{\n\t\"hostname\": \"%s\","
"\n\t\"latest_alarm_log_unique_id\": %u,"
"\n\t\"status\": %s,"
@ -377,17 +377,17 @@ void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) {
health_alarms2json_fill_alarms(host, wb, all, health_rrdcalc2json_nolock);
// rrdhost_rdlock(host);
// buffer_strcat(wb, "\n\t},\n\t\"templates\": {");
// RRDCALCTEMPLATE *rt;
// for(rt = host->templates; rt ; rt = rt->next)
// health_rrdcalctemplate2json_nolock(wb, rt);
// rrdhost_unlock(host);
buffer_strcat(wb, "\n\t}\n}\n");
rrdhost_unlock(host);
}
void health_alarms_values2json(RRDHOST *host, BUFFER *wb, int all) {
rrdhost_rdlock(host);
buffer_sprintf(wb, "{\n\t\"hostname\": \"%s\","
"\n\t\"alarms\": {\n",
rrdhost_hostname(host));
@ -395,7 +395,6 @@ void health_alarms_values2json(RRDHOST *host, BUFFER *wb, int all) {
health_alarms2json_fill_alarms(host, wb, all, health_rrdcalc_values2json_nolock);
buffer_strcat(wb, "\n\t}\n}\n");
rrdhost_unlock(host);
}
static int have_recent_alarm(RRDHOST *host, uint32_t alarm_id, time_t mark)

19
health/health_log.c
View file

@ -182,12 +182,15 @@ static inline ssize_t health_alarm_log_read(RRDHOST *host, FILE *fp, const char
size_t line = 0, len = 0;
ssize_t loaded = 0, updated = 0, errored = 0, duplicate = 0;
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
DICTIONARY *all_rrdcalcs = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
DICTIONARY *all_rrdcalcs = dictionary_create(
DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(host, rc)
foreach_rrdcalc_in_rrdhost_read(host, rc) {
dictionary_set(all_rrdcalcs, rrdcalc_name(rc), rc, sizeof(*rc));
}
foreach_rrdcalc_in_rrdhost_done(rc);
netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
while((s = fgets_trim_len(buf, 65536, fp, &len))) {
host->health_log_entries_written++;
@ -394,11 +397,11 @@ static inline ssize_t health_alarm_log_read(RRDHOST *host, FILE *fp, const char
}
}
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
dictionary_destroy(all_rrdcalcs);
all_rrdcalcs = NULL;
netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
freez(buf);
if(!host->health_max_unique_id) host->health_max_unique_id = (uint32_t)now_realtime_sec();
@ -510,7 +513,7 @@ inline ALARM_ENTRY* health_create_alarm_entry(
return ae;
}
inline void health_alarm_log(
inline void health_alarm_log_add_entry(
RRDHOST *host,
ALARM_ENTRY *ae
) {
@ -568,8 +571,6 @@ inline void health_alarm_log_free_one_nochecks_nounlink(ALARM_ENTRY *ae) {
}
inline void health_alarm_log_free(RRDHOST *host) {
rrdhost_check_wrlock(host);
netdata_rwlock_wrlock(&host->health_log.alarm_log_rwlock);
ALARM_ENTRY *ae;

1
libnetdata/Makefile.am
View file

@ -25,6 +25,7 @@ SUBDIRS = \
socket \
statistical \
storage_number \
string \
threads \
url \
worker_utilization \

18
libnetdata/dictionary/README.md
View file

@ -18,7 +18,7 @@ Dictionaries provide an interface to:
- **Delete** an item from the dictionary (provided its `name`)
- **Traverse** the list of items in the dictionary
Dictionaries are **ordered**, meaning that the order they have been added is preserved while traversing them. The caller may reverse this order by passing the flag `DICTIONARY_FLAG_ADD_IN_FRONT` when creating the dictionary.
Dictionaries are **ordered**, meaning that the order they have been added is preserved while traversing them. The caller may reverse this order by passing the flag `DICT_OPTION_ADD_IN_FRONT` when creating the dictionary.
Dictionaries guarantee **uniqueness** of all items added to them, meaning that only one item with a given name can exist in the dictionary at any given time.
@ -35,21 +35,21 @@ In **clone** mode, the dictionary guarantees that all operations on the dictiona
1.`dictionary_register_insert_callback()` that will be called just after the insertion of an item to the dictionary, or after the replacement of the value of a dictionary item (but while the dictionary is write-locked - if locking is enabled).
2. `dictionary_register_delete_callback()` that will be called just prior to the deletion of an item from the dictionary, or prior to the replacement of the value of a dictionary item (but while the dictionary is write-locked - if locking is enabled).
3. `dictionary_register_conflict_callback()` that will be called when `DICTIONARY_FLAG_DONT_OVERWRITE_VALUE` is set and another value is attempted to be inserted for the same key.
3. `dictionary_register_conflict_callback()` that will be called when `DICT_OPTION_DONT_OVERWRITE_VALUE` is set and another value is attempted to be inserted for the same key.
In **link** mode, the name and/or the value are just linked to the dictionary item, and it is the user's responsibility to free the memory they use after an item is deleted from the dictionary or when the dictionary is destroyed.
By default, **clone** mode is used for both the name and the value.
To use **link** mode for names, add `DICTIONARY_FLAG_NAME_LINK_DONT_CLONE` to the flags when creating the dictionary.
To use **link** mode for names, add `DICT_OPTION_NAME_LINK_DONT_CLONE` to the flags when creating the dictionary.
To use **link** mode for values, add `DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE` to the flags when creating the dictionary.
To use **link** mode for values, add `DICT_OPTION_VALUE_LINK_DONT_CLONE` to the flags when creating the dictionary.
## Locks
The dictionary allows both **single-threaded** operation (no locks - faster) and **multi-threaded** operation utilizing a read-write lock.
The default is **multi-threaded**. To enable **single-threaded** add `DICTIONARY_FLAG_SINGLE_THREADED` to the flags when creating the dictionary.
The default is **multi-threaded**. To enable **single-threaded** add `DICT_OPTION_SINGLE_THREADED` to the flags when creating the dictionary.
## Hash table operations
@ -72,7 +72,7 @@ This call is used to:
- **add** an item to the dictionary.
- **reset** the value of an existing item in the dictionary.
If **resetting** is not desired, add `DICTIONARY_FLAG_DONT_OVERWRITE_VALUE` to the flags when creating the dictionary. In this case, `dictionary_set()` will return the value of the original item found in the dictionary instead of resetting it and the value passed to the call will be ignored. Optionally a conflict callback function can be registered, to manipulate (probably merge or extend) the original value, based on the new value attempted to be added to the dictionary.
If **resetting** is not desired, add `DICT_OPTION_DONT_OVERWRITE_VALUE` to the flags when creating the dictionary. In this case, `dictionary_set()` will return the value of the original item found in the dictionary instead of resetting it and the value passed to the call will be ignored. Optionally a conflict callback function can be registered, to manipulate (probably merge or extend) the original value, based on the new value attempted to be added to the dictionary.
For **multi-threaded** operation, the `dictionary_set()` calls get an exclusive write lock on the dictionary.
@ -143,7 +143,7 @@ Example:
```c
// create the dictionary
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAGS_NONE);
DICTIONARY *dict = dictionary_create(DICT_OPTION_NONE);
// add an item to it
dictionary_set(dict, "name", "value", 6);
@ -189,7 +189,7 @@ There are 4 calls:
- `dictionary_walkthrough_read()` and `dictionary_sorted_walkthrough_read()` that acquire a shared read lock, and they call a callback function for every item of the dictionary. The callback function may use the unsafe versions of the `dictionary_get()` calls to lookup other items in the dictionary, but it should not attempt to add or remove items to/from the dictionary.
- `dictionary_walkthrough_write()` and `dictionary_sorted_walkthrough_write()` that acquire an exclusive write lock, and they call a callback function for every item of the dictionary. This is to be used when items need to be added to or removed from the dictionary. The `write` versions can be used to delete any or all the items from the dictionary, including the currently working one. For the `sorted` version, all items in the dictionary maintain a reference counter, so all deletions are deferred until the sorted walkthrough finishes.**
The non sorted versions traverse the items in the same order they have been added to the dictionary (or the reverse order if the flag `DICTIONARY_FLAG_ADD_IN_FRONT` is set during dictionary creation). The sorted versions sort alphabetically the items based on their name, and then they traverse them in the sorted order.
The non sorted versions traverse the items in the same order they have been added to the dictionary (or the reverse order if the flag `DICT_OPTION_ADD_IN_FRONT` is set during dictionary creation). The sorted versions sort alphabetically the items based on their name, and then they traverse them in the sorted order.
The callback function returns an `int`. If this value is negative, traversal of the dictionary is stopped immediately and the negative value is returned to the caller. If the returned value of all callback calls is zero or positive, the walkthrough functions return the sum of the return values of all callbacks. So, if you are just interested to know how many items fall into some condition, write a callback function that returns 1 when the item satisfies that condition and 0 when it does not and the walkthrough function will return how many tested positive.
@ -241,5 +241,5 @@ Since the dictionary uses a hash table and a double linked list, if the contract
This is currently used in statsd:
- the data collection thread uses only `get` and `set`. It never uses `del`. New items are added at the front of the linked list (`DICTIONARY_FLAG_ADD_IN_FRONT`).
- the data collection thread uses only `get` and `set`. It never uses `del`. New items are added at the front of the linked list (`DICT_OPTION_ADD_IN_FRONT`).
- the flushing thread is only traversing the dictionary up to the point it last traversed it (it uses a flag for that to know where it stopped last time). It never uses `get`, `set` or `del`.

3673
libnetdata/dictionary/dictionary.c
View file

File diff suppressed because it is too large Load diff

281
libnetdata/dictionary/dictionary.h
View file

@ -13,19 +13,19 @@
* Names and Values in the dictionary can be cloned or linked.
* In clone mode, the dictionary does all the memory management.
* The default is clone for both names and values.
* Set DICTIONARY_FLAG_NAME_LINK_DONT_CLONE to link names.
* Set DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE to link names.
* Set DICT_OPTION_NAME_LINK_DONT_CLONE to link names.
* Set DICT_OPTION_VALUE_LINK_DONT_CLONE to link names.
*
* ORDERED
* Items are ordered in the order they are added (new items are appended at the end).
* You may reverse the order by setting the flag DICTIONARY_FLAG_ADD_IN_FRONT.
* You may reverse the order by setting the flag DICT_OPTION_ADD_IN_FRONT.
*
* LOOKUP
* The dictionary uses JudyHS to maintain a very fast randomly accessible hash table.
*
* MULTI-THREADED and SINGLE-THREADED
* Each dictionary may be single threaded (no locks), or multi-threaded (multiple readers or one writer).
* The default is multi-threaded. Add the flag DICTIONARY_FLAG_SINGLE_THREADED for single-threaded.
* The default is multi-threaded. Add the flag DICT_OPTION_SINGLE_THREADED for single-threaded.
*
* WALK-THROUGH and FOREACH traversal
* The dictionary can be traversed on read or write mode, either with a callback (walkthrough) or with
@ -35,110 +35,184 @@
*
*/
#ifdef DICTIONARY_INTERNALS
#define DICTFE_CONST
#define DICT_ITEM_CONST
#else
#define DICTFE_CONST const
#define DICT_ITEM_CONST const
#endif
typedef struct dictionary DICTIONARY;
typedef struct dictionary_item DICTIONARY_ITEM;
typedef enum dictionary_flags {
DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
typedef enum dictionary_options {
DICT_OPTION_NONE = 0, // the default is the opposite of all below
DICT_OPTION_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
DICT_OPTION_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
DICT_OPTION_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
DICT_OPTION_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
DICT_OPTION_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
} DICT_OPTIONS;
// to change the value of the following, you also need to change the corresponding #defines in dictionary.c
DICTIONARY_FLAG_RESERVED1 = (1 << 28), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
DICTIONARY_FLAG_RESERVED2 = (1 << 29), // reserved for DICTIONARY_FLAG_DESTROYED
DICTIONARY_FLAG_RESERVED3 = (1 << 30), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
} DICTIONARY_FLAGS;
struct dictionary_stats {
const char *name; // the name of the category
struct {
size_t active; // the number of active dictionaries
size_t deleted; // the number of dictionaries queued for destruction
} dictionaries;
struct {
long entries; // active items in the dictionary
long pending_deletion; // pending deletion items in the dictionary
long referenced; // referenced items in the dictionary
} items;
struct {
size_t creations; // dictionary creations
size_t destructions; // dictionary destructions
size_t flushes; // dictionary flushes
size_t traversals; // dictionary foreach
size_t walkthroughs; // dictionary walkthrough
size_t garbage_collections; // dictionary garbage collections
size_t searches; // item searches
size_t inserts; // item inserts
size_t resets; // item resets
size_t deletes; // item deletes
} ops;
struct {
size_t inserts; // number of times the insert callback is called
size_t conflicts; // number of times the insert conflict is called
size_t reacts; // number of times the insert react is called
size_t deletes; // number of times the insert delete is called
} callbacks;
// memory
struct {
long indexed; // bytes of keys indexed (indication of the index size)
long values; // bytes of caller structures
long dict; // bytes of the structures dictionary needs
} memory;
// spin locks
struct {
size_t use; // number of times a reference to item had to spin to acquire it or ignore it
size_t search; // number of times a successful search result had to be thrown away
size_t insert; // number of times an insertion to the hash table had to be repeated
} spin_locks;
};
// Create a dictionary
#ifdef NETDATA_INTERNAL_CHECKS
#define dictionary_create(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
#define dictionary_create_advanced(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
extern DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file);
#define dictionary_create(options) dictionary_create_advanced_with_trace(options, NULL, __FUNCTION__, __LINE__, __FILE__)
#define dictionary_create_advanced(options, stats) dictionary_create_advanced_with_trace(options, stats, __FUNCTION__, __LINE__, __FILE__)
extern DICTIONARY *dictionary_create_advanced_with_trace(DICT_OPTIONS options, struct dictionary_stats *stats, const char *function, size_t line, const char *file);
#else
#define dictionary_create(flags) dictionary_create_advanced(flags, 0);
extern DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size);
#define dictionary_create(options) dictionary_create_advanced(options, NULL);
extern DICTIONARY *dictionary_create_advanced(DICT_OPTIONS options, struct dictionary_stats *stats);
#endif
extern void *dictionary_scratchpad(DICTIONARY *dict);
// Create a view on a dictionary
#ifdef NETDATA_INTERNAL_CHECKS
#define dictionary_create_view(master) dictionary_create_view_with_trace(master, __FUNCTION__, __LINE__, __FILE__)
extern DICTIONARY *dictionary_create_view_with_trace(DICTIONARY *master, const char *function, size_t line, const char *file);
#else
extern DICTIONARY *dictionary_create_view(DICTIONARY *master);
#endif
// an insert callback to be called just after an item is added to the dictionary
// this callback is called while the dictionary is write locked!
extern void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const char *name, void *value, void *data), void *data);
extern void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
// a delete callback to be called just before an item is deleted forever
// this callback is called while the dictionary is write locked!
extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const char *name, void *value, void *data), void *data);
extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
// a merge callback to be called when DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
// a merge callback to be called when DICT_OPTION_DONT_OVERWRITE_VALUE
// and an item is already found in the dictionary - the dictionary does nothing else in this case
// the old_value will remain in the dictionary - the new_value is ignored
extern void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data), void *data);
// The callback should return true if the value has been updated (it increases the dictionary version).
extern void dictionary_register_conflict_callback(DICTIONARY *dict, bool (*conflict_callback)(const DICTIONARY_ITEM *item, void *old_value, void *new_value, void *data), void *data);
// a reaction callback to be called after every item insertion or conflict
// after the constructors have finished and the items are fully available for use
// and the dictionary is not write locked anymore
extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data);
extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
// Destroy a dictionary
// returns the number of bytes freed
// the returned value will not include name and value sizes if DICTIONARY_FLAG_WITH_STATISTICS is not set
// Returns the number of bytes freed
// The returned value will not include name/key sizes
// Registered delete callbacks will be run for each item in the dictionary.
extern size_t dictionary_destroy(DICTIONARY *dict);
// Empties a dictionary
// Referenced items will survive, but are not offered anymore.
// Registered delete callbacks will be run for each item in the dictionary.
extern void dictionary_flush(DICTIONARY *dict);
extern void dictionary_version_increment(DICTIONARY *dict);
// ----------------------------------------------------------------------------
// Set an item in the dictionary
//
// - if an item with the same name does not exist, create one
// - if an item with the same name exists, then:
// a) if DICTIONARY_FLAG_DONT_OVERWRITE_VALUE is set, just return the existing value (ignore the new value)
// a) if DICT_OPTION_DONT_OVERWRITE_VALUE is set, just return the existing value (ignore the new value)
// else b) reset the value to the new value passed at the call
//
// When DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE is set, the value is linked, otherwise it is copied
// When DICTIONARY_FLAG_NAME_LINK_DONT_CLONE is set, the name is linked, otherwise it is copied
// When DICT_OPTION_VALUE_LINK_DONT_CLONE is set, the value is linked, otherwise it is copied
// When DICT_OPTION_NAME_LINK_DONT_CLONE is set, the name is linked, otherwise it is copied
//
// When neither DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE nor DICTIONARY_FLAG_NAME_LINK_DONT_CLONE are set, all the
// When neither DICT_OPTION_VALUE_LINK_DONT_CLONE nor DICT_OPTION_NAME_LINK_DONT_CLONE are set, all the
// memory management for names and values is done by the dictionary.
//
// Passing NULL as value, the dictionary will callocz() the newly allocated value, otherwise it will copy it.
// Passing 0 as value_len, the dictionary will set the value to NULL (no allocations for value will be made).
extern void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len);
#define dictionary_set(dict, name, value, value_len) dictionary_set_advanced(dict, name, -1, value, value_len, NULL)
extern void *dictionary_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data);
#define dictionary_set_and_acquire_item(dict, name, value, value_len) dictionary_set_and_acquire_item_advanced(dict, name, -1, value, value_len, NULL)
extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data);
// set an item in a dictionary view
#define dictionary_view_set_and_acquire_item(dict, name, master_item) dictionary_view_set_and_acquire_item_advanced(dict, name, -1, master_item)
extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_view_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item);
#define dictionary_view_set(dict, name, master_item) dictionary_view_set_advanced(dict, name, -1, master_item)
extern void *dictionary_view_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item);
// ----------------------------------------------------------------------------
// Get an item from the dictionary
// If it returns NULL, the item is not found
extern void *dictionary_get(DICTIONARY *dict, const char *name);
#define dictionary_get(dict, name) dictionary_get_advanced(dict, name, -1)
extern void *dictionary_get_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
#define dictionary_get_and_acquire_item(dict, name) dictionary_get_and_acquire_item_advanced(dict, name, -1)
extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_get_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
// ----------------------------------------------------------------------------
// Delete an item from the dictionary
// returns 0 if the item was found and has been deleted
// returns -1 if the item was not found in the index
extern int dictionary_del(DICTIONARY *dict, const char *name);
// returns true if the item was found and has been deleted
// returns false if the item was not found in the index
extern DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name);
extern DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name);
#define dictionary_del(dict, name) dictionary_del_advanced(dict, name, -1)
extern bool dictionary_del_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
extern DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
extern DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len);
// ----------------------------------------------------------------------------
// reference counters management
extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item);
extern void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item);
extern void dictionary_acquired_item_release(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item);
extern DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item);
extern const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item);
extern void *dictionary_acquired_item_value(DICTIONARY_ITEM *item);
extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item);
// UNSAFE functions, without locks
// to be used when the user is traversing with the right lock type
// Read lock is acquired by dictionary_walktrhough_read() and dfe_start_read()
// Write lock is acquired by dictionary_walktrhough_write() and dfe_start_write()
// For code readability, please use these macros:
#define dictionary_get_having_read_lock(dict, name) dictionary_get_unsafe(dict, name)
#define dictionary_get_having_write_lock(dict, name) dictionary_get_unsafe(dict, name)
#define dictionary_set_having_write_lock(dict, name, value, value_len) dictionary_set_unsafe(dict, name, value, value_len)
#define dictionary_del_having_write_lock(dict, name) dictionary_del_unsafe(dict, name)
extern const char *dictionary_acquired_item_name(DICT_ITEM_CONST DICTIONARY_ITEM *item);
extern void *dictionary_acquired_item_value(DICT_ITEM_CONST DICTIONARY_ITEM *item);
extern void *dictionary_get_unsafe(DICTIONARY *dict, const char *name);
extern void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
extern int dictionary_del_unsafe(DICTIONARY *dict, const char *name);
// ----------------------------------------------------------------------------
// Traverse (walk through) the items of the dictionary.
// The order of traversal is currently the order of insertion.
//
@ -153,12 +227,13 @@ extern int dictionary_del_unsafe(DICTIONARY *dict, const char *name);
//
#define dictionary_walkthrough_read(dict, callback, data) dictionary_walkthrough_rw(dict, 'r', callback, data)
#define dictionary_walkthrough_write(dict, callback, data) dictionary_walkthrough_rw(dict, 'w', callback, data)
extern int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *value, void *data), void *data);
extern int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
#define dictionary_sorted_walkthrough_read(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'r', callback, data)
#define dictionary_sorted_walkthrough_write(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'w', callback, data)
int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *entry, void *data), void *data);
int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data);
// ----------------------------------------------------------------------------
// Traverse with foreach
//
// Use like this:
@ -173,84 +248,60 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
// You can only delete the current item from inside a dfe_start_write() - you can add as many as you want.
//
#ifdef DICTIONARY_INTERNALS
#define DICTFE_CONST
#else
#define DICTFE_CONST const
#endif
#define DICTIONARY_LOCK_READ 'r'
#define DICTIONARY_LOCK_WRITE 'w'
#define DICTIONARY_LOCK_REENTRANT 'z'
#define DICTIONARY_LOCK_NONE 'u'
typedef DICTFE_CONST struct dictionary_foreach {
DICTIONARY *dict; // the dictionary upon we work
DICTIONARY_ITEM *item; // the item we work on, to remember the position we are at
// this can be used with dictionary_acquired_item_dup() to
// acquire the currently working item.
DICTFE_CONST char *name; // the dictionary name of the last item used
void *value; // the dictionary value of the last item used
// same as the return value of dictfe_start() and dictfe_next()
// the following are for internal use only - to keep track of the point we are
size_t counter; // counts the number of iterations made, starting from zero
char rw; // the lock mode 'r' or 'w'
usec_t started_ut; // the time the caller started iterating (now_realtime_usec())
DICTIONARY *dict; // the dictionary upon we work
void *last_item; // the item we work on, to remember the position we are at
} DICTFE;
#define dfe_start_read(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_READ)
#define dfe_start_write(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_WRITE)
#define dfe_start_reentrant(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_REENTRANT)
#define dfe_start_unsafe(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_NONE)
#define dfe_start_rw(dict, value, mode) \
do { \
DICTFE value ## _dfe = {}; \
const char *value ## _name; (void)(value ## _name); (void)(value); \
for((value) = dictionary_foreach_start_rw(&value ## _dfe, (dict), (mode)), ( value ## _name ) = value ## _dfe.name; \
(value ## _dfe.name) ;\
(value) = dictionary_foreach_next(&value ## _dfe), ( value ## _name ) = value ## _dfe.name) \
#define dfe_start_rw(dict, value, mode) \
do { \
DICTFE value ## _dfe = {}; \
(void)(value); /* needed to avoid warning when looping without using this */ \
for((value) = dictionary_foreach_start_rw(&value ## _dfe, (dict), (mode)); \
(value ## _dfe.item) ; \
(value) = dictionary_foreach_next(&value ## _dfe)) \
{
#define dfe_done(value) \
} \
dictionary_foreach_done(&value ## _dfe); \
#define dfe_done(value) \
} \
dictionary_foreach_done(&value ## _dfe); \
} while(0)
extern void * dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw);
extern void * dictionary_foreach_next(DICTFE *dfe);
extern usec_t dictionary_foreach_done(DICTFE *dfe);
// Get statistics about the dictionary
extern long int dictionary_stats_allocated_memory(DICTIONARY *dict);
extern long int dictionary_stats_entries(DICTIONARY *dict);
extern size_t dictionary_stats_version(DICTIONARY *dict);
extern size_t dictionary_stats_inserts(DICTIONARY *dict);
extern size_t dictionary_stats_searches(DICTIONARY *dict);
extern size_t dictionary_stats_deletes(DICTIONARY *dict);
extern size_t dictionary_stats_resets(DICTIONARY *dict);
extern size_t dictionary_stats_walkthroughs(DICTIONARY *dict);
extern size_t dictionary_stats_referenced_items(DICTIONARY *dict);
extern int dictionary_unittest(size_t entries);
extern void *dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw);
extern void *dictionary_foreach_next(DICTFE *dfe);
extern void dictionary_foreach_done(DICTFE *dfe);
// ----------------------------------------------------------------------------
// STRING implementation
// Get statistics about the dictionary
typedef struct netdata_string STRING;
extern STRING *string_strdupz(const char *str);
extern STRING *string_dup(STRING *string);
extern void string_freez(STRING *string);
extern size_t string_strlen(STRING *string);
extern const char *string2str(STRING *string) NEVERNULL;
extern size_t dictionary_version(DICTIONARY *dict);
extern size_t dictionary_entries(DICTIONARY *dict);
extern size_t dictionary_referenced_items(DICTIONARY *dict);
extern long int dictionary_stats_for_registry(DICTIONARY *dict);
// keep common prefix/suffix and replace everything else with [x]
extern STRING *string_2way_merge(STRING *a, STRING *b);
// for all cases that the caller does not provide a stats structure, this is where they are accumulated.
extern struct dictionary_stats dictionary_stats_category_other;
static inline int string_cmp(STRING *s1, STRING *s2) {
// STRINGs are deduplicated, so the same strings have the same pointer
// when they differ, we do the typical strcmp() comparison
return (s1 == s2)?0:strcmp(string2str(s1), string2str(s2));
}
extern void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases);
extern int dictionary_unittest(size_t entries);
// ----------------------------------------------------------------------------
// THREAD CACHE

1
libnetdata/libnetdata.h
View file

@ -410,6 +410,7 @@ extern char *netdata_configured_host_prefix;
#include "config/appconfig.h"
#include "log/log.h"
#include "procfile/procfile.h"
#include "string/string.h"
#include "dictionary/dictionary.h"
#if defined(HAVE_LIBBPF) && !defined(__cplusplus)
#include "ebpf/ebpf.h"

6
libnetdata/locks/README.md
View file

@ -49,7 +49,7 @@ The library maintains a linked-list of all the lock holders (one entry per threa
If any call is expected to pause the caller (ie the caller is attempting a read lock while there is a write lock in place and vice versa), the library will log something like this:
```
RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) WANTS a 'W' lock (while holding 1 rwlocks and 1 mutexes).
RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) WANTS a 'W' lock (while holding 1 rwlocks and 1 mutexes).
There are 7 readers and 0 writers are holding the lock:
=> 1: RW_LOCK: process 4190091 'WEB_SERVER[static14]' (function web_client_api_request_v1_data() 526@web/api/web_api_v1.c) is having 1 'R' lock for 709847 usec.
=> 2: RW_LOCK: process 4190079 'WEB_SERVER[static6]' (function web_client_api_request_v1_data() 526@web/api/web_api_v1.c) is having 1 'R' lock for 709869 usec.
@ -63,9 +63,9 @@ There are 7 readers and 0 writers are holding the lock:
And each of the above is paired with a `GOT` log, like this:
```
RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) GOT a 'W' lock (while holding 2 rwlocks and 1 mutexes).
RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) GOT a 'W' lock (while holding 2 rwlocks and 1 mutexes).
There are 0 readers and 1 writers are holding the lock:
=> 1: RW_LOCK: process 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) is having 1 'W' lock for 36 usec.
=> 1: RW_LOCK: process 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) is having 1 'W' lock for 36 usec.
```
Keep in mind that the lock and log are not atomic. The list of callers is indicative (and sometimes just empty because the original holders of the lock, unlocked it until we had the chance to print their names).

11
libnetdata/log/log.c
View file

@ -850,6 +850,13 @@ void error_int( const char *prefix, const char *file __maybe_unused, const char
log_unlock();
}
#ifdef NETDATA_INTERNAL_CHECKS
static void crash_netdata(void) {
// make Netdata core dump
abort();
}
#endif
void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) {
// save a copy of errno - just in case this function generates a new error
int __errno = errno;
@ -897,6 +904,10 @@ void fatal_int( const char *file, const char *function, const unsigned long line
snprintfz(action_result, 60, "%s:%s", program_name, strncmp(thread_tag, "STREAM_RECEIVER", strlen("STREAM_RECEIVER")) ? thread_tag : "[x]");
send_statistics("FATAL", action_result, action_data);
#ifdef NETDATA_INTERNAL_CHECKS
crash_netdata();
#endif
netdata_cleanup_and_exit(1);
}

2
libnetdata/required_dummies.h
View file

@ -33,6 +33,8 @@ int health_variable_lookup(STRING *variable, struct rrdcalc *rc, NETDATA_DOUBLE
};
#endif
void rrdset_thread_rda_free(void){};
// required by get_system_cpus()
char *netdata_configured_host_prefix = "";

8
libnetdata/string/Makefile.am Normal file
View file

@ -0,0 +1,8 @@
# SPDX-License-Identifier: GPL-3.0-or-later
AUTOMAKE_OPTIONS = subdir-objects
MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
dist_noinst_DATA = \
README.md \
$(NULL)

20
libnetdata/string/README.md Normal file
View file

@ -0,0 +1,20 @@
<!--
custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/string/README.md
-->
# STRING
STRING provides a way to allocate and free text strings, while de-duplicating them.
It can be used similarly to libc string functions:
- `strdup()` and `strdupz()` become `string_strdupz()`.
- `strlen()` becomes `string_strlen()` (and it does not walkthrough the bytes of the string).
- `free()` and `freez()` become `string_freez()`.
There is also a special `string_dup()` function that increases the reference counter of a STRING, avoiding the
index lookup to find it.
Once there is a `STRING *`, the actual `const char *` can be accessed with `string2str()`.
All STRING should be constant. Changing the contents of a `const char *` that has been acquired by `string2str()` should never happen.

595
libnetdata/string/string.c Normal file
View file

@ -0,0 +1,595 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#include "../libnetdata.h"
#include <Judy.h>
typedef int32_t REFCOUNT;
// ----------------------------------------------------------------------------
// STRING implementation - dedup all STRING
struct netdata_string {
uint32_t length; // the string length including the terminating '\0'
REFCOUNT refcount; // how many times this string is used
// We use a signed number to be able to detect duplicate frees of a string.
// If at any point this goes below zero, we have a duplicate free.
const char str[]; // the string itself, is appended to this structure
};
static struct string_hashtable {
Pvoid_t JudyHSArray; // the Judy array - hashtable
netdata_rwlock_t rwlock; // the R/W lock to protect the Judy array
long int entries; // the number of entries in the index
long int active_references; // the number of active references alive
long int memory; // the memory used, without the JudyHS index
size_t inserts; // the number of successful inserts to the index
size_t deletes; // the number of successful deleted from the index
size_t searches; // the number of successful searches in the index
size_t duplications; // when a string is referenced
size_t releases; // when a string is unreferenced
#ifdef NETDATA_INTERNAL_CHECKS
// internal statistics
size_t found_deleted_on_search;
size_t found_available_on_search;
size_t found_deleted_on_insert;
size_t found_available_on_insert;
size_t spins;
#endif
} string_base = {
.JudyHSArray = NULL,
.rwlock = NETDATA_RWLOCK_INITIALIZER,
};
#ifdef NETDATA_INTERNAL_CHECKS
#define string_internal_stats_add(var, val) __atomic_add_fetch(&string_base.var, val, __ATOMIC_RELAXED)
#else
#define string_internal_stats_add(var, val) do {;} while(0)
#endif
#define string_stats_atomic_increment(var) __atomic_add_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
#define string_stats_atomic_decrement(var) __atomic_sub_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases) {
*inserts = string_base.inserts;
*deletes = string_base.deletes;
*searches = string_base.searches;
*entries = (size_t)string_base.entries;
*references = (size_t)string_base.active_references;
*memory = (size_t)string_base.memory;
*duplications = string_base.duplications;
*releases = string_base.releases;
}
#define string_entry_acquire(se) __atomic_add_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
#define string_entry_release(se) __atomic_sub_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
static inline bool string_entry_check_and_acquire(STRING *se) {
REFCOUNT expected, desired, count = 0;
do {
count++;
expected = __atomic_load_n(&se->refcount, __ATOMIC_SEQ_CST);
if(expected <= 0) {
// We cannot use this.
// The reference counter reached value zero,
// so another thread is deleting this.
string_internal_stats_add(spins, count - 1);
return false;
}
desired = expected + 1;
}
while(!__atomic_compare_exchange_n(&se->refcount, &expected, desired, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST));
string_internal_stats_add(spins, count - 1);
// statistics
// string_base.active_references is altered at the in string_strdupz() and string_freez()
string_stats_atomic_increment(duplications);
return true;
}
STRING *string_dup(STRING *string) {
if(unlikely(!string)) return NULL;
#ifdef NETDATA_INTERNAL_CHECKS
if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) <= 0))
fatal("STRING: tried to %s() a string that is freed (it has %d references).", __FUNCTION__, string->refcount);
#endif
string_entry_acquire(string);
// statistics
string_stats_atomic_increment(active_references);
string_stats_atomic_increment(duplications);
return string;
}
// Search the index and return an ACQUIRED string entry, or NULL
static inline STRING *string_index_search(const char *str, size_t length) {
STRING *string;
// Find the string in the index
// With a read-lock so that multiple readers can use the index concurrently.
netdata_rwlock_rdlock(&string_base.rwlock);
Pvoid_t *Rc;
Rc = JudyHSGet(string_base.JudyHSArray, (void *)str, length);
if(likely(Rc)) {
// found in the hash table
string = *Rc;
if(string_entry_check_and_acquire(string)) {
// we can use this entry
string_internal_stats_add(found_available_on_search, 1);
}
else {
// this entry is about to be deleted by another thread
// do not touch it, let it go...
string = NULL;
string_internal_stats_add(found_deleted_on_search, 1);
}
}
else {
// not found in the hash table
string = NULL;
}
string_stats_atomic_increment(searches);
netdata_rwlock_unlock(&string_base.rwlock);
return string;
}
// Insert a string to the index and return an ACQUIRED string entry,
// or NULL if the call needs to be retried (a deleted entry with the same key is still in the index)
// The returned entry is ACQUIRED, and it can either be:
// 1. a new item inserted, or
// 2. an item found in the index that is not currently deleted
static inline STRING *string_index_insert(const char *str, size_t length) {
STRING *string;
netdata_rwlock_wrlock(&string_base.rwlock);
STRING **ptr;
{
JError_t J_Error;
Pvoid_t *Rc = JudyHSIns(&string_base.JudyHSArray, (void *)str, length, &J_Error);
if (unlikely(Rc == PJERR)) {
fatal(
"STRING: Cannot insert entry with name '%s' to JudyHS, JU_ERRNO_* == %u, ID == %d",
str,
JU_ERRNO(&J_Error),
JU_ERRID(&J_Error));
}
ptr = (STRING **)Rc;
}
if (likely(*ptr == 0)) {
// a new item added to the index
size_t mem_size = sizeof(STRING) + length;
string = mallocz(mem_size);
strcpy((char *)string->str, str);
string->length = length;
string->refcount = 1;
*ptr = string;
string_base.inserts++;
string_base.entries++;
string_base.memory += (long)mem_size;
}
else {
// the item is already in the index
string = *ptr;
if(string_entry_check_and_acquire(string)) {
// we can use this entry
string_internal_stats_add(found_available_on_insert, 1);
}
else {
// this entry is about to be deleted by another thread
// do not touch it, let it go...
string = NULL;
string_internal_stats_add(found_deleted_on_insert, 1);
}
string_stats_atomic_increment(searches);
}
netdata_rwlock_unlock(&string_base.rwlock);
return string;
}
// delete an entry from the index
static inline void string_index_delete(STRING *string) {
netdata_rwlock_wrlock(&string_base.rwlock);
#ifdef NETDATA_INTERNAL_CHECKS
if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) != 0))
fatal("STRING: tried to delete a string at %s() that is already freed (it has %d references).", __FUNCTION__, string->refcount);
#endif
bool deleted = false;
if (likely(string_base.JudyHSArray)) {
JError_t J_Error;
int ret = JudyHSDel(&string_base.JudyHSArray, (void *)string->str, string->length, &J_Error);
if (unlikely(ret == JERR)) {
error(
"STRING: Cannot delete entry with name '%s' from JudyHS, JU_ERRNO_* == %u, ID == %d",
string->str,
JU_ERRNO(&J_Error),
JU_ERRID(&J_Error));
} else
deleted = true;
}
if (unlikely(!deleted))
error("STRING: tried to delete '%s' that is not in the index. Ignoring it.", string->str);
else {
size_t mem_size = sizeof(STRING) + string->length;
string_base.deletes++;
string_base.entries--;
string_base.memory -= (long)mem_size;
freez(string);
}
netdata_rwlock_unlock(&string_base.rwlock);
}
STRING *string_strdupz(const char *str) {
if(unlikely(!str || !*str)) return NULL;
size_t length = strlen(str) + 1;
STRING *string = string_index_search(str, length);
while(!string) {
// The search above did not find anything,
// We loop here, because during insert we may find an entry that is being deleted by another thread.
// So, we have to let it go and retry to insert it again.
string = string_index_insert(str, length);
}
// statistics
string_stats_atomic_increment(active_references);
return string;
}
void string_freez(STRING *string) {
if(unlikely(!string)) return;
REFCOUNT refcount = string_entry_release(string);
#ifdef NETDATA_INTERNAL_CHECKS
if(unlikely(refcount < 0))
fatal("STRING: tried to %s() a string that is already freed (it has %d references).", __FUNCTION__, string->refcount);
#endif
if(unlikely(refcount == 0))
string_index_delete(string);
// statistics
string_stats_atomic_decrement(active_references);
string_stats_atomic_increment(releases);
}
size_t string_strlen(STRING *string) {
if(unlikely(!string)) return 0;
return string->length - 1;
}
const char *string2str(STRING *string) {
if(unlikely(!string)) return "";
return string->str;
}
STRING *string_2way_merge(STRING *a, STRING *b) {
static STRING *X = NULL;
if(unlikely(!X)) {
X = string_strdupz("[x]");
}
if(unlikely(a == b)) return string_dup(a);
if(unlikely(a == X)) return string_dup(a);
if(unlikely(b == X)) return string_dup(b);
if(unlikely(!a)) return string_dup(X);
if(unlikely(!b)) return string_dup(X);
size_t alen = string_strlen(a);
size_t blen = string_strlen(b);
size_t length = alen + blen + string_strlen(X) + 1;
char buf1[length + 1], buf2[length + 1], *dst1;
const char *s1, *s2;
s1 = string2str(a);
s2 = string2str(b);
dst1 = buf1;
for( ; *s1 && *s2 && *s1 == *s2 ;s1++, s2++)
*dst1++ = *s1;
*dst1 = '\0';
if(*s1 != '\0' || *s2 != '\0') {
*dst1++ = '[';
*dst1++ = 'x';
*dst1++ = ']';
s1 = &(string2str(a))[alen - 1];
s2 = &(string2str(b))[blen - 1];
char *dst2 = &buf2[length];
*dst2 = '\0';
for (; *s1 && *s2 && *s1 == *s2; s1--, s2--)
*(--dst2) = *s1;
strcpy(dst1, dst2);
}
return string_strdupz(buf1);
}
// ----------------------------------------------------------------------------
// STRING unit test
struct thread_unittest {
int join;
int dups;
};
static void *string_thread(void *arg) {
struct thread_unittest *tu = arg;
for(; 1 ;) {
if(__atomic_load_n(&tu->join, __ATOMIC_RELAXED))
break;
STRING *s = string_strdupz("string thread checking 1234567890");
for(int i = 0; i < tu->dups ; i++)
string_dup(s);
for(int i = 0; i < tu->dups ; i++)
string_freez(s);
string_freez(s);
}
return arg;
}
static char **string_unittest_generate_names(size_t entries) {
char **names = mallocz(sizeof(char *) * entries);
for(size_t i = 0; i < entries ;i++) {
char buf[25 + 1] = "";
snprintfz(buf, 25, "name.%zu.0123456789.%zu \t !@#$%%^&*(),./[]{}\\|~`", i, entries / 2 + i);
names[i] = strdupz(buf);
}
return names;
}
static void string_unittest_free_char_pp(char **pp, size_t entries) {
for(size_t i = 0; i < entries ;i++)
freez(pp[i]);
freez(pp);
}
int string_unittest(size_t entries) {
size_t errors = 0;
fprintf(stderr, "Generating %zu names and values...\n", entries);
char **names = string_unittest_generate_names(entries);
// check string
{
long int string_entries_starting = string_base.entries;
fprintf(stderr, "\nChecking strings...\n");
STRING *s1 = string_strdupz("hello unittest");
STRING *s2 = string_strdupz("hello unittest");
if(s1 != s2) {
errors++;
fprintf(stderr, "ERROR: duplicating strings are not deduplicated\n");
}
else
fprintf(stderr, "OK: duplicating string are deduplicated\n");
STRING *s3 = string_dup(s1);
if(s3 != s1) {
errors++;
fprintf(stderr, "ERROR: cloning strings are not deduplicated\n");
}
else
fprintf(stderr, "OK: cloning string are deduplicated\n");
if(s1->refcount != 3) {
errors++;
fprintf(stderr, "ERROR: string refcount is not 3\n");
}
else
fprintf(stderr, "OK: string refcount is 3\n");
STRING *s4 = string_strdupz("world unittest");
if(s4 == s1) {
errors++;
fprintf(stderr, "ERROR: string is sharing pointers on different strings\n");
}
else
fprintf(stderr, "OK: string is properly handling different strings\n");
usec_t start_ut, end_ut;
STRING **strings = mallocz(entries * sizeof(STRING *));
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
strings[i] = string_strdupz(names[i]);
}
end_ut = now_realtime_usec();
fprintf(stderr, "Created %zu strings in %llu usecs\n", entries, end_ut - start_ut);
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
strings[i] = string_dup(strings[i]);
}
end_ut = now_realtime_usec();
fprintf(stderr, "Cloned %zu strings in %llu usecs\n", entries, end_ut - start_ut);
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
strings[i] = string_strdupz(string2str(strings[i]));
}
end_ut = now_realtime_usec();
fprintf(stderr, "Found %zu existing strings in %llu usecs\n", entries, end_ut - start_ut);
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
string_freez(strings[i]);
}
end_ut = now_realtime_usec();
fprintf(stderr, "Released %zu referenced strings in %llu usecs\n", entries, end_ut - start_ut);
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
string_freez(strings[i]);
}
end_ut = now_realtime_usec();
fprintf(stderr, "Released (again) %zu referenced strings in %llu usecs\n", entries, end_ut - start_ut);
start_ut = now_realtime_usec();
for(size_t i = 0; i < entries ;i++) {
string_freez(strings[i]);
}
end_ut = now_realtime_usec();
fprintf(stderr, "Freed %zu strings in %llu usecs\n", entries, end_ut - start_ut);
freez(strings);
if(string_base.entries != string_entries_starting + 2) {
errors++;
fprintf(stderr, "ERROR: strings dictionary should have %ld items but it has %ld\n", string_entries_starting + 2, string_base.entries);
}
else
fprintf(stderr, "OK: strings dictionary has 2 items\n");
}
// check 2-way merge
{
struct testcase {
char *src1; char *src2; char *expected;
} tests[] = {
{ "", "", ""},
{ "a", "", "[x]"},
{ "", "a", "[x]"},
{ "a", "a", "a"},
{ "abcd", "abcd", "abcd"},
{ "foo_cs", "bar_cs", "[x]_cs"},
{ "cp_UNIQUE_INFIX_cs", "cp_unique_infix_cs", "cp_[x]_cs"},
{ "cp_UNIQUE_INFIX_ci_unique_infix_cs", "cp_unique_infix_ci_UNIQUE_INFIX_cs", "cp_[x]_cs"},
{ "foo[1234]", "foo[4321]", "foo[[x]]"},
{ NULL, NULL, NULL },
};
for (struct testcase *tc = &tests[0]; tc->expected != NULL; tc++) {
STRING *src1 = string_strdupz(tc->src1);
STRING *src2 = string_strdupz(tc->src2);
STRING *expected = string_strdupz(tc->expected);
STRING *result = string_2way_merge(src1, src2);
if (string_cmp(result, expected) != 0) {
fprintf(stderr, "string_2way_merge(\"%s\", \"%s\") -> \"%s\" (expected=\"%s\")\n",
string2str(src1),
string2str(src2),
string2str(result),
string2str(expected));
errors++;
}
string_freez(src1);
string_freez(src2);
string_freez(expected);
string_freez(result);
}
}
// threads testing of string
{
struct thread_unittest tu = {
.dups = 1,
.join = 0,
};
#ifdef NETDATA_INTERNAL_CHECKS
size_t ofound_deleted_on_search = string_base.found_deleted_on_search,
ofound_available_on_search = string_base.found_available_on_search,
ofound_deleted_on_insert = string_base.found_deleted_on_insert,
ofound_available_on_insert = string_base.found_available_on_insert,
ospins = string_base.spins;
#endif
size_t oinserts, odeletes, osearches, oentries, oreferences, omemory, oduplications, oreleases;
string_statistics(&oinserts, &odeletes, &osearches, &oentries, &oreferences, &omemory, &oduplications, &oreleases);
time_t seconds_to_run = 5;
int threads_to_create = 2;
fprintf(
stderr,
"Checking string concurrency with %d threads for %ld seconds...\n",
threads_to_create,
seconds_to_run);
// check string concurrency
netdata_thread_t threads[threads_to_create];
tu.join = 0;
for (int i = 0; i < threads_to_create; i++) {
char buf[100 + 1];
snprintf(buf, 100, "string%d", i);
netdata_thread_create(
&threads[i], buf, NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE, string_thread, &tu);
}
sleep_usec(seconds_to_run * USEC_PER_SEC);
__atomic_store_n(&tu.join, 1, __ATOMIC_RELAXED);
for (int i = 0; i < threads_to_create; i++) {
void *retval;
netdata_thread_join(threads[i], &retval);
}
size_t inserts, deletes, searches, sentries, references, memory, duplications, releases;
string_statistics(&inserts, &deletes, &searches, &sentries, &references, &memory, &duplications, &releases);
fprintf(stderr, "inserts %zu, deletes %zu, searches %zu, entries %zu, references %zu, memory %zu, duplications %zu, releases %zu\n",
inserts - oinserts, deletes - odeletes, searches - osearches, sentries - oentries, references - oreferences, memory - omemory, duplications - oduplications, releases - oreleases);
#ifdef NETDATA_INTERNAL_CHECKS
size_t found_deleted_on_search = string_base.found_deleted_on_search,
found_available_on_search = string_base.found_available_on_search,
found_deleted_on_insert = string_base.found_deleted_on_insert,
found_available_on_insert = string_base.found_available_on_insert,
spins = string_base.spins;
fprintf(stderr, "on insert: %zu ok + %zu deleted\non search: %zu ok + %zu deleted\nspins: %zu\n",
found_available_on_insert - ofound_available_on_insert,
found_deleted_on_insert - ofound_deleted_on_insert,
found_available_on_search - ofound_available_on_search,
found_deleted_on_search - ofound_deleted_on_search,
spins - ospins
);
#endif
}
string_unittest_free_char_pp(names, entries);
fprintf(stderr, "\n%zu errors found\n", errors);
return errors ? 1 : 0;
}

30
libnetdata/string/string.h Normal file
View file

@ -0,0 +1,30 @@
#ifndef NETDATA_STRING_H
#define NETDATA_STRING_H 1
#include "../libnetdata.h"
// ----------------------------------------------------------------------------
// STRING implementation
typedef struct netdata_string STRING;
extern STRING *string_strdupz(const char *str);
extern STRING *string_dup(STRING *string);
extern void string_freez(STRING *string);
extern size_t string_strlen(STRING *string);
extern const char *string2str(STRING *string) NEVERNULL;
// keep common prefix/suffix and replace everything else with [x]
extern STRING *string_2way_merge(STRING *a, STRING *b);
static inline int string_cmp(STRING *s1, STRING *s2) {
// STRINGs are deduplicated, so the same strings have the same pointer
// when they differ, we do the typical strcmp() comparison
return (s1 == s2)?0:strcmp(string2str(s1), string2str(s2));
}
extern void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases);
extern int string_unittest(size_t entries);
#endif

29
libnetdata/threads/threads.c
View file

@ -29,26 +29,35 @@ const char *netdata_thread_tag(void) {
// ----------------------------------------------------------------------------
// compatibility library functions
static __thread pid_t gettid_cached_tid = 0;
pid_t gettid(void) {
pid_t tid = 0;
if(likely(gettid_cached_tid > 0))
return gettid_cached_tid;
#ifdef __FreeBSD__
return (pid_t)pthread_getthreadid_np();
tid = (pid_t)pthread_getthreadid_np();
#elif defined(__APPLE__)
#if (defined __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1060)
uint64_t curthreadid;
pthread_threadid_np(NULL, &curthreadid);
return (pid_t)curthreadid;
tid = (pid_t)curthreadid;
#else /* __MAC_OS_X_VERSION_MIN_REQUIRED */
return (pid_t)pthread_self;
tid = (pid_t)pthread_self;
#endif /* __MAC_OS_X_VERSION_MIN_REQUIRED */
#else /* __APPLE__*/
return (pid_t)syscall(SYS_gettid);
tid = (pid_t)syscall(SYS_gettid);
#endif /* __FreeBSD__, __APPLE__*/
gettid_cached_tid = tid;
return tid;
}
// ----------------------------------------------------------------------------
@ -97,6 +106,8 @@ void netdata_threads_init_after_fork(size_t stacksize) {
// ----------------------------------------------------------------------------
// netdata_thread_create
extern void rrdset_thread_rda_free(void);
static void thread_cleanup(void *ptr) {
if(netdata_thread != ptr) {
NETDATA_THREAD *info = (NETDATA_THREAD *)ptr;
@ -106,6 +117,7 @@ static void thread_cleanup(void *ptr) {
if(!(netdata_thread->options & NETDATA_THREAD_OPTION_DONT_LOG_CLEANUP))
info("thread with task id %d finished", gettid());
rrdset_thread_rda_free();
thread_cache_destroy();
freez((void *)netdata_thread->tag);
@ -215,11 +227,18 @@ int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THR
// ----------------------------------------------------------------------------
// netdata_thread_cancel
#ifdef NETDATA_INTERNAL_CHECKS
int netdata_thread_cancel_with_trace(netdata_thread_t thread, int line, const char *file, const char *function) {
#else
int netdata_thread_cancel(netdata_thread_t thread) {
#endif
int ret = pthread_cancel(thread);
if(ret != 0)
#ifdef NETDATA_INTERNAL_CHECKS
error("cannot cancel thread. pthread_cancel() failed with code %d at %d@%s, function %s()", ret, line, file, function);
#else
error("cannot cancel thread. pthread_cancel() failed with code %d.", ret);
#endif
return ret;
}

7
libnetdata/threads/threads.h
View file

@ -28,7 +28,14 @@ extern size_t netdata_threads_init(void);
extern void netdata_threads_init_after_fork(size_t stacksize);
extern int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THREAD_OPTIONS options, void *(*start_routine) (void *), void *arg);
#ifdef NETDATA_INTERNAL_CHECKS
#define netdata_thread_cancel(thread) netdata_thread_cancel_with_trace(thread, __LINE__, __FILE__, __FUNCTION__)
extern int netdata_thread_cancel_with_trace(netdata_thread_t thread, int line, const char *file, const char *function);
#else
extern int netdata_thread_cancel(netdata_thread_t thread);
#endif
extern int netdata_thread_join(netdata_thread_t thread, void **retval);
extern int netdata_thread_detach(pthread_t thread);

2
ml/Dimension.h
View file

@ -44,7 +44,7 @@ public:
RRDDIM *getAnomalyRateRD() const { return AnomalyRateRD; }
void setAnomalyRateRDName(const char *Name) const {
rrddim_set_name(AnomalyRateRD->rrdset, AnomalyRateRD, Name);
rrddim_reset_name(AnomalyRateRD->rrdset, AnomalyRateRD, Name);
}
virtual ~RrdDimension() {}

8
registry/registry.c
View file

@ -108,9 +108,7 @@ static int registry_json_person_url_callback(void *entry, void *data) {
}
// callback for rendering MACHINE_URLs
static int registry_json_machine_url_callback(const char *name, void *entry, void *data) {
(void)name;
static int registry_json_machine_url_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
REGISTRY_MACHINE_URL *mu = (REGISTRY_MACHINE_URL *)entry;
struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data;
struct web_client *w = c->w;
@ -443,8 +441,8 @@ void registry_statistics(void) {
}
else rrdset_next(stm);
rrddim_set(stm, "persons", registry.persons_memory + dictionary_stats_allocated_memory(registry.persons));
rrddim_set(stm, "machines", registry.machines_memory + dictionary_stats_allocated_memory(registry.machines));
rrddim_set(stm, "persons", registry.persons_memory + dictionary_stats_for_registry(registry.persons));
rrddim_set(stm, "machines", registry.machines_memory + dictionary_stats_for_registry(registry.machines));
rrddim_set(stm, "urls", registry.urls_memory);
rrddim_set(stm, "persons_urls", registry.persons_urls_memory);
rrddim_set(stm, "machines_urls", registry.machines_urls_memory);

11
registry/registry_db.c
View file

@ -11,9 +11,7 @@ int registry_db_should_be_saved(void) {
// ----------------------------------------------------------------------------
// INTERNAL FUNCTIONS FOR SAVING REGISTRY OBJECTS
static int registry_machine_save_url(const char *name, void *entry, void *file) {
(void)name;
static int registry_machine_save_url(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
REGISTRY_MACHINE_URL *mu = entry;
FILE *fp = file;
@ -32,8 +30,7 @@ static int registry_machine_save_url(const char *name, void *entry, void *file)
return ret;
}
static int registry_machine_save(const char *name, void *entry, void *file) {
(void)name;
static int registry_machine_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
REGISTRY_MACHINE *m = entry;
FILE *fp = file;
@ -79,9 +76,7 @@ static inline int registry_person_save_url(void *entry, void *file) {
return ret;
}
static inline int registry_person_save(const char *name, void *entry, void *file) {
(void)name;
static inline int registry_person_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
REGISTRY_PERSON *p = entry;
FILE *fp = file;

18
registry/registry_init.c
View file

@ -76,8 +76,8 @@ int registry_init(void) {
netdata_mutex_init(&registry.lock);
// create dictionaries
registry.persons = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
registry.machines = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
registry.persons = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
registry.machines = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
avl_init(&registry.registry_urls_root_index, registry_url_compare);
// load the registry database
@ -93,9 +93,7 @@ int registry_init(void) {
return 0;
}
static int machine_urls_delete_callback(const char *name, void *entry, void *data) {
(void)name;
static int machine_urls_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)data;
(void)m;
@ -110,10 +108,7 @@ static int machine_urls_delete_callback(const char *name, void *entry, void *dat
return 1;
}
static int machine_delete_callback(const char *name, void *entry, void *data) {
(void)name;
(void)data;
static int machine_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data __maybe_unused) {
REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)entry;
int ret = dictionary_walkthrough_read(m->machine_urls, machine_urls_delete_callback, m);
@ -122,10 +117,7 @@ static int machine_delete_callback(const char *name, void *entry, void *data) {
return ret + 1;
}
static int registry_person_del_callback(const char *name, void *entry, void *d) {
(void)name;
(void)d;
static int registry_person_del_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *d __maybe_unused) {
REGISTRY_PERSON *p = (REGISTRY_PERSON *)entry;
debug(D_REGISTRY, "Registry: registry_person_del('%s'): deleting person", p->guid);

2
registry/registry_internals.h
View file

@ -8,7 +8,7 @@
#define REGISTRY_URL_FLAGS_DEFAULT 0x00
#define REGISTRY_URL_FLAGS_EXPIRED 0x01
#define REGISTRY_DICTIONARY_FLAGS (DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE | DICTIONARY_FLAG_NAME_LINK_DONT_CLONE | DICTIONARY_FLAG_SINGLE_THREADED)
#define REGISTRY_DICTIONARY_OPTIONS (DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_SINGLE_THREADED)
// ----------------------------------------------------------------------------
// COMMON structures

10
registry/registry_machine.c
View file

@ -25,9 +25,9 @@ REGISTRY_MACHINE_URL *registry_machine_url_allocate(REGISTRY_MACHINE *m, REGISTR
debug(D_REGISTRY, "registry_machine_url_allocate('%s', '%s'): indexing URL in machine", m->guid, u->url);
registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
dictionary_set(m->machine_urls, u->url, mu, sizeof(REGISTRY_MACHINE_URL));
registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
registry_url_link(u);
@ -42,7 +42,7 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
strncpyz(m->guid, machine_guid, GUID_LEN);
debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating dictionary of urls", machine_guid);
m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
m->first_t = m->last_t = (uint32_t)when;
m->usages = 0;
@ -50,9 +50,9 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
registry.machines_memory += sizeof(REGISTRY_MACHINE);
registry.machines_count++;
registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
dictionary_set(registry.machines, m->guid, m, sizeof(REGISTRY_MACHINE));
registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
return m;
}

30
streaming/rrdpush.c
View file

@ -205,7 +205,7 @@ void rrdpush_send_clabels(RRDHOST *host, RRDSET *st) {
// Send the current chart definition.
// Assumes that collector thread has already called sender_start for mutex / buffer state.
static inline void rrdpush_send_chart_definition_nolock(RRDSET *st) {
static inline void rrdpush_send_chart_definition(RRDSET *st) {
RRDHOST *host = st->rrdhost;
rrdset_flag_set(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
@ -260,26 +260,15 @@ static inline void rrdpush_send_chart_definition_nolock(RRDSET *st) {
, rd->multiplier
, rd->divisor
, rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)?"obsolete":""
, rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)?"hidden":""
, rrddim_flag_check(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)?"noreset":""
, rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN)?"hidden":""
, rrddim_option_check(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS)?"noreset":""
);
rd->exposed = 1;
}
rrddim_foreach_done(rd);
// send the chart local custom variables
RRDSETVAR *rs;
for(rs = st->variables; rs ;rs = rs->next) {
if(unlikely(rs->type == RRDVAR_TYPE_CALCULATED && rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
NETDATA_DOUBLE *value = (NETDATA_DOUBLE *) rs->value;
buffer_sprintf(
host->sender->build
, "VARIABLE CHART %s = " NETDATA_DOUBLE_FORMAT "\n"
, string2str(rs->variable)
, *value
);
}
}
rrdsetvar_print_to_streaming_custom_chart_variables(st, host->sender->build);
st->upstream_resync_time = st->last_collected_time.tv_sec + (remote_clock_resync_iterations * st->update_every);
}
@ -301,6 +290,7 @@ static inline bool rrdpush_send_chart_metrics_nolock(RRDSET *st, struct sender_s
count_of_dimensions_written++;
}
}
rrddim_foreach_done(rd);
buffer_strcat(host->sender->build, "END\n");
return count_of_dimensions_written != 0;
@ -315,11 +305,9 @@ bool rrdset_push_chart_definition_now(RRDSET *st) {
if(unlikely(!host->rrdpush_send_enabled || !should_send_chart_matching(st)))
return false;
rrdset_rdlock(st);
sender_start(host->sender);
rrdpush_send_chart_definition_nolock(st);
rrdpush_send_chart_definition(st);
sender_commit(host->sender);
rrdset_unlock(st);
return true;
}
@ -380,13 +368,13 @@ void rrdset_done_push(RRDSET *st) {
host->rrdpush_sender_error_shown = 0;
}
if(dictionary_stats_entries(st->rrddim_root_index) == 0)
if(dictionary_entries(st->rrddim_root_index) == 0)
return;
sender_start(host->sender);
if(need_to_send_chart_definition(st))
rrdpush_send_chart_definition_nolock(st);
rrdpush_send_chart_definition(st);
if(rrdpush_send_chart_metrics_nolock(st, host->sender)) {
// signal the sender there are more data

2
streaming/rrdpush.h
View file

@ -178,7 +178,7 @@ extern void rrdpush_claimed_id(RRDHOST *host);
extern int rrdpush_receiver_thread_spawn(struct web_client *w, char *url);
extern void rrdpush_sender_thread_stop(RRDHOST *host);
extern void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, RRDVAR *rv);
extern void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
extern void log_stream_connection(const char *client_ip, const char *client_port, const char *api_key, const char *machine_guid, const char *host, const char *msg);
extern int connect_to_one_of_destinations(
struct rrdpush_destinations *destinations,

32
streaming/sender.c
View file

@ -124,33 +124,31 @@ static inline void rrdpush_sender_thread_close_socket(RRDHOST *host) {
}
}
static inline void rrdpush_sender_add_host_variable_to_buffer_nolock(RRDHOST *host, RRDVAR *rv) {
NETDATA_DOUBLE *value = (NETDATA_DOUBLE *)rv->value;
static inline void rrdpush_sender_add_host_variable_to_buffer_nolock(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
buffer_sprintf(
host->sender->build
, "VARIABLE HOST %s = " NETDATA_DOUBLE_FORMAT "\n"
, rrdvar_name(rv)
, *value
, rrdvar_name(rva)
, rrdvar2number(rva)
);
debug(D_STREAM, "RRDVAR pushed HOST VARIABLE %s = " NETDATA_DOUBLE_FORMAT, rrdvar_name(rv), *value);
debug(D_STREAM, "RRDVAR pushed HOST VARIABLE %s = " NETDATA_DOUBLE_FORMAT, rrdvar_name(rva), rrdvar2number(rva));
}
void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, RRDVAR *rv) {
void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
if(host->rrdpush_send_enabled && host->rrdpush_sender_spawn && __atomic_load_n(&host->rrdpush_sender_connected, __ATOMIC_SEQ_CST)) {
sender_start(host->sender);
rrdpush_sender_add_host_variable_to_buffer_nolock(host, rv);
rrdpush_sender_add_host_variable_to_buffer_nolock(host, rva);
sender_commit(host->sender);
}
}
static int rrdpush_sender_thread_custom_host_variables_callback(const char *name __maybe_unused, void *rrdvar_ptr, void *host_ptr) {
RRDVAR *rv = (RRDVAR *)rrdvar_ptr;
static int rrdpush_sender_thread_custom_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar_ptr __maybe_unused, void *host_ptr) {
const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
RRDHOST *host = (RRDHOST *)host_ptr;
if(unlikely(rv->options & RRDVAR_OPTION_CUSTOM_HOST_VAR && rv->type == RRDVAR_TYPE_CALCULATED)) {
if(unlikely(rrdvar_flags(rv) & RRDVAR_FLAG_CUSTOM_HOST_VAR && rrdvar_type(rv) == RRDVAR_TYPE_CALCULATED)) {
rrdpush_sender_add_host_variable_to_buffer_nolock(host, rv);
// return 1, so that the traversal will return the number of variables sent
@ -163,7 +161,7 @@ static int rrdpush_sender_thread_custom_host_variables_callback(const char *name
static void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
sender_start(host->sender);
int ret = rrdvar_walkthrough_read(host->rrdvar_root_index, rrdpush_sender_thread_custom_host_variables_callback, host);
int ret = rrdvar_walkthrough_read(host->rrdvars, rrdpush_sender_thread_custom_host_variables_callback, host);
(void)ret;
sender_commit(host->sender);
@ -173,24 +171,18 @@ static void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
// resets all the chart, so that their definitions
// will be resent to the central netdata
static void rrdpush_sender_thread_reset_all_charts(RRDHOST *host) {
rrdhost_rdlock(host);
RRDSET *st;
rrdset_foreach_read(st, host) {
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
st->upstream_resync_time = 0;
rrdset_rdlock(st);
RRDDIM *rd;
rrddim_foreach_read(rd, st)
rd->exposed = 0;
rrdset_unlock(st);
rrddim_foreach_done(rd);
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
}
static inline void rrdpush_sender_thread_data_flush(RRDHOST *host) {

4
tests/profile/benchmark-dictionary.c
View file

@ -19,8 +19,8 @@ void netdata_cleanup_and_exit(int ret) { exit(ret); }
int main(int argc, char **argv) {
if(argc || argv) {;}
// DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_WITH_STATISTICS);
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_WITH_STATISTICS);
// DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED|DICT_OPTION_WITH_STATISTICS);
DICTIONARY *dict = dictionary_create(DICT_OPTION_STATS);
if(!dict) fatal("Cannot create dictionary.");
struct rusage start, end;

14
web/api/badges/web_buffer_svg.c
View file

@ -893,6 +893,10 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
int group = RRDR_GROUPING_AVERAGE;
uint32_t options = 0x00000000;
const RRDCALC_ACQUIRED *rca = NULL;
RRDCALC *rc = NULL;
RRDSET *st = NULL;
while(url) {
char *value = mystrsep(&url, "&");
if(!value || !*value) continue;
@ -957,7 +961,7 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
int scale = (scale_str && *scale_str)?str2i(scale_str):100;
RRDSET *st = rrdset_find(host, chart);
st = rrdset_find(host, chart);
if(!st) st = rrdset_find_byname(host, chart);
if(!st) {
buffer_no_cacheable(w->response.data);
@ -967,9 +971,10 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
}
st->last_accessed_time = now_realtime_sec();
RRDCALC *rc = NULL;
if(alarm) {
rc = rrdcalc_find(st, alarm);
rca = rrdcalc_from_rrdset_get(st, alarm);
rc = rrdcalc_acquired_to_rrdcalc(rca);
if (!rc) {
buffer_no_cacheable(w->response.data);
buffer_svg(w->response.data, "alarm not found", NAN, "", NULL, NULL, -1, scale, 0, -1, -1, NULL, NULL);
@ -1143,7 +1148,8 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
);
}
cleanup:
cleanup:
rrdcalc_from_rrdset_release(st, rca);
buffer_free(dimensions);
return ret;
}

21
web/api/exporters/shell/allmetrics_shell.c
View file

@ -25,7 +25,6 @@ static inline size_t shell_name_copy(char *d, const char *s, size_t usable) {
void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_string, BUFFER *wb) {
analytics_log_shell();
SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
rrdhost_rdlock(host);
// for each chart
RRDSET *st;
@ -39,8 +38,6 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
buffer_sprintf(wb, "\n# chart: %s (name: %s)\n", rrdset_id(st), rrdset_name(st));
if(rrdset_is_available_for_viewers(st)) {
rrdset_rdlock(st);
// for each dimension
RRDDIM *rd;
rrddim_foreach_read(rd, st) {
@ -55,22 +52,23 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
else {
if(rd->multiplier < 0 || rd->divisor < 0) n = -n;
n = roundndd(n);
if(!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)) total += n;
if(!rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN)) total += n;
buffer_sprintf(wb, "NETDATA_%s_%s=\"" NETDATA_DOUBLE_FORMAT_ZERO "\" # %s\n", chart, dimension, n, rrdset_units(st));
}
}
}
rrddim_foreach_done(rd);
total = roundndd(total);
buffer_sprintf(wb, "NETDATA_%s_VISIBLETOTAL=\"" NETDATA_DOUBLE_FORMAT_ZERO "\" # %s\n", chart, total, rrdset_units(st));
rrdset_unlock(st);
}
}
rrdset_foreach_done(st);
buffer_strcat(wb, "\n# NETDATA ALARMS RUNNING\n");
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(!rc->rrdset) continue;
char chart[SHELL_ELEMENT_MAX + 1];
@ -90,8 +88,8 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
buffer_sprintf(wb, "NETDATA_ALARM_%s_%s_STATUS=\"%s\"\n", chart, alarm, rrdcalc_status2string(rc->status));
}
foreach_rrdcalc_in_rrdhost_done(rc);
rrdhost_unlock(host);
simple_pattern_free(filter);
}
@ -100,7 +98,6 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_string, BUFFER *wb) {
analytics_log_json();
SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
rrdhost_rdlock(host);
buffer_strcat(wb, "{");
@ -114,8 +111,6 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
continue;
if(rrdset_is_available_for_viewers(st)) {
rrdset_rdlock(st);
buffer_sprintf(
wb,
"%s\n"
@ -132,7 +127,7 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
rrdset_family(st),
rrdset_context(st),
rrdset_units(st),
(int64_t)rrdset_last_entry_t_nolock(st));
(int64_t)rrdset_last_entry_t(st));
chart_counter++;
dimension_counter = 0;
@ -161,14 +156,14 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
dimension_counter++;
}
}
rrddim_foreach_done(rd);
buffer_strcat(wb, "\n\t\t}\n\t}");
rrdset_unlock(st);
}
}
rrdset_foreach_done(st);
buffer_strcat(wb, "\n}");
rrdhost_unlock(host);
simple_pattern_free(filter);
}

15
web/api/formatters/charts2json.c
View file

@ -69,7 +69,6 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
);
c = 0;
rrdhost_rdlock(host);
rrdset_foreach_read(st, host) {
if ((!show_archived && rrdset_is_available_for_viewers(st)) || (show_archived && rrdset_is_archived(st))) {
if(c) buffer_strcat(wb, ",");
@ -82,13 +81,14 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
st->last_accessed_time = now;
}
}
rrdset_foreach_done(st);
RRDCALC *rc;
foreach_rrdcalc_in_rrdhost(host, rc) {
foreach_rrdcalc_in_rrdhost_read(host, rc) {
if(rc->rrdset)
alarms++;
}
rrdhost_unlock(host);
foreach_rrdcalc_in_rrdhost_done(rc);
buffer_sprintf(wb
, "\n\t}"
@ -150,9 +150,7 @@ struct array_printer {
BUFFER *wb;
};
static int print_collector_callback(const char *name, void *entry, void *data) {
(void)name;
static int print_collector_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
struct array_printer *ap = (struct array_printer *)data;
BUFFER *wb = ap->wb;
struct collector *col=(struct collector *) entry;
@ -167,12 +165,11 @@ static int print_collector_callback(const char *name, void *entry, void *data) {
}
void chartcollectors2json(RRDHOST *host, BUFFER *wb) {
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
RRDSET *st;
char name[500];
time_t now = now_realtime_sec();
rrdhost_rdlock(host);
rrdset_foreach_read(st, host) {
if (rrdset_is_available_for_viewers(st)) {
struct collector col = {
@ -184,7 +181,7 @@ void chartcollectors2json(RRDHOST *host, BUFFER *wb) {
st->last_accessed_time = now;
}
}
rrdhost_unlock(host);
rrdset_foreach_done(st);
struct array_printer ap = {
.c = 0,
.wb = wb

25
web/api/formatters/json_wrapper.c
View file

@ -7,9 +7,7 @@ struct value_output {
BUFFER *wb;
};
static int value_list_output(const char *name, void *entry, void *data) {
(void)name;
static int value_list_output_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
struct value_output *ap = (struct value_output *)data;
BUFFER *wb = ap->wb;
char *output = (char *) entry;
@ -64,8 +62,6 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
sq[0] = '"';
}
if (should_lock)
rrdset_rdlock(r->st);
buffer_sprintf(wb, "{\n"
" %sapi%s: 1,\n"
" %sid%s: %s%s%s,\n"
@ -83,8 +79,8 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
, kq, kq, sq, context_mode && temp_rd?rrdset_context(r->st):rrdset_name(r->st), sq
, kq, kq, r->update_every
, kq, kq, r->st->update_every
, kq, kq, (uint32_t) (context_param_list ? context_param_list->first_entry_t : rrdset_first_entry_t_nolock(r->st))
, kq, kq, (uint32_t) (context_param_list ? context_param_list->last_entry_t : rrdset_last_entry_t_nolock(r->st))
, kq, kq, (uint32_t) (context_param_list ? context_param_list->first_entry_t : rrdset_first_entry_t(r->st))
, kq, kq, (uint32_t) (context_param_list ? context_param_list->last_entry_t : rrdset_last_entry_t(r->st))
, kq, kq, (uint32_t)r->before
, kq, kq, (uint32_t)r->after
, kq, kq, sq, web_client_api_request_v1_data_group_to_string(group_method), sq
@ -94,9 +90,6 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
buffer_sprintf(wb, "%s,\n %sdimension_names%s: [", sq, kq, kq);
if (should_lock)
rrdset_unlock(r->st);
for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) {
if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue;
if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue;
@ -147,37 +140,37 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
struct value_output co = {.c = 0, .wb = wb};
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
snprintfz(name, RRD_ID_LENGTH_MAX * 2, "%s:%s", rrddim_id(rd), rrddim_name(rd));
int len = snprintfz(output, RRD_ID_LENGTH_MAX * 2 + 7, "[\"%s\",\"%s\"]", rrddim_id(rd), rrddim_name(rd));
dictionary_set(dict, name, output, len+1);
}
dictionary_walkthrough_read(dict, value_list_output, &co);
dictionary_walkthrough_read(dict, value_list_output_callback, &co);
dictionary_destroy(dict);
co.c = 0;
buffer_sprintf(wb, "],\n %sfull_chart_list%s: [", kq, kq);
dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
int len = snprintfz(output, RRD_ID_LENGTH_MAX * 2 + 7, "[\"%s\",\"%s\"]", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset));
snprintfz(name, RRD_ID_LENGTH_MAX * 2, "%s:%s", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset));
dictionary_set(dict, name, output, len + 1);
}
dictionary_walkthrough_read(dict, value_list_output, &co);
dictionary_walkthrough_read(dict, value_list_output_callback, &co);
dictionary_destroy(dict);
RRDSET *st;
co.c = 0;
buffer_sprintf(wb, "],\n %sfull_chart_labels%s: [", kq, kq);
dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
st = rd->rrdset;
if (st->rrdlabels)
rrdlabels_walkthrough_read(st->rrdlabels, fill_formatted_callback, dict);
}
dictionary_walkthrough_read(dict, value_list_output, &co);
dictionary_walkthrough_read(dict, value_list_output_callback, &co);
dictionary_destroy(dict);
buffer_strcat(wb, "],\n");
}

11
web/api/formatters/rrd2json.c
View file

@ -104,13 +104,11 @@ void build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_lis
(*param_list)->rd = NULL;
}
RRDDIM *rd1;
st->last_accessed_time = now_realtime_sec();
rrdset_rdlock(st);
(*param_list)->first_entry_t = MIN((*param_list)->first_entry_t, rrdset_first_entry_t_nolock(st));
(*param_list)->last_entry_t = MAX((*param_list)->last_entry_t, rrdset_last_entry_t_nolock(st));
(*param_list)->first_entry_t = MIN((*param_list)->first_entry_t, rrdset_first_entry_t(st));
(*param_list)->last_entry_t = MAX((*param_list)->last_entry_t, rrdset_last_entry_t(st));
RRDDIM *rd1;
rrddim_foreach_read(rd1, st) {
RRDDIM *rd = onewayalloc_memdupz(owa, rd1, sizeof(RRDDIM));
rd->id = string_dup(rd1->id);
@ -124,8 +122,7 @@ void build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_lis
rd->next = (*param_list)->rd;
(*param_list)->rd = rd;
}
rrdset_unlock(st);
rrddim_foreach_done(rd1);
}
void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb) {

Some files were not shown because too many files have changed in this diff Show more