archive/netdata_netdata
0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-16 18:37:50 +00:00
Code Issues Projects Releases Wiki Activity

Rrdcontext ()

Browse source 
* type checking on dictionary return values

* first STRING implementation, used by DICTIONARY and RRDLABEL

* enable AVL compilation of STRING

* Initial functions to store context info

* Call simple test functions

* Add host_id when getting charts

* Allow host to be null and in this case it will process the localhost

* Simplify init
Do not use strdupz - link directly to sqlite result set

* Init the database during startup

* make it compile - no functionality yet

* intermediate commit

* intermidiate

* first interface to sql

* loading instances

* check if we need to update cloud

* comparison of rrdcontext on conflict

* merge context titles

* rrdcontext public interface; statistics on STRING; scratchpad on DICTIONARY

* dictionaries maintain version numbers; rrdcontext api

* cascading changes

* first operational cleanup

* string unittest

* proper cleanup of referenced dictionaries

* added rrdmetrics

* rrdmetric starting retention

* Add fields to context
Adjuct context creation and delete

* Memory cleanup

* Fix get context list
Fix memory double free in tests
Store context with two hosts

* calculated retention

* rrdcontext retention with collection

* Persist database and shutdown

* loading all from sql

* Get chart list and dimension list changes

* fully working attempt 1

* fully working attempt 2

* missing archived flag from log

* fixed archived / collected

* operational

* proper cleanup

* cleanup - implemented all interface functions - dictionary react callback triggers after the dictionary is unlocked

* track all reasons for changes

* proper tracking of reasons of changes

* fully working thread

* better versioning of contexts

* fix string indexing with AVL

* running version per context vs hub version; ifdef dbengine

* added option to disable rrdmetrics

* release old context when a chart changes context

* cleanup properly

* renamed config

* cleanup contexts; general cleanup;

* deletion inline with dequeue; lots of cleanup; child connected/disconnected

* ml should start after rrdcontext

* added missing NULL to ri->rrdset; rrdcontext flags are now only changed under a mutex lock

* fix buggy STRING under AVL

* Rework database initialization
Add migration logic to the context database

* fix data race conditions during context deletion

* added version hash algorithm

* fix string over AVL

* update aclk-schemas

* compile new ctx related protos

* add ctx stream message utils

* add context messages

* add dummy rx message handlers

* add the new topics

* add ctx capability

* add helper functions to send the new messages

* update cmake build to not fail

* update topic names

* handle rrdcontext_enabled

* add more functions

* fatal on OOM cases instead of return NULL

* silence unknown query type error

* fully working attempt 1

* fully working attempt 2

* allow compiling without ACLK

* added family to the context

* removed excess character in UUID

* smarter merging of titles and families

* Database migration code to add family
Add family to SQL_CHART_DATA and VERSIONED_CONTEXT_DATA

* add family to context message

* enable ctx in communication

* hardcoded enabled contexts

* Add hard code for CTX

* add update node collectors to json

* add context message log

* fix log about last_time_t

* fix collected flags for queued items

* prevent crash on charts cleanup

* fix bug in AVL indexing of dictionaries; make sure react callback of dictionaries has a reference counter, which is acquired while the dictionary is locked

* fixed dictionary unittest

* strict policy to cleanup and garbage collector

* fix db rotation and garbage collection timings

* remove deadlock

* proper garbage collection - a lot faster retention recalculation

* Added not NULL in database columns
Remove migration code for context -- we will ship with version 1 of the table schema
Added define for query in tests to detect localhost

* Use UUID_STR_LEN instead of GUID_LEN + 1
Use realistic timestamps when adding test data in the database

* Add NULL checks for passed parameters

* Log deleted context when compiled with NETDATA_INTERNAL_CHECKS

* Error checking for null host id

* add missing ContextsCheckpoint log convertor

* Fix spelling in VACCUM

* Hold additional information for host -- prepare to load archived hosts on startup

* Make sure claim id is valid

* is_get_claimed is actually get the current claim id

* Simplify ctx get chart list query

* remove env negotiation

* fix string unittest when there are some strings already in the index

* propagate live-retention flag upstream; cleanup all update reasons; updated instances logging; automated attaching started/stopped collecting flags;

* first implementation of /api/v1/contexts

* full contexts API; updated swagger

* disabled debugging; rrdcontext enabled by default

* final cleanup and renaming of global variables

* return current time on currently collected contexts, charts and dimensions

* added option "deepscan" to the API to have the server refresh the retention and recalculate the contexts on the fly

* fixed identation of yaml

* Add constrains to the host table

* host->node_id may not be available

* new capabilities

* lock the context while rendering json

* update aclk-schemas

* added permanent labels to all charts about plugin, module and family; added labels to all proc plugin modules

* always add the labels

* allow merging of families down to [x]

* dont show uuids by default, added option to enable them; response is now accepting after,before to show only data for a specific timeframe; deleted items are only shown when "deleted" is requested; hub version is now shown when "queue" is requested

* Use the localhost claim id

* Fix to handle host constrains better

* cgroups: add "k8s." prefix to chart context in k8s

* Improve sqlite metadata version migration check

* empty values set to "[none]"; fix labels unit test to reflect that

* Check if we reached the version we want first (address CODACY report re: Array index 'i' is used before limits check)

* Rewrite condition to address CODACY report (Redundant condition: t->filter_callback. '!A || (A && B)' is equivalent to '!A || B')

* Properly unlock context

* fixed memory leak on rrdcontexts - it was not freeing all dictionaries in rrdhost; added wait of up to 100ms on dictionary_destroy() to give time to dictionaries to release their items before destroying them

* fixed memory leak on rrdlabels not freed on rrdinstances

* fixed leak when dimensions and charts are redefined

* Mark entries for charts and dimensions as submitted to the cloud 3600 seconds after their creation
Mark entries for charts and dimensions as updated (confirmed by the cloud) 1800 seconds after their submission

* renamed struct string

* update cgroups alarms

* fixed codacy suggestions

* update dashboard info

* fix k8s_cgroup_10s_received_packets_storm alarm

* added filtering options to /api/v1/contexts and /api/v1/context

* fix eslint

* fix eslint

* Fix pointer binding for host / chart uuids

* Fix cgroups unit tests

* fixed non-retention updates not propagated upstream

* removed non-fatal fatals

* Remove context from 2 way string merge.

* Move string_2way_merge to dictionary.c

* Add 2-way string merge tests.

* split long lines

* fix indentation in netdata-swagger.yaml

* update netdata-swagger.json

* yamllint please

* remove the deleted flag when a context is collected

* fix yaml warning in swagger

* removed non-fatal fatals

* charts should now be able to switch contexts

* allow deletion of unused metrics, instances and contexts

* keep the queued flag

* cleanup old rrdinstance labels

* dont hide objects when there is no filter; mark objects as deleted when there are no sub-objects

* delete old instances once they changed context

* delete all instances and contexts that do not have sub-objects

* more precise transitions

* Load archived hosts on startup (part 1)

* update the queued time every time

* disable by default; dedup deleted dimensions after snapshot

* Load archived hosts on startup (part 2)

* delayed processing of events until charts are being collected

* remove dont-trigger flag when object is collected

* polish all triggers given the new dont_process flag

* Remove always true condition
Enums for readbility / create_host_callback only if ACLK is enabled (for now)

* Skip retention message if context streaming is enabled
Add messages in the access log if context streaming is enabled

* Check for node id being a UUID that can be parsed
Improve error check / reporting when loading archived hosts and creating ACLK sync threads

* collected, archived, deleted are now mutually exclusive

* Enable the "orphan" handling for now
Remove dead code
Fix memory leak on free host

* Queue charts and dimensions will be no-op if host is set to stream contexts

* removed unused parameter and made sure flags are set on rrdcontext insert

* make the rrdcontext thread abort mid-work when exiting

* Skip chart hash computation and storage if contexts streaming is enabled

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Timo <timotej@netdata.cloud>
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
Co-authored-by: Vasilis Kalintiris <vasilis@netdata.cloud>
This commit is contained in:
Costa Tsaousis 2022-07-24 22:33:09 +03:00 • committed by GitHub
parent 94040523c7
commit 291b978282
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
82 changed files with 6994 additions and 795 deletions

12
CMakeLists.txt
View file

@ -710,6 +710,8 @@ set(RRD_PLUGIN_FILES
database/rrdcalc.h
database/rrdcalctemplate.c
database/rrdcalctemplate.h
database/rrdcontext.c
database/rrdcontext.h
database/rrddim.c
database/rrddimvar.c
database/rrddimvar.h
@ -729,6 +731,8 @@ set(RRD_PLUGIN_FILES
database/ram/rrddim_mem.h
database/sqlite/sqlite_functions.c
database/sqlite/sqlite_functions.h
database/sqlite/sqlite_context.c
database/sqlite/sqlite_context.h
database/sqlite/sqlite_db_migration.c
database/sqlite/sqlite_db_migration.h
database/sqlite/sqlite_aclk.c
@ -881,6 +885,8 @@ set(ACLK_FILES
aclk/aclk_charts_api.h
aclk/aclk_alarm_api.c
aclk/aclk_alarm_api.h
aclk/aclk_contexts_api.c
aclk/aclk_contexts_api.h
mqtt_websockets/src/mqtt_wss_client.c
mqtt_websockets/src/include/mqtt_wss_client.h
mqtt_websockets/src/mqtt_wss_log.c
@ -917,6 +923,10 @@ set(ACLK_FILES
aclk/schema-wrappers/capability.h
aclk/schema-wrappers/proto_2_json.cc
aclk/schema-wrappers/proto_2_json.h
aclk/schema-wrappers/context_stream.cc
aclk/schema-wrappers/context_stream.h
aclk/schema-wrappers/context.cc
aclk/schema-wrappers/context.h
aclk/schema-wrappers/schema_wrappers.h
aclk/schema-wrappers/schema_wrapper_utils.cc
aclk/schema-wrappers/schema_wrapper_utils.h
@ -1231,6 +1241,8 @@ set(ACLK_PROTO_DEFS
aclk/aclk-schemas/proto/nodeinstance/connection/v1/connection.proto
aclk/aclk-schemas/proto/nodeinstance/create/v1/creation.proto
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto
aclk/aclk-schemas/proto/context/v1/context.proto
aclk/aclk-schemas/proto/context/v1/stream.proto
)
PROTOBUF_ACLK_GENERATE_CPP(ACLK_PROTO_BUILT_SRCS ACLK_PROTO_BUILT_HDRS ${ACLK_PROTO_DEFS})

24
Makefile.am
View file

@ -434,6 +434,8 @@ RRD_PLUGIN_FILES = \
database/rrdcalc.h \
database/rrdcalctemplate.c \
database/rrdcalctemplate.h \
database/rrdcontext.c \
database/rrdcontext.h \
database/rrddim.c \
database/rrddimvar.c \
database/rrddimvar.h \
@ -453,6 +455,8 @@ RRD_PLUGIN_FILES = \
database/ram/rrddim_mem.h \
database/sqlite/sqlite_functions.c \
database/sqlite/sqlite_functions.h \
database/sqlite/sqlite_context.c \
database/sqlite/sqlite_context.h \
database/sqlite/sqlite_db_migration.c \
database/sqlite/sqlite_db_migration.h \
database/sqlite/sqlite_aclk.c \
@ -706,6 +710,8 @@ ACLK_FILES = \
aclk/aclk_charts_api.h \
aclk/aclk_alarm_api.c \
aclk/aclk_alarm_api.h \
aclk/aclk_contexts_api.c \
aclk/aclk_contexts_api.h \
aclk/schema-wrappers/connection.cc \
aclk/schema-wrappers/connection.h \
aclk/schema-wrappers/node_connection.cc \
@ -729,6 +735,10 @@ ACLK_FILES = \
aclk/schema-wrappers/schema_wrappers.h \
aclk/schema-wrappers/schema_wrapper_utils.cc \
aclk/schema-wrappers/schema_wrapper_utils.h \
aclk/schema-wrappers/context_stream.cc \
aclk/schema-wrappers/context_stream.h \
aclk/schema-wrappers/context.cc \
aclk/schema-wrappers/context.h \
$(NULL)
mqtt_websockets/src/mqtt_wss_client.$(OBJEXT) : CFLAGS += -Wno-unused-result
@ -746,6 +756,8 @@ ACLK_PROTO_DEFINITIONS = \
aclk/aclk-schemas/proto/nodeinstance/connection/v1/connection.proto \
aclk/aclk-schemas/proto/nodeinstance/create/v1/creation.proto \
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto \
aclk/aclk-schemas/proto/context/v1/context.proto \
aclk/aclk-schemas/proto/context/v1/stream.proto \
$(NULL)
dist_noinst_DATA += $(ACLK_PROTO_DEFINITIONS)
@ -774,6 +786,10 @@ ACLK_PROTO_BUILT_FILES = aclk/aclk-schemas/proto/agent/v1/connection.pb.cc \
aclk/aclk-schemas/proto/alarm/v1/stream.pb.h \
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.cc \
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.h \
aclk/aclk-schemas/proto/context/v1/context.pb.cc \
aclk/aclk-schemas/proto/context/v1/context.pb.h \
aclk/aclk-schemas/proto/context/v1/stream.pb.cc \
aclk/aclk-schemas/proto/context/v1/stream.pb.h \
$(NULL)
BUILT_SOURCES += $(ACLK_PROTO_BUILT_FILES)
@ -828,6 +844,14 @@ aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.cc \
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.h: aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto
$(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
aclk/aclk-schemas/proto/context/v1/context.pb.cc \
aclk/aclk-schemas/proto/context/v1/context.pb.h: aclk/aclk-schemas/proto/context/v1/context.proto
$(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
aclk/aclk-schemas/proto/context/v1/stream.pb.cc \
aclk/aclk-schemas/proto/context/v1/stream.pb.h: aclk/aclk-schemas/proto/context/v1/stream.proto
$(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
endif #ENABLE_ACLK
ACLK_ALWAYS_BUILD_FILES = \

2
aclk/aclk-schemas

@ -1 +1 @@
Subproject commit fa46ccca237a9bdb613b3b1f2809a25b7b45c7c4
Subproject commit 3252118bd547640251356629f0df05eaf952ac39

32
aclk/aclk.c
View file

@ -141,12 +141,12 @@ static int wait_till_cloud_enabled()
static int wait_till_agent_claimed(void)
{
//TODO prevent malloc and freez
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
while (likely(!agent_id)) {
sleep_usec(USEC_PER_SEC * 1);
if (netdata_exit)
return 1;
agent_id = is_agent_claimed();
agent_id = get_agent_claimid();
}
freez(agent_id);
return 0;
@ -769,6 +769,16 @@ void aclk_host_state_update(RRDHOST *host, int cmd)
};
node_state_update.node_id = mallocz(UUID_STR_LEN);
uuid_unparse_lower(node_id, (char*)node_state_update.node_id);
struct capability caps[] = {
{ .name = "proto", .version = 1, .enabled = 1 },
{ .name = "ml", .version = ml_capable(localhost), .enabled = ml_enabled(host) },
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
{ .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
{ .name = NULL, .version = 0, .enabled = 0 }
};
node_state_update.capabilities = caps;
rrdhost_aclk_state_lock(localhost);
node_state_update.claim_id = localhost->aclk_state.claimed_id;
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@ -801,6 +811,20 @@ void aclk_send_node_instances()
};
node_state_update.node_id = mallocz(UUID_STR_LEN);
uuid_unparse_lower(list->node_id, (char*)node_state_update.node_id);
char host_id[UUID_STR_LEN];
uuid_unparse_lower(list->host_id, host_id);
RRDHOST *host = rrdhost_find_by_guid(host_id, 0);
struct capability caps[] = {
{ .name = "proto", .version = 1, .enabled = 1 },
{ .name = "ml", .version = ml_capable(localhost), .enabled = host ? ml_enabled(host) : 0 },
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
{ .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
{ .name = NULL, .version = 0, .enabled = 0 }
};
node_state_update.capabilities = caps;
rrdhost_aclk_state_lock(localhost);
node_state_update.claim_id = localhost->aclk_state.claimed_id;
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@ -913,7 +937,7 @@ char *ng_aclk_state(void)
);
buffer_sprintf(wb, "Protocol Used: Protobuf\nMQTT Version: %d\nClaimed: ", use_mqtt_5 ? 5 : 3);
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
if (agent_id == NULL)
buffer_strcat(wb, "No\n");
else {
@ -1079,7 +1103,7 @@ char *ng_aclk_state_json(void)
json_object_array_add(grp, tmp);
json_object_object_add(msg, "protocols-supported", grp);
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
tmp = json_object_new_boolean(agent_id != NULL);
json_object_object_add(msg, "agent-claimed", tmp);

1
aclk/aclk_api.c
View file

@ -16,6 +16,7 @@ int aclk_disable_runtime = 0;
int aclk_stats_enabled;
int use_mqtt_5 = 0;
int aclk_ctx_based = 0;
#define ACLK_IMPL_KEY_NAME "aclk implementation"

1
aclk/aclk_api.h
View file

@ -20,6 +20,7 @@ extern int aclk_stats_enabled;
extern int aclk_alert_reloaded;
extern int use_mqtt_5;
extern int aclk_ctx_based;
#ifdef ENABLE_ACLK
void *aclk_starter(void *ptr);

23
aclk/aclk_contexts_api.c Normal file
View file

@ -0,0 +1,23 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#include "aclk_query_queue.h"
#include "aclk_contexts_api.h"
void aclk_send_contexts_snapshot(contexts_snapshot_t data)
{
aclk_query_t query = aclk_query_new(PROTO_BIN_MESSAGE);
query->data.bin_payload.topic = ACLK_TOPICID_CTXS_SNAPSHOT;
query->data.bin_payload.payload = contexts_snapshot_2bin(data, &query->data.bin_payload.size);
query->data.bin_payload.msg_name = "ContextsSnapshot";
QUEUE_IF_PAYLOAD_PRESENT(query);
}
void aclk_send_contexts_updated(contexts_updated_t data)
{
aclk_query_t query = aclk_query_new(PROTO_BIN_MESSAGE);
query->data.bin_payload.topic = ACLK_TOPICID_CTXS_UPDATED;
query->data.bin_payload.payload = contexts_updated_2bin(data, &query->data.bin_payload.size);
query->data.bin_payload.msg_name = "ContextsUpdated";
QUEUE_IF_PAYLOAD_PRESENT(query);
}

12
aclk/aclk_contexts_api.h Normal file
View file

@ -0,0 +1,12 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#ifndef ACLK_CONTEXTS_API_H
#define ACLK_CONTEXTS_API_H
#include "schema-wrappers/schema_wrappers.h"
void aclk_send_contexts_snapshot(contexts_snapshot_t data);
void aclk_send_contexts_updated(contexts_updated_t data);
#endif /* ACLK_CONTEXTS_API_H */

9
aclk/aclk_otp.c
View file

@ -493,7 +493,7 @@ int aclk_get_mqtt_otp(RSA *p_key, char **mqtt_id, char **mqtt_usr, char **mqtt_p
unsigned char *challenge;
int challenge_bytes;
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
if (agent_id == NULL) {
error("Agent was not claimed - cannot perform challenge/response");
return 1;
@ -836,7 +836,7 @@ int aclk_get_env(aclk_env_t *env, const char* aclk_hostname, int aclk_port) {
req.request_type = HTTP_REQ_GET;
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
if (agent_id == NULL)
{
error("Agent was not claimed - cannot perform challenge/response");
@ -844,7 +844,10 @@ int aclk_get_env(aclk_env_t *env, const char* aclk_hostname, int aclk_port) {
return 1;
}
buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
if (rrdcontext_enabled)
buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto,ctx&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
else
buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
freez(agent_id);

1
aclk/aclk_query.c
View file

@ -278,6 +278,7 @@ const char *aclk_query_get_name(aclk_query_type_t qt)
case ALARM_PROVIDE_CFG: return "provide_alarm_config";
case ALARM_SNAPSHOT: return "alarm_snapshot";
case UPDATE_NODE_COLLECTORS: return "update_node_collectors";
case PROTO_BIN_MESSAGE: return "generic_binary_proto_message";
default:
error_report("Unknown query type used %d", (int) qt);
return "unknown";

1
aclk/aclk_query_queue.c
View file

@ -122,6 +122,7 @@ void aclk_query_free(aclk_query_t query)
case ALARM_PROVIDE_CFG:
case ALARM_SNAPSHOT:
case UPDATE_NODE_COLLECTORS:
case PROTO_BIN_MESSAGE:
if (!use_mqtt_5)
freez(query->data.bin_payload.payload);
break;

1
aclk/aclk_query_queue.h
View file

@ -23,6 +23,7 @@ typedef enum {
ALARM_PROVIDE_CFG,
ALARM_SNAPSHOT,
UPDATE_NODE_COLLECTORS,
PROTO_BIN_MESSAGE,
ACLK_QUERY_TYPE_COUNT // always keep this as last
} aclk_query_type_t;

47
aclk/aclk_rx_msgs.c
View file

@ -289,6 +289,15 @@ int create_node_instance_result(const char *msg, size_t msg_len)
}
}
struct capability caps[] = {
{ .name = "proto", .version = 1, .enabled = 1 },
{ .name = "ml", .version = ml_capable(localhost), .enabled = host ? ml_enabled(host) : 0 },
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
{ .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
{ .name = NULL, .version = 0, .enabled = 0 }
};
node_state_update.capabilities = caps;
rrdhost_aclk_state_lock(localhost);
node_state_update.claim_id = localhost->aclk_state.claimed_id;
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@ -313,6 +322,7 @@ int send_node_instances(const char *msg, size_t msg_len)
int stream_charts_and_dimensions(const char *msg, size_t msg_len)
{
aclk_ctx_based = 0;
stream_charts_and_dims_t res = parse_stream_charts_and_dims(msg, msg_len);
if (!res.claim_id || !res.node_id) {
error("Error parsing StreamChartsAndDimensions msg");
@ -426,6 +436,41 @@ int handle_disconnect_req(const char *msg, size_t msg_len)
return 0;
}
int contexts_checkpoint(const char *msg, size_t msg_len)
{
aclk_ctx_based = 1;
struct ctxs_checkpoint *cmd = parse_ctxs_checkpoint(msg, msg_len);
if (!cmd)
return 1;
rrdcontext_hub_checkpoint_command(cmd);
freez(cmd->claim_id);
freez(cmd->node_id);
freez(cmd);
return 0;
}
int stop_streaming_contexts(const char *msg, size_t msg_len)
{
if (!aclk_ctx_based) {
error_report("Received StopStreamingContexts message but context based communication was not enabled (Cloud violated the protocol). Ignoring message");
return 1;
}
struct stop_streaming_ctxs *cmd = parse_stop_streaming_ctxs(msg, msg_len);
if (!cmd)
return 1;
rrdcontext_hub_stop_streaming_command(cmd);
freez(cmd->claim_id);
freez(cmd->node_id);
freez(cmd);
return 0;
}
typedef struct {
const char *name;
simple_hash_t name_hash;
@ -444,6 +489,8 @@ new_cloud_rx_msg_t rx_msgs[] = {
{ .name = "SendAlarmConfiguration", .name_hash = 0, .fnc = send_alarm_configuration },
{ .name = "SendAlarmSnapshot", .name_hash = 0, .fnc = send_alarm_snapshot },
{ .name = "DisconnectReq", .name_hash = 0, .fnc = handle_disconnect_req },
{ .name = "ContextsCheckpoint", .name_hash = 0, .fnc = contexts_checkpoint },
{ .name = "StopStreamingContexts", .name_hash = 0, .fnc = stop_streaming_contexts },
{ .name = NULL, .name_hash = 0, .fnc = NULL },
};

1
aclk/aclk_tx_msgs.c
View file

@ -248,6 +248,7 @@ uint16_t aclk_send_agent_connection_update(mqtt_wss_client client, int reachable
{ .name = "ml", .version = 1, .enabled = ml_enabled(localhost) },
#endif
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
{ .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
{ .name = NULL, .version = 0, .enabled = 0 }
};

4
aclk/aclk_util.c
View file

@ -124,6 +124,8 @@ struct topic_name {
{ .id = ACLK_TOPICID_ALARM_CONFIG, .name = "alarm-config" },
{ .id = ACLK_TOPICID_ALARM_SNAPSHOT, .name = "alarm-snapshot" },
{ .id = ACLK_TOPICID_NODE_COLLECTORS, .name = "node-instance-collectors" },
{ .id = ACLK_TOPICID_CTXS_SNAPSHOT, .name = "contexts-snapshot" },
{ .id = ACLK_TOPICID_CTXS_UPDATED, .name = "contexts-updated" },
{ .id = ACLK_TOPICID_UNKNOWN, .name = NULL }
};
@ -147,6 +149,8 @@ enum aclk_topics compulsory_topics[] = {
ACLK_TOPICID_ALARM_CONFIG,
ACLK_TOPICID_ALARM_SNAPSHOT,
ACLK_TOPICID_NODE_COLLECTORS,
ACLK_TOPICID_CTXS_SNAPSHOT,
ACLK_TOPICID_CTXS_UPDATED,
ACLK_TOPICID_UNKNOWN
};

4
aclk/aclk_util.h
View file

@ -88,7 +88,9 @@ enum aclk_topics {
ACLK_TOPICID_ALARM_HEALTH = 15,
ACLK_TOPICID_ALARM_CONFIG = 16,
ACLK_TOPICID_ALARM_SNAPSHOT = 17,
ACLK_TOPICID_NODE_COLLECTORS = 18
ACLK_TOPICID_NODE_COLLECTORS = 18,
ACLK_TOPICID_CTXS_SNAPSHOT = 19,
ACLK_TOPICID_CTXS_UPDATED = 20
};
const char *aclk_get_topic(enum aclk_topics topic);

125
aclk/schema-wrappers/context.cc Normal file
View file

@ -0,0 +1,125 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#include "proto/context/v1/context.pb.h"
#include "libnetdata/libnetdata.h"
#include "schema_wrapper_utils.h"
#include "context.h"
using namespace context::v1;
// ContextsSnapshot
contexts_snapshot_t contexts_snapshot_new(const char *claim_id, const char *node_id, uint64_t version)
{
ContextsSnapshot *ctxs_snap = new ContextsSnapshot;
if (ctxs_snap == NULL)
fatal("Cannot allocate ContextsSnapshot object. OOM");
ctxs_snap->set_claim_id(claim_id);
ctxs_snap->set_node_id(node_id);
ctxs_snap->set_version(version);
return ctxs_snap;
}
void contexts_snapshot_delete(contexts_snapshot_t snapshot)
{
delete (ContextsSnapshot *)snapshot;
}
void contexts_snapshot_set_version(contexts_snapshot_t ctxs_snapshot, uint64_t version)
{
((ContextsSnapshot *)ctxs_snapshot)->set_version(version);
}
static void fill_ctx_updated(ContextUpdated *ctx, struct context_updated *c_ctx)
{
ctx->set_id(c_ctx->id);
ctx->set_version(c_ctx->version);
ctx->set_first_entry(c_ctx->first_entry);
ctx->set_last_entry(c_ctx->last_entry);
ctx->set_deleted(c_ctx->deleted);
ctx->set_title(c_ctx->title);
ctx->set_priority(c_ctx->priority);
ctx->set_chart_type(c_ctx->chart_type);
ctx->set_units(c_ctx->units);
ctx->set_family(c_ctx->family);
}
void contexts_snapshot_add_ctx_update(contexts_snapshot_t ctxs_snapshot, struct context_updated *ctx_update)
{
ContextsSnapshot *ctxs_snap = (ContextsSnapshot *)ctxs_snapshot;
ContextUpdated *ctx = ctxs_snap->add_contexts();
fill_ctx_updated(ctx, ctx_update);
}
char *contexts_snapshot_2bin(contexts_snapshot_t ctxs_snapshot, size_t *len)
{
ContextsSnapshot *ctxs_snap = (ContextsSnapshot *)ctxs_snapshot;
*len = PROTO_COMPAT_MSG_SIZE_PTR(ctxs_snap);
char *bin = (char*)mallocz(*len);
if (!ctxs_snap->SerializeToArray(bin, *len)) {
freez(bin);
delete ctxs_snap;
return NULL;
}
delete ctxs_snap;
return bin;
}
// ContextsUpdated
contexts_updated_t contexts_updated_new(const char *claim_id, const char *node_id, uint64_t version_hash, uint64_t created_at)
{
ContextsUpdated *ctxs_updated = new ContextsUpdated;
if (ctxs_updated == NULL)
fatal("Cannot allocate ContextsUpdated object. OOM");
ctxs_updated->set_claim_id(claim_id);
ctxs_updated->set_node_id(node_id);
ctxs_updated->set_version_hash(version_hash);
ctxs_updated->set_created_at(created_at);
return ctxs_updated;
}
void contexts_updated_delete(contexts_updated_t ctxs_updated)
{
delete (ContextsUpdated *)ctxs_updated;
}
void contexts_updated_update_version_hash(contexts_updated_t ctxs_updated, uint64_t version_hash)
{
((ContextsUpdated *)ctxs_updated)->set_version_hash(version_hash);
}
void contexts_updated_add_ctx_update(contexts_updated_t ctxs_updated, struct context_updated *ctx_update)
{
ContextsUpdated *ctxs_update = (ContextsUpdated *)ctxs_updated;
ContextUpdated *ctx = ctxs_update->add_contextupdates();
if (ctx == NULL)
fatal("Cannot allocate ContextUpdated object. OOM");
fill_ctx_updated(ctx, ctx_update);
}
char *contexts_updated_2bin(contexts_updated_t ctxs_updated, size_t *len)
{
ContextsUpdated *ctxs_update = (ContextsUpdated *)ctxs_updated;
*len = PROTO_COMPAT_MSG_SIZE_PTR(ctxs_update);
char *bin = (char*)mallocz(*len);
if (!ctxs_update->SerializeToArray(bin, *len)) {
freez(bin);
delete ctxs_update;
return NULL;
}
delete ctxs_update;
return bin;
}

53
aclk/schema-wrappers/context.h Normal file
View file

@ -0,0 +1,53 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#ifndef ACLK_SCHEMA_WRAPPER_CONTEXT_H
#define ACLK_SCHEMA_WRAPPER_CONTEXT_H
#include <stdint.h>
#include <sys/types.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef void* contexts_updated_t;
typedef void* contexts_snapshot_t;
struct context_updated {
// context id
const char *id;
uint64_t version;
uint64_t first_entry;
uint64_t last_entry;
int deleted;
const char *title;
uint64_t priority;
const char *chart_type;
const char *units;
const char *family;
};
// ContextS Snapshot related
contexts_snapshot_t contexts_snapshot_new(const char *claim_id, const char *node_id, uint64_t version);
void contexts_snapshot_delete(contexts_snapshot_t ctxs_snapshot);
void contexts_snapshot_set_version(contexts_snapshot_t ctxs_snapshot, uint64_t version);
void contexts_snapshot_add_ctx_update(contexts_snapshot_t ctxs_snapshot, struct context_updated *ctx_update);
char *contexts_snapshot_2bin(contexts_snapshot_t ctxs_snapshot, size_t *len);
// ContextS Updated related
contexts_updated_t contexts_updated_new(const char *claim_id, const char *node_id, uint64_t version_hash, uint64_t created_at);
void contexts_updated_delete(contexts_updated_t ctxs_updated);
void contexts_updated_update_version_hash(contexts_updated_t ctxs_updated, uint64_t version_hash);
void contexts_updated_add_ctx_update(contexts_updated_t ctxs_updated, struct context_updated *ctx_update);
char *contexts_updated_2bin(contexts_updated_t ctxs_updated, size_t *len);
#ifdef __cplusplus
}
#endif
#endif /* ACLK_SCHEMA_WRAPPER_CONTEXT_H */

42
aclk/schema-wrappers/context_stream.cc Normal file
View file

@ -0,0 +1,42 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#include "proto/context/v1/stream.pb.h"
#include "context_stream.h"
#include "libnetdata/libnetdata.h"
struct stop_streaming_ctxs *parse_stop_streaming_ctxs(const char *data, size_t len)
{
context::v1::StopStreamingContexts msg;
struct stop_streaming_ctxs *res;
if (!msg.ParseFromArray(data, len))
return NULL;
res = (struct stop_streaming_ctxs *)callocz(1, sizeof(struct stop_streaming_ctxs));
res->claim_id = strdupz(msg.claim_id().c_str());
res->node_id = strdupz(msg.node_id().c_str());
return res;
}
struct ctxs_checkpoint *parse_ctxs_checkpoint(const char *data, size_t len)
{
context::v1::ContextsCheckpoint msg;
struct ctxs_checkpoint *res;
if (!msg.ParseFromArray(data, len))
return NULL;
res = (struct ctxs_checkpoint *)callocz(1, sizeof(struct ctxs_checkpoint));
res->claim_id = strdupz(msg.claim_id().c_str());
res->node_id = strdupz(msg.node_id().c_str());
res->version_hash = msg.version_hash();
return res;
}

36
aclk/schema-wrappers/context_stream.h Normal file
View file

@ -0,0 +1,36 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#ifndef ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H
#define ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H
#ifdef __cplusplus
extern "C" {
#endif
struct stop_streaming_ctxs {
char *claim_id;
char *node_id;
// we omit reason as there is only one defined at this point
// as soon as there is more than one defined in StopStreaminContextsReason
// we should add it
// 0 - RATE_LIMIT_EXCEEDED
};
struct stop_streaming_ctxs *parse_stop_streaming_ctxs(const char *data, size_t len);
struct ctxs_checkpoint {
char *claim_id;
char *node_id;
uint64_t version_hash;
};
struct ctxs_checkpoint *parse_ctxs_checkpoint(const char *data, size_t len);
#ifdef __cplusplus
}
#endif
#endif /* ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H */

9
aclk/schema-wrappers/node_connection.cc
View file

@ -28,6 +28,15 @@ char *generate_node_instance_connection(size_t *len, const node_instance_connect
timestamp->set_seconds(tv.tv_sec);
timestamp->set_nanos(tv.tv_usec * 1000);
if (data->capabilities) {
struct capability *capa = data->capabilities;
while (capa->name) {
aclk_lib::v1::Capability *proto_capa = msg.add_capabilities();
capability_set(proto_capa, capa);
capa++;
}
}
*len = PROTO_COMPAT_MSG_SIZE(msg);
char *bin = (char*)malloc(*len);
if (bin)

3
aclk/schema-wrappers/node_connection.h
View file

@ -3,6 +3,8 @@
#ifndef ACLK_SCHEMA_WRAPPER_NODE_CONNECTION_H
#define ACLK_SCHEMA_WRAPPER_NODE_CONNECTION_H
#include "capability.h"
#ifdef __cplusplus
extern "C" {
#endif
@ -17,6 +19,7 @@ typedef struct {
int64_t session_id;
int32_t hops;
struct capability *capabilities;
} node_instance_connection_t;
char *generate_node_instance_connection(size_t *len, const node_instance_connection_t *data);

12
aclk/schema-wrappers/proto_2_json.cc
View file

@ -11,6 +11,8 @@
#include "proto/nodeinstance/connection/v1/connection.pb.h"
#include "proto/nodeinstance/create/v1/creation.pb.h"
#include "proto/nodeinstance/info/v1/info.pb.h"
#include "proto/context/v1/stream.pb.h"
#include "proto/context/v1/context.pb.h"
#include "libnetdata/libnetdata.h"
@ -45,6 +47,12 @@ static google::protobuf::Message *msg_name_to_protomsg(const char *msgname)
return new alarms::v1::AlarmSnapshot;
if (!strcmp(msgname, "AlarmLogEntry"))
return new alarms::v1::AlarmLogEntry;
if (!strcmp(msgname, "UpdateNodeCollectors"))
return new nodeinstance::info::v1::UpdateNodeCollectors;
if (!strcmp(msgname, "ContextsUpdated"))
return new context::v1::ContextsUpdated;
if (!strcmp(msgname, "ContextsSnapshot"))
return new context::v1::ContextsSnapshot;
//rx side
if (!strcmp(msgname, "CreateNodeInstanceResult"))
@ -67,6 +75,10 @@ static google::protobuf::Message *msg_name_to_protomsg(const char *msgname)
return new alarms::v1::SendAlarmSnapshot;
if (!strcmp(msgname, "DisconnectReq"))
return new agent::v1::DisconnectReq;
if (!strcmp(msgname, "ContextsCheckpoint"))
return new context::v1::ContextsCheckpoint;
if (!strcmp(msgname, "StopStreamingContexts"))
return new context::v1::StopStreamingContexts;
return NULL;
}

2
aclk/schema-wrappers/schema_wrappers.h
View file

@ -14,5 +14,7 @@
#include "alarm_stream.h"
#include "node_info.h"
#include "capability.h"
#include "context_stream.h"
#include "context.h"
#endif /* SCHEMA_WRAPPERS_H */

2
claim/claim.c
View file

@ -31,7 +31,7 @@ static char *claiming_errors[] = {
/* Retrieve the claim id for the agent.
* Caller owns the string.
*/
char *is_agent_claimed()
char *get_agent_claimid()
{
char *result;
rrdhost_aclk_state_lock(localhost);

2
claim/claim.h
View file

@ -9,7 +9,7 @@ extern char *claiming_pending_arguments;
extern struct config cloud_config;
void claim_agent(char *claiming_arguments);
char *is_agent_claimed(void);
char *get_agent_claimid(void);
void load_claiming_state(void);
void load_cloud_conf(int silent);

87
collectors/cgroups.plugin/sys_fs_cgroup.c
View file

@ -768,6 +768,12 @@ struct cgroup_network_interface {
struct cgroup_network_interface *next;
};
enum cgroups_container_orchestrator {
CGROUPS_ORCHESTRATOR_UNSET,
CGROUPS_ORCHESTRATOR_UNKNOWN,
CGROUPS_ORCHESTRATOR_K8S
};
// *** WARNING *** The fields are not thread safe. Take care of safe usage.
struct cgroup {
uint32_t options;
@ -791,6 +797,8 @@ struct cgroup {
DICTIONARY *chart_labels;
int container_orchestrator;
struct cpuacct_stat cpuacct_stat;
struct cpuacct_usage cpuacct_usage;
struct cpuacct_cpu_throttling cpuacct_cpu_throttling;
@ -935,6 +943,10 @@ static inline int is_cgroup_systemd_service(struct cgroup *cg) {
}
// ---------------------------------------------------------------------------------------------
static int k8s_is_kubepod(struct cgroup *cg) {
return cg->container_orchestrator == CGROUPS_ORCHESTRATOR_K8S;
}
static int k8s_is_container(const char *id) {
// examples:
// https://github.com/netdata/netdata/blob/0fc101679dcd12f1cb8acdd07bb4c85d8e553e53/collectors/cgroups.plugin/cgroup-name.sh#L121-L147
@ -1690,7 +1702,8 @@ static inline void read_cgroup_network_interfaces(struct cgroup *cg) {
info("CGROUP: cgroup '%s' has network interface '%s' as '%s'", cg->id, i->host_device, i->container_device);
// register a device rename to proc_net_dev.c
netdev_rename_device_add(i->host_device, i->container_device, cg->chart_id, cg->chart_labels);
netdev_rename_device_add(
i->host_device, i->container_device, cg->chart_id, cg->chart_labels, k8s_is_kubepod(cg) ? "k8s." : "");
}
}
@ -2589,6 +2602,14 @@ static inline void discovery_process_first_time_seen_cgroup(struct cgroup *cg) {
char comm[TASK_COMM_LEN];
if (cg->container_orchestrator == CGROUPS_ORCHESTRATOR_UNSET) {
if (strstr(cg->id, "kubepods")) {
cg->container_orchestrator = CGROUPS_ORCHESTRATOR_K8S;
} else {
cg->container_orchestrator = CGROUPS_ORCHESTRATOR_UNKNOWN;
}
}
if (is_inside_k8s && !k8s_get_container_first_proc_comm(cg->id, comm)) {
// container initialization may take some time when CPU % is high
// seen on GKE: comm is '6' before 'runc:[2:INIT]' (dunno if it could be another number)
@ -3782,7 +3803,7 @@ void update_cgroup_charts(int update_every) {
, "cpu"
, NULL
, "cpu"
, "cgroup.cpu"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu" : "cgroup.cpu"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -3855,7 +3876,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_limit"
, NULL
, "cpu"
, "cgroup.cpu_limit"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_limit" : "cgroup.cpu_limit"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -3908,7 +3929,7 @@ void update_cgroup_charts(int update_every) {
, "throttled"
, NULL
, "cpu"
, "cgroup.throttled"
, k8s_is_kubepod(cg) ? "k8s.cgroup.throttled" : "cgroup.throttled"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -3934,7 +3955,7 @@ void update_cgroup_charts(int update_every) {
, "throttled_duration"
, NULL
, "cpu"
, "cgroup.throttled_duration"
, k8s_is_kubepod(cg) ? "k8s.cgroup.throttled_duration" : "cgroup.throttled_duration"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -3962,7 +3983,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_shares"
, NULL
, "cpu"
, "cgroup.cpu_shares"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_shares" : "cgroup.cpu_shares"
, title
, "shares"
, PLUGIN_CGROUPS_NAME
@ -3993,7 +4014,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_per_core"
, NULL
, "cpu"
, "cgroup.cpu_per_core"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_per_core" : "cgroup.cpu_per_core"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4029,7 +4050,7 @@ void update_cgroup_charts(int update_every) {
, "mem"
, NULL
, "mem"
, "cgroup.mem"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem" : "cgroup.mem"
, title
, "MiB"
, PLUGIN_CGROUPS_NAME
@ -4089,7 +4110,7 @@ void update_cgroup_charts(int update_every) {
, "writeback"
, NULL
, "mem"
, "cgroup.writeback"
, k8s_is_kubepod(cg) ? "k8s.cgroup.writeback" : "cgroup.writeback"
, title
, "MiB"
, PLUGIN_CGROUPS_NAME
@ -4124,7 +4145,7 @@ void update_cgroup_charts(int update_every) {
, "mem_activity"
, NULL
, "mem"
, "cgroup.mem_activity"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem_activity" : "cgroup.mem_activity"
, title
, "MiB/s"
, PLUGIN_CGROUPS_NAME
@ -4155,7 +4176,7 @@ void update_cgroup_charts(int update_every) {
, "pgfaults"
, NULL
, "mem"
, "cgroup.pgfaults"
, k8s_is_kubepod(cg) ? "k8s.cgroup.pgfaults" : "cgroup.pgfaults"
, title
, "MiB/s"
, PLUGIN_CGROUPS_NAME
@ -4187,7 +4208,7 @@ void update_cgroup_charts(int update_every) {
, "mem_usage"
, NULL
, "mem"
, "cgroup.mem_usage"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem_usage" : "cgroup.mem_usage"
, title
, "MiB"
, PLUGIN_CGROUPS_NAME
@ -4254,7 +4275,7 @@ void update_cgroup_charts(int update_every) {
, "mem_usage_limit"
, NULL
, "mem"
, "cgroup.mem_usage_limit"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem_usage_limit": "cgroup.mem_usage_limit"
, title
, "MiB"
, PLUGIN_CGROUPS_NAME
@ -4286,7 +4307,7 @@ void update_cgroup_charts(int update_every) {
, "mem_utilization"
, NULL
, "mem"
, "cgroup.mem_utilization"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem_utilization" : "cgroup.mem_utilization"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4335,7 +4356,7 @@ void update_cgroup_charts(int update_every) {
, "mem_failcnt"
, NULL
, "mem"
, "cgroup.mem_failcnt"
, k8s_is_kubepod(cg) ? "k8s.cgroup.mem_failcnt" : "cgroup.mem_failcnt"
, title
, "count"
, PLUGIN_CGROUPS_NAME
@ -4365,7 +4386,7 @@ void update_cgroup_charts(int update_every) {
, "io"
, NULL
, "disk"
, "cgroup.io"
, k8s_is_kubepod(cg) ? "k8s.cgroup.io" : "cgroup.io"
, title
, "KiB/s"
, PLUGIN_CGROUPS_NAME
@ -4397,7 +4418,7 @@ void update_cgroup_charts(int update_every) {
, "serviced_ops"
, NULL
, "disk"
, "cgroup.serviced_ops"
, k8s_is_kubepod(cg) ? "k8s.cgroup.serviced_ops" : "cgroup.serviced_ops"
, title
, "operations/s"
, PLUGIN_CGROUPS_NAME
@ -4429,7 +4450,7 @@ void update_cgroup_charts(int update_every) {
, "throttle_io"
, NULL
, "disk"
, "cgroup.throttle_io"
, k8s_is_kubepod(cg) ? "k8s.cgroup.throttle_io" : "cgroup.throttle_io"
, title
, "KiB/s"
, PLUGIN_CGROUPS_NAME
@ -4461,7 +4482,7 @@ void update_cgroup_charts(int update_every) {
, "throttle_serviced_ops"
, NULL
, "disk"
, "cgroup.throttle_serviced_ops"
, k8s_is_kubepod(cg) ? "k8s.cgroup.throttle_serviced_ops" : "cgroup.throttle_serviced_ops"
, title
, "operations/s"
, PLUGIN_CGROUPS_NAME
@ -4493,7 +4514,7 @@ void update_cgroup_charts(int update_every) {
, "queued_ops"
, NULL
, "disk"
, "cgroup.queued_ops"
, k8s_is_kubepod(cg) ? "k8s.cgroup.queued_ops" : "cgroup.queued_ops"
, title
, "operations"
, PLUGIN_CGROUPS_NAME
@ -4525,7 +4546,7 @@ void update_cgroup_charts(int update_every) {
, "merged_ops"
, NULL
, "disk"
, "cgroup.merged_ops"
, k8s_is_kubepod(cg) ? "k8s.cgroup.merged_ops" : "cgroup.merged_ops"
, title
, "operations/s"
, PLUGIN_CGROUPS_NAME
@ -4563,7 +4584,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_some_pressure"
, NULL
, "cpu"
, "cgroup.cpu_some_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_some_pressure" : "cgroup.cpu_some_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4587,7 +4608,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_some_pressure_stall_time"
, NULL
, "cpu"
, "cgroup.cpu_some_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_some_pressure_stall_time" : "cgroup.cpu_some_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -4615,7 +4636,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_full_pressure"
, NULL
, "cpu"
, "cgroup.cpu_full_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_full_pressure" : "cgroup.cpu_full_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4639,7 +4660,7 @@ void update_cgroup_charts(int update_every) {
, "cpu_full_pressure_stall_time"
, NULL
, "cpu"
, "cgroup.cpu_full_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_full_pressure_stall_time" : "cgroup.cpu_full_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -4670,7 +4691,7 @@ void update_cgroup_charts(int update_every) {
, "mem_some_pressure"
, NULL
, "mem"
, "cgroup.memory_some_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.memory_some_pressure" : "cgroup.memory_some_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4694,7 +4715,7 @@ void update_cgroup_charts(int update_every) {
, "memory_some_pressure_stall_time"
, NULL
, "mem"
, "cgroup.memory_some_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.memory_some_pressure_stall_time" : "cgroup.memory_some_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -4724,7 +4745,7 @@ void update_cgroup_charts(int update_every) {
, "mem_full_pressure"
, NULL
, "mem"
, "cgroup.memory_full_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.memory_full_pressure" : "cgroup.memory_full_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4749,7 +4770,7 @@ void update_cgroup_charts(int update_every) {
, "memory_full_pressure_stall_time"
, NULL
, "mem"
, "cgroup.memory_full_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.memory_full_pressure_stall_time" : "cgroup.memory_full_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -4780,7 +4801,7 @@ void update_cgroup_charts(int update_every) {
, "io_some_pressure"
, NULL
, "disk"
, "cgroup.io_some_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.io_some_pressure" : "cgroup.io_some_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4804,7 +4825,7 @@ void update_cgroup_charts(int update_every) {
, "io_some_pressure_stall_time"
, NULL
, "disk"
, "cgroup.io_some_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.io_some_pressure_stall_time" : "cgroup.io_some_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME
@ -4833,7 +4854,7 @@ void update_cgroup_charts(int update_every) {
, "io_full_pressure"
, NULL
, "disk"
, "cgroup.io_full_pressure"
, k8s_is_kubepod(cg) ? "k8s.cgroup.io_full_pressure" : "cgroup.io_full_pressure"
, title
, "percentage"
, PLUGIN_CGROUPS_NAME
@ -4857,7 +4878,7 @@ void update_cgroup_charts(int update_every) {
, "io_full_pressure_stall_time"
, NULL
, "disk"
, "cgroup.io_full_pressure_stall_time"
, k8s_is_kubepod(cg) ? "k8s.cgroup.io_full_pressure_stall_time" : "cgroup.io_full_pressure_stall_time"
, title
, "ms"
, PLUGIN_CGROUPS_NAME

3
collectors/cgroups.plugin/tests/test_doubles.c
View file

@ -132,12 +132,13 @@ void update_pressure_charts(struct pressure_charts *charts)
}
void netdev_rename_device_add(
const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels)
const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels, const char *ctx_prefix)
{
UNUSED(host_device);
UNUSED(container_device);
UNUSED(container_name);
UNUSED(labels);
UNUSED(ctx_prefix);
}
void netdev_rename_device_del(const char *host_device)

12
collectors/cups.plugin/cups_plugin.c
View file

@ -162,12 +162,12 @@ struct job_metrics *get_job_metrics(char *dest) {
reset_job_metrics(NULL, &new_job_metrics, NULL);
jm = dictionary_set(dict_dest_job_metrics, dest, &new_job_metrics, sizeof(struct job_metrics));
printf("CHART cups.job_num_%s '' 'Active job number of destination %s' jobs '%s' cups.job_num stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
printf("CHART cups.job_num_%s '' 'Active jobs of %s' jobs '%s' cups.job_num stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
printf("CHART cups.job_size_%s '' 'Active job size of destination %s' KB '%s' cups.job_size stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
printf("CHART cups.job_size_%s '' 'Active jobs size of %s' KB '%s' cups.job_size stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
@ -196,12 +196,12 @@ int collect_job_metrics(const char *name, void *entry, void *data) {
"END\n",
name, jm->size_pending, jm->size_held, jm->size_processing);
} else {
printf("CHART cups.job_num_%s '' 'Active job number of destination %s' jobs '%s' cups.job_num stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
printf("CHART cups.job_num_%s '' 'Active jobs of %s' jobs '%s' cups.job_num stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
printf("CHART cups.job_size_%s '' 'Active job size of destination %s' KB '%s' cups.job_size stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
printf("CHART cups.job_size_%s '' 'Active jobs size of %s' KB '%s' cups.job_size stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
@ -387,12 +387,12 @@ int main(int argc, char **argv) {
printf("DIMENSION acceptingjobs '' absolute 1 1\n");
printf("DIMENSION shared '' absolute 1 1\n");
printf("CHART cups.job_num '' 'Total active job number' jobs overview cups.job_num stacked 100002 %i\n", netdata_update_every);
printf("CHART cups.job_num '' 'Active jobs' jobs overview cups.job_num stacked 100002 %i\n", netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");
printf("CHART cups.job_size '' 'Total active job size' KB overview cups.job_size stacked 100003 %i\n", netdata_update_every);
printf("CHART cups.job_size '' 'Active jobs size' KB overview cups.job_size stacked 100003 %i\n", netdata_update_every);
printf("DIMENSION pending '' absolute 1 1\n");
printf("DIMENSION held '' absolute 1 1\n");
printf("DIMENSION processing '' absolute 1 1\n");

4
collectors/diskspace.plugin/plugin_diskspace.c
View file

@ -431,7 +431,9 @@ static inline void do_disk_space_stats(struct mountinfo *mi, int update_every) {
};
mp.chart_labels = rrdlabels_create();
rrdlabels_add(mp.chart_labels, "fstype", mi->filesystem, RRDLABEL_SRC_AUTO);
rrdlabels_add(mp.chart_labels, "mount_point", mi->mount_point, RRDLABEL_SRC_AUTO);
rrdlabels_add(mp.chart_labels, "filesystem", mi->filesystem, RRDLABEL_SRC_AUTO);
rrdlabels_add(mp.chart_labels, "mount_root", mi->root, RRDLABEL_SRC_AUTO);
m = dictionary_set(dict_mountpoints, mi->mount_point, &mp, sizeof(struct mount_point_metadata));

7
collectors/proc.plugin/plugin_proc.h
View file

@ -54,7 +54,12 @@ extern unsigned long long zfs_arcstats_shrinkable_cache_size_bytes;
// netdev renames
extern void netdev_rename_device_add(
const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels);
const char *host_device,
const char *container_device,
const char *container_name,
DICTIONARY *labels,
const char *ctx_prefix);
extern void netdev_rename_device_del(const char *host_device);
#include "proc_self_mountinfo.h"

125
collectors/proc.plugin/proc_diskstats.c
View file

@ -830,6 +830,30 @@ static struct disk *get_disk(unsigned long major, unsigned long minor, char *dis
return d;
}
static void add_labels_to_disk(struct disk *d, RRDSET *st) {
rrdlabels_add(st->state->chart_labels, "device", d->disk, RRDLABEL_SRC_AUTO);
rrdlabels_add(st->state->chart_labels, "mount_point", d->mount_point, RRDLABEL_SRC_AUTO);
switch (d->type) {
default:
case DISK_TYPE_UNKNOWN:
rrdlabels_add(st->state->chart_labels, "device_type", "unknown", RRDLABEL_SRC_AUTO);
break;
case DISK_TYPE_PHYSICAL:
rrdlabels_add(st->state->chart_labels, "device_type", "physical", RRDLABEL_SRC_AUTO);
break;
case DISK_TYPE_PARTITION:
rrdlabels_add(st->state->chart_labels, "device_type", "partition", RRDLABEL_SRC_AUTO);
break;
case DISK_TYPE_VIRTUAL:
rrdlabels_add(st->state->chart_labels, "device_type", "virtual", RRDLABEL_SRC_AUTO);
break;
}
}
int do_proc_diskstats(int update_every, usec_t dt) {
static procfile *ff = NULL;
@ -1067,6 +1091,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_io_reads = rrddim_add(d->st_io, "reads", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
d->rd_io_writes = rrddim_add(d->st_io, "writes", NULL, d->sector_size * -1, 1024, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_io);
}
else rrdset_next(d->st_io);
@ -1094,8 +1120,9 @@ int do_proc_diskstats(int update_every, usec_t dt) {
, RRDSET_TYPE_AREA
);
d->rd_io_discards =
rrddim_add(d->st_ext_io, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
d->rd_io_discards = rrddim_add(d->st_ext_io, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_ext_io);
} else
rrdset_next(d->st_ext_io);
@ -1130,6 +1157,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_ops_reads = rrddim_add(d->st_ops, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_ops_writes = rrddim_add(d->st_ops, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_ops);
}
else rrdset_next(d->st_ops);
@ -1162,7 +1191,10 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_ops_discards = rrddim_add(d->st_ext_ops, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
if (do_fl_stats)
d->rd_ops_flushes = rrddim_add(d->st_ext_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
} else
add_labels_to_disk(d, d->st_ext_ops);
}
else
rrdset_next(d->st_ext_ops);
last_discards = rrddim_set_by_pointer(d->st_ext_ops, d->rd_ops_discards, discards);
@ -1196,6 +1228,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_qops, RRDSET_FLAG_DETAIL);
d->rd_qops_operations = rrddim_add(d->st_qops, "operations", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_qops);
}
else rrdset_next(d->st_qops);
@ -1228,6 +1262,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_backlog, RRDSET_FLAG_DETAIL);
d->rd_backlog_backlog = rrddim_add(d->st_backlog, "backlog", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_backlog);
}
else rrdset_next(d->st_backlog);
@ -1259,8 +1295,9 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_busy, RRDSET_FLAG_DETAIL);
d->rd_busy_busy =
rrddim_add(d->st_busy, "busy", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_busy_busy = rrddim_add(d->st_busy, "busy", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_busy);
}
else rrdset_next(d->st_busy);
@ -1288,6 +1325,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_util, RRDSET_FLAG_DETAIL);
d->rd_util_utilization = rrddim_add(d->st_util, "utilization", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_util);
}
else rrdset_next(d->st_util);
@ -1326,6 +1365,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_mops_reads = rrddim_add(d->st_mops, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_mops_writes = rrddim_add(d->st_mops, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_mops);
}
else rrdset_next(d->st_mops);
@ -1358,7 +1399,10 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_ext_mops, RRDSET_FLAG_DETAIL);
d->rd_mops_discards = rrddim_add(d->st_ext_mops, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
} else
add_labels_to_disk(d, d->st_ext_mops);
}
else
rrdset_next(d->st_ext_mops);
rrddim_set_by_pointer(d->st_ext_mops, d->rd_mops_discards, mdiscards);
@ -1391,6 +1435,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_iotime_reads = rrddim_add(d->st_iotime, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_iotime_writes = rrddim_add(d->st_iotime, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_iotime);
}
else rrdset_next(d->st_iotime);
@ -1422,9 +1468,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_iotime_discards = rrddim_add(d->st_ext_iotime, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
if (do_fl_stats)
d->rd_iotime_flushes =
rrddim_add(d->st_ext_iotime, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
} else
d->rd_iotime_flushes = rrddim_add(d->st_ext_iotime, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_ext_iotime);
}
else
rrdset_next(d->st_ext_iotime);
last_discardms = rrddim_set_by_pointer(d->st_ext_iotime, d->rd_iotime_discards, discardms);
@ -1465,6 +1513,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_await_reads = rrddim_add(d->st_await, "reads", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
d->rd_await_writes = rrddim_add(d->st_await, "writes", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_await);
}
else rrdset_next(d->st_await);
@ -1494,18 +1544,22 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_await_discards = rrddim_add(d->st_ext_await, "discards", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
if (do_fl_stats)
d->rd_await_flushes =
rrddim_add(d->st_ext_await, "flushes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
} else
d->rd_await_flushes = rrddim_add(d->st_ext_await, "flushes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_ext_await);
}
else
rrdset_next(d->st_ext_await);
rrddim_set_by_pointer(
d->st_ext_await, d->rd_await_discards,
(discards - last_discards) ? (discardms - last_discardms) / (discards - last_discards) : 0);
if (do_fl_stats)
rrddim_set_by_pointer(
d->st_ext_await, d->rd_await_flushes,
(flushes - last_flushes) ? (flushms - last_flushms) / (flushes - last_flushes) : 0);
rrdset_done(d->st_ext_await);
}
@ -1534,6 +1588,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_avgsz_reads = rrddim_add(d->st_avgsz, "reads", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
d->rd_avgsz_writes = rrddim_add(d->st_avgsz, "writes", NULL, d->sector_size * -1, 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_avgsz);
}
else rrdset_next(d->st_avgsz);
@ -1561,9 +1617,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_ext_avgsz, RRDSET_FLAG_DETAIL);
d->rd_avgsz_discards =
rrddim_add(d->st_ext_avgsz, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
} else
d->rd_avgsz_discards = rrddim_add(d->st_ext_avgsz, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_ext_avgsz);
}
else
rrdset_next(d->st_ext_avgsz);
rrddim_set_by_pointer(
@ -1599,8 +1657,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
rrdset_flag_set(d->st_svctm, RRDSET_FLAG_DETAIL);
d->rd_svctm_svctm = rrddim_add(d->st_svctm, "svctm", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_svctm);
}
else rrdset_next(d->st_svctm);
else
rrdset_next(d->st_svctm);
rrddim_set_by_pointer(d->st_svctm, d->rd_svctm_svctm, ((reads - last_reads) + (writes - last_writes)) ? (busy_ms - last_busy_ms) / ((reads - last_reads) + (writes - last_writes)) : 0);
rrdset_done(d->st_svctm);
@ -1705,8 +1766,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_bcache_hit_ratio_1hour = rrddim_add(d->st_bcache_hit_ratio, "1hour", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
d->rd_bcache_hit_ratio_1day = rrddim_add(d->st_bcache_hit_ratio, "1day", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
d->rd_bcache_hit_ratio_total = rrddim_add(d->st_bcache_hit_ratio, "ever", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_bcache_hit_ratio);
}
else rrdset_next(d->st_bcache_hit_ratio);
else
rrdset_next(d->st_bcache_hit_ratio);
rrddim_set_by_pointer(d->st_bcache_hit_ratio, d->rd_bcache_hit_ratio_5min, stats_five_minute_cache_hit_ratio);
rrddim_set_by_pointer(d->st_bcache_hit_ratio, d->rd_bcache_hit_ratio_1hour, stats_hour_cache_hit_ratio);
@ -1735,8 +1799,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_bcache_rate_congested = rrddim_add(d->st_bcache_rates, "congested", NULL, 1, 1024, RRD_ALGORITHM_ABSOLUTE);
d->rd_bcache_rate_writeback = rrddim_add(d->st_bcache_rates, "writeback", NULL, -1, 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_bcache_rates);
}
else rrdset_next(d->st_bcache_rates);
else
rrdset_next(d->st_bcache_rates);
rrddim_set_by_pointer(d->st_bcache_rates, d->rd_bcache_rate_writeback, writeback_rate);
rrddim_set_by_pointer(d->st_bcache_rates, d->rd_bcache_rate_congested, cache_congested);
@ -1761,8 +1828,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
);
d->rd_bcache_dirty_size = rrddim_add(d->st_bcache_size, "dirty", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_bcache_size);
}
else rrdset_next(d->st_bcache_size);
else
rrdset_next(d->st_bcache_size);
rrddim_set_by_pointer(d->st_bcache_size, d->rd_bcache_dirty_size, dirty_data);
rrdset_done(d->st_bcache_size);
@ -1786,8 +1856,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
);
d->rd_bcache_available_percent = rrddim_add(d->st_bcache_usage, "avail", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_disk(d, d->st_bcache_usage);
}
else rrdset_next(d->st_bcache_usage);
else
rrdset_next(d->st_bcache_usage);
rrddim_set_by_pointer(d->st_bcache_usage, d->rd_bcache_available_percent, cache_available_percent);
rrdset_done(d->st_bcache_usage);
@ -1813,8 +1886,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_bcache_cache_read_races = rrddim_add(d->st_bcache_cache_read_races, "races", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_bcache_cache_io_errors = rrddim_add(d->st_bcache_cache_read_races, "errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_bcache_cache_read_races);
}
else rrdset_next(d->st_bcache_cache_read_races);
else
rrdset_next(d->st_bcache_cache_read_races);
rrddim_set_by_pointer(d->st_bcache_cache_read_races, d->rd_bcache_cache_read_races, cache_read_races);
rrddim_set_by_pointer(d->st_bcache_cache_read_races, d->rd_bcache_cache_io_errors, cache_io_errors);
@ -1849,8 +1925,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_bcache_misses = rrddim_add(d->st_bcache, "misses", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_bcache_miss_collisions = rrddim_add(d->st_bcache, "collisions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_bcache_readaheads = rrddim_add(d->st_bcache, "readaheads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_bcache);
}
else rrdset_next(d->st_bcache);
else
rrdset_next(d->st_bcache);
rrddim_set_by_pointer(d->st_bcache, d->rd_bcache_hits, stats_total_cache_hits);
rrddim_set_by_pointer(d->st_bcache, d->rd_bcache_misses, stats_total_cache_misses);
@ -1884,6 +1963,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
d->rd_bcache_bypass_hits = rrddim_add(d->st_bcache_bypass, "hits", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
d->rd_bcache_bypass_misses = rrddim_add(d->st_bcache_bypass, "misses", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_disk(d, d->st_bcache_bypass);
}
else rrdset_next(d->st_bcache_bypass);

4
collectors/proc.plugin/proc_interrupts.c
View file

@ -225,6 +225,10 @@ int do_proc_interrupts(int update_every, usec_t dt) {
, update_every
, RRDSET_TYPE_STACKED
);
char core[50+1];
snprintfz(core, 50, "cpu%d", c);
rrdlabels_add(core_st[c]->state->chart_labels, "cpu", core, RRDLABEL_SRC_AUTO);
}
else rrdset_next(core_st[c]);

38
collectors/proc.plugin/proc_mdstat.c
View file

@ -77,6 +77,11 @@ static inline void make_chart_obsolete(char *name, const char *id_modifier)
}
}
static void add_labels_to_mdstat(struct raid *raid, RRDSET *st) {
rrdlabels_add(st->state->chart_labels, "device", raid->name, RRDLABEL_SRC_AUTO);
rrdlabels_add(st->state->chart_labels, "raid_level", raid->level, RRDLABEL_SRC_AUTO);
}
int do_proc_mdstat(int update_every, usec_t dt)
{
(void)dt;
@ -407,7 +412,8 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(st_mdstat_health);
} else
}
else
rrdset_next(st_mdstat_health);
if (!redundant_num) {
@ -458,7 +464,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_STACKED);
rrdset_isnot_obsolete(raid->st_disks);
} else
add_labels_to_mdstat(raid, raid->st_disks);
}
else
rrdset_next(raid->st_disks);
if (unlikely(!raid->rd_inuse && !(raid->rd_inuse = rrddim_find_active(raid->st_disks, "inuse"))))
@ -495,7 +504,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(raid->st_mismatch_cnt);
} else
add_labels_to_mdstat(raid, raid->st_mismatch_cnt);
}
else
rrdset_next(raid->st_mismatch_cnt);
if (unlikely(!raid->rd_mismatch_cnt && !(raid->rd_mismatch_cnt = rrddim_find_active(raid->st_mismatch_cnt, "count"))))
@ -529,7 +541,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(raid->st_operation);
} else
add_labels_to_mdstat(raid, raid->st_operation);
}
else
rrdset_next(raid->st_operation);
if(unlikely(!raid->rd_check && !(raid->rd_check = rrddim_find_active(raid->st_operation, "check"))))
@ -569,7 +584,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
update_every, RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(raid->st_finish);
} else
add_labels_to_mdstat(raid, raid->st_finish);
}
else
rrdset_next(raid->st_finish);
if(unlikely(!raid->rd_finish_in && !(raid->rd_finish_in = rrddim_find_active(raid->st_finish, "finish_in"))))
@ -601,7 +619,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(raid->st_speed);
} else
add_labels_to_mdstat(raid, raid->st_speed);
}
else
rrdset_next(raid->st_speed);
if (unlikely(!raid->rd_speed && !(raid->rd_speed = rrddim_find_active(raid->st_speed, "speed"))))
@ -635,7 +656,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
RRDSET_TYPE_LINE);
rrdset_isnot_obsolete(raid->st_nonredundant);
} else
add_labels_to_mdstat(raid, raid->st_nonredundant);
}
else
rrdset_next(raid->st_nonredundant);
if (unlikely(!raid->rd_nonredundant && !(raid->rd_nonredundant = rrddim_find_active(raid->st_nonredundant, "available"))))

57
collectors/proc.plugin/proc_net_dev.c
View file

@ -333,6 +333,7 @@ static struct netdev_rename {
const char *container_device;
const char *container_name;
const char *ctx_prefix;
DICTIONARY *chart_labels;
@ -355,7 +356,13 @@ static struct netdev_rename *netdev_rename_find(const char *host_device, uint32_
}
// other threads can call this function to register a rename to a netdev
void netdev_rename_device_add(const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels) {
void netdev_rename_device_add(
const char *host_device,
const char *container_device,
const char *container_name,
DICTIONARY *labels,
const char *ctx_prefix)
{
netdata_mutex_lock(&netdev_rename_mutex);
uint32_t hash = simple_hash(host_device);
@ -365,6 +372,7 @@ void netdev_rename_device_add(const char *host_device, const char *container_dev
r->host_device = strdupz(host_device);
r->container_device = strdupz(container_device);
r->container_name = strdupz(container_name);
r->ctx_prefix = strdupz(ctx_prefix);
r->chart_labels = rrdlabels_create();
rrdlabels_migrate_to_these(r->chart_labels, labels);
r->hash = hash;
@ -415,6 +423,7 @@ void netdev_rename_device_del(const char *host_device) {
freez((void *) r->host_device);
freez((void *) r->container_name);
freez((void *) r->container_device);
freez((void *) r->ctx_prefix);
rrdlabels_destroy(r->chart_labels);
freez((void *) r);
break;
@ -471,18 +480,30 @@ static inline void netdev_rename_cgroup(struct netdev *d, struct netdev_rename *
snprintfz(buffer, RRD_ID_LENGTH_MAX, "net_mtu_%s", r->container_device);
d->chart_id_net_mtu = strdupz(buffer);
d->chart_ctx_net_bytes = strdupz("cgroup.net_net");
d->chart_ctx_net_compressed = strdupz("cgroup.net_compressed");
d->chart_ctx_net_drops = strdupz("cgroup.net_drops");
d->chart_ctx_net_errors = strdupz("cgroup.net_errors");
d->chart_ctx_net_events = strdupz("cgroup.net_events");
d->chart_ctx_net_fifo = strdupz("cgroup.net_fifo");
d->chart_ctx_net_packets = strdupz("cgroup.net_packets");
d->chart_ctx_net_speed = strdupz("cgroup.net_speed");
d->chart_ctx_net_duplex = strdupz("cgroup.net_duplex");
d->chart_ctx_net_operstate = strdupz("cgroup.net_operstate");
d->chart_ctx_net_carrier = strdupz("cgroup.net_carrier");
d->chart_ctx_net_mtu = strdupz("cgroup.net_mtu");
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_net", r->ctx_prefix);
d->chart_ctx_net_bytes = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_compressed", r->ctx_prefix);
d->chart_ctx_net_compressed = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_drops", r->ctx_prefix);
d->chart_ctx_net_drops = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_errors", r->ctx_prefix);
d->chart_ctx_net_errors = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_events", r->ctx_prefix);
d->chart_ctx_net_events = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_fifo", r->ctx_prefix);
d->chart_ctx_net_fifo = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_packets", r->ctx_prefix);
d->chart_ctx_net_packets = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_speed", r->ctx_prefix);
d->chart_ctx_net_speed = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_duplex", r->ctx_prefix);
d->chart_ctx_net_duplex = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_operstate", r->ctx_prefix);
d->chart_ctx_net_operstate = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_carrier", r->ctx_prefix);
d->chart_ctx_net_carrier = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_mtu", r->ctx_prefix);
d->chart_ctx_net_mtu = strdupz(buffer);
snprintfz(buffer, RRD_ID_LENGTH_MAX, "net %s", r->container_device);
d->chart_family = strdupz(buffer);
@ -743,11 +764,13 @@ int do_proc_net_dev(int update_every, usec_t dt) {
snprintfz(buffer, FILENAME_MAX, path_to_sys_devices_virtual_net, d->name);
if (likely(access(buffer, R_OK) == 0)) {
d->virtual = 1;
rrdlabels_add(d->chart_labels, "type", "virtual", RRDLABEL_SRC_AUTO);
} else {
d->virtual = 0;
rrdlabels_add(d->chart_labels, "type", "real", RRDLABEL_SRC_AUTO);
rrdlabels_add(d->chart_labels, "interface_type", "virtual", RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
}
else {
d->virtual = 0;
rrdlabels_add(d->chart_labels, "interface_type", "real", RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
}
rrdlabels_add(d->chart_labels, "device", name, RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
if(likely(!d->virtual)) {
// set the filename to get the interface speed

278
collectors/proc.plugin/proc_net_wireless.c
View file

@ -198,6 +198,10 @@ static void configure_device(int do_status, int do_quality, int do_discarded_pac
wireless_dev->chart_id_net_missed_beacon = strdupz(buffer);
}
static void add_labels_to_wireless(struct netwireless *w, RRDSET *st) {
rrdlabels_add(st->state->chart_labels, "device", w->name, RRDLABEL_SRC_AUTO);
}
int do_proc_net_wireless(int update_every, usec_t dt)
{
UNUSED(dt);
@ -209,21 +213,11 @@ int do_proc_net_wireless(int update_every, usec_t dt)
char filename[FILENAME_MAX + 1];
snprintfz(filename, FILENAME_MAX, "%s%s", netdata_configured_host_prefix, "/proc/net/wireless");
proc_net_wireless_filename = config_get(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,"filename to monitor",
filename);
do_status = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
"status for all interfaces", CONFIG_BOOLEAN_AUTO);
do_quality = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
"quality for all interfaces", CONFIG_BOOLEAN_AUTO);
do_discarded_packets = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
"discarded packets for all interfaces",
CONFIG_BOOLEAN_AUTO);
do_beacon = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
"missed beacon for all interface", CONFIG_BOOLEAN_AUTO);
proc_net_wireless_filename = config_get(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,"filename to monitor", filename);
do_status = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "status for all interfaces", CONFIG_BOOLEAN_AUTO);
do_quality = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "quality for all interfaces", CONFIG_BOOLEAN_AUTO);
do_discarded_packets = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "discarded packets for all interfaces", CONFIG_BOOLEAN_AUTO);
do_beacon = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "missed beacon for all interface", CONFIG_BOOLEAN_AUTO);
}
if (unlikely(!ff)) {
@ -255,25 +249,28 @@ int do_proc_net_wireless(int update_every, usec_t dt)
wireless_dev->status = str2kernel_uint_t(procfile_lineword(ff, l, 1));
if (unlikely(!wireless_dev->st_status)) {
wireless_dev->st_status = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_status,
NULL,
wireless_dev->name,
"wireless.status",
"Internal status reported by interface.",
"status",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_status = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_status,
NULL,
wireless_dev->name,
"wireless.status",
"Internal status reported by interface.",
"status",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_status, RRDSET_FLAG_DETAIL);
wireless_dev->rd_status = rrddim_add(wireless_dev->st_status, "status", NULL, 1,
1, RRD_ALGORITHM_ABSOLUTE);
} else {
rrdset_next(wireless_dev->st_status);
wireless_dev->rd_status = rrddim_add(wireless_dev->st_status, "status", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_wireless(wireless_dev, wireless_dev->st_status);
}
else
rrdset_next(wireless_dev->st_status);
rrddim_set_by_pointer(wireless_dev->st_status, wireless_dev->rd_status,
(collected_number)wireless_dev->status);
@ -286,78 +283,81 @@ int do_proc_net_wireless(int update_every, usec_t dt)
wireless_dev->noise = str2ndd(procfile_lineword(ff, l, 4), NULL);
if (unlikely(!wireless_dev->st_link)) {
wireless_dev->st_link = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_link,
NULL,
wireless_dev->name,
"wireless.link_quality",
"Overall quality of the link. This is an aggregate value, and depends on the driver and hardware.",
"value",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 1,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_link = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_link,
NULL,
wireless_dev->name,
"wireless.link_quality",
"Overall quality of the link. This is an aggregate value, and depends on the driver and hardware.",
"value",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 1,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_link, RRDSET_FLAG_DETAIL);
wireless_dev->rd_link = rrddim_add(wireless_dev->st_link, "link_quality", NULL, 1, 1,
RRD_ALGORITHM_ABSOLUTE);
} else {
rrdset_next(wireless_dev->st_link);
wireless_dev->rd_link = rrddim_add(wireless_dev->st_link, "link_quality", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_wireless(wireless_dev, wireless_dev->st_link);
}
else
rrdset_next(wireless_dev->st_link);
if (unlikely(!wireless_dev->st_level)) {
wireless_dev->st_level = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_level,
NULL,
wireless_dev->name,
"wireless.signal_level",
"The signal level is the wireless signal power level received by the wireless client. The closer the value is to 0, the stronger the signal.",
"dBm",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 2,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_level = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_level,
NULL,
wireless_dev->name,
"wireless.signal_level",
"The signal level is the wireless signal power level received by the wireless client. The closer the value is to 0, the stronger the signal.",
"dBm",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 2,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_level, RRDSET_FLAG_DETAIL);
wireless_dev->rd_level = rrddim_add(wireless_dev->st_level, "signal_level", NULL, 1, 1,
RRD_ALGORITHM_ABSOLUTE);
} else {
rrdset_next(wireless_dev->st_level);
wireless_dev->rd_level = rrddim_add(wireless_dev->st_level, "signal_level", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_wireless(wireless_dev, wireless_dev->st_level);
}
else
rrdset_next(wireless_dev->st_level);
if (unlikely(!wireless_dev->st_noise)) {
wireless_dev->st_noise = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_noise,
NULL,
wireless_dev->name,
"wireless.noise_level",
"The noise level indicates the amount of background noise in your environment. The closer the value to 0, the greater the noise level.",
"dBm",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 3,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_noise = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_noise,
NULL,
wireless_dev->name,
"wireless.noise_level",
"The noise level indicates the amount of background noise in your environment. The closer the value to 0, the greater the noise level.",
"dBm",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 3,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_noise, RRDSET_FLAG_DETAIL);
wireless_dev->rd_noise = rrddim_add(wireless_dev->st_noise, "noise_level", NULL, 1, 1,
RRD_ALGORITHM_ABSOLUTE);
} else {
rrdset_next(wireless_dev->st_noise);
}
wireless_dev->rd_noise = rrddim_add(wireless_dev->st_noise, "noise_level", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
rrddim_set_by_pointer(wireless_dev->st_link, wireless_dev->rd_link,
(collected_number)wireless_dev->link);
add_labels_to_wireless(wireless_dev, wireless_dev->st_noise);
}
else
rrdset_next(wireless_dev->st_noise);
rrddim_set_by_pointer(wireless_dev->st_link, wireless_dev->rd_link, (collected_number)wireless_dev->link);
rrdset_done(wireless_dev->st_link);
rrddim_set_by_pointer(wireless_dev->st_level, wireless_dev->rd_level,
(collected_number)wireless_dev->level);
rrddim_set_by_pointer(wireless_dev->st_level, wireless_dev->rd_level, (collected_number)wireless_dev->level);
rrdset_done(wireless_dev->st_level);
rrddim_set_by_pointer(wireless_dev->st_noise, wireless_dev->rd_noise,
(collected_number)wireless_dev->noise);
rrddim_set_by_pointer(wireless_dev->st_noise, wireless_dev->rd_noise, (collected_number)wireless_dev->noise);
rrdset_done(wireless_dev->st_noise);
}
@ -369,49 +369,38 @@ int do_proc_net_wireless(int update_every, usec_t dt)
wireless_dev->misc = str2kernel_uint_t(procfile_lineword(ff, l, 9));
if (unlikely(!wireless_dev->st_discarded_packets)) {
wireless_dev->st_discarded_packets = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_discarded_packets,
NULL,
wireless_dev->name,
"wireless.discarded_packets",
"Packet discarded in the wireless adapter due to \"wireless\" specific problems.",
"packets/s",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 4,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_discarded_packets = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_discarded_packets,
NULL,
wireless_dev->name,
"wireless.discarded_packets",
"Packet discarded in the wireless adapter due to \"wireless\" specific problems.",
"packets/s",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 4,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_discarded_packets, RRDSET_FLAG_DETAIL);
wireless_dev->rd_nwid = rrddim_add(wireless_dev->st_discarded_packets, "nwid", NULL, 1,
1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_crypt = rrddim_add(wireless_dev->st_discarded_packets, "crypt", NULL, 1,
1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_frag = rrddim_add(wireless_dev->st_discarded_packets, "frag", NULL, 1,
1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_retry = rrddim_add(wireless_dev->st_discarded_packets, "retry", NULL, 1,
1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_misc = rrddim_add(wireless_dev->st_discarded_packets, "misc", NULL, 1,
1, RRD_ALGORITHM_INCREMENTAL);
} else {
rrdset_next(wireless_dev->st_discarded_packets);
wireless_dev->rd_nwid = rrddim_add(wireless_dev->st_discarded_packets, "nwid", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_crypt = rrddim_add(wireless_dev->st_discarded_packets, "crypt", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_frag = rrddim_add(wireless_dev->st_discarded_packets, "frag", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_retry = rrddim_add(wireless_dev->st_discarded_packets, "retry", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
wireless_dev->rd_misc = rrddim_add(wireless_dev->st_discarded_packets, "misc", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_wireless(wireless_dev, wireless_dev->st_discarded_packets);
}
else
rrdset_next(wireless_dev->st_discarded_packets);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_nwid,
(collected_number)wireless_dev->nwid);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_crypt,
(collected_number)wireless_dev->crypt);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_frag,
(collected_number)wireless_dev->frag);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_retry,
(collected_number)wireless_dev->retry);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_misc,
(collected_number)wireless_dev->misc);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_nwid, (collected_number)wireless_dev->nwid);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_crypt, (collected_number)wireless_dev->crypt);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_frag, (collected_number)wireless_dev->frag);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_retry, (collected_number)wireless_dev->retry);
rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_misc, (collected_number)wireless_dev->misc);
rrdset_done(wireless_dev->st_discarded_packets);
}
@ -420,28 +409,31 @@ int do_proc_net_wireless(int update_every, usec_t dt)
wireless_dev->missed_beacon = str2kernel_uint_t(procfile_lineword(ff, l, 10));
if (unlikely(!wireless_dev->st_missed_beacon)) {
wireless_dev->st_missed_beacon = rrdset_create_localhost("wireless",
wireless_dev->chart_id_net_missed_beacon,
NULL,
wireless_dev->name,
"wireless.missed_beacons",
"Number of missed beacons.",
"frames/s",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 5,
update_every,
RRDSET_TYPE_LINE);
wireless_dev->st_missed_beacon = rrdset_create_localhost(
"wireless",
wireless_dev->chart_id_net_missed_beacon,
NULL,
wireless_dev->name,
"wireless.missed_beacons",
"Number of missed beacons.",
"frames/s",
PLUGIN_PROC_NAME,
PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
NETDATA_CHART_PRIO_WIRELESS_IFACE + 5,
update_every,
RRDSET_TYPE_LINE);
rrdset_flag_set(wireless_dev->st_missed_beacon, RRDSET_FLAG_DETAIL);
wireless_dev->rd_missed_beacon = rrddim_add(wireless_dev->st_missed_beacon, "missed_beacons",
NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
} else {
rrdset_next(wireless_dev->st_missed_beacon);
}
wireless_dev->rd_missed_beacon = rrddim_add(wireless_dev->st_missed_beacon, "missed_beacons", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
add_labels_to_wireless(wireless_dev, wireless_dev->st_missed_beacon);
}
else
rrdset_next(wireless_dev->st_missed_beacon);
rrddim_set_by_pointer(wireless_dev->st_missed_beacon, wireless_dev->rd_missed_beacon, (collected_number)wireless_dev->missed_beacon);
rrddim_set_by_pointer(wireless_dev->st_missed_beacon, wireless_dev->rd_missed_beacon,
(collected_number)wireless_dev->missed_beacon);
rrdset_done(wireless_dev->st_missed_beacon);
}

12
collectors/proc.plugin/proc_pagetypeinfo.c
View file

@ -242,15 +242,14 @@ int do_proc_pagetypeinfo(int update_every, usec_t dt) {
// "Node" + NUMA-NodeID + ZoneName + TypeName
char setname[4+1+MAX_ZONETYPE_NAME+1+MAX_PAGETYPE_NAME +1];
snprintfz(setname, MAX_ZONETYPE_NAME + MAX_PAGETYPE_NAME, "Node %d %s %s",
pgl->node, pgl->zone, pgl->type);
snprintfz(setname, MAX_ZONETYPE_NAME + MAX_PAGETYPE_NAME, "Node %d %s %s", pgl->node, pgl->zone, pgl->type);
st_nodezonetype[p] = rrdset_create_localhost(
"mem"
, setid
, NULL
, "pagetype"
, NULL
, "mem.pagetype"
, setname
, "B"
, PLUGIN_PROC_NAME
@ -259,6 +258,13 @@ int do_proc_pagetypeinfo(int update_every, usec_t dt) {
, update_every
, RRDSET_TYPE_STACKED
);
char node[50+1];
snprintfz(node, 50, "node%d", pgl->node);
rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_id", node, RRDLABEL_SRC_AUTO);
rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_zone", pgl->zone, RRDLABEL_SRC_AUTO);
rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_type", pgl->type, RRDLABEL_SRC_AUTO);
for (o = 0; o < pageorders_cnt; o++) {
char dimid[3+1];
snprintfz(dimid, 3, "%lu", o);

4
collectors/proc.plugin/proc_softirqs.c
View file

@ -217,6 +217,10 @@ int do_proc_softirqs(int update_every, usec_t dt) {
, update_every
, RRDSET_TYPE_STACKED
);
char core[50+1];
snprintfz(core, 50, "cpu%d", c);
rrdlabels_add(core_st[c]->state->chart_labels, "cpu", core, RRDLABEL_SRC_AUTO);
}
else
rrdset_next(core_st[c]);

4
collectors/proc.plugin/proc_stat.c
View file

@ -1039,6 +1039,10 @@ int do_proc_stat(int update_every, usec_t dt) {
, RRDSET_TYPE_STACKED
);
char corebuf[50+1];
snprintfz(corebuf, 50, "cpu%zu", core);
rrdlabels_add(cpuidle_charts[core].st->state->chart_labels, "cpu", corebuf, RRDLABEL_SRC_AUTO);
char cpuidle_dim_id[RRD_ID_LENGTH_MAX + 1];
cpuidle_charts[core].active_time_rd = rrddim_add(cpuidle_charts[core].st, "active", "C0 (active)", 1, 1, RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL);
for(state = 0; state < cpuidle_charts[core].cpuidle_state_len; state++) {

4
collectors/proc.plugin/sys_block_zram.c
View file

@ -75,6 +75,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
, RRDSET_TYPE_AREA);
d->rd_compr_data_size = rrddim_add(d->st_usage, "compressed", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
d->rd_metadata_size = rrddim_add(d->st_usage, "metadata", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(d->st_usage->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_savings.%s", name);
d->st_savings = rrdset_create_localhost(
@ -92,6 +93,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
, RRDSET_TYPE_AREA);
d->rd_savings_size = rrddim_add(d->st_savings, "savings", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
d->rd_original_size = rrddim_add(d->st_savings, "original", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(d->st_savings->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_ratio.%s", name);
d->st_comp_ratio = rrdset_create_localhost(
@ -108,6 +110,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
, update_every
, RRDSET_TYPE_LINE);
d->rd_comp_ratio = rrddim_add(d->st_comp_ratio, "ratio", NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(d->st_comp_ratio->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_efficiency.%s", name);
d->st_alloc_efficiency = rrdset_create_localhost(
@ -124,6 +127,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
, update_every
, RRDSET_TYPE_LINE);
d->rd_alloc_efficiency = rrddim_add(d->st_alloc_efficiency, "percent", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
rrdlabels_add(d->st_alloc_efficiency->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
}
static int init_devices(DICTIONARY *devices, unsigned int zram_id, int update_every) {

8
collectors/proc.plugin/sys_class_power_supply.c
View file

@ -112,6 +112,10 @@ void power_supply_free(struct power_supply *ps) {
}
}
static void add_labels_to_power_supply(struct power_supply *ps, RRDSET *st) {
rrdlabels_add(st->state->chart_labels, "device", ps->name, RRDLABEL_SRC_AUTO);
}
int do_sys_class_power_supply(int update_every, usec_t dt) {
(void)dt;
static int do_capacity = -1, do_property[3] = {-1};
@ -358,6 +362,8 @@ int do_sys_class_power_supply(int update_every, usec_t dt) {
, update_every
, RRDSET_TYPE_LINE
);
add_labels_to_power_supply(ps, ps->capacity->st);
}
else
rrdset_next(ps->capacity->st);
@ -389,6 +395,8 @@ int do_sys_class_power_supply(int update_every, usec_t dt) {
, update_every
, RRDSET_TYPE_LINE
);
add_labels_to_power_supply(ps, pr->st);
}
else
rrdset_next(pr->st);

2
collectors/proc.plugin/sys_devices_system_node.c
View file

@ -115,6 +115,8 @@ int do_proc_sys_devices_system_node(int update_every, usec_t dt) {
, RRDSET_TYPE_LINE
);
rrdlabels_add(m->numastat_st->state->chart_labels, "numa_node", m->name, RRDLABEL_SRC_AUTO);
rrdset_flag_set(m->numastat_st, RRDSET_FLAG_DETAIL);
rrddim_add(m->numastat_st, "numa_hit", "hit", 1, 1, RRD_ALGORITHM_INCREMENTAL);

13
collectors/proc.plugin/sys_fs_btrfs.c
View file

@ -448,6 +448,11 @@ static inline int find_all_btrfs_pools(const char *path) {
return 0;
}
static void add_labels_to_btrfs(BTRFS_NODE *n, RRDSET *st) {
rrdlabels_add(st->state->chart_labels, "device", n->id, RRDLABEL_SRC_AUTO);
rrdlabels_add(st->state->chart_labels, "device_label", n->label, RRDLABEL_SRC_AUTO);
}
int do_sys_fs_btrfs(int update_every, usec_t dt) {
static int initialized = 0
, do_allocation_disks = CONFIG_BOOLEAN_AUTO
@ -579,6 +584,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
node->rd_allocation_disks_metadata_used = rrddim_add(node->st_allocation_disks, "meta_used", "meta used", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_disks_system_free = rrddim_add(node->st_allocation_disks, "sys_free", "sys free", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_disks_system_used = rrddim_add(node->st_allocation_disks, "sys_used", "sys used", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_btrfs(node, node->st_allocation_disks);
}
else rrdset_next(node->st_allocation_disks);
@ -632,6 +639,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
node->rd_allocation_data_free = rrddim_add(node->st_allocation_data, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_data_used = rrddim_add(node->st_allocation_data, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_btrfs(node, node->st_allocation_data);
}
else rrdset_next(node->st_allocation_data);
@ -676,6 +685,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
node->rd_allocation_metadata_free = rrddim_add(node->st_allocation_metadata, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_metadata_used = rrddim_add(node->st_allocation_metadata, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_metadata_reserved = rrddim_add(node->st_allocation_metadata, "reserved", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_btrfs(node, node->st_allocation_metadata);
}
else rrdset_next(node->st_allocation_metadata);
@ -720,6 +731,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
node->rd_allocation_system_free = rrddim_add(node->st_allocation_system, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
node->rd_allocation_system_used = rrddim_add(node->st_allocation_system, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
add_labels_to_btrfs(node, node->st_allocation_system);
}
else rrdset_next(node->st_allocation_system);

2
daemon/analytics.c
View file

@ -541,7 +541,7 @@ void analytics_gather_mutable_meta_data(void)
analytics_set_data(
&analytics_data.netdata_config_is_parent, (localhost->next || configured_as_parent()) ? "true" : "false");
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
analytics_set_data(&analytics_data.netdata_host_agent_claimed, claim_id ? "true" : "false");
freez(claim_id);

1
daemon/global_statistics.c
View file

@ -996,6 +996,7 @@ static struct worker_utilization all_workers_utilization[] = {
{ .name = "TC", .family = "workers plugin tc", .priority = 1000000 },
{ .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
{ .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
{ .name = "RRDCONTEXT", .family = "workers aclk contexts", .priority = 1000000 },
// has to be terminated with a NULL
{ .name = NULL, .family = NULL }

11
daemon/main.c
View file

@ -64,6 +64,7 @@ void netdata_cleanup_and_exit(int ret) {
rrdeng_exit(multidb_ctx[tier]);
#endif
}
sql_close_context_database();
sql_close_database();
// unlink the pid
@ -732,6 +733,12 @@ static void get_netdata_configured_variables() {
config_set_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
}
// --------------------------------------------------------------------
// rrdcontext
rrdcontext_enabled = config_get_boolean(CONFIG_SECTION_CLOUD, "rrdcontexts", rrdcontext_enabled);
// --------------------------------------------------------------------
// get various system parameters
@ -978,6 +985,7 @@ int main(int argc, char **argv) {
// No call to load the config file on this code-path
post_conf_load(&user);
get_netdata_configured_variables();
rrdcontext_enabled = CONFIG_BOOLEAN_NO;
default_rrd_update_every = 1;
default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
default_health_enabled = 0;
@ -1010,6 +1018,9 @@ int main(int argc, char **argv) {
else if(strcmp(optarg, "mctest") == 0) {
return mc_unittest();
}
else if(strcmp(optarg, "ctxtest") == 0) {
return ctx_unittest();
}
else if(strcmp(optarg, "dicttest") == 0) {
return dictionary_unittest(10000);
}

10
daemon/static_threads.c
View file

@ -135,6 +135,16 @@ const struct netdata_static_thread static_threads_common[] = {
},
#endif
{
.name = "rrdcontext",
.config_section = NULL,
.config_name = NULL,
.enabled = 1,
.thread = NULL,
.init_routine = NULL,
.start_routine = rrdcontext_main
},
{NULL, NULL, NULL, 0, NULL, NULL, NULL}
};

3
daemon/unit_test.c
View file

@ -1772,6 +1772,7 @@ static RRDHOST *dbengine_rrdhost_find_or_create(char *name)
, default_rrdpush_api_key
, default_rrdpush_send_charts_matching
, NULL
, 0
);
}
@ -2279,7 +2280,7 @@ void generate_dbengine_dataset(unsigned history_seconds)
}
freez(thread_info);
rrd_wrlock();
rrdhost_free(host);
rrdhost_free(host, 1);
rrd_unlock();
}

1
database/engine/rrdengine.c
View file

@ -887,6 +887,7 @@ static void after_delete_old_data(struct rrdengine_worker_config* wc)
wc->cleanup_thread_deleting_files = 0;
aclk_data_rotated();
rrdcontext_db_rotation();
/* interrupt event loop */
uv_stop(wc->loop);

30
database/engine/rrdengineapi.c
View file

@ -623,6 +623,36 @@ int rrdeng_metric_latest_time_by_uuid(uuid_t *dim_uuid, time_t *first_entry_t, t
return 1;
}
int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t)
{
struct page_cache *pg_cache;
struct rrdengine_instance *ctx;
Pvoid_t *PValue;
struct pg_cache_page_index *page_index = NULL;
ctx = (struct rrdengine_instance *)si;
if (unlikely(!ctx)) {
error("DBENGINE: invalid STORAGE INSTANCE to %s()", __FUNCTION__);
return 1;
}
pg_cache = &ctx->pg_cache;
uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, dim_uuid, sizeof(uuid_t));
if (likely(NULL != PValue)) {
page_index = *PValue;
}
uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
if (likely(page_index)) {
*first_entry_t = page_index->oldest_time / USEC_PER_SEC;
*last_entry_t = page_index->latest_time / USEC_PER_SEC;
return 0;
}
return 1;
}
/* Also gets a reference for the page */
void *rrdeng_create_page(struct rrdengine_instance *ctx, uuid_t *id, struct rrdeng_page_descr **ret_descr)
{

1
database/engine/rrdengineapi.h
View file

@ -76,5 +76,6 @@ extern int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *db
extern int rrdeng_exit(struct rrdengine_instance *ctx);
extern void rrdeng_prepare_exit(struct rrdengine_instance *ctx);
extern int rrdeng_metric_latest_time_by_uuid(uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t, int tier);
extern int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t);
#endif /* NETDATA_RRDENGINEAPI_H */

59
database/rrd.h
View file

@ -49,6 +49,7 @@ struct pg_cache_page_index;
#include "streaming/rrdpush.h"
#include "aclk/aclk_rrdhost_state.h"
#include "sqlite/sqlite_health.h"
#include "rrdcontext.h"
extern int storage_tiers;
extern int storage_tiers_grouping_iterations[RRD_STORAGE_TIERS];
@ -196,15 +197,16 @@ typedef enum rrddim_flags {
#define rrddim_flag_clear(rd, flag) __atomic_and_fetch(&((rd)->flags), ~(flag), __ATOMIC_SEQ_CST)
typedef enum rrdlabel_source {
RRDLABEL_SRC_AUTO = (1 << 0), // set when Netdata found the label by some automation
RRDLABEL_SRC_CONFIG = (1 << 1), // set when the user configured the label
RRDLABEL_SRC_K8S = (1 << 2), // set when this label is found from k8s (RRDLABEL_SRC_AUTO should also be set)
RRDLABEL_SRC_ACLK = (1 << 3), // set when this label is found from ACLK (RRDLABEL_SRC_AUTO should also be set)
RRDLABEL_SRC_AUTO = (1 << 0), // set when Netdata found the label by some automation
RRDLABEL_SRC_CONFIG = (1 << 1), // set when the user configured the label
RRDLABEL_SRC_K8S = (1 << 2), // set when this label is found from k8s (RRDLABEL_SRC_AUTO should also be set)
RRDLABEL_SRC_ACLK = (1 << 3), // set when this label is found from ACLK (RRDLABEL_SRC_AUTO should also be set)
// more sources can be added here
RRDLABEL_FLAG_OLD = (1 << 30), // marks set for rrdlabels internal use - they are not exposed outside rrdlabels
RRDLABEL_FLAG_NEW = (1 << 31) //
RRDLABEL_FLAG_PERMANENT = (1 << 29), // set when this label should never be removed (can be overwritten though)
RRDLABEL_FLAG_OLD = (1 << 30), // marks for rrdlabels internal use - they are not exposed outside rrdlabels
RRDLABEL_FLAG_NEW = (1 << 31) // marks for rrdlabels internal use - they are not exposed outside rrdlabels
} RRDLABEL_SRC;
extern DICTIONARY *rrdlabels_create(void);
@ -222,7 +224,7 @@ extern int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)
extern void rrdlabels_log_to_buffer(DICTIONARY *labels, BUFFER *wb);
extern bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_pattern_txt);
extern bool rrdlabels_match_simple_pattern_parsed(DICTIONARY *labels, SIMPLE_PATTERN *pattern, char equal);
extern void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size));
extern int rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size));
extern void rrdlabels_migrate_to_these(DICTIONARY *dst, DICTIONARY *src);
extern void rrdlabels_copy(DICTIONARY *dst, DICTIONARY *src);
@ -284,7 +286,7 @@ struct rrddim {
struct rrddim_tier *tiers[RRD_STORAGE_TIERS]; // our tiers of databases
size_t collections_counter; // the number of times we added values to this rrdim
size_t collections_counter; // the number of times we added values to this rrddim
collected_number collected_value_max; // the absolute maximum of the collected value
NETDATA_DOUBLE calculated_value; // the current calculated value, after applying the algorithm - resets to zero after being used
@ -296,11 +298,12 @@ struct rrddim {
// the *_volume members are used to calculate the accuracy of the rounding done by the
// storage number - they are printed to debug.log when debug is enabled for a set.
NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
struct rrddim *next; // linking of dimensions within the same data set
struct rrdset *rrdset;
RRDMETRIC_ACQUIRED *rrdmetric; // the rrdmetric of this dimension
// ------------------------------------------------------------------------
// members for checking the data when loading from disk
@ -531,6 +534,9 @@ struct rrdset {
char *context; // the template of this data set
uint32_t hash_context; // the hash of the chart's context
RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
RRDSET_TYPE chart_type; // line, area, stacked
int update_every; // every how many seconds is this updated?
@ -644,17 +650,17 @@ extern bool rrdset_memory_load_or_create_map_save(RRDSET *st_on_file, RRD_MEMORY
// and may lead to missing information.
typedef enum rrdhost_flags {
RRDHOST_FLAG_ORPHAN = 1 << 0, // this host is orphan (not receiving data)
RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = 1 << 1, // delete files of obsolete charts
RRDHOST_FLAG_DELETE_ORPHAN_HOST = 1 << 2, // delete the entire host when orphan
RRDHOST_FLAG_EXPORTING_SEND = 1 << 3, // send it to external databases
RRDHOST_FLAG_EXPORTING_DONT_SEND = 1 << 4, // don't send it to external databases
RRDHOST_FLAG_ARCHIVED = 1 << 5, // The host is archived, no collected charts yet
RRDHOST_FLAG_MULTIHOST = 1 << 6, // Host belongs to localhost/megadb
RRDHOST_FLAG_PENDING_FOREACH_ALARMS = 1 << 7, // contains dims with uninitialized foreach alarms
RRDHOST_FLAG_STREAM_LABELS_UPDATE = 1 << 8,
RRDHOST_FLAG_STREAM_LABELS_STOP = 1 << 9,
RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
RRDHOST_FLAG_MULTIHOST = (1 << 6), // Host belongs to localhost/megadb
RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
} RRDHOST_FLAGS;
#define rrdhost_flag_check(host, flag) (__atomic_load_n(&((host)->flags), __ATOMIC_SEQ_CST) & (flag))
@ -925,6 +931,9 @@ struct rrdhost {
STORAGE_INSTANCE *storage_instance[RRD_STORAGE_TIERS]; // the database instances of the storage tiers
RRDCONTEXTS *rrdctx_queue;
RRDCONTEXTS *rrdctx;
uuid_t host_uuid; // Global GUID for this host
uuid_t *node_id; // Cloud node_id
@ -1000,6 +1009,7 @@ extern RRDHOST *rrdhost_find_or_create(
, char *rrdpush_api_key
, char *rrdpush_send_charts_matching
, struct rrdhost_system_info *system_info
, bool is_archived
);
extern void rrdhost_update(RRDHOST *host
@ -1083,7 +1093,7 @@ extern void rrdhost_cleanup_all(void);
extern void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host);
extern void rrdhost_system_info_free(struct rrdhost_system_info *system_info);
extern void rrdhost_free(RRDHOST *host);
extern void rrdhost_free(RRDHOST *host, bool force);
extern void rrdhost_save_charts(RRDHOST *host);
extern void rrdhost_delete_charts(RRDHOST *host);
extern void rrd_cleanup_obsolete_charts();
@ -1291,7 +1301,7 @@ extern RRDSET *rrdset_index_del_name(RRDHOST *host, RRDSET *st);
extern void rrdset_free(RRDSET *st);
extern void rrdset_reset(RRDSET *st);
extern void rrdset_save(RRDSET *st);
extern void rrdset_delete(RRDSET *st);
extern void rrdset_delete_files(RRDSET *st);
extern void rrdset_delete_obsolete_dimensions(RRDSET *st);
extern RRDHOST *rrdhost_create(
@ -1299,7 +1309,7 @@ extern RRDHOST *rrdhost_create(
const char *abbrev_timezone, int32_t utc_offset,const char *tags, const char *program_name, const char *program_version,
int update_every, long entries, RRD_MEMORY_MODE memory_mode, unsigned int health_enabled, unsigned int rrdpush_enabled,
char *rrdpush_destination, char *rrdpush_api_key, char *rrdpush_send_charts_matching, struct rrdhost_system_info *system_info,
int is_localhost); //TODO: Remove , int is_archived);
int is_localhost, bool is_archived);
#endif /* NETDATA_RRD_INTERNALS */
@ -1317,6 +1327,7 @@ extern int get_tier_grouping(int tier);
#include "database/engine/rrdengineapi.h"
#endif
#include "sqlite/sqlite_functions.h"
#include "sqlite/sqlite_context.h"
#include "sqlite/sqlite_aclk.h"
#include "sqlite/sqlite_aclk_chart.h"
#include "sqlite/sqlite_aclk_alert.h"

2830
database/rrdcontext.c Normal file
View file

File diff suppressed because it is too large Load diff

91
database/rrdcontext.h Normal file
View file

@ -0,0 +1,91 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#ifndef NETDATA_RRDCONTEXT_H
#define NETDATA_RRDCONTEXT_H 1
// ----------------------------------------------------------------------------
// RRDMETRIC
typedef struct rrdmetric_acquired RRDMETRIC_ACQUIRED;
// ----------------------------------------------------------------------------
// RRDINSTANCE
typedef struct rrdinstance_acquired RRDINSTANCE_ACQUIRED;
// ----------------------------------------------------------------------------
// RRDCONTEXT
typedef struct rrdcontexts_dictionary RRDCONTEXTS;
typedef struct rrdcontext_acquired RRDCONTEXT_ACQUIRED;
// ----------------------------------------------------------------------------
#include "rrd.h"
// ----------------------------------------------------------------------------
// public API for rrdhost
extern void rrdhost_load_rrdcontext_data(RRDHOST *host);
extern void rrdhost_create_rrdcontexts(RRDHOST *host);
extern void rrdhost_destroy_rrdcontexts(RRDHOST *host);
extern void rrdcontext_host_child_connected(RRDHOST *host);
extern void rrdcontext_host_child_disconnected(RRDHOST *host);
typedef enum {
RRDCONTEXT_OPTION_NONE = 0,
RRDCONTEXT_OPTION_SHOW_METRICS = (1 << 0),
RRDCONTEXT_OPTION_SHOW_INSTANCES = (1 << 1),
RRDCONTEXT_OPTION_SHOW_LABELS = (1 << 2),
RRDCONTEXT_OPTION_SHOW_QUEUED = (1 << 3),
RRDCONTEXT_OPTION_SHOW_FLAGS = (1 << 4),
RRDCONTEXT_OPTION_SHOW_DELETED = (1 << 5),
RRDCONTEXT_OPTION_DEEPSCAN = (1 << 6),
RRDCONTEXT_OPTION_SHOW_UUIDS = (1 << 7),
RRDCONTEXT_OPTION_SKIP_ID = (1 << 31), // internal use
} RRDCONTEXT_TO_JSON_OPTIONS;
#define RRDCONTEXT_OPTIONS_ALL (RRDCONTEXT_OPTION_SHOW_METRICS|RRDCONTEXT_OPTION_SHOW_INSTANCES|RRDCONTEXT_OPTION_SHOW_LABELS|RRDCONTEXT_OPTION_SHOW_QUEUED|RRDCONTEXT_OPTION_SHOW_FLAGS|RRDCONTEXT_OPTION_SHOW_DELETED|RRDCONTEXT_OPTION_SHOW_UUIDS)
extern int rrdcontext_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, const char *context, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions);
extern int rrdcontexts_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions);
// ----------------------------------------------------------------------------
// public API for rrddims
extern void rrdcontext_updated_rrddim(RRDDIM *rd);
extern void rrdcontext_removed_rrddim(RRDDIM *rd);
extern void rrdcontext_updated_rrddim_algorithm(RRDDIM *rd);
extern void rrdcontext_updated_rrddim_multiplier(RRDDIM *rd);
extern void rrdcontext_updated_rrddim_divisor(RRDDIM *rd);
extern void rrdcontext_updated_rrddim_flags(RRDDIM *rd);
extern void rrdcontext_collected_rrddim(RRDDIM *rd);
// ----------------------------------------------------------------------------
// public API for rrdsets
extern void rrdcontext_updated_rrdset(RRDSET *st);
extern void rrdcontext_removed_rrdset(RRDSET *st);
extern void rrdcontext_updated_rrdset_name(RRDSET *st);
extern void rrdcontext_updated_rrdset_flags(RRDSET *st);
extern void rrdcontext_collected_rrdset(RRDSET *st);
// ----------------------------------------------------------------------------
// public API for ACLK
extern void rrdcontext_hub_checkpoint_command(void *cmd);
extern void rrdcontext_hub_stop_streaming_command(void *cmd);
// ----------------------------------------------------------------------------
// public API for threads
extern int rrdcontext_enabled;
extern void rrdcontext_db_rotation(void);
extern void *rrdcontext_main(void *);
#endif // NETDATA_RRDCONTEXT_H

12
database/rrddim.c
View file

@ -73,6 +73,7 @@ inline int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm)
rd->exposed = 0;
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdcontext_updated_rrddim_algorithm(rd);
return 1;
}
@ -85,6 +86,7 @@ inline int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multip
rd->exposed = 0;
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdcontext_updated_rrddim_multiplier(rd);
return 1;
}
@ -97,6 +99,7 @@ inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor)
rd->exposed = 0;
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdcontext_updated_rrddim_divisor(rd);
return 1;
}
@ -199,6 +202,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
}
rrdset_unlock(st);
rrdcontext_updated_rrddim(rd);
return rd;
}
@ -344,6 +348,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
ml_new_dimension(rd);
rrdset_unlock(st);
rrdcontext_updated_rrddim(rd);
return(rd);
}
@ -352,6 +357,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
void rrddim_free(RRDSET *st, RRDDIM *rd)
{
rrdcontext_removed_rrddim(rd);
ml_delete_dimension(rd);
debug(D_RRD_CALLS, "rrddim_free() %s.%s", st->name, rd->name);
@ -446,6 +452,7 @@ int rrddim_hide(RRDSET *st, const char *id) {
rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
rrdcontext_updated_rrddim_flags(rd);
return 0;
}
@ -463,6 +470,7 @@ int rrddim_unhide(RRDSET *st, const char *id) {
rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
rrdcontext_updated_rrddim_flags(rd);
return 0;
}
@ -475,12 +483,14 @@ inline void rrddim_is_obsolete(RRDSET *st, RRDDIM *rd) {
}
rrddim_flag_set(rd, RRDDIM_FLAG_OBSOLETE);
rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
rrdcontext_updated_rrddim_flags(rd);
}
inline void rrddim_isnot_obsolete(RRDSET *st __maybe_unused, RRDDIM *rd) {
debug(D_RRD_CALLS, "rrddim_isnot_obsolete() for chart %s, dimension %s", st->name, rd->name);
rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
rrdcontext_updated_rrddim_flags(rd);
}
// ----------------------------------------------------------------------------
@ -489,6 +499,8 @@ inline void rrddim_isnot_obsolete(RRDSET *st __maybe_unused, RRDDIM *rd) {
inline collected_number rrddim_set_by_pointer(RRDSET *st __maybe_unused, RRDDIM *rd, collected_number value) {
debug(D_RRD_CALLS, "rrddim_set_by_pointer() for chart %s, dimension %s, value " COLLECTED_NUMBER_FORMAT, st->name, rd->name, value);
rrdcontext_collected_rrddim(rd);
now_realtime_timeval(&rd->last_collected_time);
rd->collected_value = value;
rd->updated = 1;

176
database/rrdhost.c
View file

@ -187,7 +187,8 @@ RRDHOST *rrdhost_create(const char *hostname,
char *rrdpush_api_key,
char *rrdpush_send_charts_matching,
struct rrdhost_system_info *system_info,
int is_localhost
int is_localhost,
bool archived
) {
debug(D_RRDHOST, "Host '%s': adding with guid '%s'", hostname, guid);
@ -207,8 +208,7 @@ RRDHOST *rrdhost_create(const char *hostname,
host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
host->health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
host->sender = mallocz(sizeof(*host->sender));
sender_init(host->sender, host);
sender_init(host);
netdata_mutex_init(&host->receiver_lock);
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
@ -335,19 +335,21 @@ RRDHOST *rrdhost_create(const char *hostname,
if(t != host) {
error("Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.", host->hostname, host->machine_guid, t->hostname, t->machine_guid);
rrdhost_free(host);
rrdhost_free(host, 1);
return NULL;
}
if (likely(!uuid_parse(host->machine_guid, host->host_uuid))) {
int rc = sql_store_host(&host->host_uuid, hostname, registry_hostname, update_every, os, timezone, tags,
host->system_info ? host->system_info->hops : 0);
if (unlikely(rc))
error_report("Failed to store machine GUID to the database");
int rc;
if (!archived) {
rc = sql_store_host_info(host);
if (unlikely(rc))
error_report("Failed to store machine GUID to the database");
}
sql_load_node_id(host);
if (host->health_enabled) {
if (!file_is_migrated(host->health_log_filename)) {
int rc = sql_create_health_log_table(host);
rc = sql_create_health_log_table(host);
if (unlikely(rc)) {
error_report("Failed to create health log table in the database");
health_alarm_log_load(host);
@ -402,7 +404,7 @@ RRDHOST *rrdhost_create(const char *hostname,
error(
"Host '%s': cannot initialize host with machine guid '%s'. Failed to initialize DB engine at '%s'.",
host->hostname, host->machine_guid, host->cache_dir);
rrdhost_free(host);
rrdhost_free(host, 1);
host = NULL;
//rrd_hosts_available++; //TODO: maybe we want this?
@ -447,8 +449,6 @@ RRDHOST *rrdhost_create(const char *hostname,
host->system_info->mc_version = enable_metric_correlations ? metric_correlations_version : 0;
}
ml_new_host(host);
info("Host '%s' (at registry as '%s') with guid '%s' initialized"
", os '%s'"
", timezone '%s'"
@ -491,6 +491,11 @@ RRDHOST *rrdhost_create(const char *hostname,
rrd_hosts_available++;
rrdhost_load_rrdcontext_data(host);
if (!archived)
ml_new_host(host);
else
rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
return host;
}
@ -571,6 +576,14 @@ void rrdhost_update(RRDHOST *host
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
rrdhost_flag_clear(host, RRDHOST_FLAG_ARCHIVED);
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
if (host->rrdpush_send_destination)
host->destinations = destinations_init(host->rrdpush_send_destination);
host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
if(host->health_enabled) {
int r;
char filename[FILENAME_MAX + 1];
@ -606,6 +619,8 @@ void rrdhost_update(RRDHOST *host
}
}
rrd_hosts_available++;
ml_new_host(host);
rrdhost_load_rrdcontext_data(host);
info("Host %s is not in archived mode anymore", host->hostname);
}
@ -632,6 +647,7 @@ RRDHOST *rrdhost_find_or_create(
, char *rrdpush_api_key
, char *rrdpush_send_charts_matching
, struct rrdhost_system_info *system_info
, bool archived
) {
debug(D_RRDHOST, "Searching for host '%s' with guid '%s'", hostname, guid);
@ -641,7 +657,7 @@ RRDHOST *rrdhost_find_or_create(
/* If a legacy memory mode instantiates all dbengine state must be discarded to avoid inconsistencies */
error("Archived host '%s' has memory mode '%s', but the wanted one is '%s'. Discarding archived state.",
host->hostname, rrd_memory_mode_name(host->rrd_memory_mode), rrd_memory_mode_name(mode));
rrdhost_free(host);
rrdhost_free(host, 1);
host = NULL;
}
if(!host) {
@ -666,6 +682,7 @@ RRDHOST *rrdhost_find_or_create(
, rrdpush_send_charts_matching
, system_info
, 0
, archived
);
}
else {
@ -705,6 +722,7 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
if(host != protected_host
&& host != localhost
&& rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)
&& !rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)
&& !host->receiver
&& host->senders_disconnected_time
&& host->senders_disconnected_time + rrdhost_free_orphan_time < now)
@ -733,7 +751,7 @@ restart_after_removal:
else
rrdhost_save_charts(host);
rrdhost_free(host);
rrdhost_free(host, 0);
goto restart_after_removal;
}
}
@ -750,6 +768,10 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
info("Skipping SQLITE metadata initialization since memory mode is not dbengine");
}
if (unlikely(sql_init_context_database(0))) {
error_report("Failed to initialize context metadata database");
}
#ifdef ENABLE_DBENGINE
storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
if(storage_tiers < 1) {
@ -895,6 +917,7 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
, default_rrdpush_send_charts_matching
, system_info
, 1
, 0
);
if (unlikely(!localhost)) {
rrd_unlock();
@ -903,8 +926,8 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
if (likely(system_info))
migrate_localhost(&localhost->host_uuid);
sql_aclk_sync_init();
rrd_unlock();
sql_aclk_sync_init();
web_client_api_v1_management_init();
return localhost==NULL;
@ -988,19 +1011,12 @@ void rrdhost_system_info_free(struct rrdhost_system_info *system_info) {
}
void destroy_receiver_state(struct receiver_state *rpt);
void rrdhost_free(RRDHOST *host) {
if(!host) return;
info("Freeing all memory for host '%s'...", host->hostname);
void stop_streaming_sender(RRDHOST *host)
{
if (unlikely(!host->sender))
return;
rrd_check_wrlock(); // make sure the RRDs are write locked
rrdhost_wrlock(host);
ml_delete_host(host);
rrdhost_unlock(host);
// ------------------------------------------------------------------------
// clean up streaming
rrdpush_sender_thread_stop(host); // stop a possibly running thread
cbuffer_free(host->sender->buffer);
buffer_free(host->sender->build);
@ -1010,40 +1026,45 @@ void rrdhost_free(RRDHOST *host) {
#endif
freez(host->sender);
host->sender = NULL;
if (netdata_exit) {
netdata_mutex_lock(&host->receiver_lock);
if (host->receiver) {
if (!host->receiver->exited)
netdata_thread_cancel(host->receiver->thread);
netdata_mutex_unlock(&host->receiver_lock);
struct receiver_state *rpt = host->receiver;
while (host->receiver && !rpt->exited)
sleep_usec(50 * USEC_PER_MS);
// If the receiver detached from the host then its thread will destroy the state
if (host->receiver == rpt)
destroy_receiver_state(host->receiver);
}
else
netdata_mutex_unlock(&host->receiver_lock);
}
}
void stop_streaming_receiver(RRDHOST *host)
{
netdata_mutex_lock(&host->receiver_lock);
if (host->receiver) {
if (!host->receiver->exited)
netdata_thread_cancel(host->receiver->thread);
netdata_mutex_unlock(&host->receiver_lock);
struct receiver_state *rpt = host->receiver;
while (host->receiver && !rpt->exited)
sleep_usec(50 * USEC_PER_MS);
// If the receiver detached from the host then its thread will destroy the state
if (host->receiver == rpt)
destroy_receiver_state(host->receiver);
} else
netdata_mutex_unlock(&host->receiver_lock);
}
void rrdhost_free(RRDHOST *host, bool force) {
if(!host) return;
if (netdata_exit || force)
info("Freeing all memory for host '%s'...", host->hostname);
rrd_check_wrlock(); // make sure the RRDs are write locked
rrdhost_wrlock(host);
ml_delete_host(host);
rrdhost_unlock(host);
// ------------------------------------------------------------------------
// clean up streaming
stop_streaming_sender(host);
if (netdata_exit || force)
stop_streaming_receiver(host);
rrdhost_wrlock(host); // lock this RRDHOST
#ifdef ENABLE_ACLK
struct aclk_database_worker_config *wc = host->dbsync_worker;
if (wc && !netdata_exit) {
struct aclk_database_cmd cmd;
memset(&cmd, 0, sizeof(cmd));
cmd.opcode = ACLK_DATABASE_ORPHAN_HOST;
struct aclk_completion compl ;
init_aclk_completion(&compl );
cmd.completion = &compl ;
aclk_database_enq_cmd(wc, &cmd);
wait_for_aclk_completion(&compl );
destroy_aclk_completion(&compl );
}
#endif
// ------------------------------------------------------------------------
// release its children resources
@ -1095,13 +1116,34 @@ void rrdhost_free(RRDHOST *host) {
}
#endif
if (!netdata_exit && !force) {
info("Setting archive mode for host '%s'...", host->hostname);
rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
rrdhost_unlock(host);
return;
}
#ifdef ENABLE_ACLK
struct aclk_database_worker_config *wc = host->dbsync_worker;
if (wc && !netdata_exit) {
struct aclk_database_cmd cmd;
memset(&cmd, 0, sizeof(cmd));
cmd.opcode = ACLK_DATABASE_ORPHAN_HOST;
struct aclk_completion compl ;
init_aclk_completion(&compl );
cmd.completion = &compl ;
aclk_database_enq_cmd(wc, &cmd);
wait_for_aclk_completion(&compl );
destroy_aclk_completion(&compl );
}
#endif
// ------------------------------------------------------------------------
// remove it from the indexes
if(rrdhost_index_del(host) != host)
error("RRDHOST '%s' removed from index, deleted the wrong entry.", host->hostname);
// ------------------------------------------------------------------------
// unlink it from the host
@ -1118,8 +1160,6 @@ void rrdhost_free(RRDHOST *host) {
else error("Request to free RRDHOST '%s': cannot find it", host->hostname);
}
// ------------------------------------------------------------------------
// free it
@ -1138,6 +1178,12 @@ void rrdhost_free(RRDHOST *host) {
freez(host->varlib_dir);
freez(host->rrdpush_send_api_key);
freez(host->rrdpush_send_destination);
struct rrdpush_destinations *tmp_destination;
while (host->destinations) {
tmp_destination = host->destinations->next;
freez(host->destinations);
host->destinations = tmp_destination;
}
freez(host->health_default_exec);
freez(host->health_default_recipient);
freez(host->health_log_filename);
@ -1149,6 +1195,8 @@ void rrdhost_free(RRDHOST *host) {
netdata_rwlock_destroy(&host->rrdhost_rwlock);
freez(host->node_id);
rrdhost_destroy_rrdcontexts(host);
freez(host);
#ifdef ENABLE_ACLK
if (wc)
@ -1160,8 +1208,8 @@ void rrdhost_free(RRDHOST *host) {
void rrdhost_free_all(void) {
rrd_wrlock();
/* Make sure child-hosts are released before the localhost. */
while(localhost->next) rrdhost_free(localhost->next);
rrdhost_free(localhost);
while(localhost->next) rrdhost_free(localhost->next, 1);
rrdhost_free(localhost, 1);
rrd_unlock();
}
@ -1342,7 +1390,7 @@ void rrdhost_delete_charts(RRDHOST *host) {
rrdset_foreach_write(st, host) {
rrdset_rdlock(st);
rrdset_delete(st);
rrdset_delete_files(st);
rrdset_unlock(st);
}
@ -1370,7 +1418,7 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
rrdset_rdlock(st);
if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE))
rrdset_delete(st);
rrdset_delete_files(st);
else if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS))
rrdset_delete_obsolete_dimensions(st);
else
@ -1518,7 +1566,7 @@ restart_after_removal:
rrdset_rdlock(st);
if(rrdhost_delete_obsolete_charts)
rrdset_delete(st);
rrdset_delete_files(st);
else
rrdset_save(st);

94
database/rrdlabels.c
View file

@ -369,11 +369,12 @@ __attribute__((constructor)) void initialize_labels_keys_char_map(void) {
}
static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, size_t dst_size, unsigned char *char_map, bool utf) {
static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, size_t dst_size, unsigned char *char_map, bool utf, const char *empty) {
if(unlikely(!dst_size)) return 0;
if(unlikely(!src || !*src)) {
*dst = '\0';
return 0;
strncpyz((char *)dst, empty, dst_size);
dst[dst_size - 1] = '\0';
return strlen((char *)dst);
}
unsigned char *d = dst;
@ -447,27 +448,33 @@ static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, s
// check if dst is all underscores and empty it if it is
d = dst;
while(*d == '_') d++;
if(!*d) {
if(unlikely(*d == '\0')) {
*dst = '\0';
mblen = 0;
}
if(unlikely(*dst == '\0')) {
strncpyz((char *)dst, empty, dst_size);
dst[dst_size - 1] = '\0';
return strlen((char *)dst);
}
return mblen;
}
static inline size_t rrdlabels_sanitize_name(char *dst, const char *src, size_t dst_size) {
return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_names_char_map, 0);
return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_names_char_map, 0, "");
}
static inline size_t rrdlabels_sanitize_value(char *dst, const char *src, size_t dst_size) {
return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_values_char_map, 1);
return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_values_char_map, 1, "[none]");
}
// ----------------------------------------------------------------------------
// rrdlabels_create()
typedef struct rrdlabel {
const char *value;
STRING *label_value;
RRDLABEL_SRC label_source;
} RRDLABEL;
@ -476,9 +483,9 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data)
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
RRDLABEL *lb = (RRDLABEL *)value;
// allocate our own memory for the value
lb->value = strdupz(lb->value);
lb->label_source |= RRDLABEL_FLAG_NEW;
// label_value is already allocated by the STRING
lb->label_source |= RRDLABEL_FLAG_NEW;
lb->label_source &= ~RRDLABEL_FLAG_OLD;
}
static void rrdlabel_delete_callback(const char *name, void *value, void *data) {
@ -486,8 +493,8 @@ static void rrdlabel_delete_callback(const char *name, void *value, void *data)
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
RRDLABEL *lb = (RRDLABEL *)value;
freez((void *)lb->value);
lb->value = NULL;
string_freez(lb->label_value);
lb->label_value = NULL;
}
static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) {
@ -496,17 +503,22 @@ static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *n
RRDLABEL *lbold = (RRDLABEL *)oldvalue;
RRDLABEL *lbnew = (RRDLABEL *)newvalue;
if(strcmp(lbold->value, lbnew->value) == 0) {
if(lbold->label_value == lbnew->label_value || strcmp(string2str(lbold->label_value), string2str(lbnew->label_value)) == 0) {
// they are the same
lbold->label_source |= lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_OLD;
lbold->label_source |= lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_OLD;
lbold->label_source &= ~RRDLABEL_FLAG_NEW;
// free the new one
string_freez(lbnew->label_value);
}
else {
// they are different
freez((void *)lbold->value);
lbold->value = strdupz(lbnew->value);
lbold->label_source = lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_NEW;
string_freez(lbold->label_value);
lbold->label_value = lbnew->label_value;
lbold->label_source = lbnew->label_source;
lbold->label_source |= RRDLABEL_FLAG_NEW;
lbold->label_source &= ~RRDLABEL_FLAG_OLD;
}
}
@ -536,7 +548,7 @@ static void labels_add_already_sanitized(DICTIONARY *dict, const char *key, cons
RRDLABEL tmp = {
.label_source = ls,
.value = value
.label_value = string_strdupz(value)
};
dictionary_set(dict, key, &tmp, sizeof(RRDLABEL));
}
@ -611,11 +623,11 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) {
// rrdlabels_get_to_buffer_or_null()
void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) {
void *acquired_item = dictionary_acquire_item(labels, key);
RRDLABEL *lb = dictionary_acquired_item_value(labels, acquired_item);
DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
if(lb && lb->value)
buffer_sprintf(wb, "%s%s%s", quote, lb->value, quote);
if(lb && lb->label_value)
buffer_sprintf(wb, "%s%s%s", quote, string2str(lb->label_value), quote);
else
buffer_strcat(wb, null);
@ -652,7 +664,7 @@ static int remove_not_old_not_new_callback(const char *name, void *value, void *
DICTIONARY *dict = (DICTIONARY *)data;
RRDLABEL *lb = (RRDLABEL *)value;
if(!(lb->label_source & RRDLABEL_FLAG_OLD) && !(lb->label_source & RRDLABEL_FLAG_NEW)) {
if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) {
dictionary_del_having_write_lock(dict, name);
return 1;
}
@ -681,7 +693,7 @@ static int labels_walkthrough_callback(const char *name, void *value, void *data
if(ls & RRDLABEL_FLAG_NEW) ls &= ~RRDLABEL_FLAG_NEW;
if(ls & RRDLABEL_FLAG_OLD) ls &= ~RRDLABEL_FLAG_OLD;
return d->callback(name, lb->value, ls, d->data);
return d->callback(name, string2str(lb->label_value), ls, d->data);
}
int rrdlabels_walkthrough_read(DICTIONARY *labels, int (*callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *data) {
@ -708,7 +720,7 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const
static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) {
DICTIONARY *dst = (DICTIONARY *)data;
RRDLABEL *lb = (RRDLABEL *)value;
labels_add_already_sanitized(dst, name, lb->value, lb->label_source);
labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source);
return 1;
}
@ -761,7 +773,7 @@ static int simple_pattern_match_name_and_value_callback(const char *name, void *
size_t len = RRDLABELS_MAX_NAME_LENGTH + RRDLABELS_MAX_VALUE_LENGTH + 2; // +1 for =, +1 for \0
char tmp[len], *dst = &tmp[0];
const char *v = lb->value;
const char *v = string2str(lb->label_value);
// copy the name
while(*name) *dst++ = *name++;
@ -822,7 +834,7 @@ static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value,
BUFFER *wb = (BUFFER *)data;
RRDLABEL *lb = (RRDLABEL *)value;
buffer_sprintf(wb, "Label: %s: \"%s\" (", name, lb->value);
buffer_sprintf(wb, "Label: %s: \"%s\" (", name, string2str(lb->label_value));
size_t sources = 0;
if(lb->label_source & RRDLABEL_SRC_AUTO) {
@ -878,7 +890,7 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
char n[n_size];
char v[v_size];
const char *nn = name, *vv = lb->value;
const char *nn = name, *vv = string2str(lb->label_value);
if(t->name_sanitizer) {
t->name_sanitizer(n, name, n_size);
@ -886,11 +898,11 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
}
if(t->value_sanitizer) {
t->value_sanitizer(v, lb->value, v_size);
t->value_sanitizer(v, string2str(lb->label_value), v_size);
vv = v;
}
if(!t->filter_callback || (t->filter_callback && t->filter_callback(name, lb->value, lb->label_source, t->filter_data))) {
if(!t->filter_callback || t->filter_callback(name, string2str(lb->label_value), lb->label_source, t->filter_data)) {
buffer_sprintf(t->wb, "%s%s%s%s%s%s%s%s%s", t->count++?t->between_them:"", t->before_each, t->quote, nn, t->quote, t->equal, t->quote, vv, t->quote);
return 1;
}
@ -898,7 +910,7 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
return 0;
}
void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size)) {
int rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size)) {
struct labels_to_buffer tmp = {
.wb = wb,
.filter_callback = filter_callback,
@ -911,7 +923,7 @@ void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each
.between_them = between_them,
.count = 0
};
dictionary_sorted_walkthrough_read(labels, label_to_buffer_callback, (void *)&tmp);
return dictionary_walkthrough_read(labels, label_to_buffer_callback, (void *)&tmp);
}
static int chart_label_store_to_sql_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
@ -1024,12 +1036,12 @@ int rrdlabels_unittest_add_pairs() {
errors += rrdlabels_unittest_add_a_pair("tag:a", "tag", "a");
// test empty values
errors += rrdlabels_unittest_add_a_pair("tag", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag:", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag:\"\"", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag:''", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag:\r\n", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag\r\n", "tag", "");
errors += rrdlabels_unittest_add_a_pair("tag", "tag", "[none]");
errors += rrdlabels_unittest_add_a_pair("tag:", "tag", "[none]");
errors += rrdlabels_unittest_add_a_pair("tag:\"\"", "tag", "[none]");
errors += rrdlabels_unittest_add_a_pair("tag:''", "tag", "[none]");
errors += rrdlabels_unittest_add_a_pair("tag:\r\n", "tag", "[none]");
errors += rrdlabels_unittest_add_a_pair("tag\r\n", "tag", "[none]");
// test UTF-8 in values
errors += rrdlabels_unittest_add_a_pair("tag: country:Ελλάδα", "tag", "country:Ελλάδα");
@ -1107,7 +1119,7 @@ int rrdlabels_unittest_sanitize_value(const char *src, const char *expected) {
int rrdlabels_unittest_sanitization() {
int errors = 0;
errors += rrdlabels_unittest_sanitize_value("", "");
errors += rrdlabels_unittest_sanitize_value("", "[none]");
errors += rrdlabels_unittest_sanitize_value("1", "1");
errors += rrdlabels_unittest_sanitize_value(" hello world ", "hello world");

26
database/rrdset.c
View file

@ -186,6 +186,7 @@ int rrdset_set_name(RRDSET *st, const char *name) {
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_IGNORE);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdcontext_updated_rrdset_name(st);
return 2;
}
@ -204,6 +205,7 @@ inline void rrdset_is_obsolete(RRDSET *st) {
// the chart will not get more updates (data collection)
// so, we have to push its definition now
rrdset_push_chart_definition_now(st);
rrdcontext_updated_rrdset_flags(st);
}
}
@ -216,6 +218,7 @@ inline void rrdset_isnot_obsolete(RRDSET *st) {
// the chart will be pushed upstream automatically
// due to data collection
rrdcontext_updated_rrdset_flags(st);
}
}
@ -251,6 +254,7 @@ inline void rrdset_update_heterogeneous_flag(RRDSET *st) {
}
rrdset_flag_clear(st, RRDSET_FLAG_HETEROGENEOUS);
rrdcontext_updated_rrdset_flags(st);
}
// ----------------------------------------------------------------------------
@ -383,6 +387,9 @@ void rrdset_free(RRDSET *st) {
rrdset_unlock(st);
// this has to be after the dimensions are freed
rrdcontext_removed_rrdset(st);
// ------------------------------------------------------------------------
// free it
@ -419,7 +426,7 @@ void rrdset_save(RRDSET *st) {
rrddim_memory_file_save(rd);
}
void rrdset_delete(RRDSET *st) {
void rrdset_delete_files(RRDSET *st) {
RRDDIM *rd;
rrdset_check_rdlock(st);
@ -480,6 +487,14 @@ static inline RRDSET *rrdset_find_on_create(RRDHOST *host, const char *fullid) {
return NULL;
}
static inline void rrdset_update_permanent_labels(RRDSET *st) {
if(!st->state || !st->state->chart_labels) return;
rrdlabels_add(st->state->chart_labels, "_collect_plugin", st->plugin_name, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
rrdlabels_add(st->state->chart_labels, "_collect_module", st->module_name, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
rrdlabels_add(st->state->chart_labels, "_instance_family", st->family, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
}
RRDSET *rrdset_create_custom(
RRDHOST *host
, const char *type
@ -659,8 +674,11 @@ RRDSET *rrdset_create_custom(
}
}
/* Fall-through during switch from archived to active so that the host lock is taken and health is linked */
if (!changed_from_archived_to_active)
if (!changed_from_archived_to_active) {
rrdset_update_permanent_labels(st);
rrdcontext_updated_rrdset(st);
return st;
}
}
rrdhost_wrlock(host);
@ -680,6 +698,7 @@ RRDSET *rrdset_create_custom(
rrdhost_unlock(host);
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
rrdcontext_updated_rrdset(st);
return st;
}
@ -748,6 +767,7 @@ RRDSET *rrdset_create_custom(
netdata_rwlock_init(&st->rrdset_rwlock);
st->state->chart_labels = rrdlabels_create();
rrdset_update_permanent_labels(st);
if(name && *name && rrdset_set_name(st, name))
// we did set the name
@ -789,6 +809,7 @@ RRDSET *rrdset_create_custom(
compute_chart_hash(st);
rrdhost_unlock(host);
rrdcontext_updated_rrdset(st);
return(st);
}
@ -1280,6 +1301,7 @@ void rrdset_done(RRDSET *st) {
if(unlikely(netdata_exit)) return;
debug(D_RRD_CALLS, "rrdset_done() for chart %s", st->name);
rrdcontext_collected_rrdset(st);
RRDDIM *rd;

95
database/sqlite/sqlite_aclk.c
View file

@ -258,6 +258,68 @@ void aclk_sync_exit_all()
uv_mutex_unlock(&aclk_async_lock);
}
#ifdef ENABLE_ACLK
enum {
IDX_HOST_ID,
IDX_HOSTNAME,
IDX_REGISTRY,
IDX_UPDATE_EVERY,
IDX_OS,
IDX_TIMEZONE,
IDX_TAGS,
IDX_HOPS,
IDX_MEMORY_MODE,
IDX_ABBREV_TIMEZONE,
IDX_UTC_OFFSET,
IDX_PROGRAM_NAME,
IDX_PROGRAM_VERSION,
IDX_ENTRIES,
IDX_HEALTH_ENABLED,
};
static int create_host_callback(void *data, int argc, char **argv, char **column)
{
UNUSED(data);
UNUSED(argc);
UNUSED(column);
char guid[UUID_STR_LEN];
uuid_unparse_lower(*(uuid_t *)argv[IDX_HOST_ID], guid);
struct rrdhost_system_info *system_info = callocz(1, sizeof(struct rrdhost_system_info));
system_info->hops = str2i((const char *) argv[IDX_HOPS]);
sql_build_host_system_info((uuid_t *)argv[IDX_HOST_ID], system_info);
RRDHOST *host = rrdhost_find_or_create(
(const char *) argv[IDX_HOSTNAME]
, (const char *) argv[IDX_REGISTRY]
, guid
, (const char *) argv[IDX_OS]
, (const char *) argv[IDX_TIMEZONE]
, (const char *) argv[IDX_ABBREV_TIMEZONE]
, argv[IDX_UTC_OFFSET] ? str2uint32_t(argv[IDX_UTC_OFFSET]) : 0
, (const char *) argv[IDX_TAGS]
, (const char *) (argv[IDX_PROGRAM_NAME] ? argv[IDX_PROGRAM_NAME] : "unknown")
, (const char *) (argv[IDX_PROGRAM_VERSION] ? argv[IDX_PROGRAM_VERSION] : "unknown")
, argv[3] ? str2i(argv[IDX_UPDATE_EVERY]) : 1
, argv[13] ? str2i(argv[IDX_ENTRIES]) : 0
, RRD_MEMORY_MODE_DBENGINE
, 0 // health
, 0 // rrdpush enabled
, NULL //destination
, NULL // api key
, NULL // send charts matching
, system_info
, 1
);
char node_str[UUID_STR_LEN] = "<none>";
uuid_unparse_lower(*host->node_id, node_str);
internal_error(true, "Adding archived host \"%s\" with GUID \"%s\" node id = \"%s\"", host->hostname, host->machine_guid, node_str);
return 0;
}
#endif
int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
{
char uuid_str[GUID_LEN + 1];
@ -267,10 +329,11 @@ int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
uuid_unparse_lower(*((uuid_t *) argv[0]), uuid_str);
if (rrdhost_find_by_guid(uuid_str, 0) == localhost)
RRDHOST *host = rrdhost_find_by_guid(uuid_str, 0);
if (host == localhost)
return 0;
sql_create_aclk_table(NULL, (uuid_t *) argv[0], (uuid_t *) argv[1]);
sql_create_aclk_table(host, (uuid_t *) argv[0], (uuid_t *) argv[1]);
return 0;
}
@ -303,8 +366,23 @@ void sql_aclk_sync_init(void)
info("SQLite aclk sync initialization completed");
fatal_assert(0 == uv_mutex_init(&aclk_async_lock));
if (likely(rrdcontext_enabled == CONFIG_BOOLEAN_YES)) {
rc = sqlite3_exec(db_meta, "SELECT host_id, hostname, registry_hostname, update_every, os, "
"timezone, tags, hops, memory_mode, abbrev_timezone, utc_offset, program_name, "
"program_version, entries, health_enabled FROM host WHERE hops >0;",
create_host_callback, NULL, &err_msg);
if (rc != SQLITE_OK) {
error_report("SQLite error when loading archived hosts, rc = %d (%s)", rc, err_msg);
sqlite3_free(err_msg);
}
}
rc = sqlite3_exec(db_meta, "SELECT ni.host_id, ni.node_id FROM host h, node_instance ni WHERE "
"h.host_id = ni.host_id AND ni.node_id IS NOT NULL;", aclk_start_sync_thread, NULL, NULL);
"h.host_id = ni.host_id AND ni.node_id IS NOT NULL;", aclk_start_sync_thread, NULL, &err_msg);
if (rc != SQLITE_OK) {
error_report("SQLite error when starting ACLK sync threads, rc = %d (%s)", rc, err_msg);
sqlite3_free(err_msg);
}
#endif
return;
}
@ -803,6 +881,17 @@ void sql_maint_aclk_sync_database(struct aclk_database_worker_config *wc, struct
buffer_sprintf(sql,"DELETE FROM aclk_alert_%s WHERE date_submitted IS NOT NULL AND "
"date_cloud_ack < unixepoch()-%d;", wc->uuid_str, ACLK_DELETE_ACK_ALERTS_INTERNAL);
db_execute(buffer_tostring(sql));
buffer_flush(sql);
buffer_sprintf(sql,"UPDATE aclk_chart_%s SET status = NULL, date_submitted=unixepoch() WHERE "
"date_submitted IS NULL AND date_created < unixepoch()-%d;", wc->uuid_str, ACLK_AUTO_MARK_SUBMIT_INTERVAL);
db_execute(buffer_tostring(sql));
buffer_flush(sql);
buffer_sprintf(sql,"UPDATE aclk_chart_%s SET date_updated = unixepoch() WHERE date_updated IS NULL"
" AND date_submitted IS NOT NULL AND date_submitted < unixepoch()-%d;",
wc->uuid_str, ACLK_AUTO_MARK_UPDATED_INTERVAL);
db_execute(buffer_tostring(sql));
buffer_free(sql);
return;

2
database/sqlite/sqlite_aclk.h
View file

@ -22,6 +22,8 @@
#define ACLK_DATABASE_ROTATION_INTERVAL (3600)
#define ACLK_DELETE_ACK_INTERNAL (600)
#define ACLK_DELETE_ACK_ALERTS_INTERNAL (86400)
#define ACLK_AUTO_MARK_SUBMIT_INTERVAL (3600)
#define ACLK_AUTO_MARK_UPDATED_INTERVAL (1800)
#define ACLK_SYNC_QUERY_SIZE 512
struct aclk_completion {

6
database/sqlite/sqlite_aclk_alert.c
View file

@ -224,7 +224,7 @@ void aclk_push_alert_event(struct aclk_database_worker_config *wc, struct aclk_d
return;
}
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -449,7 +449,7 @@ void aclk_push_alarm_health_log(struct aclk_database_worker_config *wc, struct a
#else
int rc;
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -911,7 +911,7 @@ void aclk_push_alert_snapshot_event(struct aclk_database_worker_config *wc, stru
if (unlikely(!wc->alerts_snapshot_id))
return;
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;

30
database/sqlite/sqlite_aclk_chart.c
View file

@ -143,7 +143,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
int rc = 0;
CHECK_SQLITE_CONNECTION(db_meta);
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
RRDSET *st = cmd.data;
@ -222,7 +222,7 @@ void aclk_process_dimension_deletion(struct aclk_database_worker_config *wc, str
if (uuid_parse(wc->host_guid, host_id))
return;
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (!claim_id)
return;
@ -281,7 +281,7 @@ int aclk_add_dimension_event(struct aclk_database_worker_config *wc, struct aclk
struct aclk_chart_dimension_data *aclk_cd_data = cmd.data;
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (!claim_id)
goto cleanup;
@ -308,7 +308,7 @@ void aclk_send_chart_event(struct aclk_database_worker_config *wc, struct aclk_d
return;
}
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -736,7 +736,7 @@ void aclk_start_streaming(char *node_id, uint64_t sequence_id, time_t created_at
wc->chart_reset_count);
chart_reset_t chart_reset;
chart_reset.claim_id = is_agent_claimed();
chart_reset.claim_id = get_agent_claimid();
if (chart_reset.claim_id) {
chart_reset.node_id = node_id;
chart_reset.reason = SEQ_ID_NOT_EXISTS;
@ -825,7 +825,12 @@ void aclk_update_retention(struct aclk_database_worker_config *wc)
if (!aclk_connected)
return;
char *claim_id = is_agent_claimed();
if (wc->host && rrdhost_flag_check(wc->host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
internal_error(true, "Skipping aclk_update_retention for host %s because context streaming is enabled", wc->host->hostname);
return;
}
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -1060,6 +1065,10 @@ void sql_get_last_chart_sequence(struct aclk_database_worker_config *wc)
void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
{
RRDHOST *host = rd->rrdset->rrdhost;
if (likely(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)))
return;
int live = !last_updated;
if (likely(rd->aclk_live_status == live))
@ -1076,7 +1085,7 @@ void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
if (unlikely(!wc))
return;
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -1120,7 +1129,7 @@ void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
void aclk_send_dimension_update(RRDDIM *rd)
{
char *claim_id = is_agent_claimed();
char *claim_id = get_agent_claimid();
if (unlikely(!claim_id))
return;
@ -1290,6 +1299,11 @@ void sql_check_chart_liveness(RRDSET *st) {
// ST is read locked
int queue_chart_to_aclk(RRDSET *st)
{
RRDHOST *host = st->rrdhost;
if (likely(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)))
return 0;
return sql_queue_chart_payload((struct aclk_database_worker_config *) st->rrdhost->dbsync_worker,
st, ACLK_DATABASE_ADD_CHART);
}

5
database/sqlite/sqlite_aclk_node.c
View file

@ -39,7 +39,7 @@ void sql_build_node_collectors(struct aclk_database_worker_config *wc)
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
upd_node_collectors.node_id = wc->node_id;
upd_node_collectors.claim_id = is_agent_claimed();
upd_node_collectors.claim_id = get_agent_claimid();
upd_node_collectors.node_collectors = collectors_from_charts(wc->host, dict);
aclk_update_node_collectors(&upd_node_collectors);
@ -68,7 +68,7 @@ void sql_build_node_info(struct aclk_database_worker_config *wc, struct aclk_dat
rrd_rdlock();
node_info.node_id = wc->node_id;
node_info.claim_id = is_agent_claimed();
node_info.claim_id = get_agent_claimid();
node_info.machine_guid = wc->host_guid;
node_info.child = (wc->host != localhost);
node_info.ml_info.ml_capable = ml_capable(localhost);
@ -78,6 +78,7 @@ void sql_build_node_info(struct aclk_database_worker_config *wc, struct aclk_dat
{ .name = "proto", .version = 1, .enabled = 1 },
{ .name = "ml", .version = ml_capable(localhost), .enabled = ml_enabled(wc->host) },
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
{ .name = "ctx", .version = 1, .enabled = rrdcontext_enabled},
{ .name = NULL, .version = 0, .enabled = 0 }
};
node_info.node_instance_capabilities = instance_caps;

598
database/sqlite/sqlite_context.c Normal file
View file

@ -0,0 +1,598 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#include "sqlite_functions.h"
#include "sqlite_context.h"
#include "sqlite_db_migration.h"
#define DB_CONTEXT_METADATA_VERSION 1
const char *database_context_config[] = {
"CREATE TABLE IF NOT EXISTS context (host_id BLOB, id TEXT NOT NULL, version INT NOT NULL, title TEXT NOT NULL, " \
"chart_type TEXT NOT NULL, unit TEXT NOT NULL, priority INT NOT NULL, first_time_t INT NOT NULL, "
"last_time_t INT NOT NULL, deleted INT NOT NULL, "
"family TEXT, PRIMARY KEY (host_id, id));",
NULL
};
const char *database_context_cleanup[] = {
"VACUUM;",
NULL
};
sqlite3 *db_context_meta = NULL;
/*
* Initialize the SQLite database
* Return 0 on success
*/
int sql_init_context_database(int memory)
{
char sqlite_database[FILENAME_MAX + 1];
int rc;
if (likely(!memory))
snprintfz(sqlite_database, FILENAME_MAX, "%s/context-meta.db", netdata_configured_cache_dir);
else
strcpy(sqlite_database, ":memory:");
rc = sqlite3_open(sqlite_database, &db_context_meta);
if (rc != SQLITE_OK) {
error_report("Failed to initialize database at %s, due to \"%s\"", sqlite_database, sqlite3_errstr(rc));
sqlite3_close(db_context_meta);
db_context_meta = NULL;
return 1;
}
info("SQLite database %s initialization", sqlite_database);
char buf[1024 + 1] = "";
const char *list[2] = { buf, NULL };
int target_version = DB_CONTEXT_METADATA_VERSION;
if (likely(!memory))
target_version = perform_context_database_migration(db_context_meta, DB_CONTEXT_METADATA_VERSION);
// https://www.sqlite.org/pragma.html#pragma_auto_vacuum
// PRAGMA schema.auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL;
snprintfz(buf, 1024, "PRAGMA auto_vacuum=%s;", config_get(CONFIG_SECTION_SQLITE, "auto vacuum", "INCREMENTAL"));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_synchronous
// PRAGMA schema.synchronous = 0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA;
snprintfz(buf, 1024, "PRAGMA synchronous=%s;", config_get(CONFIG_SECTION_SQLITE, "synchronous", "NORMAL"));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_journal_mode
// PRAGMA schema.journal_mode = DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
snprintfz(buf, 1024, "PRAGMA journal_mode=%s;", config_get(CONFIG_SECTION_SQLITE, "journal mode", "WAL"));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_temp_store
// PRAGMA temp_store = 0 | DEFAULT | 1 | FILE | 2 | MEMORY;
snprintfz(buf, 1024, "PRAGMA temp_store=%s;", config_get(CONFIG_SECTION_SQLITE, "temp store", "MEMORY"));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_journal_size_limit
// PRAGMA schema.journal_size_limit = N ;
snprintfz(buf, 1024, "PRAGMA journal_size_limit=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "journal size limit", 16777216));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_cache_size
// PRAGMA schema.cache_size = pages;
// PRAGMA schema.cache_size = -kibibytes;
snprintfz(buf, 1024, "PRAGMA cache_size=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "cache size", -2000));
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
snprintfz(buf, 1024, "PRAGMA user_version=%d;", target_version);
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
snprintfz(buf, 1024, "ATTACH DATABASE \"%s/netdata-meta.db\" as meta;", netdata_configured_cache_dir);
if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
if (init_database_batch(db_context_meta, DB_CHECK_NONE, 0, &database_context_config[0]))
return 1;
if (init_database_batch(db_context_meta, DB_CHECK_NONE, 0, &database_context_cleanup[0]))
return 1;
return 0;
}
/*
* Close the sqlite database
*/
void sql_close_context_database(void)
{
int rc;
if (unlikely(!db_context_meta))
return;
info("Closing context SQLite database");
rc = sqlite3_close_v2(db_context_meta);
if (unlikely(rc != SQLITE_OK))
error_report("Error %d while closing the context SQLite database, %s", rc, sqlite3_errstr(rc));
return;
}
//
// Fetching data
//
#define CTX_GET_CHART_LIST "SELECT c.chart_id, c.type||'.'||c.id, c.name, c.context, c.title, c.unit, c.priority, " \
"c.update_every, c.chart_type, c.family FROM meta.chart c WHERE c.host_id = @host_id; "
void ctx_get_chart_list(uuid_t *host_uuid, void (*dict_cb)(SQL_CHART_DATA *, void *), void *data)
{
int rc;
sqlite3_stmt *res = NULL;
if (unlikely(!host_uuid)) {
internal_error(true, "Requesting context chart list without host_id");
return;
}
rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_CHART_LIST, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to fetch chart list");
return;
}
rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind host_id to fetch the chart list");
goto skip_load;
}
SQL_CHART_DATA chart_data = { 0 };
while (sqlite3_step(res) == SQLITE_ROW) {
uuid_copy(chart_data.chart_id, *((uuid_t *)sqlite3_column_blob(res, 0)));
chart_data.id = (char *) sqlite3_column_text(res, 1);
chart_data.name = (char *) sqlite3_column_text(res, 2);
chart_data.context = (char *) sqlite3_column_text(res, 3);
chart_data.title = (char *) sqlite3_column_text(res, 4);
chart_data.units = (char *) sqlite3_column_text(res, 5);
chart_data.priority = sqlite3_column_int(res, 6);
chart_data.update_every = sqlite3_column_int(res, 7);
chart_data.chart_type = sqlite3_column_int(res, 8);
chart_data.family = (char *) sqlite3_column_text(res, 9);
dict_cb(&chart_data, data);
}
skip_load:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement that fetches chart label data, rc = %d", rc);
}
// Dimension list
#define CTX_GET_DIMENSION_LIST "SELECT d.dim_id, d.id, d.name FROM meta.dimension d WHERE d.chart_id = @id;"
void ctx_get_dimension_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_DIMENSION_DATA *, void *), void *data)
{
int rc;
sqlite3_stmt *res = NULL;
rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_DIMENSION_LIST, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to fetch chart dimension data");
return;
}
rc = sqlite3_bind_blob(res, 1, chart_uuid, sizeof(*chart_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind chart_id to fetch dimension list");
goto failed;
}
SQL_DIMENSION_DATA dimension_data;
while (sqlite3_step(res) == SQLITE_ROW) {
uuid_copy(dimension_data.dim_id, *((uuid_t *)sqlite3_column_blob(res, 0)));
dimension_data.id = (char *) sqlite3_column_text(res, 1);
dimension_data.name = (char *) sqlite3_column_text(res, 2);
dict_cb(&dimension_data, data);
}
failed:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement that fetches the chart dimension list, rc = %d", rc);
}
// LABEL LIST
#define CTX_GET_LABEL_LIST "SELECT l.label_key, l.label_value, l.source_type FROM meta.chart_label l WHERE l.chart_id = @id;"
void ctx_get_label_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_CLABEL_DATA *, void *), void *data)
{
int rc;
sqlite3_stmt *res = NULL;
rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_LABEL_LIST, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to fetch chart lanbels");
return;
}
rc = sqlite3_bind_blob(res, 1, chart_uuid, sizeof(*chart_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind chart_id to fetch chart labels");
goto failed;
}
SQL_CLABEL_DATA label_data;
while (sqlite3_step(res) == SQLITE_ROW) {
label_data.label_key = (char *) sqlite3_column_text(res, 0);
label_data.label_value = (char *) sqlite3_column_text(res, 1);
label_data.label_source = sqlite3_column_int(res, 2);
dict_cb(&label_data, data);
}
failed:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement that fetches chart label data, rc = %d", rc);
return;
}
// CONTEXT LIST
#define CTX_GET_CONTEXT_LIST "SELECT id, version, title, chart_type, unit, priority, first_time_t, " \
"last_time_t, deleted, family FROM context c WHERE c.host_id = @host_id;"
void ctx_get_context_list(uuid_t *host_uuid, void (*dict_cb)(VERSIONED_CONTEXT_DATA *, void *), void *data)
{
if (unlikely(!host_uuid))
return;
int rc;
sqlite3_stmt *res = NULL;
rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_CONTEXT_LIST, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to fetch stored context list");
return;
}
VERSIONED_CONTEXT_DATA context_data = {0};
rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind host_id to fetch versioned context data");
goto failed;
}
while (sqlite3_step(res) == SQLITE_ROW) {
context_data.id = (char *) sqlite3_column_text(res, 0);
context_data.version = sqlite3_column_int64(res, 1);
context_data.title = (char *) sqlite3_column_text(res, 2);
context_data.chart_type = (char *) sqlite3_column_text(res, 3);
context_data.units = (char *) sqlite3_column_text(res, 4);
context_data.priority = sqlite3_column_int64(res, 5);
context_data.first_time_t = sqlite3_column_int64(res, 6);
context_data.last_time_t = sqlite3_column_int64(res, 7);
context_data.deleted = sqlite3_column_int(res, 8);
context_data.family = (char *) sqlite3_column_text(res, 9);
dict_cb(&context_data, data);
}
failed:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement that fetches stored context versioned data, rc = %d", rc);
}
//
// Storing Data
//
#define CTX_STORE_CONTEXT "INSERT OR REPLACE INTO context " \
"(host_id, id, version, title, chart_type, unit, priority, first_time_t, last_time_t, deleted, family) " \
"VALUES (@host_id, @context, @version, @title, @chart_type, @unit, @priority, @first_time_t, @last_time_t, @deleted, @family);"
int ctx_store_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data)
{
int rc, rc_stored = 1;
sqlite3_stmt *res = NULL;
if (unlikely(!host_uuid || !context_data || !context_data->id))
return 0;
rc = sqlite3_prepare_v2(db_context_meta, CTX_STORE_CONTEXT, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to store context");
return 1;
}
rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind host_uuid to store context details");
goto skip_store;
}
rc = bind_text_null(res, 2, context_data->id, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context to store details");
goto skip_store;
}
rc = sqlite3_bind_int64(res, 3, (time_t) context_data->version);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind first_time_t to store context details");
goto skip_store;
}
rc = bind_text_null(res, 4, context_data->title, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context to store details");
goto skip_store;
}
rc = bind_text_null(res, 5, context_data->chart_type, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context to store details");
goto skip_store;
}
rc = bind_text_null(res, 6, context_data->units, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context to store details");
goto skip_store;
}
rc = sqlite3_bind_int64(res, 7, (time_t) context_data->priority);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind first_time_t to store context details");
goto skip_store;
}
rc = sqlite3_bind_int64(res, 8, (time_t) context_data->first_time_t);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind first_time_t to store context details");
goto skip_store;
}
rc = sqlite3_bind_int64(res, 9, (time_t) context_data->last_time_t);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind last_time_t to store context details");
goto skip_store;
}
rc = sqlite3_bind_int(res, 10, (time_t) context_data->deleted);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind last_time_t to store context details");
goto skip_store;
}
rc = bind_text_null(res, 11, context_data->family, 1);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context to store details");
goto skip_store;
}
rc_stored = execute_insert(res);
if (rc_stored != SQLITE_DONE)
error_report("Failed store context details for context %s, rc = %d", context_data->id, rc_stored);
skip_store:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement that stores context details, rc = %d", rc);
return (rc_stored != SQLITE_DONE);
}
// Delete a context
#define CTX_DELETE_CONTEXT "DELETE FROM context WHERE host_id = @host_id AND id = @context;"
int ctx_delete_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data)
{
int rc, rc_stored = 1;
sqlite3_stmt *res = NULL;
if (unlikely(!context_data || !context_data->id))
return 0;
rc = sqlite3_prepare_v2(db_context_meta, CTX_DELETE_CONTEXT, -1, &res, 0);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to delete context");
return 1;
}
rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind host_id to delete context data");
goto skip_delete;
}
rc = sqlite3_bind_text(res, 2, context_data->id, -1, SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to bind context id for data deletion");
goto skip_delete;
}
rc_stored = execute_insert(res);
if (rc_stored != SQLITE_DONE)
error_report("Failed to delete context %s, rc = %d", context_data->id, rc_stored);
#ifdef NETDATA_INTERNAL_CHECKS
else {
char host_uuid_str[UUID_STR_LEN];
uuid_unparse_lower(*host_uuid, host_uuid_str);
info("%s: Deleted context %s under host %s", __FUNCTION__ , context_data->id, host_uuid_str);
}
#endif
skip_delete:
rc = sqlite3_finalize(res);
if (rc != SQLITE_OK)
error_report("Failed to finalize statement where deleting a context, rc = %d", rc);
return (rc_stored != SQLITE_DONE);
}
//
// TESTING FUNCTIONS
//
static void dict_ctx_get_label_list_cb(SQL_CLABEL_DATA *label_data_ptr, void *data)
{
(void)data;
SQL_CLABEL_DATA *label_data = label_data_ptr;
info(" LABEL %d %s = %s", label_data->label_source, label_data->label_key, label_data->label_value);
}
static void dict_ctx_get_dimension_list_cb(SQL_DIMENSION_DATA *dimension_data_ptr, void *data)
{
(void)data;
SQL_DIMENSION_DATA *dimension_data = dimension_data_ptr;
char uuid_str[UUID_STR_LEN];
uuid_unparse_lower(dimension_data->dim_id, uuid_str);
info(" Dimension %s = %s", uuid_str, dimension_data->id);
}
static void dict_ctx_get_chart_list_cb(SQL_CHART_DATA *chart_data, void *data)
{
(void)data;
char uuid_str[UUID_STR_LEN];
uuid_unparse_lower(chart_data->chart_id, uuid_str);
info("OK GOT %s ID = %s NAME = %s CONTEXT = %s", uuid_str, chart_data->id, chart_data->name, chart_data->context);
ctx_get_label_list(&chart_data->chart_id, dict_ctx_get_label_list_cb, NULL);
ctx_get_dimension_list(&chart_data->chart_id, dict_ctx_get_dimension_list_cb, NULL);
}
static void dict_ctx_get_context_list_cb(VERSIONED_CONTEXT_DATA *context_data, void *data)
{
(void)data;
info(" Context id = %s "
"version = %lu "
"title = %s "
"chart_type = %s "
"units = %s "
"priority = %lu "
"first time = %lu "
"last time = %lu "
"deleted = %d"
"family = %s",
context_data->id,
context_data->version,
context_data->title,
context_data->chart_type,
context_data->units,
context_data->priority,
context_data->first_time_t,
context_data->last_time_t,
context_data->deleted,
context_data->family);
}
static int localhost_uuid_cb(void *data, int argc, char **argv, char **column)
{
uuid_t *uuid = data;
UNUSED(argc);
UNUSED(column);
uuid_copy(*uuid, * (uuid_t *) argv[0]);
return 0;
}
#define SQL_FIND_LOCALHOST "SELECT host_id FROM meta.host WHERE hops = 0;"
int ctx_unittest(void)
{
uuid_t host_uuid;
uuid_t host_uuid1;
uuid_generate(host_uuid1);
char *err_msg;
sql_init_context_database(1);
int rc = sqlite3_exec(db_context_meta, SQL_FIND_LOCALHOST, localhost_uuid_cb, (void *) &host_uuid, &err_msg);
if (rc != SQLITE_OK) {
info("Failed to discover localhost UUID rc = %d -- %s", rc, err_msg);
sqlite3_free(err_msg);
}
ctx_get_chart_list(&host_uuid, dict_ctx_get_chart_list_cb, NULL);
// Store a context
VERSIONED_CONTEXT_DATA context_data;
context_data.id = strdupz("cpu.cpu");
context_data.title = strdupz("TestContextTitle");
context_data.units= strdupz("TestContextUnits");
context_data.chart_type = strdupz("TestContextChartType");
context_data.family = strdupz("TestContextFamily");
context_data.priority = 50000;
context_data.deleted = 0;
context_data.first_time_t = 1657781000;
context_data.last_time_t = 1657781100;
context_data.version = now_realtime_usec();
if (likely(!ctx_store_context(&host_uuid, &context_data)))
info("Entry %s inserted", context_data.id);
else
info("Entry %s not inserted", context_data.id);
if (likely(!ctx_store_context(&host_uuid1, &context_data)))
info("Entry %s inserted", context_data.id);
else
info("Entry %s not inserted", context_data.id);
// This will change end time
context_data.first_time_t = 1657781000;
context_data.last_time_t = 1657782001;
if (likely(!ctx_update_context(&host_uuid, &context_data)))
info("Entry %s updated", context_data.id);
else
info("Entry %s not updated", context_data.id);
info("List context start after insert");
ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
info("List context end after insert");
// This will change start time
context_data.first_time_t = 1657782000;
context_data.last_time_t = 1657782001;
if (likely(!ctx_update_context(&host_uuid, &context_data)))
info("Entry %s updated", context_data.id);
else
info("Entry %s not updated", context_data.id);
// This will list one entry
info("List context start after insert");
ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
info("List context end after insert");
info("List context start after insert");
ctx_get_context_list(&host_uuid1, dict_ctx_get_context_list_cb, NULL);
info("List context end after insert");
// This will delete the entry
if (likely(!ctx_delete_context(&host_uuid, &context_data)))
info("Entry %s deleted", context_data.id);
else
info("Entry %s not deleted", context_data.id);
freez((void *)context_data.id);
freez((void *)context_data.title);
freez((void *)context_data.chart_type);
freez((void *)context_data.family);
// The list should be empty
info("List context start after delete");
ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
info("List context end after delete");
sql_close_context_database();
return 0;
}

68
database/sqlite/sqlite_context.h Normal file
View file

@ -0,0 +1,68 @@
// SPDX-License-Identifier: GPL-3.0-or-later
#ifndef NETDATA_SQLITE_CONTEXT_H
#define NETDATA_SQLITE_CONTEXT_H
#include "daemon/common.h"
#include "sqlite3.h"
typedef struct ctx_chart {
uuid_t chart_id;
const char *id;
const char *name;
const char *context;
const char *title;
const char *units;
const char *family;
int chart_type;
int priority;
int update_every;
} SQL_CHART_DATA;
typedef struct ctx_dimension {
uuid_t dim_id;
char *id;
char *name;
} SQL_DIMENSION_DATA;
typedef struct ctx_label {
char *label_key;
char *label_value;
int label_source;
} SQL_CLABEL_DATA;
// Structure to store or delete
typedef struct versioned_context_data {
uint64_t version; // the version of this context as EPOCH in seconds
const char *id; // the id of the context
const char *title; // the title of the context
const char *chart_type; // the chart_type of the context
const char *units; // the units of the context
const char *family; // the family of the context
uint64_t priority; // the chart priority of the context
uint64_t first_time_t; // the first entry in the database, in seconds
uint64_t last_time_t; // the last point in the database, in seconds
bool deleted; // true when this is deleted
} VERSIONED_CONTEXT_DATA;
extern void ctx_get_context_list(uuid_t *host_uuid, void (*dict_cb)(VERSIONED_CONTEXT_DATA *, void *), void *data);
extern void ctx_get_chart_list(uuid_t *host_uuid, void (*dict_cb)(SQL_CHART_DATA *, void *), void *data);
extern void ctx_get_label_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_CLABEL_DATA *, void *), void *data);
extern void ctx_get_dimension_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_DIMENSION_DATA *, void *), void *data);
extern int ctx_store_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data);
#define ctx_update_context(host_uuid, context_data) ctx_store_context(host_uuid, context_data)
extern int ctx_delete_context(uuid_t *host_id, VERSIONED_CONTEXT_DATA *context_data);
extern int sql_init_context_database(int memory);
extern void sql_close_context_database(void);
extern int ctx_unittest(void);
#endif //NETDATA_SQLITE_CONTEXT_H

119
database/sqlite/sqlite_db_migration.c
View file

@ -11,6 +11,25 @@ static int return_int_cb(void *data, int argc, char **argv, char **column)
return 0;
}
static int table_exists_in_database(const char *table)
{
char *err_msg = NULL;
char sql[128];
int exists = 0;
snprintf(sql, 127, "select 1 from sqlite_schema where type = 'table' and name = '%s';", table);
int rc = sqlite3_exec(db_meta, sql, return_int_cb, (void *) &exists, &err_msg);
if (rc != SQLITE_OK) {
info("Error checking table existence; %s", err_msg);
sqlite3_free(err_msg);
}
return exists;
}
static int column_exists_in_table(const char *table, const char *column)
{
char *err_msg = NULL;
@ -34,14 +53,34 @@ const char *database_migrate_v1_v2[] = {
NULL
};
const char *database_migrate_v2_v3[] = {
"ALTER TABLE host ADD memory_mode INT NOT NULL DEFAULT 0;",
"ALTER TABLE host ADD abbrev_timezone TEXT NOT NULL DEFAULT '';",
"ALTER TABLE host ADD utc_offset INT NOT NULL DEFAULT 0;",
"ALTER TABLE host ADD program_name TEXT NOT NULL DEFAULT 'unknown';",
"ALTER TABLE host ADD program_version TEXT NOT NULL DEFAULT 'unknown';",
"ALTER TABLE host ADD entries INT NOT NULL DEFAULT 0;",
"ALTER TABLE host ADD health_enabled INT NOT NULL DEFAULT 0;",
NULL
};
static int do_migration_v1_v2(sqlite3 *database, const char *name)
{
UNUSED(database);
UNUSED(name);
info("Running database migration %s", name);
info("Running \"%s\" database migration", name);
if (!column_exists_in_table("host", "hops"))
return init_database_batch(DB_CHECK_NONE, 0, &database_migrate_v1_v2[0]);
if (table_exists_in_database("host") && !column_exists_in_table("host", "hops"))
return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v1_v2[0]);
return 0;
}
static int do_migration_v2_v3(sqlite3 *database, const char *name)
{
UNUSED(name);
info("Running \"%s\" database migration", name);
if (table_exists_in_database("host") && !column_exists_in_table("host", "memory_mode"))
return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v2_v3[0]);
return 0;
}
@ -54,12 +93,51 @@ static int do_migration_noop(sqlite3 *database, const char *name)
return 0;
}
static struct database_func_migration_list {
typedef struct database_func_migration_list {
char *name;
int (*func)(sqlite3 *database, const char *name);
} migration_action[] = {
} DATABASE_FUNC_MIGRATION_LIST;
static int migrate_database(sqlite3 *database, int target_version, char *db_name, DATABASE_FUNC_MIGRATION_LIST *migration_list)
{
int user_version = 0;
char *err_msg = NULL;
int rc = sqlite3_exec(database, "PRAGMA user_version;", return_int_cb, (void *) &user_version, &err_msg);
if (rc != SQLITE_OK) {
info("Error checking the %s database version; %s", db_name, err_msg);
sqlite3_free(err_msg);
}
if (likely(user_version == target_version)) {
info("%s database version is %d (no migration needed)", db_name, target_version);
return target_version;
}
info("Database version is %d, current version is %d. Running migration for %s ...", user_version, target_version, db_name);
for (int i = user_version; i < target_version && migration_list[i].func; i++) {
rc = (migration_list[i].func)(database, migration_list[i].name);
if (unlikely(rc)) {
error_report("Database %s migration from version %d to version %d failed", db_name, i, i + 1);
return i;
}
}
return target_version;
}
DATABASE_FUNC_MIGRATION_LIST migration_action[] = {
{.name = "v0 to v1", .func = do_migration_noop},
{.name = "v1 to v2", .func = do_migration_v1_v2},
{.name = "v2 to v3", .func = do_migration_v2_v3},
// the terminator of this array
{.name = NULL, .func = NULL}
};
DATABASE_FUNC_MIGRATION_LIST context_migration_action[] = {
{.name = "v0 to v1", .func = do_migration_noop},
// the terminator of this array
{.name = NULL, .func = NULL}
};
@ -67,27 +145,10 @@ static struct database_func_migration_list {
int perform_database_migration(sqlite3 *database, int target_version)
{
int user_version = 0;
char *err_msg = NULL;
int rc = sqlite3_exec(database, "PRAGMA user_version;", return_int_cb, (void *) &user_version, &err_msg);
if (rc != SQLITE_OK) {
info("Error checking the database version; %s", err_msg);
sqlite3_free(err_msg);
}
if (likely(user_version == target_version)) {
info("Metadata database version is %d", target_version);
return target_version;
}
info("Database version is %d, current version is %d. Running migration ...", user_version, target_version);
for (int i = user_version; migration_action[i].func && i < target_version; i++) {
rc = (migration_action[i].func)(database, migration_action[i].name);
if (unlikely(rc)) {
error_report("Database migration from version %d to version %d failed", i, i + 1);
return i;
}
}
return target_version;
return migrate_database(database, target_version, "metadata", migration_action);
}
int perform_context_database_migration(sqlite3 *database, int target_version)
{
return migrate_database(database, target_version, "context", context_migration_action);
}

1
database/sqlite/sqlite_db_migration.h
View file

@ -7,5 +7,6 @@
int perform_database_migration(sqlite3 *database, int target_version);
int perform_context_database_migration(sqlite3 *database, int target_version);
#endif //NETDATA_SQLITE_DB_MIGRATION_H

159
database/sqlite/sqlite_functions.c
View file

@ -3,11 +3,18 @@
#include "sqlite_functions.h"
#include "sqlite_db_migration.h"
#define DB_METADATA_VERSION 2
#define DB_METADATA_VERSION 3
const char *database_config[] = {
"CREATE TABLE IF NOT EXISTS host(host_id blob PRIMARY KEY, hostname text, "
"registry_hostname text, update_every int, os text, timezone text, tags text, hops INT);",
"CREATE TABLE IF NOT EXISTS host(host_id BLOB PRIMARY KEY, hostname TEXT NOT NULL, "
"registry_hostname TEXT NOT NULL default 'unknown', update_every INT NOT NULL default 1, "
"os TEXT NOT NULL default 'unknown', timezone TEXT NOT NULL default 'unknown', tags TEXT NOT NULL default '',"
"hops INT NOT NULL DEFAULT 0,"
"memory_mode INT DEFAULT 0, abbrev_timezone TEXT DEFAULT '', utc_offset INT NOT NULL DEFAULT 0,"
"program_name TEXT NOT NULL DEFAULT 'unknown', program_version TEXT NOT NULL DEFAULT 'unknown', "
"entries INT NOT NULL DEFAULT 0,"
"health_enabled INT NOT NULL DEFAULT 0);",
"CREATE TABLE IF NOT EXISTS chart(chart_id blob PRIMARY KEY, host_id blob, type text, id text, name text, "
"family text, context text, title text, unit text, plugin text, module text, priority int, update_every int, "
"chart_type int, memory_mode int, history_entries);",
@ -349,13 +356,13 @@ static int attempt_database_fix()
return sql_init_database(DB_CHECK_FIX_DB | DB_CHECK_CONT, 0);
}
int init_database_batch(int rebuild, int init_type, const char *batch[])
int init_database_batch(sqlite3 *database, int rebuild, int init_type, const char *batch[])
{
int rc;
char *err_msg = NULL;
for (int i = 0; batch[i]; i++) {
debug(D_METADATALOG, "Executing %s", batch[i]);
rc = sqlite3_exec(db_meta, batch[i], 0, 0, &err_msg);
rc = sqlite3_exec(database, batch[i], 0, 0, &err_msg);
if (rc != SQLITE_OK) {
error_report("SQLite error during database %s, rc = %d (%s)", init_type ? "cleanup" : "setup", rc, err_msg);
error_report("SQLite failed statement %s", batch[i]);
@ -449,41 +456,41 @@ int sql_init_database(db_check_action_type_t rebuild, int memory)
// https://www.sqlite.org/pragma.html#pragma_auto_vacuum
// PRAGMA schema.auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL;
snprintfz(buf, 1024, "PRAGMA auto_vacuum=%s;", config_get(CONFIG_SECTION_SQLITE, "auto vacuum", "INCREMENTAL"));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_synchronous
// PRAGMA schema.synchronous = 0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA;
snprintfz(buf, 1024, "PRAGMA synchronous=%s;", config_get(CONFIG_SECTION_SQLITE, "synchronous", "NORMAL"));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_journal_mode
// PRAGMA schema.journal_mode = DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
snprintfz(buf, 1024, "PRAGMA journal_mode=%s;", config_get(CONFIG_SECTION_SQLITE, "journal mode", "WAL"));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_temp_store
// PRAGMA temp_store = 0 | DEFAULT | 1 | FILE | 2 | MEMORY;
snprintfz(buf, 1024, "PRAGMA temp_store=%s;", config_get(CONFIG_SECTION_SQLITE, "temp store", "MEMORY"));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_journal_size_limit
// PRAGMA schema.journal_size_limit = N ;
snprintfz(buf, 1024, "PRAGMA journal_size_limit=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "journal size limit", 16777216));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
// https://www.sqlite.org/pragma.html#pragma_cache_size
// PRAGMA schema.cache_size = pages;
// PRAGMA schema.cache_size = -kibibytes;
snprintfz(buf, 1024, "PRAGMA cache_size=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "cache size", -2000));
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
snprintfz(buf, 1024, "PRAGMA user_version=%d;", target_version);
if(init_database_batch(rebuild, 0, list)) return 1;
if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
if (init_database_batch(rebuild, 0, &database_config[0]))
if (init_database_batch(db_meta, rebuild, 0, &database_config[0]))
return 1;
if (init_database_batch(rebuild, 0, &database_cleanup[0]))
if (init_database_batch(db_meta, rebuild, 0, &database_cleanup[0]))
return 1;
fatal_assert(0 == uv_mutex_init(&sqlite_transaction_lock));
@ -853,6 +860,113 @@ bind_fail:
return 1;
}
//
// Store host and host system info information in the database
#define SQL_STORE_HOST_INFO "INSERT OR REPLACE INTO host " \
"(host_id, hostname, registry_hostname, update_every, os, timezone," \
"tags, hops, memory_mode, abbrev_timezone, utc_offset, program_name, program_version," \
"entries, health_enabled) " \
"values (@host_id, @hostname, @registry_hostname, @update_every, @os, @timezone, @tags, @hops, @memory_mode, " \
"@abbrev_timezone, @utc_offset, @program_name, @program_version, " \
"@entries, @health_enabled);"
int sql_store_host_info(RRDHOST *host)
{
static __thread sqlite3_stmt *res = NULL;
int rc;
if (unlikely(!db_meta)) {
if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
return 0;
error_report("Database has not been initialized");
return 1;
}
if (unlikely((!res))) {
rc = prepare_statement(db_meta, SQL_STORE_HOST_INFO, &res);
if (unlikely(rc != SQLITE_OK)) {
error_report("Failed to prepare statement to store host, rc = %d", rc);
return 1;
}
}
rc = sqlite3_bind_blob(res, 1, &host->host_uuid, sizeof(host->host_uuid), SQLITE_STATIC);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 2, host->hostname, 0);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 3, host->registry_hostname, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int(res, 4, host->rrd_update_every);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 5, host->os, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 6, host->timezone, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 7, host->tags, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int(res, 8, host->system_info ? host->system_info->hops : 0);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int(res, 9, host->rrd_memory_mode);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 10, host->abbrev_timezone, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int(res, 11, host->utc_offset);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 12, host->program_name, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = bind_text_null(res, 13, host->program_version, 1);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int64(res, 14, host->rrd_history_entries);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
rc = sqlite3_bind_int(res, 15, host->health_enabled);
if (unlikely(rc != SQLITE_OK))
goto bind_fail;
int store_rc = sqlite3_step(res);
if (unlikely(store_rc != SQLITE_DONE))
error_report("Failed to store host %s, rc = %d", host->hostname, rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement to store host %s, rc = %d", host->hostname, rc);
return !(store_rc == SQLITE_DONE);
bind_fail:
error_report("Failed to bind parameter to store host %s, rc = %d", host->hostname, rc);
rc = sqlite3_reset(res);
if (unlikely(rc != SQLITE_OK))
error_report("Failed to reset statement to store host %s, rc = %d", host->hostname, rc);
return 1;
}
/*
* Store a chart in the database
*/
@ -1884,6 +1998,11 @@ void compute_chart_hash(RRDSET *st)
unsigned int hash_len;
char priority_str[32];
if (rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
internal_error(true, "Skipping compute_chart_hash for host %s because context streaming is enabled", st->rrdhost->hostname);
return;
}
sprintf(priority_str, "%ld", st->priority);
evpctx = EVP_MD_CTX_create();
@ -2318,6 +2437,7 @@ failed:
return;
};
#define SELECT_HOST_INFO "SELECT system_key, system_value FROM host_info WHERE host_id = @host_id;"
void sql_build_host_system_info(uuid_t *host_id, struct rrdhost_system_info *system_info)
@ -2481,3 +2601,12 @@ void sql_store_host_system_info(uuid_t *host_id, const struct rrdhost_system_inf
return;
}
// Utils
int bind_text_null(sqlite3_stmt *res, int position, const char *text, bool can_be_null)
{
if (likely(text))
return sqlite3_bind_text(res, position, text, -1, SQLITE_STATIC);
if (!can_be_null)
return 1;
return sqlite3_bind_null(res, position);
}

8
database/sqlite/sqlite_functions.h
View file

@ -60,9 +60,12 @@ typedef enum db_check_action_type {
extern int sql_init_database(db_check_action_type_t rebuild, int memory);
extern void sql_close_database(void);
extern int bind_text_null(sqlite3_stmt *res, int position, const char *text, bool can_be_null);
extern int sql_store_host(uuid_t *guid, const char *hostname, const char *registry_hostname, int update_every, const char *os,
const char *timezone, const char *tags, int hops);
extern int sql_store_host_info(RRDHOST *host);
extern int sql_store_chart(
uuid_t *chart_uuid, uuid_t *host_uuid, const char *type, const char *id, const char *name, const char *family,
const char *context, const char *title, const char *units, const char *plugin, const char *module, long priority,
@ -104,7 +107,8 @@ extern void compute_chart_hash(RRDSET *st);
extern int sql_set_dimension_option(uuid_t *dim_uuid, char *option);
char *get_hostname_by_node_id(char *node_id);
void free_temporary_host(RRDHOST *host);
int init_database_batch(int rebuild, int init_type, const char *batch[]);
int init_database_batch(sqlite3 *database, int rebuild, int init_type, const char *batch[]);
void migrate_localhost(uuid_t *host_uuid);
extern void sql_store_host_system_info(uuid_t *host_id, const struct rrdhost_system_info *system_info);
extern void sql_build_host_system_info(uuid_t *host_id, struct rrdhost_system_info *system_info);
#endif //NETDATA_SQLITE_FUNCTIONS_H

70
health/health.d/cgroups.conf
View file

@ -69,3 +69,73 @@ component: Network
info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \
compared to the rate over the last minute
to: sysadmin
# ---------------------------------K8s containers--------------------------------------------
template: k8s_cgroup_10min_cpu_usage
on: k8s.cgroup.cpu_limit
class: Utilization
type: Cgroups
component: CPU
os: linux
hosts: *
lookup: average -10m unaligned
units: %
every: 1m
warn: $this > (($status >= $WARNING) ? (75) : (85))
crit: $this > (($status == $CRITICAL) ? (85) : (95))
delay: down 15m multiplier 1.5 max 1h
info: average cgroup CPU utilization over the last 10 minutes
to: sysadmin
template: k8s_cgroup_ram_in_use
on: k8s.cgroup.mem_usage
class: Utilization
type: Cgroups
component: Memory
os: linux
hosts: *
calc: ($ram) * 100 / $memory_limit
units: %
every: 10s
warn: $this > (($status >= $WARNING) ? (80) : (90))
crit: $this > (($status == $CRITICAL) ? (90) : (98))
delay: down 15m multiplier 1.5 max 1h
info: cgroup memory utilization
to: sysadmin
# check for packet storms
# 1. calculate the rate packets are received in 1m: 1m_received_packets_rate
# 2. do the same for the last 10s
# 3. raise an alarm if the later is 10x or 20x the first
# we assume the minimum packet storm should at least have
# 10000 packets/s, average of the last 10 seconds
template: k8s_cgroup_1m_received_packets_rate
on: k8s.cgroup.net_packets
class: Workload
type: Cgroups
component: Network
hosts: *
lookup: average -1m unaligned of received
units: packets
every: 10s
info: average number of packets received by the network interface $family over the last minute
template: k8s_cgroup_10s_received_packets_storm
on: k8s.cgroup.net_packets
class: Workload
type: Cgroups
component: Network
hosts: *
lookup: average -10s unaligned of received
calc: $this * 100 / (($k8s_cgroup_10s_received_packets_storm < 1000)?(1000):($k8s_cgroup_10s_received_packets_storm))
every: 10s
units: %
warn: $this > (($status >= $WARNING)?(200):(5000))
crit: $this > (($status == $CRITICAL)?(5000):(6000))
options: no-clear-notification
info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \
compared to the rate over the last minute
to: sysadmin

6
libnetdata/dictionary/README.md
View file

@ -97,7 +97,7 @@ This call is used to get the value of an item, given its name. It utilizes the `
For **multi-threaded** operation, the `dictionary_get()` call gets a shared read lock on the dictionary.
In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_get_and_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
The format is:
@ -133,7 +133,7 @@ Where:
> **IMPORTANT**<br/>There is also an **unsafe** version (without locks) of this call. This is to be used when traversing the dictionary, to delete the current item. It should never be called without an active lock on the dictionary, which can only be acquired while traversing.
### dictionary_acquire_item()
### dictionary_get_and_acquire_item()
This call can be used the search and get a dictionary item, while ensuring that it will be available for use, until `dictionary_acquired_item_release()` is called.
@ -149,7 +149,7 @@ DICTIONARY *dict = dictionary_create(DICTIONARY_FLAGS_NONE);
dictionary_set(dict, "name", "value", 6);
// find the item we added and acquire it
void *item = dictionary_acquire_item(dict, "name");
void *item = dictionary_get_and_acquire_item(dict, "name");
// extract its value
char *value = (char *)dictionary_acquired_item_value(dict, item);

804
libnetdata/dictionary/dictionary.c
View file

File diff suppressed because it is too large Load diff

87
libnetdata/dictionary/dictionary.h
View file

@ -35,26 +35,34 @@
*
*/
#ifndef DICTIONARY_INTERNALS
typedef void DICTIONARY;
#endif
typedef struct dictionary DICTIONARY;
typedef struct dictionary_item DICTIONARY_ITEM;
typedef enum dictionary_flags {
DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
// to change the value of the following, you also need to change the corresponding #defines in dictionary.c
DICTIONARY_FLAG_RESERVED1 = (1 << 29), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
DICTIONARY_FLAG_RESERVED2 = (1 << 30), // reserved for DICTIONARY_FLAG_DESTROYED
DICTIONARY_FLAG_RESERVED3 = (1 << 31), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
DICTIONARY_FLAG_RESERVED1 = (1 << 29), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
DICTIONARY_FLAG_RESERVED2 = (1 << 30), // reserved for DICTIONARY_FLAG_DESTROYED
DICTIONARY_FLAG_RESERVED3 = (1 << 31), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
} DICTIONARY_FLAGS;
// Create a dictionary
extern DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags);
#ifdef NETDATA_INTERNAL_CHECKS
#define dictionary_create(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
#define dictionary_create_advanced(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
extern DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file);
#else
#define dictionary_create(flags) dictionary_create_advanced(flags, 0);
extern DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size);
#endif
extern void *dictionary_scratchpad(DICTIONARY *dict);
// an insert callback to be called just after an item is added to the dictionary
// this callback is called while the dictionary is write locked!
@ -69,6 +77,11 @@ extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_cal
// the old_value will remain in the dictionary - the new_value is ignored
extern void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data), void *data);
// a reaction callback to be called after every item insertion or conflict
// after the constructors have finished and the items are fully available for use
// and the dictionary is not write locked anymore
extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data);
// Destroy a dictionary
// returns the number of bytes freed
// the returned value will not include name and value sizes if DICTIONARY_FLAG_WITH_STATISTICS is not set
@ -99,11 +112,18 @@ extern void *dictionary_get(DICTIONARY *dict, const char *name);
// returns -1 if the item was not found in the index
extern int dictionary_del(DICTIONARY *dict, const char *name);
extern void *dictionary_acquire_item_unsafe(DICTIONARY *dict, const char *name);
extern void *dictionary_acquire_item(DICTIONARY *dict, const char *name);
extern void *dictionary_acquired_item_value(DICTIONARY *dict, void *item);
extern void dictionary_acquired_item_release(DICTIONARY *dict, void *item);
extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, void *item);
extern DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name);
extern DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name);
extern DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
extern DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len);
extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item);
extern void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item);
extern DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item);
extern const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item);
extern void *dictionary_acquired_item_value(DICTIONARY_ITEM *item);
// UNSAFE functions, without locks
// to be used when the user is traversing with the right lock type
@ -159,6 +179,10 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
#define DICTFE_CONST const
#endif
#define DICTIONARY_LOCK_READ 'r'
#define DICTIONARY_LOCK_WRITE 'w'
#define DICTIONARY_LOCK_NONE 'u'
typedef DICTFE_CONST struct dictionary_foreach {
DICTFE_CONST char *name; // the dictionary name of the last item used
void *value; // the dictionary value of the last item used
@ -171,8 +195,8 @@ typedef DICTFE_CONST struct dictionary_foreach {
void *last_item; // the item we work on, to remember the position we are at
} DICTFE;
#define dfe_start_read(dict, value) dfe_start_rw(dict, value, 'r')
#define dfe_start_write(dict, value) dfe_start_rw(dict, value, 'w')
#define dfe_start_read(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_READ)
#define dfe_start_write(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_WRITE)
#define dfe_start_rw(dict, value, mode) \
do { \
DICTFE value ## _dfe = {}; \
@ -194,12 +218,35 @@ extern usec_t dictionary_foreach_done(DICTFE *dfe);
// Get statistics about the dictionary
extern long int dictionary_stats_allocated_memory(DICTIONARY *dict);
extern long int dictionary_stats_entries(DICTIONARY *dict);
extern size_t dictionary_stats_version(DICTIONARY *dict);
extern size_t dictionary_stats_inserts(DICTIONARY *dict);
extern size_t dictionary_stats_searches(DICTIONARY *dict);
extern size_t dictionary_stats_deletes(DICTIONARY *dict);
extern size_t dictionary_stats_resets(DICTIONARY *dict);
extern size_t dictionary_stats_walkthroughs(DICTIONARY *dict);
extern size_t dictionary_stats_referenced_items(DICTIONARY *dict);
extern int dictionary_unittest(size_t entries);
// ----------------------------------------------------------------------------
// STRING implementation
typedef struct netdata_string STRING;
extern STRING *string_strdupz(const char *str);
extern STRING *string_dup(STRING *string);
extern void string_freez(STRING *string);
extern size_t string_length(STRING *string);
extern const char *string2str(STRING *string) NEVERNULL;
// keep common prefix/suffix and replace everything else with [x]
extern STRING *string_2way_merge(STRING *a, STRING *b);
static inline int string_cmp(STRING *s1, STRING *s2) {
// STRINGs are deduplicated, so the same strings have the same pointer
if(unlikely(s1 == s2)) return 0;
// they differ, do the typical comparison
return strcmp(string2str(s1), string2str(s2));
}
#endif /* NETDATA_DICTIONARY_H */

5
streaming/receiver.c
View file

@ -517,6 +517,7 @@ static int rrdpush_receive(struct receiver_state *rpt)
, rrdpush_api_key
, rrdpush_send_charts_matching
, rpt->system_info
, 0
);
if(!rpt->host) {
@ -706,6 +707,8 @@ static int rrdpush_receive(struct receiver_state *rpt)
aclk_host_state_update(rpt->host, 1);
#endif
rrdcontext_host_child_connected(rpt->host);
size_t count = streaming_parser(rpt, &cd, fp);
log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rpt->hostname,
@ -713,6 +716,8 @@ static int rrdpush_receive(struct receiver_state *rpt)
error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).", rpt->hostname, rpt->client_ip,
rpt->client_port, count);
rrdcontext_host_child_disconnected(rpt->host);
#ifdef ENABLE_ACLK
// in case we have cloud connection we inform cloud
// new child connected

3
streaming/rrdpush.c
View file

@ -489,6 +489,9 @@ struct rrdpush_destinations *destinations_init(const char *dests) {
// The sender mutex guards thread creation, any spurious data is wiped on reconnection.
void rrdpush_sender_thread_stop(RRDHOST *host) {
if (!host->sender)
return;
netdata_mutex_lock(&host->sender->mutex);
netdata_thread_t thr = 0;

2
streaming/rrdpush.h
View file

@ -160,7 +160,7 @@ extern char *default_rrdpush_api_key;
extern char *default_rrdpush_send_charts_matching;
extern unsigned int remote_clock_resync_iterations;
extern void sender_init(struct sender_state *s, RRDHOST *parent);
extern void sender_init(RRDHOST *parent);
extern struct rrdpush_destinations *destinations_init(const char *destinations);
void sender_start(struct sender_state *s);
void sender_commit(struct sender_state *s);

20
streaming/sender.c
View file

@ -724,17 +724,21 @@ static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
netdata_mutex_unlock(&host->sender->mutex);
}
void sender_init(struct sender_state *s, RRDHOST *parent) {
memset(s, 0, sizeof(*s));
s->host = parent;
s->buffer = cbuffer_new(1024, 1024*1024);
s->build = buffer_create(1);
void sender_init(RRDHOST *parent)
{
if (parent->sender)
return;
parent->sender = callocz(1, sizeof(*parent->sender));
parent->sender->host = parent;
parent->sender->buffer = cbuffer_new(1024, 1024*1024);
parent->sender->build = buffer_create(1);
#ifdef ENABLE_COMPRESSION
s->rrdpush_compression = default_compression_enabled;
parent->sender->rrdpush_compression = default_compression_enabled;
if (default_compression_enabled)
s->compressor = create_compressor();
parent->sender->compressor = create_compressor();
#endif
netdata_mutex_init(&s->mutex);
netdata_mutex_init(&parent->sender->mutex);
}
void *rrdpush_sender_thread(void *ptr) {

302
web/api/netdata-swagger.json
View file

@ -48,7 +48,7 @@
"/chart": {
"get": {
"summary": "Get info about a specific chart",
"description": "The Chart endpoint returns detailed information about a chart.",
"description": "The chart endpoint returns detailed information about a chart.",
"parameters": [
{
"name": "chart",
@ -82,6 +82,219 @@
}
}
},
"/contexts": {
"get": {
"summary": "Get a list of all contexts available at the server",
"description": "The contexts endpoint returns a summary about all contexts stored in the netdata server.",
"parameters": [
{
"name": "options",
"in": "query",
"description": "Options that affect data generation.",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "array",
"items": {
"type": "string",
"enum": [
"full",
"all",
"charts",
"dimensions",
"labels",
"uuids",
"queue",
"flags",
"deleted",
"deepscan"
]
},
"default": [
"full"
]
}
},
{
"name": "after",
"in": "query",
"description": "limit the results on context having data after this timestamp.",
"required": false,
"schema": {
"type": "number",
"format": "integer"
}
},
{
"name": "before",
"in": "query",
"description": "limit the results on context having data before this timestamp.",
"required": false,
"schema": {
"type": "number",
"format": "integer"
}
},
{
"name": "chart_label_key",
"in": "query",
"description": "a simple pattern matching charts label keys (use comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
},
{
"name": "chart_labels_filter",
"in": "query",
"description": "a simple pattern matching charts label key and values (use colon for equality, comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
},
{
"name": "dimensions",
"in": "query",
"description": "a simple pattern matching dimensions (use comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "An array of contexts.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/context_summary"
}
}
}
}
}
}
},
"/context": {
"get": {
"summary": "Get info about a specific context",
"description": "The context endpoint returns detailed information about a given context.",
"parameters": [
{
"name": "context",
"in": "query",
"description": "The id of the context as returned by the /contexts call.",
"required": true,
"schema": {
"type": "string",
"format": "as returned by /contexts",
"default": "system.cpu"
}
},
{
"name": "options",
"in": "query",
"description": "Options that affect data generation.",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "array",
"items": {
"type": "string",
"enum": [
"full",
"all",
"charts",
"dimensions",
"labels",
"uuids",
"queue",
"flags",
"deleted",
"deepscan"
]
},
"default": [
"full"
]
}
},
{
"name": "after",
"in": "query",
"description": "limit the results on context having data after this timestamp.",
"required": false,
"schema": {
"type": "number",
"format": "integer"
}
},
{
"name": "before",
"in": "query",
"description": "limit the results on context having data before this timestamp.",
"required": false,
"schema": {
"type": "number",
"format": "integer"
}
},
{
"name": "chart_label_key",
"in": "query",
"description": "a simple pattern matching charts label keys (use comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
},
{
"name": "chart_labels_filter",
"in": "query",
"description": "a simple pattern matching charts label key and values (use colon for equality, comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
},
{
"name": "dimensions",
"in": "query",
"description": "a simple pattern matching dimensions (use comma or pipe as separator)",
"required": false,
"allowEmptyValue": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "A javascript object with detailed information about the context.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/context"
}
}
}
},
"400": {
"description": "No context id was supplied in the request."
},
"404": {
"description": "No context with the given id is found."
}
}
}
},
"/alarm_variables": {
"get": {
"summary": "List variables available to configure alarms for a chart",
@ -924,7 +1137,7 @@
"/alarms_values": {
"get": {
"summary": "Get a list of active or raised alarms on the server",
"description": "The alarms_values endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing \"?all\", all the enabled alarms are returned. This option output differs from `/alarms` in the number of variables delivered. This endpoint gives to user `id`, `value`, `last_updated` time, and alarm `status`.",
"description": "The alarms_values endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing '?all', all the enabled alarms are returned. This option output differs from `/alarms` in the number of variables delivered. This endpoint gives to user `id`, `value`, `last_updated` time, and alarm `status`.",
"parameters": [
{
"name": "all",
@ -1131,7 +1344,7 @@
"/aclk": {
"get": {
"summary": "Get information about current ACLK state",
"description": "aclk endpoint returns detailed information about current state of ACLK (Agent to Cloud communication).",
"description": "ACLK endpoint returns detailed information about current state of ACLK (Agent to Cloud communication).",
"responses": {
"200": {
"description": "JSON object with ACLK information.",
@ -1678,6 +1891,89 @@
}
}
},
"context_summary": {
"type": "object",
"properties": {
"hostname": {
"type": "string",
"description": "The hostname of the netdata server."
},
"machine_guid": {
"type": "string",
"description": "The unique installation id of this netdata server."
},
"node_id": {
"type": "string",
"description": "The unique node id of this netdata server at the hub.",
"example": "nightly"
},
"claim_id": {
"type": "string",
"description": "The unique handshake id of this netdata server and the hub."
},
"host_labels": {
"type": "object",
"description": "The host labels associated with this netdata server."
},
"context": {
"type": "object",
"description": "An object containing all the context objects available at the netdata server. This is used as an indexed array. The key of each context object is the id of the context.",
"additionalProperties": {
"$ref": "#/components/schemas/context"
}
}
}
},
"context": {
"type": "object",
"properties": {
"version": {
"type": "string",
"description": "The version of this context. The number are not sequential, but bigger numbers depict a newer object."
},
"hub_version": {
"type": "string",
"description": "The version of this context, as known by hub."
},
"family": {
"type": "string",
"description": "The family of the context. When multiple charts of a context have different families, the netdata server replaces the different parts with [x], so that the context can have only one family."
},
"title": {
"type": "string",
"description": "The title of the context. When multiple charts of a context have different titles, the netdata server replaces the different parts with [x], so that the context can have only one title."
},
"priority": {
"type": "number",
"description": "The relative priority of the context. When multiple contexts have different priorities, the minimum among them is selected as the priority of the context."
},
"units": {
"type": "string",
"description": "The unit of measurement for the values of all dimensions of the context. If multiple charts of context have different units, the latest collected is selected."
},
"chart_type": {
"type": "string",
"description": "The chart type.",
"enum": [
"line",
"area",
"stacked"
]
},
"first_time_t": {
"type": "number",
"description": "The UNIX timestamp of the first entry (the oldest) in the database."
},
"last_time_t": {
"type": "number",
"description": "The UNIX timestamp of the latest entry in the database."
},
"charts": {
"type": "object",
"description": "An object containing all the charts available for the chart. This is used as an indexed array. For each pair in the dictionary, the key is the id of the chart and the value provides all details about the chart."
}
}
},
"alarm_variables": {
"type": "object",
"properties": {

282
web/api/netdata-swagger.yaml
View file

@ -43,7 +43,7 @@ paths:
/chart:
get:
summary: Get info about a specific chart
description: The Chart endpoint returns detailed information about a chart.
description: The chart endpoint returns detailed information about a chart.
parameters:
- name: chart
in: query
@ -64,6 +64,159 @@ paths:
description: No chart id was supplied in the request.
"404":
description: No chart with the given id is found.
/contexts:
get:
summary: Get a list of all contexts available at the server
description: The contexts endpoint returns a summary about all contexts stored in the
netdata server.
parameters:
- name: options
in: query
description: Options that affect data generation.
required: false
allowEmptyValue: true
schema:
type: array
items:
type: string
enum:
- full
- all
- charts
- dimensions
- labels
- uuids
- queue
- flags
- deleted
- deepscan
default:
- full
- name: after
in: query
description: limit the results on context having data after this timestamp.
required: false
schema:
type: number
format: integer
- name: before
in: query
description: limit the results on context having data before this timestamp.
required: false
schema:
type: number
format: integer
- name: chart_label_key
in: query
description: a simple pattern matching charts label keys (use comma or pipe as separator)
required: false
allowEmptyValue: true
schema:
type: string
- name: chart_labels_filter
in: query
description: "a simple pattern matching charts label key and values (use colon for equality, comma or pipe
as separator)"
required: false
allowEmptyValue: true
schema:
type: string
- name: dimensions
in: query
description: a simple pattern matching dimensions (use comma or pipe as separator)
required: false
allowEmptyValue: true
schema:
type: string
responses:
"200":
description: An array of contexts.
content:
application/json:
schema:
$ref: "#/components/schemas/context_summary"
/context:
get:
summary: Get info about a specific context
description: The context endpoint returns detailed information about a given context.
parameters:
- name: context
in: query
description: The id of the context as returned by the /contexts call.
required: true
schema:
type: string
format: as returned by /contexts
default: system.cpu
- name: options
in: query
description: Options that affect data generation.
required: false
allowEmptyValue: true
schema:
type: array
items:
type: string
enum:
- full
- all
- charts
- dimensions
- labels
- uuids
- queue
- flags
- deleted
- deepscan
default:
- full
- name: after
in: query
description: limit the results on context having data after this timestamp.
required: false
schema:
type: number
format: integer
- name: before
in: query
description: limit the results on context having data before this timestamp.
required: false
schema:
type: number
format: integer
- name: chart_label_key
in: query
description: a simple pattern matching charts label keys (use comma or pipe as separator)
required: false
allowEmptyValue: true
schema:
type: string
- name: chart_labels_filter
in: query
description: "a simple pattern matching charts label key and values (use colon for equality, comma or pipe
as separator)"
required: false
allowEmptyValue: true
schema:
type: string
- name: dimensions
in: query
description: a simple pattern matching dimensions (use comma or pipe as separator)
required: false
allowEmptyValue: true
schema:
type: string
responses:
"200":
description: A javascript object with detailed information about the context.
content:
application/json:
schema:
$ref: "#/components/schemas/context"
"400":
description: No context id was supplied in the request.
"404":
description: No context with the given id is found.
/alarm_variables:
get:
summary: List variables available to configure alarms for a chart
@ -461,7 +614,9 @@ paths:
format: any text
- name: label_color
in: query
description: A color to be used for the background of the label side(left side) of the badge. One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character). If value wrong or not given default color will be used.
description: "A color to be used for the background of the label side(left side) of the badge.
One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character).
If value wrong or not given default color will be used."
required: false
allowEmptyValue: true
schema:
@ -486,7 +641,8 @@ paths:
description: "A color to be used for the background of the value *(right)* part of badge. You can set
multiple using a pipe with a condition each, like this:
`color<value|color:null` The following operators are
supported: >, <, >=, <=, =, :null (to check if no value exists). Each color can be specified in same manner as for `label_color` parameter.
supported: >, <, >=, <=, =, :null (to check if no value exists).
Each color can be specified in same manner as for `label_color` parameter.
Currently only integers are supported as values."
required: false
allowEmptyValue: true
@ -495,7 +651,9 @@ paths:
format: any text
- name: text_color_lbl
in: query
description: Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.
description: "Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal
color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default
color will be used."
required: false
allowEmptyValue: true
schema:
@ -517,7 +675,9 @@ paths:
format: ^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$
- name: text_color_val
in: query
description: Font color for value *(right)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.
description: "Font color for value *(right)* part of the badge. One of predefined colors or as HTML
hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value
given default color will be used."
required: false
allowEmptyValue: true
schema:
@ -565,7 +725,12 @@ paths:
format: integer
- name: fixed_width_lbl
in: query
description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's left side *(label/name)*. You must set this parameter together with `fixed_width_val` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.
description: "This parameter overrides auto-sizing of badge and creates it with fixed width.
This parameter determines the size of the label's left side *(label/name)*.
You must set this parameter together with `fixed_width_val` otherwise it will be ignored.
You should set the label/value widths wide enough to provide space for all the possible values/contents of
the badge you're requesting. In case the text cannot fit the space given it will be clipped.
The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`."
required: false
allowEmptyValue: false
schema:
@ -573,7 +738,12 @@ paths:
format: integer
- name: fixed_width_val
in: query
description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's right side *(value)*. You must set this parameter together with `fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.
description: "This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter
determines the size of the label's right side *(value)*. You must set this parameter together with
`fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to
provide space for all the possible values/contents of the badge you're requesting. In case the text cannot
fit the space given it will be clipped. The `scale` parameter still applies on the values you give to
`fixed_width_lbl` and `fixed_width_val`."
required: false
allowEmptyValue: false
schema:
@ -754,12 +924,12 @@ paths:
/alarms_values:
get:
summary: Get a list of active or raised alarms on the server
description: The alarms_values endpoint returns the list of all raised or enabled alarms on
description: "The alarms_values endpoint returns the list of all raised or enabled alarms on
the netdata server. Called without any parameters, the raised alarms in
state WARNING or CRITICAL are returned. By passing "?all", all the
state WARNING or CRITICAL are returned. By passing '?all', all the
enabled alarms are returned.
This option output differs from `/alarms` in the number of variables delivered. This endpoint gives
to user `id`, `value`, `last_updated` time, and alarm `status`.
to user `id`, `value`, `last_updated` time, and alarm `status`."
parameters:
- name: all
in: query
@ -855,16 +1025,16 @@ paths:
memory.
/manage/health:
get:
summary: Accesses the health management API to control health checks and
notifications at runtime.
description: Available from Netdata v1.12 and above, protected via bearer
summary: "Accesses the health management API to control health checks and
notifications at runtime."
description: "Available from Netdata v1.12 and above, protected via bearer
authorization. Especially useful for maintenance periods, the API allows
you to disable health checks completely, silence alarm notifications, or
Disable/Silence specific alarms that match selectors on alarm/template
name, chart, context, host and family. For the simple disable/silence
all scenarios, only the cmd parameter is required. The other parameters
are used to define alarm selectors. For more information and examples,
refer to the netdata documentation.
refer to the netdata documentation."
parameters:
- name: cmd
in: query
@ -918,8 +1088,8 @@ paths:
/aclk:
get:
summary: Get information about current ACLK state
description: aclk endpoint returns detailed information
about current state of ACLK (Agent to Cloud communication).
description: "ACLK endpoint returns detailed information
about current state of ACLK (Agent to Cloud communication)."
responses:
"200":
description: JSON object with ACLK information.
@ -929,15 +1099,15 @@ paths:
$ref: "#/components/schemas/aclk_state"
/metric_correlations:
get:
summary: Analyze all the metrics to find their correlations
description: Given two time-windows (baseline, highlight), it goes
summary: "Analyze all the metrics to find their correlations"
description: "Given two time-windows (baseline, highlight), it goes
through all the available metrics, querying both windows and tries to find
how these two windows relate to each other. It supports
multiple algorithms to do so. The result is a list of all
metrics evaluated, weighted for 0.0 (the two windows are
more different) to 1.0 (the two windows are similar).
The algorithm adjusts automatically the baseline window to be
a power of two multiple of the highlighted (1, 2, 4, 8, etc).
a power of two multiple of the highlighted (1, 2, 4, 8, etc)."
parameters:
- name: baseline_after
in: query
@ -1377,6 +1547,75 @@ components:
type: number
nullable: true
description: Chart health red threshold.
context_summary:
type: object
properties:
hostname:
type: string
description: The hostname of the netdata server.
machine_guid:
type: string
description: The unique installation id of this netdata server.
node_id:
type: string
description: The unique node id of this netdata server at the hub.
example: nightly
claim_id:
type: string
description: The unique handshake id of this netdata server and the hub.
host_labels:
type: object
description: The host labels associated with this netdata server.
context:
type: object
description: "An object containing all the context objects available at the netdata server.
This is used as an indexed array. The key of each context object is the id of the context."
additionalProperties:
$ref: "#/components/schemas/context"
context:
type: object
properties:
version:
type: string
description: "The version of this context.
The number are not sequential, but bigger numbers depict a newer object."
hub_version:
type: string
description: The version of this context, as known by hub.
family:
type: string
description: "The family of the context. When multiple charts of a context have different families,
the netdata server replaces the different parts with [x], so that the context can have only one family."
title:
type: string
description: "The title of the context. When multiple charts of a context have different titles,
the netdata server replaces the different parts with [x], so that the context can have only one title."
priority:
type: number
description: "The relative priority of the context. When multiple contexts have different priorities,
the minimum among them is selected as the priority of the context."
units:
type: string
description: "The unit of measurement for the values of all dimensions of the context. If multiple charts
of context have different units, the latest collected is selected."
chart_type:
type: string
description: The chart type.
enum:
- line
- area
- stacked
first_time_t:
type: number
description: The UNIX timestamp of the first entry (the oldest) in the database.
last_time_t:
type: number
description: The UNIX timestamp of the latest entry in the database.
charts:
type: object
description: "An object containing all the charts available for the chart. This is used as an indexed array.
For each pair in the dictionary, the key is the id of the chart and the value provides all details about
the chart."
alarm_variables:
type: object
properties:
@ -1848,8 +2087,9 @@ components:
properties:
aclk-available:
type: string
description: Describes whether this agent is capable of connection to the Cloud.
False means agent has been built without ACLK component either on purpose (user choice) or due to missing dependency.
description: "Describes whether this agent is capable of connection to the Cloud.
False means agent has been built without ACLK component either on purpose (user choice)
or due to missing dependency."
aclk-version:
type: integer
description: Describes which ACLK version is currently used.

4
web/api/queries/query.c
View file

@ -1173,8 +1173,8 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, int tier, time_t now) {
all_points_read += points;
tmp->query_ops.finalize(&handle);
internal_error(true, "DBENGINE: backfilled chart '%s', dimension '%s', tier %d, from %ld to %ld, with %zu points from tier %d",
rd->rrdset->name, rd->name, tier, after_wanted, before_wanted, points, tr);
//internal_error(true, "DBENGINE: backfilled chart '%s', dimension '%s', tier %d, from %ld to %ld, with %zu points from tier %d",
// rd->rrdset->name, rd->name, tier, after_wanted, before_wanted, points, tr);
}
rrdr_query_completed(all_points_read, all_points_read);

157
web/api/web_api_v1.c
View file

@ -374,6 +374,155 @@ inline int web_client_api_request_v1_alarm_variables(RRDHOST *host, struct web_c
return web_client_api_request_single_chart(host, w, url, health_api_v1_chart_variables2json);
}
static RRDCONTEXT_TO_JSON_OPTIONS rrdcontext_to_json_parse_options(char *o) {
RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
char *tok;
while(o && *o && (tok = mystrsep(&o, ", |"))) {
if(!*tok) continue;
if(!strcmp(tok, "full") || !strcmp(tok, "all"))
options |= RRDCONTEXT_OPTIONS_ALL;
else if(!strcmp(tok, "charts") || !strcmp(tok, "instances"))
options |= RRDCONTEXT_OPTION_SHOW_INSTANCES;
else if(!strcmp(tok, "dimensions") || !strcmp(tok, "metrics"))
options |= RRDCONTEXT_OPTION_SHOW_METRICS;
else if(!strcmp(tok, "queue"))
options |= RRDCONTEXT_OPTION_SHOW_QUEUED;
else if(!strcmp(tok, "flags"))
options |= RRDCONTEXT_OPTION_SHOW_FLAGS;
else if(!strcmp(tok, "uuids"))
options |= RRDCONTEXT_OPTION_SHOW_UUIDS;
else if(!strcmp(tok, "deleted"))
options |= RRDCONTEXT_OPTION_SHOW_DELETED;
else if(!strcmp(tok, "labels"))
options |= RRDCONTEXT_OPTION_SHOW_LABELS;
else if(!strcmp(tok, "deepscan"))
options |= RRDCONTEXT_OPTION_DEEPSCAN;
}
return options;
}
static int web_client_api_request_v1_context(RRDHOST *host, struct web_client *w, char *url) {
char *context = NULL;
RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
time_t after = 0, before = 0;
const char *chart_label_key = NULL, *chart_labels_filter = NULL;
BUFFER *dimensions = NULL;
buffer_flush(w->response.data);
while(url) {
char *value = mystrsep(&url, "&");
if(!value || !*value) continue;
char *name = mystrsep(&value, "=");
if(!name || !*name) continue;
if(!value || !*value) continue;
// name and value are now the parameters
// they are not null and not empty
if(!strcmp(name, "context") || !strcmp(name, "ctx")) context = value;
else if(!strcmp(name, "after")) after = str2l(value);
else if(!strcmp(name, "before")) before = str2l(value);
else if(!strcmp(name, "options")) options = rrdcontext_to_json_parse_options(value);
else if(!strcmp(name, "chart_label_key")) chart_label_key = value;
else if(!strcmp(name, "chart_labels_filter")) chart_labels_filter = value;
else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) {
if(!dimensions) dimensions = buffer_create(100);
buffer_strcat(dimensions, "|");
buffer_strcat(dimensions, value);
}
}
if(!context || !*context) {
buffer_sprintf(w->response.data, "No context is given at the request.");
return HTTP_RESP_BAD_REQUEST;
}
SIMPLE_PATTERN *chart_label_key_pattern = NULL;
SIMPLE_PATTERN *chart_labels_filter_pattern = NULL;
SIMPLE_PATTERN *chart_dimensions_pattern = NULL;
if(chart_label_key)
chart_label_key_pattern = simple_pattern_create(chart_label_key, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
if(chart_labels_filter)
chart_labels_filter_pattern = simple_pattern_create(chart_labels_filter, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
if(dimensions) {
chart_dimensions_pattern = simple_pattern_create(buffer_tostring(dimensions), ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
buffer_free(dimensions);
}
w->response.data->contenttype = CT_APPLICATION_JSON;
int ret = rrdcontext_to_json(host, w->response.data, after, before, options, context, chart_label_key_pattern, chart_labels_filter_pattern, chart_dimensions_pattern);
simple_pattern_free(chart_label_key_pattern);
simple_pattern_free(chart_labels_filter_pattern);
simple_pattern_free(chart_dimensions_pattern);
return ret;
}
static int web_client_api_request_v1_contexts(RRDHOST *host, struct web_client *w, char *url) {
RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
time_t after = 0, before = 0;
const char *chart_label_key = NULL, *chart_labels_filter = NULL;
BUFFER *dimensions = NULL;
buffer_flush(w->response.data);
while(url) {
char *value = mystrsep(&url, "&");
if(!value || !*value) continue;
char *name = mystrsep(&value, "=");
if(!name || !*name) continue;
if(!value || !*value) continue;
// name and value are now the parameters
// they are not null and not empty
if(!strcmp(name, "after")) after = str2l(value);
else if(!strcmp(name, "before")) before = str2l(value);
else if(!strcmp(name, "options")) options = rrdcontext_to_json_parse_options(value);
else if(!strcmp(name, "chart_label_key")) chart_label_key = value;
else if(!strcmp(name, "chart_labels_filter")) chart_labels_filter = value;
else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) {
if(!dimensions) dimensions = buffer_create(100);
buffer_strcat(dimensions, "|");
buffer_strcat(dimensions, value);
}
}
SIMPLE_PATTERN *chart_label_key_pattern = NULL;
SIMPLE_PATTERN *chart_labels_filter_pattern = NULL;
SIMPLE_PATTERN *chart_dimensions_pattern = NULL;
if(chart_label_key)
chart_label_key_pattern = simple_pattern_create(chart_label_key, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
if(chart_labels_filter)
chart_labels_filter_pattern = simple_pattern_create(chart_labels_filter, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
if(dimensions) {
chart_dimensions_pattern = simple_pattern_create(buffer_tostring(dimensions), ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
buffer_free(dimensions);
}
w->response.data->contenttype = CT_APPLICATION_JSON;
int ret = rrdcontexts_to_json(host, w->response.data, after, before, options, chart_label_key_pattern, chart_labels_filter_pattern, chart_dimensions_pattern);
simple_pattern_free(chart_label_key_pattern);
simple_pattern_free(chart_labels_filter_pattern);
simple_pattern_free(chart_dimensions_pattern);
return ret;
}
inline int web_client_api_request_v1_charts(RRDHOST *host, struct web_client *w, char *url) {
(void)url;
@ -932,8 +1081,6 @@ static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) {
buffer_strcat(wb, "\t\"mirrored_hosts\": [\n");
rrd_rdlock();
rrdhost_foreach_read(host) {
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))
continue;
if (count > 0)
buffer_strcat(wb, ",\n");
@ -945,8 +1092,6 @@ static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) {
count = 0;
rrdhost_foreach_read(host)
{
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))
continue;
if (count > 0)
buffer_strcat(wb, ",\n");
@ -1077,7 +1222,7 @@ inline int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb)
buffer_strcat(wb, "\t\"aclk-ng-available\": false,\n");
buffer_strcat(wb, "\t\"aclk-legacy-available\": false,\n");
#endif
char *agent_id = is_agent_claimed();
char *agent_id = get_agent_claimid();
if (agent_id == NULL)
buffer_strcat(wb, "\t\"agent-claimed\": false,\n");
else {
@ -1398,6 +1543,8 @@ static struct api_command {
{ "data", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_data },
{ "chart", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_chart },
{ "charts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_charts },
{ "context", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_context },
{ "contexts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_contexts },
{ "archivedcharts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_archivedcharts },
// registry checks the ACL by itself, so we allow everything

486
web/gui/dashboard_info.js
View file

@ -1284,6 +1284,41 @@ const ebpfUDPrecv = 'Number of calls to <a href="https://learn.netdata.cloud/doc
'Netdata gives a summary for this chart in <a href="#ebpf_global_udp_bandwidth_call">Network Stack</a>. ' +
'When the integration is <a href="https://learn.netdata.cloud/guides/troubleshoot/monitor-debug-applications-ebpf" target="_blank">enabled</a>, Netdata shows UDP calls per <a href="#ebpf_apps_udp_recv">application</a>.' + ebpfChartProvides
const cgroupCPULimit = 'Total CPU utilization within the configured or system-wide (if not set) limits. When the CPU utilization of a cgroup exceeds the limit for the configured period, the tasks belonging to its hierarchy will be throttled and are not allowed to run again until the next period.'
const cgroupCPU = 'Total CPU utilization within the system-wide CPU resources (all cores). The amount of time spent by tasks of the cgroup in <a href="https://en.wikipedia.org/wiki/CPU_modes#Mode_types" target="_blank">user and kernel</a> modes.'
const cgroupThrottled = 'The percentage of runnable periods when tasks in a cgroup have been throttled. The tasks have not been allowed to run because they have exhausted all of the available time as specified by their CPU quota.'
const cgroupThrottledDuration = 'The total time duration for which tasks in a cgroup have been throttled. When an application has used its allotted CPU quota for a given period, it gets throttled until the next period.'
const cgroupCPUShared = '<p>The weight of each group living in the same hierarchy, that translates into the amount of CPU it is expected to get. The percentage of CPU assigned to the cgroup is the value of shares divided by the sum of all shares in all cgroups in the same level.</p> <p>For example, tasks in two cgroups that have <b>cpu.shares</b> set to 100 will receive equal CPU time, but tasks in a cgroup that has <b>cpu.shares</b> set to 200 receive twice the CPU time of tasks in a cgroup where <b>cpu.shares</b> is set to 100.</p>'
const cgroupCPUPerCore = 'Total CPU utilization per core within the system-wide CPU resources.'
const cgroupCPUSomePressure = 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on CPU. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupCPUSomePressureStallTime = 'The amount of time some processes have been waiting for CPU time.'
const cgroupCPUFullPressure = 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on CPU resource simultaneously. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupCPUFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to CPU congestion.'
const cgroupMemUtilization = 'RAM utilization within the configured or system-wide (if not set) limits. When the RAM utilization of a cgroup exceeds the limit, OOM killer will start killing the tasks belonging to the cgroup.'
const cgroupMemUsageLimit = 'RAM usage within the configured or system-wide (if not set) limits. When the RAM usage of a cgroup exceeds the limit, OOM killer will start killing the tasks belonging to the cgroup.'
const cgroupMemUsage = 'The amount of used RAM and swap memory.'
const cgroupMem = 'Memory usage statistics. The individual metrics are described in the memory.stat section for <a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#per-memory-cgroup-local-status" target="_blank">cgroup-v1</a> and <a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory-interface-files" target="_blank">cgroup-v2</a>.'
const cgroupMemFailCnt = 'The number of memory usage hits limits.'
const cgroupWriteback = '<b>Dirty</b> is the amount of memory waiting to be written to disk. <b>Writeback</b> is how much memory is actively being written to disk.'
const cgroupMemActivity = '<p>Memory accounting statistics.</p><p><b>In</b> - a page is accounted as either mapped anon page (RSS) or cache page (Page Cache) to the cgroup. <b>Out</b> - a page is unaccounted from the cgroup.</p>'
const cgroupPgFaults = '<p>Memory <a href="https://en.wikipedia.org/wiki/Page_fault" target="_blank">page fault</a> statistics.</p><p><b>Pgfault</b> - all page faults. <b>Swap</b> - major page faults.</p>'
const cgroupMemorySomePressure = 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on memory. In this state the CPU is still doing productive work. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupMemorySomePressureStallTime = 'The amount of time some processes have been waiting due to memory congestion.'
const cgroupMemoryFullPressure = 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on memory resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupMemoryFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to memory congestion.'
const cgroupIO = 'The amount of data transferred to and from specific devices as seen by the CFQ scheduler. It is not updated when the CFQ scheduler is operating on a request queue.'
const cgroupServicedOps = 'The number of I/O operations performed on specific devices as seen by the CFQ scheduler.'
const cgroupQueuedOps = 'The number of requests queued for I/O operations.'
const cgroupMergedOps = 'The number of BIOS requests merged into requests for I/O operations.'
const cgroupThrottleIO = 'The amount of data transferred to and from specific devices as seen by the throttling policy.'
const cgroupThrottleIOServicesOps = 'The number of I/O operations performed on specific devices as seen by the throttling policy.'
const cgroupIOSomePressure = 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on I/O. In this state the CPU is still doing productive work. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupIOSomePRessureStallTime = 'The amount of time some processes have been waiting due to I/O congestion.'
const cgroupIOFullPressure = 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> line indicates the share of time in which <b>all non-idle tasks</b> are stalled on I/O resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
const cgroupIOFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to I/O congestion.'
netdataDashboard.context = {
'system.cpu': {
info: function (os) {
@ -3140,6 +3175,64 @@ netdataDashboard.context = {
info: netMTUInfo
},
'k8s.cgroup.net_net': {
mainheads: [
function (_, id) {
var iface;
try {
iface = ' ' + id.substring(id.lastIndexOf('.net_') + 5, id.length);
} catch (e) {
iface = '';
}
return netdataDashboard.gaugeChart('Received' + iface, '12%', 'received');
},
function (_, id) {
var iface;
try {
iface = ' ' + id.substring(id.lastIndexOf('.net_') + 5, id.length);
} catch (e) {
iface = '';
}
return netdataDashboard.gaugeChart('Sent' + iface, '12%', 'sent');
}
],
info: netBytesInfo
},
'k8s.cgroup.net_packets': {
info: netPacketsInfo
},
'k8s.cgroup.net_errors': {
info: netErrorsInfo
},
'k8s.cgroup.net_fifo': {
info: netFIFOInfo
},
'k8s.cgroup.net_drops': {
info: netDropsInfo
},
'k8s.cgroup.net_compressed': {
info: netCompressedInfo
},
'k8s.cgroup.net_events': {
info: netEventsInfo
},
'k8s.cgroup.net_operstate': {
info: netOperstateInfo
},
'k8s.cgroup.net_duplex': {
info: netDuplexInfo
},
'k8s.cgroup.net_carrier': {
info: netCarrierInfo
},
'k8s.cgroup.net_speed': {
info: netSpeedInfo
},
'k8s.cgroup.net_mtu': {
info: netMTUInfo
},
// ------------------------------------------------------------------------
// WIRELESS NETWORK INTERFACES
@ -4008,8 +4101,7 @@ netdataDashboard.context = {
'cgroup.cpu_limit': {
valueRange: "[0, null]",
mainheads: [
function (os, id) {
void (os);
function (_, id) {
cgroupCPULimitIsSet = 1;
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="used"'
@ -4026,15 +4118,11 @@ netdataDashboard.context = {
+ ' role="application"></div>';
}
],
info: 'Total CPU utilization within the configured or system-wide (if not set) limits. '+
'When the CPU utilization of a cgroup exceeds the limit for the configured period, '+
'the tasks belonging to its hierarchy will be throttled and are not allowed to run again until the next period.'
info: cgroupCPULimit
},
'cgroup.cpu': {
mainheads: [
function (os, id) {
void (os);
function (_, id) {
if (cgroupCPULimitIsSet === 0) {
return '<div data-netdata="' + id + '"'
+ ' data-chart-library="gauge"'
@ -4051,60 +4139,107 @@ netdataDashboard.context = {
return '';
}
],
info: 'Total CPU utilization within the system-wide CPU resources (all cores). '+
'The amount of time spent by tasks of the cgroup in '+
'<a href="https://en.wikipedia.org/wiki/CPU_modes#Mode_types" target="_blank">user and kernel</a> modes.'
info: cgroupCPU
},
'cgroup.throttled': {
info: 'The percentage of runnable periods when tasks in a cgroup have been throttled. '+
'The tasks have not been allowed to run because they have exhausted all of the available time as specified by their CPU quota.'
info: cgroupThrottled
},
'cgroup.throttled_duration': {
info: 'The total time duration for which tasks in a cgroup have been throttled. '+
'When an application has used its allotted CPU quota for a given period, it gets throttled until the next period.'
info: cgroupThrottledDuration
},
'cgroup.cpu_shares': {
info: '<p>The weight of each group living in the same hierarchy, that translates into the amount of CPU it is expected to get. '+
'The percentage of CPU assigned to the cgroup is the value of shares divided by the sum of all shares in all cgroups in the same level.</p>'+
'<p>For example, tasks in two cgroups that have <b>cpu.shares</b> set to 100 will receive equal CPU time, '+
'but tasks in a cgroup that has <b>cpu.shares</b> set to 200 receive twice the CPU time of tasks in a cgroup where <b>cpu.shares</b> is set to 100.</p>'
info: cgroupCPUShared
},
'cgroup.cpu_per_core': {
info: 'Total CPU utilization per core within the system-wide CPU resources.'
info: cgroupCPUPerCore
},
'cgroup.cpu_some_pressure': {
info: 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
'<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on CPU. ' +
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupCPUSomePressure
},
'cgroup.cpu_some_pressure_stall_time': {
info: 'The amount of time some processes have been waiting for CPU time.'
info: cgroupCPUSomePressureStallTime
},
'cgroup.cpu_full_pressure': {
info: 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
'<b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on CPU resource simultaneously. ' +
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupCPUFullPressure
},
'cgroup.cpu_full_pressure_stall_time': {
info: 'The amount of time all non-idle processes have been stalled due to CPU congestion.'
info: cgroupCPUFullPressureStallTime
},
'k8s.cgroup.cpu_limit': {
valueRange: "[0, null]",
mainheads: [
function (_, id) {
cgroupCPULimitIsSet = 1;
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="used"'
+ ' data-gauge-max-value="100"'
+ ' data-chart-library="gauge"'
+ ' data-title="CPU"'
+ ' data-units="%"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[4] + '"'
+ ' role="application"></div>';
}
],
info: cgroupCPULimit
},
'k8s.cgroup.cpu': {
mainheads: [
function (_, id) {
if (cgroupCPULimitIsSet === 0) {
return '<div data-netdata="' + id + '"'
+ ' data-chart-library="gauge"'
+ ' data-title="CPU"'
+ ' data-units="%"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[4] + '"'
+ ' role="application"></div>';
} else
return '';
}
],
info: cgroupCPU
},
'k8s.cgroup.throttled': {
info: cgroupThrottled
},
'k8s.cgroup.throttled_duration': {
info: cgroupThrottledDuration
},
'k8s.cgroup.cpu_shares': {
info: cgroupCPUShared
},
'k8s.cgroup.cpu_per_core': {
info: cgroupCPUPerCore
},
'k8s.cgroup.cpu_some_pressure': {
info: cgroupCPUSomePressure
},
'k8s.cgroup.cpu_some_pressure_stall_time': {
info: cgroupCPUSomePressureStallTime
},
'k8s.cgroup.cpu_full_pressure': {
info: cgroupCPUFullPressure
},
'k8s.cgroup.cpu_full_pressure_stall_time': {
info: cgroupCPUFullPressureStallTime
},
'cgroup.mem_utilization': {
info: 'RAM utilization within the configured or system-wide (if not set) limits. '+
'When the RAM utilization of a cgroup exceeds the limit, '+
'OOM killer will start killing the tasks belonging to the cgroup.'
info: cgroupMemUtilization
},
'cgroup.mem_usage_limit': {
mainheads: [
function (os, id) {
void (os);
function (_, id) {
cgroupMemLimitIsSet = 1;
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="used"'
@ -4122,15 +4257,11 @@ netdataDashboard.context = {
+ ' role="application"></div>';
}
],
info: 'RAM usage within the configured or system-wide (if not set) limits. '+
'When the RAM usage of a cgroup exceeds the limit, '+
'OOM killer will start killing the tasks belonging to the cgroup.'
info: cgroupMemUsageLimit
},
'cgroup.mem_usage': {
mainheads: [
function (os, id) {
void (os);
function (_, id) {
if (cgroupMemLimitIsSet === 0) {
return '<div data-netdata="' + id + '"'
+ ' data-chart-library="gauge"'
@ -4147,79 +4278,125 @@ netdataDashboard.context = {
return '';
}
],
info: 'The amount of used RAM and swap memory.'
info: cgroupMemUsage
},
'cgroup.mem': {
info: 'Memory usage statistics. '+
'The individual metrics are described in the memory.stat section for '+
'<a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#per-memory-cgroup-local-status" target="_blank">cgroup-v1 </a>'+
'and '+
'<a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory-interface-files" target="_blank">cgroup-v2</a>.'
info: cgroupMem
},
'cgroup.mem_failcnt': {
info: 'The number of memory usage hits limits.'
info: cgroupMemFailCnt
},
'cgroup.writeback': {
info: '<b>Dirty</b> is the amount of memory waiting to be written to disk. <b>Writeback</b> is how much memory is actively being written to disk.'
info: cgroupWriteback
},
'cgroup.mem_activity': {
info: '<p>Memory accounting statistics.</p>'+
'<p><b>In</b> - a page is accounted as either mapped anon page (RSS) or cache page (Page Cache) to the cgroup. '+
'<b>Out</b> - a page is unaccounted from the cgroup.</p>'
info: cgroupMemActivity
},
'cgroup.pgfaults': {
info: '<p>Memory <a href="https://en.wikipedia.org/wiki/Page_fault" target="_blank">page fault</a> statistics.</p>'+
'<p><b>Pgfault</b> - all page faults. '+
'<b>Swap</b> - major page faults.</p>'
info: cgroupPgFaults
},
'cgroup.memory_some_pressure': {
info: 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
'<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on memory. ' +
'In this state the CPU is still doing productive work. '+
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupMemorySomePressure
},
'cgroup.memory_some_pressure_stall_time': {
info: 'The amount of time some processes have been waiting due to memory congestion.'
info: cgroupMemorySomePressureStallTime
},
'cgroup.memory_full_pressure': {
info: 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
'<b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on memory resource simultaneously. ' +
'In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. '+
'This has severe impact on performance. '+
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupMemoryFullPressure
},
'cgroup.memory_full_pressure_stall_time': {
info: 'The amount of time all non-idle processes have been stalled due to memory congestion.'
info: cgroupMemoryFullPressureStallTime
},
'k8s.cgroup.mem_utilization': {
info: cgroupMemUtilization
},
'k8s.cgroup.mem_usage_limit': {
mainheads: [
function (_, id) {
cgroupMemLimitIsSet = 1;
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="used"'
+ ' data-append-options="percentage"'
+ ' data-gauge-max-value="100"'
+ ' data-chart-library="gauge"'
+ ' data-title="Memory"'
+ ' data-units="%"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[1] + '"'
+ ' role="application"></div>';
}
],
info: cgroupMemUsageLimit
},
'k8s.cgroup.mem_usage': {
mainheads: [
function (_, id) {
if (cgroupMemLimitIsSet === 0) {
return '<div data-netdata="' + id + '"'
+ ' data-chart-library="gauge"'
+ ' data-title="Memory"'
+ ' data-units="MB"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[1] + '"'
+ ' role="application"></div>';
} else
return '';
}
],
info: cgroupMemUsage
},
'k8s.cgroup.mem': {
info: cgroupMem
},
'k8s.cgroup.mem_failcnt': {
info: cgroupMemFailCnt
},
'k8s.cgroup.writeback': {
info: cgroupWriteback
},
'k8s.cgroup.mem_activity': {
info: cgroupMemActivity
},
'k8s.cgroup.pgfaults': {
info: cgroupPgFaults
},
'k8s.cgroup.memory_some_pressure': {
info: cgroupMemorySomePressure
},
'k8s.cgroup.memory_some_pressure_stall_time': {
info: cgroupMemorySomePressureStallTime
},
'k8s.cgroup.memory_full_pressure': {
info: cgroupMemoryFullPressure
},
'k8s.cgroup.memory_full_pressure_stall_time': {
info: cgroupMemoryFullPressureStallTime
},
'cgroup.io': {
info: 'The amount of data transferred to and from specific devices as seen by the CFQ scheduler. '+
'It is not updated when the CFQ scheduler is operating on a request queue.'
info: cgroupIO
},
'cgroup.serviced_ops': {
info: 'The number of I/O operations performed on specific devices as seen by the CFQ scheduler.'
info: cgroupServicedOps
},
'cgroup.queued_ops': {
info: 'The number of requests queued for I/O operations.'
info: cgroupQueuedOps
},
'cgroup.merged_ops': {
info: 'The number of BIOS requests merged into requests for I/O operations.'
info: cgroupMergedOps
},
'cgroup.throttle_io': {
mainheads: [
function (os, id) {
void (os);
function (_, id) {
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="read"'
+ ' data-chart-library="gauge"'
@ -4233,8 +4410,7 @@ netdataDashboard.context = {
+ ' data-colors="' + NETDATA.colors[2] + '"'
+ ' role="application"></div>';
},
function (os, id) {
void (os);
function (_, id) {
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="write"'
+ ' data-chart-library="gauge"'
@ -4249,214 +4425,220 @@ netdataDashboard.context = {
+ ' role="application"></div>';
}
],
info: 'The amount of data transferred to and from specific devices as seen by the throttling policy.'
info: cgroupThrottleIO
},
'cgroup.throttle_serviced_ops': {
info: 'The number of I/O operations performed on specific devices as seen by the throttling policy.'
info: cgroupThrottleIOServicesOps
},
'cgroup.io_some_pressure': {
info: 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
'<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on I/O. ' +
'In this state the CPU is still doing productive work. '+
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupIOSomePressure
},
'cgroup.io_some_pressure_stall_time': {
info: 'The amount of time some processes have been waiting due to I/O congestion.'
info: cgroupIOSomePRessureStallTime
},
'cgroup.io_full_pressure': {
info: 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
'<b>Full</b> line indicates the share of time in which <b>all non-idle tasks</b> are stalled on I/O resource simultaneously. ' +
'In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. '+
'This has severe impact on performance. '+
'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
info: cgroupIOFullPressure
},
'cgroup.io_full_pressure_stall_time': {
info: 'The amount of time all non-idle processes have been stalled due to I/O congestion.'
info: cgroupIOFullPressureStallTime
},
'k8s.cgroup.io': {
info: cgroupIO
},
'k8s.cgroup.serviced_ops': {
info: cgroupServicedOps
},
'k8s.cgroup.queued_ops': {
info: cgroupQueuedOps
},
'k8s.cgroup.merged_ops': {
info: cgroupMergedOps
},
'k8s.cgroup.throttle_io': {
mainheads: [
function (_, id) {
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="read"'
+ ' data-chart-library="gauge"'
+ ' data-title="Read Disk I/O"'
+ ' data-units="KB/s"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[2] + '"'
+ ' role="application"></div>';
},
function (_, id) {
return '<div data-netdata="' + id + '"'
+ ' data-dimensions="write"'
+ ' data-chart-library="gauge"'
+ ' data-title="Write Disk I/O"'
+ ' data-units="KB/s"'
+ ' data-gauge-adjust="width"'
+ ' data-width="12%"'
+ ' data-before="0"'
+ ' data-after="-CHART_DURATION"'
+ ' data-points="CHART_DURATION"'
+ ' data-colors="' + NETDATA.colors[3] + '"'
+ ' role="application"></div>';
}
],
info: cgroupThrottleIO
},
'k8s.cgroup.throttle_serviced_ops': {
info: cgroupThrottleIOServicesOps
},
'k8s.cgroup.io_some_pressure': {
info: cgroupIOSomePressure
},
'k8s.cgroup.io_some_pressure_stall_time': {
info: cgroupIOSomePRessureStallTime
},
'k8s.cgroup.io_full_pressure': {
info: cgroupIOFullPressure
},
'k8s.cgroup.io_full_pressure_stall_time': {
info: cgroupIOFullPressureStallTime
},
'cgroup.swap_read': {
info: ebpfSwapRead
},
'cgroup.swap_write': {
info: ebpfSwapWrite
},
'cgroup.fd_open': {
info: ebpfFileOpen
},
'cgroup.fd_open_error': {
info: ebpfFileOpenError
},
'cgroup.fd_close': {
info: ebpfFileClosed
},
'cgroup.fd_close_error': {
info: ebpfFileCloseError
},
'cgroup.vfs_unlink': {
info: ebpfVFSUnlink
},
'cgroup.vfs_write': {
info: ebpfVFSWrite
},
'cgroup.vfs_write_error': {
info: ebpfVFSWriteError
},
'cgroup.vfs_read': {
info: ebpfVFSRead
},
'cgroup.vfs_read_error': {
info: ebpfVFSReadError
},
'cgroup.vfs_write_bytes': {
info: ebpfVFSWriteBytes
},
'cgroup.vfs_read_bytes': {
info: ebpfVFSReadBytes
},
'cgroup.vfs_fsync': {
info: ebpfVFSSync
},
'cgroup.vfs_fsync_error': {
info: ebpfVFSSyncError
},
'cgroup.vfs_open': {
info: ebpfVFSOpen
},
'cgroup.vfs_open_error': {
info: ebpfVFSOpenError
},
'cgroup.vfs_create': {
info: ebpfVFSCreate
},
'cgroup.vfs_create_error': {
info: ebpfVFSCreateError
},
'cgroup.process_create': {
info: ebpfProcessCreate
},
'cgroup.thread_create': {
info: ebpfThreadCreate
},
'cgroup.task_exit': {
info: ebpfTaskExit
},
'cgroup.task_close': {
info: ebpfTaskClose
},
'cgroup.task_error': {
info: ebpfTaskError
},
'cgroup.dc_ratio': {
info: 'Percentage of file accesses that were present in the directory cache. 100% means that every file that was accessed was present in the directory cache. If files are not present in the directory cache 1) they are not present in the file system, 2) the files were not accessed before. Read more about <a href="https://www.kernel.org/doc/htmldocs/filesystems/the_directory_cache.html" target="_blank">directory cache</a>. Netdata also gives a summary for these charts in <a href="#menu_filesystem_submenu_directory_cache__eBPF_">Filesystem submenu</a>.'
},
'cgroup.shmget': {
info: ebpfSHMget
},
'cgroup.shmat': {
info: ebpfSHMat
},
'cgroup.shmdt': {
info: ebpfSHMdt
},
'cgroup.shmctl': {
info: ebpfSHMctl
},
'cgroup.outbound_conn_v4': {
info: ebpfIPV4conn
},
'cgroup.outbound_conn_v6': {
info: ebpfIPV6conn
},
'cgroup.net_bytes_send': {
info: ebpfBandwidthSent
},
'cgroup.net_bytes_recv': {
info: ebpfBandwidthRecv
},
'cgroup.net_tcp_send': {
info: ebpfTCPSendCall
},
'cgroup.net_tcp_recv': {
info: ebpfTCPRecvCall
},
'cgroup.net_retransmit': {
info: ebpfTCPRetransmit
},
'cgroup.net_udp_send': {
info: ebpfUDPsend
},
'cgroup.net_udp_recv': {
info: ebpfUDPrecv
},
'cgroup.dc_hit_ratio': {
info: ebpfDCHit
},
'cgroup.dc_reference': {
info: ebpfDCReference
},
'cgroup.dc_not_cache': {
info: ebpfDCNotCache
},
'cgroup.dc_not_found': {
info: ebpfDCNotFound
},
'cgroup.cachestat_ratio': {
info: ebpfCachestatRatio
},
'cgroup.cachestat_dirties': {
info: ebpfCachestatDirties
},
'cgroup.cachestat_hits': {
info: ebpfCachestatHits
},
'cgroup.cachestat_misses': {
info: ebpfCachestatMisses
},