* storage engine: add host context API
Add a new API to allow storage engines to manage host contexts.
* Replace single global context with per-engine global context
* Context is full managed by storage engines: a storage engine
can use no context, a global engine context, per host contexts,
or a mix of these.
* Currently, only dbengine uses contexts.
Following the current logic, legacy hosts use their own context,
while non-legacy hosts share the global context.
* storage engine: use empty function instead of null for context ops
* rrdhost: don't check return value for void call
* rrdhost: create context with host
* storage engine: move rrddim ops to rrddim_mem.{c,h}
* storage engine: don't use NULL for end-of-list marker
* storage engine: fallback to default engine
* squashed and rebased to master
* fix overflow and single character bug in sanitize; include rrd.h instead of node_info.h
* added unittest for UTF-8 multibyte sanitization
* Fix unit test compilation
* Fix CMake build
* remove double sanitizer for opentsdb; cleanup sanitize_json_string()
* rename error_description to error_message to avoid conflict with json-c
* revert last and undef error_description from json-c
* more unittests; attempt to fix protobuf map issue
* get rid of rrdlabels_get() and replace it with a safe version that writes the value to a buffer
* added dictionary sorting unittest; rrdlabels_to_buffer() now is sorted
* better sorted dictionary checking
* proper unittesting for sorted dictionaries
* call dictionary deletion callback when destroying the dictionary
* remove obsolete variable
* Fix exporting unit tests
* Fix k8s label parsing test
* workaround for cmocka and strdupz()
* Bypass cmocka memory allocation check
* Revert "Bypass cmocka memory allocation check"
This reverts commit 4c49923839.
* Revert "workaround for cmocka and strdupz()"
This reverts commit 7bebee0480.
* Bypass cmocka memory allocation checks
* respect json formatting for chart labels
* cloud sends colons
* print the value only once
* allow parenthesis in values and spaces; make stream sender send quotes for values
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Move aclk_update_retention to the proper header file
* Do a scan but avoid going through all the dimensions if we have too much to delete -- do not generate a retention message in that case
* Schedule the retention calculation to a worker
* Adjust messages in the access log
* Fix compilation errors with --disable-cloud
* Mark a chart to be exposed only if dimension is created or metadata changes
* Add a calculate liveness for the dimension for collected to non collected (live -> stale) and vice versa
* queue_dimension_to_aclk will have the rrdset and either 0 or last collected time
If 0 then it will be marked as live else it will be marked as stale and last collected time will be sent to the cloud
* Add an extra parameter to indicate if the payload check should be done in the database or it has been done already
* Queue dimension sets dimension liveness and queues the exact payload to store in the database
* Fix compilation error when --disable-cloud is specified
- Variable "hostname" going out of scope leaks the storage it points to.
- Null-checking "rd->name" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
* Try to queue dimension always when:
Trying to clean obsolete charts
If chart has been sent and liveness apparently changed
* delay rotation and skip chart check if not send to cloud
* No need to CLEAR flag during database rotation
Do not clear chart ACLK status for dimension requests
* Change payload_sent to return timestamp of submitted message
* Clear the dimension ACLK flag if we are processing all the charts again
* Check if dimension is already queued to ACLK and ignore it
If queue fails then reset it to retry
Already try to queue the dimension
* Improve dimension cleanup during the retention message calculation
* Change queue_dimension_to_aclk to return void
* If no time range for this dimension then assume it is deleted
* Start streaming for inactive nodes
* Remove dead code
* Correctly report hostname in the access log
* Schedule a dimension deletion without trying to submit a message immediately
* Enable dimension cleanup -- also delete dimension if not found in the dbengine files
Free hostname
* Add function to check a specific chart
* If a chart is not obsoleted, check if the liveness needs to be updated
* Calculate liveness based on a (constant * update_every) for each dimension
* Scan all dimensions when the retention message is constructed and update liveness if needed
* If initial state, set to computed live
* Set computed live state to dimension
* Add a maximum dimension cleanup on startup to prevent message flood
* Schedule chart updates if charts streaming is enabled
* Adjust live state for dimension
* The query executed will have a valid dimension uuid only if memory mode is dbengine
* Switch messages to ACLK RES, ACLK REQ, ACLK STA instead of OG, IN and just AC
* Lookup hostname by node id
* Record hostname when receiving an ACK for a chart sequence
* Additional log_access info
* Adjust log message when receing health log request
* Remove redundant ACK log message
* Remove duplicate log message
* Remove duplicate sql statements
* Rearrange variable definition for clarity
* Make sure node is a valid UUID (check return code)
* Switch to prepare statement when storing active charts / dimensions
* Switch to prepare statement when storing chart labels
* Switch to prepare statement when doing a node id lookup
* Switch to prepare statement when loading the node id for a host
* Improve performance by avoiding db query
* Use prepare statement when counting pending chart messages to send to the cloud
* Delay locking while preparing commands
* No need to use buffer, avoid memory allocation overhead
* Switch to prepare statement when loading pending chart updates to send to the cloud
* Queue a chart immediately to the cloud
* Do not inform the cloud immediately if a dimension stopped collecting use MAX(obsoletion time, 1.5 * update_every)
* Notify cloud immediately on dimension deletion
* Add debug messages
* Do not schedule an update if we are shutting down
* add some logging for ng arch to access.log
* change arrows to IN, OG, AC
* log also the params for aclk requests
* check for wc->host before using wc->host->hostname
* turn two messages to info
* reduce alert event logs
* used thread local variables
* Enhance the dimension delete table and adjust the trigger to include chart_id and host_id
* Add the aclk_process_dimension_deletion function
* Change variable chart_name in aclk_upd_dimension_event (it is st->id from st.type dot st.id)
* Process dimension deletion when retention updates are sent
* Do not send charts if we don't have dimensions
* Add check for uuid_parse return code
* Move retention code to the charts
* Log information about node registration and updates
* Prevent deadlock if aclk_database_enq_cmd locks for a node
* Improve message (indicate that it comes from alerts). This will be improved in a followup PR
* Disable parts that can't be used if the new cloud env is not available
* Set dimension FLAG if message has been queued
* Queue messages using the correct protocol enabled
* Cleanup unused functions
Rename functions that queue charts and dimensions
Improve the generic chart payload add function
Add a counter for pending charts/dimension payloads to avoid polling the db
Delay the retention update message until we are done with the updates
Fix full resync command to handle sequence_id = 0 correctly
Disable functions not needed when the new cloud env functionality is not compiled
* Add chart_payload count and retry count
Output information or error message if we fail to queue chart/dimension PUSH commands
Only try to queue commands if we have chart_payload_count>0
Remove the event loop shutdown opcode handle
* Improve detection of shutdown (check netdata_exit)
* Adjusting info messages
ACLK-NG supports both new and old cloud protocol. Protobuf and C++ compiler are required only for new cloud protocol.
There is no reason to skip building whole ACLK-NG when protobuf is missing.
