* rrdset - in progress
* rrdset optimal constructor; rrdset conflict
* rrdset final touches
* re-organization of rrdset object members
* prevent use-after-free
* dictionary dfe supports also counting of iterations
* rrddim managed by dictionary
* rrd.h cleanup
* DICTIONARY_ITEM now is referencing actual dictionary items in the code
* removed rrdset linked list
* Revert "removed rrdset linked list"
This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.
* removed rrdset linked list
* added comments
* Switch chart uuid to static allocation in rrdset
Remove unused functions
* rrdset_archive() and friends...
* always create rrdfamily
* enable ml_free_dimension
* rrddim_foreach done with dfe
* most custom rrddim loops replaced with rrddim_foreach
* removed accesses to rrddim->dimensions
* removed locks that are no longer needed
* rrdsetvar is now managed by the dictionary
* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853
* conflict callback of rrdsetvar now properly checks if it has to reset the variable
* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM
* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe
* dictionary walkthrough callbacks get dictionary acquired items
* dictionary reference counters that can be dupped from zero
* added advanced functions for get and del
* rrdvar managed by dictionaries
* thread safety for rrdsetvar
* faster rrdvar initialization
* rrdvar string lengths should match in all add, del, get functions
* rrdvar internals hidden from the rest of the world
* rrdvar is now acquired throughout netdata
* hide the internal structures of rrdsetvar
* rrdsetvar is now acquired through out netdata
* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata
* better error handling
* dont create variables if not initialized for health
* dont create variables if not initialized for health again
* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items
* type checking on acquired objects
* rrdcalc renaming of functions
* type checking for rrdfamily_acquired
* rrdcalc managed by dictionaries
* rrdcalc double free fix
* host rrdvars is always needed
* attempt to fix deadlock 1
* attempt to fix deadlock 2
* Remove unused variable
* attempt to fix deadlock 3
* snprintfz
* rrdcalc index in rrdset fix
* Stop storing active charts and computing chart hashes
* Remove store active chart function
* Remove compute chart hash function
* Remove sql_store_chart_hash function
* Remove store_active_dimension function
* dictionary delayed destruction
* formatting and cleanup
* zero dictionary base on rrdsetvar
* added internal error to log delayed destructions of dictionaries
* typo in rrddimvar
* added debugging info to dictionary
* debug info
* fix for rrdcalc keys being empty
* remove forgotten unlock
* remove deadlock
* Switch to metadata version 5 and drop
chart_hash
chart_hash_map
chart_active
dimension_active
v_chart_hash
* SQL cosmetic changes
* do not busy wait while destroying a referenced dictionary
* remove deadlock
* code cleanup; re-organization;
* fast cleanup and flushing of dictionaries
* number formatting fixes
* do not delete configured alerts when archiving a chart
* rrddim obsolete linked list management outside dictionaries
* removed duplicate contexts call
* fix crash when rrdfamily is not initialized
* dont keep rrddimvar referenced
* properly cleanup rrdvar
* removed some locks
* Do not attempt to cleanup chart_hash / chart_hash_map
* rrdcalctemplate managed by dictionary
* register callbacks on the right dictionary
* removed some more locks
* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread
* when looking up for an alarm look using both chart id and chart name
* host initialization a bit more modular
* init rrdlabels on host update
* preparation for dictionary views
* improved comment
* unused variables without internal checks
* service threads isolation and worker info
* more worker info in service thread
* thread cancelability debugging with internal checks
* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647
* dictionary modularization
* Remove unused SQL statement definition
* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated
* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops
* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/proc.plugin/proc_net_dev.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* fix memory leak in rrdset cache_dir
* minor dictionary changes
* dont use index locks in single threaded
* obsolete dict option
* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;
* fix jump on uninitialized value in dictionary; remove double free of cache_dir
* addressed codacy findings
* removed debugging code
* use the private refcount on dictionaries
* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;
* more dictionary statistics
* global statistics about dictionary operations, memory, items, callbacks
* dictionary support for views - missing the public API
* removed warning about unused parameter
* chart and context name for cloud
* chart and context name for cloud, again
* dictionary statistics fixed; first implementation of dictionary views - not currently used
* only the master can globally delete an item
* context needs netdata prefix
* fix context and chart it of spins
* fix for host variables when health is not enabled
* run garbage collector on item insert too
* Fix info message; remove extra "using"
* update dict unittest for new placement of garbage collector
* we need RRDHOST->rrdvars for maintaining custom host variables
* Health initialization needs the host->host_uuid
* split STRING to its own files; no code changes other than that
* initialize health unconditionally
* unit tests do not pollute the global scope with their variables
* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* rrdfamily
* rrddim
* rrdset plugin and module names
* rrdset units
* rrdset type
* rrdset family
* rrdset title
* rrdset title more
* rrdset context
* rrdcalctemplate context and removal of context hash from rrdset
* strings statistics
* rrdset name
* rearranged members of rrdset
* eliminate rrdset name hash; rrdcalc chart converted to STRING
* rrdset id, eliminated rrdset hash
* rrdcalc, alarm_entry, alert_config and some of rrdcalctemplate
* rrdcalctemplate
* rrdvar
* eval_variable
* rrddimvar and rrdsetvar
* rrdhost hostname, os and tags
* fix master commits
* added thread cache; implemented string_dup without locks
* faster thread cache
* rrdset and rrddim now use dictionaries for indexing
* rrdhost now uses dictionary
* rrdfamily now uses DICTIONARY
* rrdvar using dictionary instead of AVL
* allocate the right size to rrdvar flag members
* rrdhost remaining char * members to STRING *
* better error handling on indexing
* strings now use a read/write lock to allow parallel searches to the index
* removed AVL support from dictionaries; implemented STRING with native Judy calls
* string releases should be negative
* only 31 bits are allowed for enum flags
* proper locking on strings
* string threading unittest and fixes
* fix lgtm finding
* fixed naming
* stream chart/dimension definitions at the beginning of a streaming session
* thread stack variable is undefined on thread cancel
* rrdcontext garbage collect per host on startup
* worker control in garbage collection
* relaxed deletion of rrdmetrics
* type checking on dictfe
* netdata chart to monitor rrdcontext triggers
* Group chart label updates
* rrdcontext better handling of collected rrdsets
* rrdpush incremental transmition of definitions should use as much buffer as possible
* require 1MB per chart
* empty the sender buffer before enabling metrics streaming
* fill up to 50% of buffer
* reset signaling metrics sending
* use the shared variable for status
* use separate host flag for enabling streaming of metrics
* make sure the flag is clear
* add logging for streaming
* add logging for streaming on buffer overflow
* circular_buffer proper sizing
* removed obsolete logs
* do not execute worker jobs if not necessary
* better messages about compression disabling
* proper use of flags and updating rrdset last access time every time the obsoletion flag is flipped
* monitor stream sender used buffer ratio
* Update exporting unit tests
* no need to compare label value with strcmp
* streaming send workers now monitor bandwidth
* workers now use strings
* streaming receiver monitors incoming bandwidth
* parser shift of worker ids
* minor fixes
* Group chart label updates
* Populate context with dimensions that have data
* Fix chart id
* better shift of parser worker ids
* fix for streaming compression
* properly count received bytes
* ensure LZ4 compression ring buffer does not wrap prematurely
* do not stream empty charts; do not process empty instances in rrdcontext
* need_to_send_chart_definition() does not need an rrdset lock any more
* rrdcontext objects are collected, after data have been written to the db
* better logging of RRDCONTEXT transitions
* always set all variables needed by the worker utilization charts
* implemented double linked list for most objects; eliminated alarm indexes from rrdhost; and many more fixes
* lockless strings design - string_dup() and string_freez() are totally lockless when they dont need to touch Judy - only Judy is protected with a read/write lock
* STRING code re-organization for clarity
* thread_cache improvements; double numbers precision on worker threads
* STRING_ENTRY now shadown STRING, so no duplicate definition is required; string_length() renamed to string_strlen() to follow the paradigm of all other functions, STRING internal statistics are now only compiled with NETDATA_INTERNAL_CHECKS
* rrdhost index by hostname now cleans up; aclk queries of archieved hosts do not index hosts
* Add index to speed up database context searches
* Removed last_updated optimization (was also buggy after latest merge with master)
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* type checking on dictionary return values
* first STRING implementation, used by DICTIONARY and RRDLABEL
* enable AVL compilation of STRING
* Initial functions to store context info
* Call simple test functions
* Add host_id when getting charts
* Allow host to be null and in this case it will process the localhost
* Simplify init
Do not use strdupz - link directly to sqlite result set
* Init the database during startup
* make it compile - no functionality yet
* intermediate commit
* intermidiate
* first interface to sql
* loading instances
* check if we need to update cloud
* comparison of rrdcontext on conflict
* merge context titles
* rrdcontext public interface; statistics on STRING; scratchpad on DICTIONARY
* dictionaries maintain version numbers; rrdcontext api
* cascading changes
* first operational cleanup
* string unittest
* proper cleanup of referenced dictionaries
* added rrdmetrics
* rrdmetric starting retention
* Add fields to context
Adjuct context creation and delete
* Memory cleanup
* Fix get context list
Fix memory double free in tests
Store context with two hosts
* calculated retention
* rrdcontext retention with collection
* Persist database and shutdown
* loading all from sql
* Get chart list and dimension list changes
* fully working attempt 1
* fully working attempt 2
* missing archived flag from log
* fixed archived / collected
* operational
* proper cleanup
* cleanup - implemented all interface functions - dictionary react callback triggers after the dictionary is unlocked
* track all reasons for changes
* proper tracking of reasons of changes
* fully working thread
* better versioning of contexts
* fix string indexing with AVL
* running version per context vs hub version; ifdef dbengine
* added option to disable rrdmetrics
* release old context when a chart changes context
* cleanup properly
* renamed config
* cleanup contexts; general cleanup;
* deletion inline with dequeue; lots of cleanup; child connected/disconnected
* ml should start after rrdcontext
* added missing NULL to ri->rrdset; rrdcontext flags are now only changed under a mutex lock
* fix buggy STRING under AVL
* Rework database initialization
Add migration logic to the context database
* fix data race conditions during context deletion
* added version hash algorithm
* fix string over AVL
* update aclk-schemas
* compile new ctx related protos
* add ctx stream message utils
* add context messages
* add dummy rx message handlers
* add the new topics
* add ctx capability
* add helper functions to send the new messages
* update cmake build to not fail
* update topic names
* handle rrdcontext_enabled
* add more functions
* fatal on OOM cases instead of return NULL
* silence unknown query type error
* fully working attempt 1
* fully working attempt 2
* allow compiling without ACLK
* added family to the context
* removed excess character in UUID
* smarter merging of titles and families
* Database migration code to add family
Add family to SQL_CHART_DATA and VERSIONED_CONTEXT_DATA
* add family to context message
* enable ctx in communication
* hardcoded enabled contexts
* Add hard code for CTX
* add update node collectors to json
* add context message log
* fix log about last_time_t
* fix collected flags for queued items
* prevent crash on charts cleanup
* fix bug in AVL indexing of dictionaries; make sure react callback of dictionaries has a reference counter, which is acquired while the dictionary is locked
* fixed dictionary unittest
* strict policy to cleanup and garbage collector
* fix db rotation and garbage collection timings
* remove deadlock
* proper garbage collection - a lot faster retention recalculation
* Added not NULL in database columns
Remove migration code for context -- we will ship with version 1 of the table schema
Added define for query in tests to detect localhost
* Use UUID_STR_LEN instead of GUID_LEN + 1
Use realistic timestamps when adding test data in the database
* Add NULL checks for passed parameters
* Log deleted context when compiled with NETDATA_INTERNAL_CHECKS
* Error checking for null host id
* add missing ContextsCheckpoint log convertor
* Fix spelling in VACCUM
* Hold additional information for host -- prepare to load archived hosts on startup
* Make sure claim id is valid
* is_get_claimed is actually get the current claim id
* Simplify ctx get chart list query
* remove env negotiation
* fix string unittest when there are some strings already in the index
* propagate live-retention flag upstream; cleanup all update reasons; updated instances logging; automated attaching started/stopped collecting flags;
* first implementation of /api/v1/contexts
* full contexts API; updated swagger
* disabled debugging; rrdcontext enabled by default
* final cleanup and renaming of global variables
* return current time on currently collected contexts, charts and dimensions
* added option "deepscan" to the API to have the server refresh the retention and recalculate the contexts on the fly
* fixed identation of yaml
* Add constrains to the host table
* host->node_id may not be available
* new capabilities
* lock the context while rendering json
* update aclk-schemas
* added permanent labels to all charts about plugin, module and family; added labels to all proc plugin modules
* always add the labels
* allow merging of families down to [x]
* dont show uuids by default, added option to enable them; response is now accepting after,before to show only data for a specific timeframe; deleted items are only shown when "deleted" is requested; hub version is now shown when "queue" is requested
* Use the localhost claim id
* Fix to handle host constrains better
* cgroups: add "k8s." prefix to chart context in k8s
* Improve sqlite metadata version migration check
* empty values set to "[none]"; fix labels unit test to reflect that
* Check if we reached the version we want first (address CODACY report re: Array index 'i' is used before limits check)
* Rewrite condition to address CODACY report (Redundant condition: t->filter_callback. '!A || (A && B)' is equivalent to '!A || B')
* Properly unlock context
* fixed memory leak on rrdcontexts - it was not freeing all dictionaries in rrdhost; added wait of up to 100ms on dictionary_destroy() to give time to dictionaries to release their items before destroying them
* fixed memory leak on rrdlabels not freed on rrdinstances
* fixed leak when dimensions and charts are redefined
* Mark entries for charts and dimensions as submitted to the cloud 3600 seconds after their creation
Mark entries for charts and dimensions as updated (confirmed by the cloud) 1800 seconds after their submission
* renamed struct string
* update cgroups alarms
* fixed codacy suggestions
* update dashboard info
* fix k8s_cgroup_10s_received_packets_storm alarm
* added filtering options to /api/v1/contexts and /api/v1/context
* fix eslint
* fix eslint
* Fix pointer binding for host / chart uuids
* Fix cgroups unit tests
* fixed non-retention updates not propagated upstream
* removed non-fatal fatals
* Remove context from 2 way string merge.
* Move string_2way_merge to dictionary.c
* Add 2-way string merge tests.
* split long lines
* fix indentation in netdata-swagger.yaml
* update netdata-swagger.json
* yamllint please
* remove the deleted flag when a context is collected
* fix yaml warning in swagger
* removed non-fatal fatals
* charts should now be able to switch contexts
* allow deletion of unused metrics, instances and contexts
* keep the queued flag
* cleanup old rrdinstance labels
* dont hide objects when there is no filter; mark objects as deleted when there are no sub-objects
* delete old instances once they changed context
* delete all instances and contexts that do not have sub-objects
* more precise transitions
* Load archived hosts on startup (part 1)
* update the queued time every time
* disable by default; dedup deleted dimensions after snapshot
* Load archived hosts on startup (part 2)
* delayed processing of events until charts are being collected
* remove dont-trigger flag when object is collected
* polish all triggers given the new dont_process flag
* Remove always true condition
Enums for readbility / create_host_callback only if ACLK is enabled (for now)
* Skip retention message if context streaming is enabled
Add messages in the access log if context streaming is enabled
* Check for node id being a UUID that can be parsed
Improve error check / reporting when loading archived hosts and creating ACLK sync threads
* collected, archived, deleted are now mutually exclusive
* Enable the "orphan" handling for now
Remove dead code
Fix memory leak on free host
* Queue charts and dimensions will be no-op if host is set to stream contexts
* removed unused parameter and made sure flags are set on rrdcontext insert
* make the rrdcontext thread abort mid-work when exiting
* Skip chart hash computation and storage if contexts streaming is enabled
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Timo <timotej@netdata.cloud>
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
Co-authored-by: Vasilis Kalintiris <vasilis@netdata.cloud>
* Report unknown page only once
Get metric storage size by the page type
Verify validity of the page and skip problematic ones
* Change PAGE_SIZE to PAGE_POINT_SIZE_BYTES
* Add bitmap256 and unittests
* Fix unit test
tier_page_type array
page_type_size arrays
* Add another counter to not rely on uint8_t overflow to stop the test loop
* Tier part 1
* Tier part 2
* Tier part 3
* Tier part 4
* Tier part 5
* Fix some ML compilation errors
* fix more conflicts
* pass proper tier
* move metric_uuid from state to RRDDIM
* move aclk_live_status from state to RRDDIM
* move ml_dimension from state to RRDDIM
* abstracted the data collection interface
* support flushing for mem db too
* abstracted the query api
* abstracted latest/oldest time per metric
* cleanup
* store_metric for tier1
* fix for store_metric
* allow multiple tiers, more than 2
* state to tier
* Change storage type in db. Query param to request min, max, sum or average
* Store tier data correctly
* Fix skipping tier page type
* Add tier grouping in the tier
* Fix to handle archived charts (part 1)
* Temp fix for query granularity when requesting tier1 data
* Fix parameters in the correct order and calculate the anomaly based on the anomaly count
* Proper tiering grouping
* Anomaly calculation based on anomaly count
* force type checking on storage handles
* update cmocka tests
* fully dynamic number of storage tiers
* fix static allocation
* configure grouping for all tiers; disable tiers for unittest; disable statsd configuration for private charts mode
* use default page dt using the tiering info
* automatic selection of tier
* fix for automatic selection of tier
* working prototype of dynamic tier selection
* automatic selection of tier done right (I hope)
* ask for the proper tier value, based on the grouping function
* fixes for unittests and load_metric_next()
* fixes for lgtm findings
* minor renames
* add dbengine to page cache size setting
* add dbengine to page cache with malloc
* query engine optimized to loop as little are required based on the view_update_every
* query engine grouping methods now do not assume a constant number of points per group and they allocate memory with OWA
* report db points per tier in jsonwrap
* query planer that switches database tiers on the fly to satisfy the query for the entire timeframe
* dbegnine statistics and documentation (in progress)
* calculate average point duration in db
* handle single point pages the best we can
* handle single point pages even better
* Keep page type in the rrdeng_page_descr
* updated doc
* handle future backwards compatibility - improved statistics
* support &tier=X in queries
* enfore increasing iterations on tiers
* tier 1 is always 1 iteration
* backfilling higher tiers on first data collection
* reversed anomaly bit
* set up to 5 tiers
* natural points should only be offered on tier 0, except a specific tier is selected
* do not allow more than 65535 points of tier0 to be aggregated on any tier
* Work only on actually activated tiers
* fix query interpolation
* fix query interpolation again
* fix lgtm finding
* Activate one tier for now
* backfilling of higher tiers using raw metrics from lower tiers
* fix for crash on start when storage tiers is increased from the default
* more statistics on exit
* fix bug that prevented higher tiers to get any values; added backfilling options
* fixed the statistics log line
* removed limit of 255 iterations per tier; moved the code of freezing rd->tiers[x]->db_metric_handle
* fixed division by zero on zero points_wanted
* removed dead code
* Decide on the descr->type for the type of metric
* dont store metrics on unknown page types
* free db_metric_handle on sql based context queries
* Disable STORAGE_POINT value check in the exporting engine unit tests
* fix for db modes other than dbengine
* fix for aclk archived chart queries destroying db_metric_handles of valid rrddims
* fix left-over freez() instead of OWA freez on median queries
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* new query engine
* use Index
* Revert change that changed in-memory page indexing to start time - update_every + 1
* use internal_error() to cleanup the code
* interpolates values when generating points
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
* netdata doubles
* fix cmocka test
* fix cmocka test again
* fix left-overs of long double to NETDATA_DOUBLE
* RRDDIM detached from disk representation; db settings in [db] section of netdata.conf
* update the memory before saving
* rrdset is now detached from file structures too
* on memory mode map, update the memory mapped structures on every iteration
* allow RRD_ID_LENGTH_MAX to be changed
* granularity secs, back to update every
* fix formatting
* more formatting
* set grouping functions
* storage engine should check the validity of timestamps, not the query engine
* calculate and store in RRDR anomaly rates for every query
* anomaly rate used by volume metric correlations
* mc volume should use absolute data, to avoid cancelling effect
* return anomaly-rates in jasonwrap with jw-anomaly-rates option to data queries
* dont return null on anomaly rates
* allow passing group query options from the URL
* added countif to the query engine and used it in metric correlations
* fix configure
* fix countif and anomaly rate percentages
* added group_options to metric correlations; updated swagger
* added newline at the end of yaml file
* always check the time the highlighted window was above/below the highlighted window
* properly track time in memory queries
* error for internal checks only
* moved pack_storage_number() into the storage engines
* moved unpack_storage_number() inside the storage engines
* remove old comment
* pass unit tests
* properly detect zero or subnormal values in pack_storage_number()
* fill nulls before the value, not after
* make sure math.h is included
* workaround for isfinite()
* fix for isfinite()
* faster isfinite() alternative
* fix for faster isfinite() alternative
* next_metric() now returns end_time too
* variable step implemented in a generic way
* remove left-over variables
* ensure we always complete the wanted number of points
* fixes
* ensure no infinite loop
* mc-volume-improvements: Add information about invalid condition
* points should have a duration in the past
* removed unneeded info() line
* Fix unit tests for exporting engine
* new_point should only be checked when it is fetched from the db; better comment about the premature breaking of the main query loop
Co-authored-by: Thiago Marques <thiagoftsm@gmail.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* storage engine: add host context API
Add a new API to allow storage engines to manage host contexts.
* Replace single global context with per-engine global context
* Context is full managed by storage engines: a storage engine
can use no context, a global engine context, per host contexts,
or a mix of these.
* Currently, only dbengine uses contexts.
Following the current logic, legacy hosts use their own context,
while non-legacy hosts share the global context.
* storage engine: use empty function instead of null for context ops
* rrdhost: don't check return value for void call
* rrdhost: create context with host
* storage engine: move rrddim ops to rrddim_mem.{c,h}
* storage engine: don't use NULL for end-of-list marker
* storage engine: fallback to default engine
* Remove error (no real value)
* Add a parameter to create an in-memory database for stress testing
* Add a new parameter to the stresstest command to set the number of deisred libuv worker threads
* one way allocator to speed up context queries
* fixed a bug while expanding memory pages
* reworked for clarity and finally fixed the bug of allocating memory beyond the page size
* further optimize allocation step to minimize the number of allocations made
* implement strdup with memcpy instead of strcpy
* added documentation
* prevent an uninitialized use of owa
* added callocz() interface
* integrate onewayalloc everywhere - apart sql queries
* one way allocator is now used in context queries using archived charts in sql
* align on the size of pointers
* forgotten freez()
* removed not needed memcpys
* give unique names to global variables to avoid conflicts with system definitions
* Add timeout parameter in queries and in calling functions
* Add CANCEL flag in RRDR and code to cancel a query
* Update swagger
* Format swagger file properly
Prior to this commit both daemon/commands.c and spawn/spawn.c used to
include database/engine/rrdenginelib.h, ie. a header file that is available
only when enabling the dbengine feature.
* Replace all usages of SN_EXISTS with SN_DEFAULT_FLAGS.
* Remove references to SN_NOT_EXISTS in comments.
* Replace raw zero constant with SN_EMPTY_SLOT.
* Use get_storage_number_flags only in storage_number.{c,h}
* Compare against SN_EMPTY_SLOT to check if a storage_number exists.
This is safe because:
1. rrdset_done_interpolate() is the only place where we call store_metric(),
2. All store_metric() calls, except for one, store an SN_EMPTY_SLOT value.
3. When we are not storing an SN_EMPTY_SLOT value, the flags that we pass to
pack_storage_number() can be either SN_EXISTS *or* SN_EXISTS_RESET.
* Compare only the SN_EXISTS_RESET bit to find reset values.
* Remove get_storage_number_flags from storage_number.h
* Do not set storage_number flags outside of rrdset_done_interpolate().
This is a NFC intended to limit the scope of storage_number flags
processing to just one function.
* Set reset bit without overwriting the rest of the flags.
* Rename SN_EXISTS to SN_ANOMALY_BIT.
* Use GOTOs in pack_storage_number to return from a single place.
* Teach pack_storage_number how to handle anomalous zero values.
Up until now, a storage_number had always either the SN_EXISTS or
SN_EXISTS_RESET bit set. This meant that it was not possible for any
packed storage_number to compare equal to the SN_EMPTY_SLOT.
However, the SN_ANOMALY_BIT can be set to zero. This is fine for every
value other than the anomalous 0 value, because it would compare equal to
SN_EMPTY_SLOT. We address this issue by mapping the anomalous zero value
to SN_EXISTS_100 (a number which was not possible to generate with the
previous versions of the agent, ie. it won't exist in older dbengine files).
This change was tested manually by intentionally flipping the anomaly
bit for odd/even iterations in rrdset_done_interpolate. Prior to this
change, charts whose dimensions had 0 values, where showing up in the
dashboard as gaps (SN_EMPTY_SLOT), whereas with this commit the values
are displayed correctly.
* Implemented collector metadata logging
* Added persistent GUIDs for charts and dimensions
* Added metadata log replay and automatic compaction
* Added detection of charts with no active collector (archived)
* Added new endpoint to report archived charts via `/api/v1/archivedcharts`
* Added support for collector metadata update
Co-authored-by: Markos Fountoulakis <44345837+mfundul@users.noreply.github.com>
* fix_db_race_condition: unit test
Adjust unit test for dbengine
* fix_db_race_condition: page cache
Fix database
* fix_db_race_condition: Missing function call
This commit brings the correct function call inside rrdengine.c
* Removed support for 16-bit and 8-bit counter overflow
* Improve behaviour of counter overflow detection versus counter resets.
* Added support for signed 32-bit and 64-bit limits for counter overflows.
* Fixed signed incremental counter issues and added unit tests.
* Basic functionality for dbengine stress test.
* Fix coverity defects
* Refactored dbengine stress test to be configurable
* Added benchmark results and evaluation in dbengine documentation
* Make dbengine the default memory mode
* Fix memory corruption during deallocation of page cache
* Refactored dataset generator in order to support the upcoming self-validating stress test and multithreading.
* Fix starvation in database engine loop when the command queues are continuously populated
* Fixing disk quota limits for dbengine dataset generator
* Variable Granularity support for data collection in the dbengine.
* Variable Granularity support for data collection in the daemon.
* Added tests to validate the data being queried after having been collected by changing data collection interval
* Fix memory corruption
* Updated database engine documentation about data collection frequency behaviour
* Database engine prototype version 0
* Database engine initial integration with netdata POC
* Scalable database engine with file and memory management.
* Database engine integration with netdata
* Added MIN MAX definitions to fix alpine build of travis CI
* Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks
* DB engine disk protocol correction
* Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine
* Fix configure to require openSSL for DB engine
* Fix netdata daemon health not holding read lock when iterating chart dimensions
* Optimized query API for new DB engine and old netdata DB fallback code-path
* netdata database internal query API improvements and cleanup
* Bugfix for DB engine queries returning empty values
* Added netdata internal check for data queries for old and new DB
* Added statistics to DB engine and fixed memory corruption bug
* Added preliminary charts for DB engine statistics
* Changed DB engine ratio statistics to incremental
* Added netdata statistics charts for DB engine internal statistics
* Fix for netdata not compiling successfully when missing dbengine dependencies
* Added DB engine functional test to netdata unittest command parameter
* Implemented DB engine dataset generator based on example.random chart
* Fix build error in CI
* Support older versions of libuv1
* Fixes segmentation fault when using multiple DB engine instances concurrently
* Fix memory corruption bug
* Fixed createdataset advanced option not exiting
* Fix for DB engine not working on FreeBSD
* Support FreeBSD library paths of new dependencies
* Workaround for unsupported O_DIRECT in OS X
* Fix unittest crashing during cleanup
* Disable DB engine FS caching in Apple OS X since O_DIRECT is not available
* Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host
* Modified DB engine dataset generator to create multiple files
* Toned down overzealous page cache prefetcher
* Reduce internal memory fragmentation for page-cache data pages
* Added documentation describing the DB engine
* Documentation bugfixes
* Fixed unit tests compilation errors since last rebase
* Added note to back-up the DB engine files in documentation
* Added codacy fix.
* Support old gcc versions for atomic counters in DB engine
