archive/netdata_netdata
0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-06 22:38:55 +00:00
Code Issues Projects Releases Wiki Activity

Split database overview and configuration reference ()

Browse source 
Co-authored-by: ilyam8 <ilya@netdata.cloud>
This commit is contained in:
Fotis Voutsas 2024-11-25 16:35:52 +02:00 • committed by GitHub
parent 3ec78e4599
commit f3ac2fea1a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 181 additions and 170 deletions

2
docs/dashboards-and-charts/visualization-date-and-time-controls.md
View file

@ -81,7 +81,7 @@ beyond stored historical metrics, you'll see this message:
![image](https://user-images.githubusercontent.com/70198089/225851033-43b95164-a651-48f2-8915-6aac9739ed93.png)
At any time, [configure the internal TSDB's storage capacity](/src/database/README.md) to expand your
At any time, [configure the internal TSDB's storage capacity](/src/database/CONFIGURATION.md) to expand your
depth of historical metrics.
### Timezone selector

2
docs/deployment-guides/deployment-strategies.md
View file

@ -193,7 +193,7 @@ We also suggest that you:
For increased security, user management and access to our latest features, tools and troubleshooting solutions.
2. [Change how long Netdata stores metrics](/src/database/README.md#modes)
2. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
To control Netdata's memory use, when you have a lot of ephemeral metrics.

2
docs/netdata-agent/configuration/common-configuration-changes.md
View file

@ -19,7 +19,7 @@ changes reflected in those visualizations due to the way Netdata Cloud proxies m
### Increase the long-term metrics retention period
Read our doc on [increasing long-term metrics storage](/src/database/README.md#tiers) for details.
Read our doc on [increasing long-term metrics storage](/src/database/CONFIGURATION.md#tiers) for details.
## Modify alerts and notifications

18
docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md
View file

@ -22,8 +22,8 @@ The following table summarizes the effect of each optimization on the CPU, RAM a
| [Use streaming and replication](#use-streaming-and-replication) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| [Reduce data collection frequency](#reduce-collection-frequency) | :heavy_check_mark: | | :heavy_check_mark: |
| [Change how long Netdata stores metrics](/src/database/README.md#tiers) | | :heavy_check_mark: | :heavy_check_mark: |
| [Use a different metric storage database](/src/database/README.md) | | :heavy_check_mark: | :heavy_check_mark: |
| [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers) | | :heavy_check_mark: | :heavy_check_mark: |
| [Use a different metric storage database](/src/database/CONFIGURATION.md) | | :heavy_check_mark: | :heavy_check_mark: |
| [Disable machine learning](#disable-machine-learning) | :heavy_check_mark: | | |
| [Use a reverse proxy](#run-netdata-behind-a-proxy) | :heavy_check_mark: | | |
| [Disable/lower gzip compression for the Agent dashboard](#disablelower-gzip-compression-for-the-dashboard) | :heavy_check_mark: | | |
@ -72,8 +72,8 @@ The memory footprint of Netdata is mainly influenced by the number of metrics co
To estimate and control memory consumption, you can (either one or a combination of the following actions):
1. [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors)
2. [Change how long Netdata stores metrics](/src/database/README.md#tiers)
3. [Use a different metric storage database](/src/database/README.md).
2. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
3. [Use a different metric storage database](/src/database/CONFIGURATION.md#modes).
### Disk footprint and I/O
@ -90,11 +90,11 @@ To optimize your disk footprint in any aspect described below, you can:
To configure retention, you can:
1. [Change how long Netdata stores metrics](/src/database/README.md#tiers).
1. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
To control disk I/O:
1. [Use a different metric storage database](/src/database/README.md),
1. [Use a different metric storage database](/src/database/CONFIGURATION.md),
Minimize deployment impact on the production system by optimizing disk footprint:
@ -123,7 +123,7 @@ in `stream.conf`). On the child nodes you should add to `netdata.conf` the follo
### Use memory mode ram for the child nodes
See [using a different metric storage database](/src/database/README.md).
See [using a different metric storage database](/src/database/README.md#modes).
## Disable unneeded plugins or collectors
@ -211,11 +211,11 @@ update_every: 10
## Lower memory usage for metrics retention
See how
to [change how long Netdata stores metrics](/src/database/README.md#tiers).
to [change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
## Use a different metric storage database
Consider [using a different metric storage database](/src/database/README.md)
Consider [using a different metric storage database](/src/database/README.md#modes)
when running Netdata on IoT devices, and for children in a parent-child set up based
on [streaming and replication](/docs/observability-centralization-points/README.md).

3
docs/netdata-agent/configuration/optimizing-metrics-database/README.md
View file

@ -1,3 +0,0 @@
# Optimizing Metrics Database Overview
This section contains documentation to help you understand how the metrics DB works, understand the key features and configure them to suit your needs.

2
docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md
View file

@ -36,7 +36,7 @@ gantt
**Configuring dbengine mode and retention**:
- Enable dbengine mode: The dbengine mode is already the default, so no configuration change is necessary. For reference, the dbengine mode can be configured by setting `[db].mode` to `dbengine` in `netdata.conf`.
- Adjust retention (optional): see [Change how long Netdata stores metrics](/src/database/README.md#tiers).
- Adjust retention (optional): see [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
## `ram`

100
src/daemon/config/README.md
View file

@ -64,10 +64,10 @@ Please note that your data history will be lost if you have modified `history` p
| setting | default | info |
|:----------------------------------:|:-------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#netdata-process-scheduling-policy) |
| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#netdata-process-scheduling-policy) |
| OOM score | `0` | |
| glibc malloc arena max for plugins | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
| glibc malloc arena max for Netdata | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
| glibc malloc arena max for plugins | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
| glibc malloc arena max for Netdata | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
| hostname | auto-detected | The hostname of the computer running Netdata. |
| host access prefix | empty | This is used in docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43). |
| timezone | auto-detected | The timezone retrieved from the environment variable |
@ -76,22 +76,22 @@ Please note that your data history will be lost if you have modified `history` p
### [db] section options
| setting | default | info |
|:---------------------------------------------:|:-------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |
| retention | `3600` | Used with `mode = ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/src/database/README.md) for more information. |
| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |
| dbengine page cache size | `32MiB` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |
| dbengine tier **`N`** retention size | `1GiB` | The disk space dedicated to metrics storage, per tier. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
| dbengine tier **`N`** retention time | `14d`, `3mo`, `1y`, `1y`, `1y` | The database retention, expressed in time. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). |
| dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. <br /> `N belongs to [1..4]` |
| dbengine tier back fill | `new` | Specifies the strategy of recreating missing data on higher database Tiers.<br /> `new`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). <br /> `none`: No back filling is applied. <br /> `N belongs to [1..4]` |
| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. For more information check [Memory Deduplication - Kernel Same Page Merging - KSM](/src/database/README.md#ksm) |
| cleanup obsolete charts after | `1h` | See [monitoring ephemeral containers](/src/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
| gap when lost iterations above | `1` | |
| cleanup orphan hosts after | `1h` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
| setting | default | info |
|:---------------------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |
| retention | `3600` | Used with `mode = ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md) for more information. |
| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |
| dbengine page cache size | `32MiB` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |
| dbengine tier **`N`** retention size | `1GiB` | The disk space dedicated to metrics storage, per tier. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
| dbengine tier **`N`** retention time | `14d`, `3mo`, `1y`, `1y`, `1y` | The database retention, expressed in time. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). |
| dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. <br /> `N belongs to [1..4]` |
| dbengine tier back fill | `new` | Specifies the strategy of recreating missing data on higher database Tiers.<br /> `new`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). <br /> `none`: No back filling is applied. <br /> `N belongs to [1..4]` |
| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. |
| cleanup obsolete charts after | `1h` | See [monitoring ephemeral containers](/src/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
| gap when lost iterations above | `1` | |
| cleanup orphan hosts after | `1h` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
> ### Info
>
@ -103,7 +103,7 @@ Please note that your data history will be lost if you have modified `history` p
|:-------------------:|:------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| config | `/etc/netdata` | The directory configuration files are kept. |
| stock config | `/usr/lib/netdata/conf.d` | |
| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#log-files) are kept. |
| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#log-files) are kept. |
| web | `/usr/share/netdata/web` | The directory the web static files are kept. |
| cache | `/var/cache/netdata` | The directory the memory database will be stored if and when Netdata exits. Netdata will re-read the database when it will start again, to continue from the same point. |
| lib | `/var/lib/netdata` | Contains the alert log and the Netdata instance GUID. |
@ -112,25 +112,25 @@ Please note that your data history will be lost if you have modified `history` p
| plugins | `"/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"` | The directory plugin programs are kept. This setting supports multiple directories, space separated. If any directory path contains spaces, enclose it in single or double quotes. |
| health config | `/etc/netdata/health.d` | The directory containing the user alert configuration files, to override the stock configurations |
| stock health config | `/usr/lib/netdata/conf.d/health.d` | Contains the stock alert configuration files for each collector |
| registry | `/opt/netdata/var/lib/netdata/registry` | Contains the [registry](/src/registry/README.md) database and GUID that uniquely identifies each Netdata Agent |
| registry | `/opt/netdata/var/lib/netdata/registry` | Contains the [registry](/src/registry/README.md) database and GUID that uniquely identifies each Netdata Agent |
### [logs] section options
There are additional configuration options for the logs. For more info, see [Netdata Logging](/src/libnetdata/log/README.md).
| setting | default | info |
|:----------------------------------:|:-----------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| debug flags | `0x0000000000000000` | Bitmap of debug options to enable. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
| debug | `/var/log/netdata/debug.log` | The filename to save debug information. This file will not be created if debugging is not enabled. You can also set it to `syslog` to send the debug messages to syslog, or `off` to disable this log. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
| error | `/var/log/netdata/error.log` | The filename to save error messages for Netdata daemon and all plugins (`stderr` is sent here for all Netdata programs, including the plugins). You can also set it to `syslog` to send the errors to syslog, or `off` to disable this log. |
| access | `/var/log/netdata/access.log` | The filename to save the log of web clients accessing Netdata charts. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. |
| collector | `journal` | The filename to save the log of Netdata collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| health | `journal` | The filename to save the log of Netdata health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| daemon | `journal` | The filename to save the log of Netdata daemon. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| facility | `daemon` | A facility keyword is used to specify the type of system that is logging the message. |
| logs flood protection period | `1m` | Length of period during which the number of errors should not exceed the `errors to trigger flood protection`. |
| logs to trigger flood protection | `1000` | Number of errors written to the log in `errors flood protection period` sec before flood protection is activated. |
| level | `info` | Controls which log messages are logged, with error being the most important. Supported values: `info` and `error`. |
| setting | default | info |
|:--------------------------------:|:-----------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| debug flags | `0x0000000000000000` | Bitmap of debug options to enable. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
| debug | `/var/log/netdata/debug.log` | The filename to save debug information. This file will not be created if debugging is not enabled. You can also set it to `syslog` to send the debug messages to syslog, or `off` to disable this log. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
| error | `/var/log/netdata/error.log` | The filename to save error messages for Netdata daemon and all plugins (`stderr` is sent here for all Netdata programs, including the plugins). You can also set it to `syslog` to send the errors to syslog, or `off` to disable this log. |
| access | `/var/log/netdata/access.log` | The filename to save the log of web clients accessing Netdata charts. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. |
| collector | `journal` | The filename to save the log of Netdata collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| health | `journal` | The filename to save the log of Netdata health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| daemon | `journal` | The filename to save the log of Netdata daemon. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
| facility | `daemon` | A facility keyword is used to specify the type of system that is logging the message. |
| logs flood protection period | `1m` | Length of period during which the number of errors should not exceed the `errors to trigger flood protection`. |
| logs to trigger flood protection | `1000` | Number of errors written to the log in `errors flood protection period` sec before flood protection is activated. |
| level | `info` | Controls which log messages are logged, with error being the most important. Supported values: `info` and `error`. |
### [environment variables] section options
@ -160,14 +160,14 @@ monitoring](/src/health/README.md).
[Alert notifications](/src/health/notifications/README.md) are configured in `health_alarm_notify.conf`.
| setting | default | info |
|:--------------------------------------:|:------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| enabled | `yes` | Set to `no` to disable all alerts and notifications |
| in memory max health log entries | 1000 | Size of the alert history held in RAM |
| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |
| run at least every | `10s` | Controls how often all alert conditions should be evaluated. |
| postpone alarms during hibernation for | `1m` | Prevents false alerts. May need to be increased if you get alerts during hibernation. |
| health log retention | `5d` | Specifies the history of alert events (in seconds) kept in the Agent's sqlite database. |
| setting | default | info |
|:--------------------------------------:|:------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| enabled | `yes` | Set to `no` to disable all alerts and notifications |
| in memory max health log entries | 1000 | Size of the alert history held in RAM |
| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |
| run at least every | `10s` | Controls how often all alert conditions should be evaluated. |
| postpone alarms during hibernation for | `1m` | Prevents false alerts. May need to be increased if you get alerts during hibernation. |
| health log retention | `5d` | Specifies the history of alert events (in seconds) kept in the Agent's sqlite database. |
| enabled alarms | * | Defines which alerts to load from both user and stock directories. This is a [simple pattern](/src/libnetdata/simple_pattern/README.md) list of alert or template names. Can be used to disable specific alerts. For example, `enabled alarms = !oom_kill *` will load all alerts except `oom_kill`. |
### [web] section options
@ -183,11 +183,11 @@ under `collectors/python.d.plugin` will be disabled.
Additionally, there will be the following options:
| setting | default | info |
|:-------------------------------:|:---------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| enable running new plugins | `yes` | When set to `yes`, Netdata will enable detected plugins, even if they are not configured explicitly. Setting this to `no` will only enable plugins explicitly configured in this file with a `yes` |
| check for new plugins every | 60 | The time in seconds to check for new plugins in the plugins directory. This allows having other applications dynamically creating plugins for Netdata. |
| checks | `no` | This is a debugging plugin for the internal latency |
| setting | default | info |
|:---------------------------:|:-------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| enable running new plugins | `yes` | When set to `yes`, Netdata will enable detected plugins, even if they are not configured explicitly. Setting this to `no` will only enable plugins explicitly configured in this file with a `yes` |
| check for new plugins every | 60 | The time in seconds to check for new plugins in the plugins directory. This allows having other applications dynamically creating plugins for Netdata. |
| checks | `no` | This is a debugging plugin for the internal latency |
### [registry] section options
@ -214,10 +214,10 @@ for all internal Netdata plugins.
External plugins will have only 2 options at `netdata.conf`:
| setting | default | info |
|:---------------:|:--------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| setting | default | info |
|:---------------:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| update every | the value of `[global].update every` setting | The frequency in seconds the plugin should collect values. For more information check the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). |
| command options | - | Additional command line options to pass to the plugin. |
| command options | - | Additional command line options to pass to the plugin. |
External plugins that need additional configuration may support a dedicated file in `/etc/netdata`. Check their
documentation.

108
src/database/CONFIGURATION.md Normal file
View file

@ -0,0 +1,108 @@
# Database Configuration Reference
This document provides details on configuring the Agent's Database. For a deeper understanding of the Database components, see the [Database overview](/src/database/README.md).
## Modes
Use [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf` and set your preferred mode:
```text
[db]
# dbengine, ram, none
mode = dbengine
```
## Tiers
### Retention Settings
> **Important**
>
> In a Parent-Child setup, these settings manage the entire storage space used by the Parent for storing metrics collected both by itself and its Children.
You can fine-tune retention for each tier by setting a time limit or size limit. Setting a limit to 0 disables it. This enables the following retention strategies:
| Setting | Retention Behavior |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Size Limit = 0, Time Limit > 0 | **Time based:** data is stored for a specific duration regardless of disk usage |
| Time Limit = 0, Size Limit > 0 | **Space based:** data is stored with a disk space limit, regardless of time |
| Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first |
You can change these limits using [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf`:
```text
[db]
mode = dbengine
storage tiers = 3
# Tier 0, per second data. Set to 0 for no limit.
dbengine tier 0 retention size = 1GiB
dbengine tier 0 retention time = 14d
# Tier 1, per minute data. Set to 0 for no limit.
dbengine tier 1 retention size = 1GiB
dbengine tier 1 retention time = 3mo
# Tier 2, per hour data. Set to 0 for no limit.
dbengine tier 2 retention size = 1GiB
dbengine tier 2 retention time = 2y
```
### Legacy configuration
<details><summary>v1.99.0 and prior</summary>
Netdata prior to v2 supports the following configuration options in `netdata.conf`.
They have the same defaults as the latest v2, but the unit of each value is given in the option name, not at the value.
```text
storage tiers = 3
# Tier 0, per second data. Set to 0 for no limit.
dbengine tier 0 disk space MB = 1024
dbengine tier 0 retention days = 14
# Tier 1, per minute data. Set to 0 for no limit.
dbengine tier 1 disk space MB = 1024
dbengine tier 1 retention days = 90
# Tier 2, per hour data. Set to 0 for no limit.
dbengine tier 2 disk space MB = 1024
dbengine tier 2 retention days = 730
```
</details>
<details><summary>v1.45.6 and prior</summary>
Netdata versions prior to v1.46.0 relied on disk space-based retention.
**Default Retention Limits**:
| Tier | Resolution | Size Limit |
|:----:|:-------------------:|:----------:|
| 0 | high (per second) | 256 MB |
| 1 | middle (per minute) | 128 MB |
| 2 | low (per hour) | 64 GiB |
You can change these limits in `netdata.conf`:
```text
[db]
mode = dbengine
storage tiers = 3
# Tier 0, per second data
dbengine multihost disk space MB = 256
# Tier 1, per minute data
dbengine tier 1 multihost disk space MB = 1024
# Tier 2, per hour data
dbengine tier 2 multihost disk space MB = 1024
```
</details>
## Cache sizes
There are two cache sizes that can be configured in `netdata.conf` to better optimize the Database:
1. `[db].dbengine page cache size`: controls the size of the cache that keeps metric data on memory.
2. `[db].dbengine extent cache size`: controls the size of the cache that keeps in memory compressed data blocks.
Both of them are dynamically adjusted to use some of the total memory computed above. The configuration in `netdata.conf` allows providing additional memory to them, increasing their caching efficiency.

102
src/database/README.md
View file

@ -1,6 +1,6 @@
# Database
Netdata stores detailed metrics at one-second granularity using its Database engine.
Netdata stores detailed metrics at one-second granularity using its Database engine. This document provides an overview of the various elements of the Database, if you want to configure it, check the [configuration reference page](/src/database/CONFIGURATION.md)
## Modes
@ -17,14 +17,6 @@ The default `dbengine` mode is optimized for:
For resource-constrained environments, particularly Child nodes in Centralization setups, consider using `ram`.
Use [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf` and set your preferred mode:
```text
[db]
# dbengine, ram, none
mode = dbengine
```
## Tiers
Netdata offers a granular approach to data retention, allowing you to manage storage based on both **time** and **disk space**. This provides greater control and helps you optimize storage usage for your specific needs.
@ -43,99 +35,13 @@ Netdata offers a granular approach to data retention, allowing you to manage sto
With these defaults, Netdata requires approximately 4 GiB of storage space (including metadata).
### Retention Settings
> **Important**
>
> In a Parent-Child setup, these settings manage the entire storage space used by the Parent for storing metrics collected both by itself and its Children.
You can fine-tune retention for each tier by setting a time limit or size limit. Setting a limit to 0 disables it. This enables the following retention strategies:
| Setting | Retention Behavior |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Size Limit = 0, Time Limit > 0 | **Time based:** data is stored for a specific duration regardless of disk usage |
| Time Limit = 0, Size Limit > 0 | **Space based:** data is stored with a disk space limit, regardless of time |
| Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first |
You can change these limits using [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf`:
```text
[db]
mode = dbengine
storage tiers = 3
# Tier 0, per second data. Set to 0 for no limit.
dbengine tier 0 retention size = 1GiB
dbengine tier 0 retention time = 14d
# Tier 1, per minute data. Set to 0 for no limit.
dbengine tier 1 retention size = 1GiB
dbengine tier 1 retention time = 3mo
# Tier 2, per hour data. Set to 0 for no limit.
dbengine tier 2 retention size = 1GiB
dbengine tier 2 retention time = 2y
```
### Monitoring Retention Utilization
Netdata provides a visual representation of storage utilization for both the time and space limits across all Tiers under "Netdata" -> "dbengine retention" on the dashboard. This chart shows exactly how your storage space (disk space limits) and time (time limits) are used for metric retention.
### Legacy configuration
<details><summary>v1.99.0 and prior</summary>
Netdata prior to v2 supports the following configuration options in `netdata.conf`.
They have the same defaults as the latest v2, but the unit of each value is given in the option name, not at the value.
```text
storage tiers = 3
# Tier 0, per second data. Set to 0 for no limit.
dbengine tier 0 disk space MB = 1024
dbengine tier 0 retention days = 14
# Tier 1, per minute data. Set to 0 for no limit.
dbengine tier 1 disk space MB = 1024
dbengine tier 1 retention days = 90
# Tier 2, per hour data. Set to 0 for no limit.
dbengine tier 2 disk space MB = 1024
dbengine tier 2 retention days = 730
```
</details>
<details><summary>v1.45.6 and prior</summary>
Netdata versions prior to v1.46.0 relied on disk space-based retention.
**Default Retention Limits**:
| Tier | Resolution | Size Limit |
|:----:|:-------------------:|:----------:|
| 0 | high (per second) | 256 MB |
| 1 | middle (per minute) | 128 MB |
| 2 | low (per hour) | 64 GiB |
You can change these limits in `netdata.conf`:
```text
[db]
mode = dbengine
storage tiers = 3
# Tier 0, per second data
dbengine multihost disk space MB = 256
# Tier 1, per minute data
dbengine tier 1 multihost disk space MB = 1024
# Tier 2, per hour data
dbengine tier 2 multihost disk space MB = 1024
```
</details>
## Cache sizes
There are two cache sizes that can be configured in `netdata.conf` to better optimize the Database:
There are two cache sizes that can be used to optimize the Database:
1. `[db].dbengine page cache size`: this is the main cache that keeps metrics data into memory. When data is not found in it, the extent cache is consulted, and if not found in that too, they are loaded from the disk.
2. `[db].dbengine extent cache size`: this is the compressed extent cache. It keeps in memory compressed data blocks, as they appear on disk, to avoid reading them again. Data found in the extent cache but not in the main cache have to be uncompressed to be queried.
Both of them are dynamically adjusted to use some of the total memory computed above. The configuration in `netdata.conf` allows providing additional memory to them, increasing their caching efficiency.
1. **Page cache size**: The main cache that keeps metrics data into memory. When data is not found in it, the extent cache is consulted, and if not found in that too, they are loaded from the disk.
2. **Extent cache size**: The compressed extent cache. It keeps in memory compressed data blocks, as they appear on disk, to avoid reading them again. Data found in the extent cache but not in the main cache have to be uncompressed to be queried.

4
src/database/engine/README.md
View file

@ -115,7 +115,7 @@ When the Netdata Agent starts, during the first data collection of each metric,
data from lower tiers, so that the aggregation they provide will be accurate.
Configuring how the number of tiers and the disk space allocated to each tier is how you can
[change how long netdata stores metrics](/src/database/README.md#tiers).
[change how long netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
### Data loss
@ -127,7 +127,7 @@ multiple other Netdata Agents.
## Memory requirements and retention
See [change how long netdata stores metrics](/src/database/README.md#tiers)
See [change how long netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
#### Exceptions

2
src/health/guides/dbengine/10min_dbengine_global_flushing_errors.md
View file

@ -9,5 +9,5 @@ faster disks. This alert is triggered in critical state when the number deleted
### Useful resources
[Read more about Netdata DB engine](/src/database/README.md/engine)
[Read more about Netdata DB engine](/src/database/engine/README.md)

2
src/health/guides/dbengine/10min_dbengine_global_flushing_warnings.md
View file

@ -11,5 +11,5 @@ This alert is triggered in warn state when the number of `dbengine` dirty pages
### Useful resources
[Read more about Netdata DB engine](/src/database/README.md/engine)
[Read more about Netdata DB engine](/src/database/engine/README.md)

2
src/health/guides/dbengine/10min_dbengine_global_fs_errors.md
View file

@ -10,5 +10,5 @@ This alert is triggered in warning state when the number of filesystem errors is
### Useful resources
[Read more about Netdata DB engine](/src/database/README.md/engine)
[Read more about Netdata DB engine](/src/database/engine/README.md)

2
src/health/guides/dbengine/10min_dbengine_global_io_errors.md
View file

@ -10,5 +10,5 @@ This alert is triggered in critical state when the number of IO errors is greate
### Useful resources
[Read more about Netdata DB engine](/src/database/README.md/engine)
[Read more about Netdata DB engine](/src/database/engine/README.md)