0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-14 01:29:11 +00:00
netdata_netdata/netdata-installer.sh
Dimitris P 4e512411ec
Logs Management ()
This PR adds the logs-management external plugin. 
See the included README for an extensive list of features. 
-------------------------------------------------------------------------------------

* Add proper status return in JSON response of functions

* Add column info to functions

* Escape special characters when returning JSON response

* Add proper functions help and defaults. Fix help not working

* Add 'logs_management_meta' object in functions results

* Fix compiler warnings

* Replace tabs with 3 spaces in web_client_api_request_v1_logsmanagement_sources()

* Add 'sources' in functions to display list of log sources

* Update functions column values for logs

* Update chart titles and remove '/s' from units

* Add support for compound queries in circular buffers

* Refactor circ_buff_search() to get rid of circ_buff_search_compound()

* Fix incorrect docker events nano timestamp padding

* Fixed botched rebasing

* Replace get_unix_time_ms() with now_realtime_msec()

* Remove binary generation from Fluent-Bit lib build

* Fix compiler warnings due to new timestamp type

* Remove STDIN and STDOUT support from Fluent-Bit library

* Initial support for FLB_KMSG kernel logs collection

* Add kernel logs charts

* Add kernel logs subsystem and device charts

* Skip collection of pre-existing logs in kmsg ring buffer

* Add example of custom kmsg charts

* Add extra initialization error logs

* Fix bug of Docker Events collector failure disabling whole logs management engine

* Remove reduntant FLB output code

* Remove some obsolete TODO comments

* Remove some commented out error/debug prints

* Disable some Fluent-Bit config options not required

* Make circular buffer spare items option configurable

* Add DB mode configuration option

* Replace p_file_infos_arr->data[i] with p_file_info in db_api.c

* Remove db_loop due to all function calls being synchronous

* Add initial README.md

* Add DB mode = none changes

* Add a simple webpage to visualize log query results

* Add support for source selection to logs_query.html

* Add option to query multiple log sources

* Mark non-queryable sources as such in logs_query.html

* Add option to use either GET or functions request in logs_query.html

* Install logs_query.html when running stress tests

* Update README.md requirements

* Change installer behavior to build logs management by default

* Disable logs management at runtime by default

* Add global db mode configuration in 'logs management' config section

* Split logsmanagement.conf into required & optional sections

* Remove --enable-logsmanagement from stress test script

* Add global config option for 'circular buffer max size MiB'

* Add global config option for 'circular buffer drop logs if full'

* Update 'General Configuration' in README.md

* Add global config option for remaining optional settings

* Add systemd collector requirements to TOC

* README: Convert general configuration to table

* README: Fix previous botched commit

* Enable logs management by default when building for stress testing

* Move logging to collector.log from error.log

* Fix contenttype compilation errors

* Move logging to collector.log in plugin_logsmanagement.c

* Rename 'rows' to 'records' in charts

* Add Netdata error.log parsing

* Add more dashboard descriptions

* Sanitize chart ids

* Attempt to fix failing CI

* Update README.md

* Update README.md

* Another attempt to fix CI failures

* Fix undefined reference to 'uv_sleep' on certain platforms

* Support FLB forward input and FLB output plugins.

Squashed commit of the following:

commit 55e2bf4fb34a2e02ffd0b280790197310a5299f3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 16:41:09 2023 +0300

    Remove error.log from stock config

commit bbdc62c2c9727359bc3c8ef8c33ee734d0039be7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 16:37:48 2023 +0300

    Add cleanup of Fluent Bit outputs in p_file_info_destroy()

commit 09b0aa4268ec1ccef160c99c5d5f31b6388edd28
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 14:34:17 2023 +0300

    Some code and config cleanup

commit 030d074667d5ee2cad10f85cd836ca90e29346ad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 13:04:08 2023 +0300

    Enable additional Fluent Bit output plugins for shared library

commit 490aa5d44caa38042521d24c6b886b8b4a59a73c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 01:33:19 2023 +0300

    Add initialization of Fluent Bit user-configured outputs

commit c96e9fe9cea96549aa5eae09d0deeb130da02793
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Apr 4 23:13:16 2023 +0100

    Complete read of parameters for FLB outputs config

commit 00988897f9b86d1ecc5c141b19df7ad7d74f7e96
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 19:43:31 2023 +0100

    Update README.md

commit 6deea5399c2707942aeaa51408f999ca45dfd351
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 16:02:28 2023 +0100

    Refactor Syslog_parser_config_t and add Flb_socket_config_t

commit 7bf998a4c298bbd489ef735c56a6e85a137772c9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 14:19:57 2023 +0100

    Update README.md

commit c353d194b12c54f134936072ebaded0424d73cc0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Mar 31 14:52:57 2023 +0100

    Update README.md

commit 6be726eaff3738ba7884de799aa52949833af65a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Mar 31 13:06:29 2023 +0100

    Update README. Fix docker_events streaming

commit 6aabfb0f1ef0529a7a0ecbaf940bc0952bf42518
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 21:27:45 2023 +0100

    Fix stuck in infinite loop bug for FLB_GENERIC, FLB_WEB_LOG and FLB_SERIAL remote log sources

commit eea6346b708cc7a5ce6e2249366870f4924eabae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 21:04:12 2023 +0100

    Remove callback that searches for streamed p_file_info match

commit bc9c5a523b0b0ab5588adbff391a43ba8d9a0cdf
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 15:51:39 2023 +0100

    Basic streaming works

commit 4c80f59f0214bc07895f0b2edca47cb02bc06420
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Mar 28 22:05:22 2023 +0100

    WIP

commit eeb37a71b602fb0738fe8077ccddc0a8ce632304
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Mar 27 22:52:09 2023 +0100

    Add generic forward streaming input

commit 1459b91847c80c4d97de96b75b00771039458ad6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 23 18:50:14 2023 +0000

    FLB_FORWARD: WIP

* Add number of logs per item in DB and in queries response

* Fix wrong number of lines stored in DB for web logs

* Refactor number of logs parsers and charts code

* Add option to toggle number of collected logs metrics and charts

* Disable kmsg log collector by default

* Fix logs_query.html to work with any server ip

* Fix regressed wrong number of web log lines bug

* Change query quota type from size_t to long long

* Update alpine version when searching for fts-dev requirements

* Update query results to return both requested and actual quota

* Fix bug of circ buffs not being read if head == read but not empty

* Squashed commit of the following:

commit 34edb316a737f3edcffcf8fa88a3801599011495
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu May 4 20:02:36 2023 +0100

    Comment out some debug prints

commit 51b9b87a88516186530f5b4b65f785b543fefe8c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 28 19:21:54 2023 +0100

    Fix wrong filenames in BLOBS_TABLE after rotation

commit 6055fc2893b48661af324f20ee61511a40abbc02
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 28 12:22:04 2023 +0100

    Add chart showing number of circular buffer items

commit 0bb5210b0847f4b7596f633ec96fc10aa8ebc791
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Apr 25 16:47:29 2023 +0300

    Various fixes.

    Fix num_lines calculation.
    Add debug prints for circ buffers.
    Remove circ buff spare items option.
    Fix calculation of circ buff memory consumption.
    Add buff_realloc_rwlock for db_mode = none case.
    Fix circ buff read to be done correctly when buff is full.

commit f494af8c95be84404c7d854494d26da3bcbd3ad7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 21 16:03:50 2023 +0300

    Fix freez() on non-malloced address

commit cce6d09e9cf9b847aface7309643e2c0a6041390
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 21 15:41:25 2023 +0300

    Add option to dynamically expand circ buffs when full

* Use log timestamps when possible, instead of collection timestamps.
Also, add config options for Fluent Bit engine and remove tail_plugin.

Squashed commit of the following:

commit b16a02eb6e3a90565c90e0a274b87b123e7b18e5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 16 19:38:57 2023 +0100

    Add Fluent Bit service config options to netdata.conf. Add monitoring of new log file fluentbit.log

commit ab77c286294548ea62a3879ac0f8b8bbfe6a0687
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 21:25:17 2023 +0100

    Remove some debug prints

commit 46d64ad2434e69b1d20720297aec1ddb869e1f84
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 21:19:32 2023 +0100

    Fix null values in charts

commit 8ec96821d6a882f28cbd19244ebdfc86c807d2f4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 17:43:04 2023 +0100

    Update README.md to reflect log timestamp changes

commit 079a91858cf9db2f74711581235bc17eb97c7dad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:23:14 2023 +0100

    Add configurable option for 'update timeout'

commit 72b5e2505d4657fcbb5ccb6eeee00c45eb0b51ff
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:05:08 2023 +0100

    Revert logsmanagement.conf to logs-manag-master one

commit 70d0ea6f8d272fff318aa3095d90a78dcc3411a7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:02:00 2023 +0100

    Fix bug of circ buff items not marked as done

commit 5716420838771edb7842be4669bf96235b15cf71
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:01:41 2023 +0100

    Fix do_custom_charts_update() to work for all log sources

commit a8def8f53fd25c3efa56ef27e267df3261913a8e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 18:20:20 2023 +0100

    Remove GENERIC and WEB_LOG cases. Remove tail_plugin.c/h. Remove generic_parser().

commit 1cf05966e33491dbeb9b877f18d1ea8643aabeba
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 16:54:59 2023 +0100

    Fix FLB_GENERIC and FLB_SERIAL to work with new timestamp logic

commit df3266810531f1af5f99b666fbf44c503b304a39
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 14:55:04 2023 +0100

    Get rid of *_collect() functions and restructure plugin_logsmanagement workers

commit 3eee069842f3257fffe60dacfc274363bc43491c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 14:28:33 2023 +0100

    Fix wrong order of #define _XOPEN_SOURCE 700 in parser.c

commit 941aa80cb55d5a7d6fe8926da930d9803be52312
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu May 11 22:27:39 2023 +0100

    Update plugin_logsmanagement_web_log to use new timestamp logic and to support delayed logs. Refactor req_method metrics code.

commit 427a7d0e2366d43cb5eab7daa1ed82dfc3bc8bc8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 20:26:08 2023 +0100

    Update plugin_logsmanagement_kernel to use new timestamp logic and to support delayed charts

commit a7e95a6d3e5c8b62531b671fd3ec7b8a3196b5bb
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 15:22:14 2023 +0100

    Update plugin_logsmanagement_systemd to use new timestamp logic and support delayed charts

commit 48237ac2ce49c82abdf2783952fd9f0ef05d72e1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 13:29:44 2023 +0100

    Refactor number of collected logs chart update code

commit a933c8fcae61c23fa0ec6d0074526ac5d243cf16
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 8 22:11:19 2023 +0100

    Update plugin_logsmanagement_docker_ev to use new timestamp logic and support delayed charts

commit 5d8db057155affd5cb721399a639d75a81801b7f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 5 15:18:06 2023 +0100

    Change some Fluent Bit collectors to use log timestamps instead of collection timestamps

* Remove some unused defines and typedefs

* Improve flb_init()

* Update file-level doxygen. Add SPDX license declaration.

* Better handling of termination of Fluent Bit

* Better handling of DB errors. Various fixes.

Squashed commit of the following:

commit f55feea1274c3857eda1e9d899743db6e3eb5bf5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Jun 6 13:28:00 2023 +0100

    Fix web log parsing in case of lines terminated by \r

commit 9e05758a4ecfac57a0db14757cff9536deda51d8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 20:42:05 2023 +0100

    Fix warnings due to -Wformat-truncation=2

commit 63477666fa42446d74693aae542580d4e1e81f03
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 16:48:45 2023 +0100

    Autodiscovery of Netdata error.log based on netdata_configured_log_dir

commit cab5e6d6061f4259172bbf72666e8b4a3a35dd66
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 16:24:39 2023 +0100

    Replace Forward config default string literals with macros

commit 4213398031dbb53afbc943d76bf7df202d12bf6f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 15:56:29 2023 +0100

    Proper cleanup of flb_lib_out_cb *callback in case of error

commit f76fd7cc7bc2d0241e4d3517f61ae192d4246300
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 15:36:07 2023 +0100

    Proper termination of Forward input and respective log sources in case of error

commit 3739fd96c29e13298eb3a6e943a63172cdf39d5f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 21:19:56 2023 +0100

    Merge db_search() and db_search_compound()

commit fcface90cb0a6df3c3a2de5e1908b1b3467dd579
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 19:17:26 2023 +0100

    Proper error handling in db_search() and db_search_compound(). Refactor the code too.

commit c10667ebee2510a1af77114b3a7e18a0054b5dae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 14:23:34 2023 +0100

    Update DB mode and dir when switching to db_mode_none

commit d37d4c3d79333bb9fa430650c13ad625458620e8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 12:56:13 2023 +0100

    Fix flb_stop() SIGSEGV

commit 892e231c68775ff1a1f052d292d26384f1ef54b1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 30 21:14:58 2023 +0100

    Switch to db_writer_db_mode_none if db_writer_db_mode_full encounters error

commit f7a0c2135ff61d3a5b0460ec5964eb6bce164bd6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 29 21:41:21 2023 +0100

    Complete error handling changes to db_init(). Add some const type qualifiers. Refactor some code for readability

commit 13dbeac936d22958394cb1aaec394384f5a93fdd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 29 17:14:17 2023 +0100

    More error handling changes in db_init(). Change some global default settings if stress testing.

commit eb0691c269cd09054190bf0ee9c4e9247b4a2548
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 26 23:29:12 2023 +0100

    Better handling of db writer threads errors. Add db timings charts

* Fix mystrsep() replaced by strsep_skip_consecutive_separators()

* Fix older GCC failure due to label before declaration

* Fix failed builds when using libuv <= v1.19

* Fix some Codacy warnings

* Fix warning: implicit declaration of function ‘strsep’

* Use USEC_PER_SEC instead of 1000000ULL

* Use UUID_STR_LEN instead of GUID_LEN + 1

* Combine multiple 'ln -sf' Docker instructions to one

* Update README with systemd development libraries requirement

* Comment out mallocz() success checkes in parser_csv()

* Fix shellcheck warnings

* Remove asserts for empty SYSLOG_IDENTIFIER or PID

* Fix FreeBSD failing builds

* Fix some more shellcheck warnings

* Update Alpine fts-dev required packages

* First changes to use web log timestamp for correct metrics timings

* Initial work to add test_parse_web_log_line() unit test

* Complete test_parse_web_log_line() tests

* Improve parse_web_log_line() for better handling of \n, \r, double quotes etc.

* Fix 'Invalid TIME error when timezone sign is negative

* Add more logs to compression unit test case

* Misc read_last_line() improvements

* Fix failing test_auto_detect_web_log_parser_config() when test case terminated without '\n'

* Remove unused preprocessor macro

* Factor out setup of parse_config_expected_num_fields

* Add test for count_fields()

* Add unit test for read_last_line()

* Fix a read_last_line() bug

* Remove PLUGIN[logsmanagement] static thread and update charts synchronously, right before data buffering

* Fix web log parser potential SIGSEGV

* Fix web log metrics bug where they could show delayed by 1 collection interval

* WIP: Add multiline support to kmsg logs and fix metric timings

* Fix kmsg subsystem and device parsing and metrics

* Add option 'use log timestamp' to select between log timestamps or collection timestamps

* Add 'Getting Started' docs section

* Move logs management functions code to separate source files

* Add 'Nginx access.log' chart description

* Remove logsmanagement.plugin source files

* Fix some memory leaks

* Improve cleanup of logsmanagement_main()

* Fix a potential memory leak of fwd_input_out_cb

* Better termination and cleanup of main_loop and its handles

* Fix main_db_dir access() check bug

* Avoid uv_walk() SIGSEGV

* Remove main_db_dir access() check

* Better termination and cleanup of DB code

* Remove flb_socket_config_destroy() that could cause a segmentation fault

* Disable unique client IPs - all-time chart by default

* Update README.md

* Fix debug() -> netdata_log_debug()

* Fix read_last_line()

* Fix timestamp sign adjustment and wrong unit tests

* Change WEB_CLIENT_ACL_DASHBOARD to WEB_CLIENT_ACL_DASHBOARD_ACLK_WEBRTC

* Do not parse web log timestamps if 'use_log_timestamp = no'

* Add Logs Management back into buildinfo.c

* Update README.md

* Do not build Fluent Bit executable binary

* Change logs rate chart to RRDSET_TYPE_LINE

* Add kludge to prevent metrics breaking due to out of order logs

* Fix wrong flb_tmp_buff_cpy_timer expiration

* Refactor initialization of input plugin for local log sources.

* Rename FLB_GENERIC collector to FLB_TAIL.

* Switch 'Netdata fluentbit.log' to disabled by default

* Add 'use inotify' configuration option

* Update  in README.md

* Add docker event actions metrics

* Update README.md to include event action chart

* Remove commented out PLUGIN[logsmanagement] code block

* Fix some warnings

* Add documentation for outgoing log streaming and exporting

* Fix some code block formatting in README.md

* Refactor code related to error status of log query results and add new invalid timestamp case

* Reduce query mem allocs and fix end timestamp == 0 bug

* Add support for duplicate timestamps in db_search()

* Add support for duplicate timestamps in circ_buff_search()

* Fix docker events contexts

* Various query fixes prior to reverse order search.

- Add reverse qsort() function in circ buffers.
- Fix issues to properly support of duplicate timestamps.
- Separate requested from actual timestamps in query parameters.
- Rename results buffer variable name to be consistent between DB and
  buffers.
- Remove default start and end timestamp from functions.
- Improve handling of invalid quotas provided by users.
- Rename 'until' timestamp name to 'to'.
- Increase default quota to 10MB from 1MB.
- Allow start timestamp to be > than end timestamp.

* Complete descending timestamp search for circular buffers

* Complete descending timestamp search for DB

* Remove MEASURE_QUERY_TIME code block

* Complete descending timestamp search when data resides in both DB and circular buffers

* Use pointer instead of copying res_hdr in query results

* Refactor web log timezone parsing to use static memory allocation

* Add stats for CPU user & system time per MiB of query results

* Micro-optimization to slightly speed up queries

* More micro-optimizations and some code cleanup

* Remove LOGS_QUERY_DATA_FORMAT_NEW_LINE option

* Escape iscntrl() chars at collection rather at query

* Reduce number of buffer_strcat() calls

* Complete descending timestamp order queries for web_api_v1

* Complete descending timestamp order queries for functions

* Fix functions query timings to match web_api_v1 ones

* Add MQTT message collector

Squashed commit of the following:

commit dbe515372ee04880b1841ef7800abe9385b12e1c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 15:18:46 2023 +0100

    Update README.md with MQTT information

commit c0b5dbcb7cdef8c6fbd5e72e7bdd08957a0fd3de
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 14:59:36 2023 +0100

    Tidy up before merge

commit 9a69c4f17eac858532918a8f850a770b12710f80
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 12:54:33 2023 +0100

    Fix issue with duplicate Log_Source_Path in DB, introduced in commit e417af3

commit 48213e9713216d62fca8a5bc1bbc41a3883fdc14
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sat Aug 19 05:05:36 2023 +0100

    WIP

commit e417af3b947f11bd61e3255306bc95953863998d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 17 18:03:39 2023 +0100

    Update functions logsmanagement help output

* Inhibit Fluent Bit build warnings

* Fix missing allow_subpaths value in api_commands_v1[].

* Fix missing HTTP_RESP_BACKEND_FETCH_FAILED error

* Fix an enum print warning

* Remove systemd-devel requirement from README and fix codacy warnings

* Update Alpine versions for musl-fts-dev

* Update Fluent Bit to v2.1.8

Squashed commit of the following:

commit faf6fc4b7919cc2611124acc67cb1973ce705530
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Aug 25 17:13:30 2023 +0100

    Fix wrong default CORE_STACK_SIZE on Alpine

commit a810238fe7830ce626f6d57245d68035b29723f7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Aug 25 00:40:02 2023 +0100

    Update Fluent Bit patches for musl

commit 8bed3b611dba94a053e22c2b4aa1d46f7787d9b4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 24 21:54:38 2023 +0100

    Fix an edge case crash when web log method is '-'

commit b29b48ea230363142697f9749508cd926e18ee19
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 24 16:26:13 2023 +0100

    Disable FLB_OUT_CALYPTIA to fix Alpine dlsym() error

commit eabe0d0523ffe98ff881675c21b0763a49c05f16
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 21:25:54 2023 +0100

    Add 'use inotify = no' troubleshooting Q&A in README

commit 7f7ae85bdb0def63b4fc05ab88f6572db948e0e7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 18:06:36 2023 +0100

    Update README.md links to latest version

commit 610c5ac7b920d4a1dfe364ad48f1ca14a0acc346
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 16:23:30 2023 +0100

    Update flb_parser_create() definition

commit f99608ff524b6f3462264e626a1073f9c2fdfdf5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 16:23:04 2023 +0100

    Add new config.cmake options

commit 446b0d564626055a0a125f525d0bd3754184b830
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 12:21:25 2023 +0100

    Update Fluent Bit submodule to v2.1.8

* Add logs_management_unittest() to CI 'unittest'

* Remove obsolete query testing files

* Patch Fluent Bit log format to match netdata's format

* Update README with instructions on how to monitor Podman events logs

* Fix core dump in case of flb_lib_path dlopen()

* Fix some potential compiler warnings

* Fix queries crash if logs manag engine not running

* Much faster termination of LOGS MANAGEMENT

* Add facets support and other minor fixes.

logsmanagement_function_execute_cb() is replaced by
logsmanagement_function_facets() which adds facets support to logs
management queries.

Internal query results header now includes additional fields
(log_source, log_type, basename, filename, chartname), that are used as facets.

Queries now support timeout as a query parameter.

A web log timestamp bug is fixed (by using timegm() instead of mktime().

web_api_v1 logsmanagement API is only available in debugging now.

Squashed commit of the following:

commit 32cf0381283029d793ec3af30d96e6cd77ee9149
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 19 16:21:32 2023 +0300

    Tidy up

commit f956b5846451c6b955a150b5d071947037e935f0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 19 13:30:54 2023 +0300

    Add more accepted params. Add data_only option. Add if_modified_since option.

commit 588c2425c60dcdd14349b7b346467dba32fda4e9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 18 18:39:50 2023 +0300

    Add timeout to queries

commit da0f055fc47a36d9af4b7cc4cefb8eb6630e36d9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 19:17:16 2023 +0300

    Fix histogram

commit 7149890974e0d26420ec1c5cfe1023801dc973fa
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 17:58:52 2023 +0300

    Add keyword query using simple patterns and fix descending timestamp values

commit 0bd068c5a76e694b876027e9fa5af6f333ab825b
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 13:54:05 2023 +0300

    Add basename, filename, chartname as facets

commit 023c2b5f758b2479a0e48da575cd59500a1373b6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 13:26:06 2023 +0300

    Add info and sources functions options

commit ab4d555b7d445f7291af474847bd9177d3726a76
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 12:54:37 2023 +0300

    Fix facet id filter

commit a69c9e2732f5a6da1764bb57d1c06d8d65979225
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 12:07:13 2023 +0300

    WIP: Add facet id filters

commit 3c02b5de81fa8a20c712863c347539a52936ddd8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:19:17 2023 +0300

    Add log source and log type to circ buff query results header

commit 8ca98672c4911c126e50f3cbdd69ac363abdb33d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:18:13 2023 +0300

    Fix logsmanagement facet function after master rebasing

commit 3f1517ad56cda2473a279a8d130bec869fc2cbb8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:14:25 2023 +0300

    Restrict /logsmanagement to ACL_DEV_OPEN_ACCESS only

commit 8ca98d69b08d006c682997268d5d2523ddde6be0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 14:40:22 2023 +0300

    Fix incorrectly parsed timestamps due to DST

commit f9b0848037b29c7fcc46da951ca5cd9eb129066f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 11 13:42:18 2023 +0300

    Add logs_management_meta object to facet query results

commit babc978f6c97107aaf8b337d8d31735d61761b6a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 11 13:03:52 2023 +0300

    Query all sources if no arguments provided

commit 486d56de87af56aae6c0dc5d165341418222ce8b
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 7 18:38:04 2023 +0300

    Add log_source and log_type (only for DB logs) as facets. Add relative time support

commit b564c12843d355c4da6436af358d5f352cb58bfe
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 7 13:47:20 2023 +0300

    Working facet with descending timestamps

commit 68c6a5c64e8425cf28ec16adfb0c50289caa82a9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Wed Sep 6 01:55:51 2023 +0300

    WIP

* Fix linking errors

* Convert logs management to external plugin.

Squashed commit of the following:

commit 16da6ba70ebde0859aed734087f04af497ce3a77
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:44:12 2023 +0100

    Use higher value of update every from netdata.conf or logsmanagement.d.conf

commit 88cc3497c403e07686e9fc0876ebb0c610a1404c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:43:02 2023 +0100

    Tidy up

commit c3fca57aac169842637d210269519612b1a91e28
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:02:04 2023 +0100

    Use external update_every from agent, if available

commit f7470708ba82495b03297cdf8962a09b16617ddd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 17:40:46 2023 +0100

    Re-enable debug logs

commit b34f5ac6a2228361ab41df7d7e5e713f724368c0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:49:20 2023 +0100

    Remove old API calls from web_api_v1.c/h

commit 7fbc1e699a7785ec837233b9562199ee6c7684da
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:32:04 2023 +0100

    Add proper termination of stats charts thread

commit 4c0fc05c8b14593bd7a0aa68f75a8a1205e04db4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:31:36 2023 +0100

    Add tests for logsmanag_config functions

commit 4dfdacb55707ab46ed6c2d5ce538ac012574b27e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 22:01:19 2023 +0100

    Remove unused headers from logsmanagement.c

commit b324ef396207c5c32e40ea9ad462bf374470b230
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 21:56:26 2023 +0100

    Remove inline from get_X_dir() functions

commit e9656e8121b66cd7ef8b5daaa5d27a134427aa35
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 21:50:32 2023 +0100

    Proper termination when a signal is received

commit b09eec147bdeffae7b268b6335f6ba89f084e050
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 20:12:13 2023 +0100

    Refactor logs management config code in separate source files

commit 014b46a5008fd296f7d25854079c518d018abdec
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 14:54:47 2023 +0100

    Fix p_file_info_destroy() crash

commit e0bdfd182513bb8d5d4b4b5b8a4cc248ccf2d64e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 14:18:27 2023 +0100

    Code refactoring and cleanup

commit 6a61cb6e2fd3a535db150b01d9450f44b3e27b30
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Oct 20 14:08:43 2023 +0100

    Fix 'source:all' queries

commit 45b516aaf819ac142353e323209b7d01e487393f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 19 21:51:05 2023 +0100

    Working 'source:...' queries and regular data queries (but not 'source:all')

commit 8064b0ee71c63da9803f79424802f860e96326e5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 19 15:34:23 2023 +0100

    Fix issue due to p_file_info_destroy()

commit a0aacc9cd00cea60218c9bfd2b9f164918a1e3de
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 17 22:06:34 2023 +0100

    Work on facet API changes

commit 480584ff9040c07e996b14efb4d21970a347633f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 16 21:43:06 2023 +0100

    Add stats charts, running as separate thread

commit 34d582dbe4bf2d8d048afab41681e337705bc611
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 16 16:24:02 2023 +0100

    Add SSL cipher charts

commit ced27ee4e2c981d291f498244f2eef2556a074fb
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sun Oct 15 21:33:29 2023 +0100

    Add Response code family, Response code, Response code type, SSL protocol charts

commit 40c4a1d91892d49b1e4e18a1c3c43258ded4014d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sat Oct 14 00:48:48 2023 +0100

    Add more web log charts

commit 890ed3ff97153dd18d15df2d1b57a181bc498ca8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Oct 13 22:14:11 2023 +0100

    Add web log vhosts and ports charts

commit 84733b6b1d353aff70687603019443610a8500c3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 12 21:40:16 2023 +0100

    Add systemd charts

commit 14673501e8f48560956f53d5b670bbe801b8f2ae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Wed Oct 11 00:28:43 2023 +0100

    Add MQTT charts

commit 366eb63b0a27dde6f0f8ba65120f34c18c1b21fd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 21:46:19 2023 +0100

    Complete kmsg changes. Reduce mem usage. Fix a dictionary key size bug

commit 3d0216365a526ffbc9ce13a20c45447bfccb47d9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 19:18:41 2023 +0100

    Add kmsg Subsystem charts

commit e61af4bb130a5cf5a5a78133f1e44b2b4c457b24
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 16:21:29 2023 +0100

    Fix bug of wrong kmsg timestamps in case of use_log_timestamp == 0

commit 03d22e0b26bddf249aab431a4f977bbd5cde98ca
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 16:20:47 2023 +0100

    Add kmsg charts, except for Subsystem and Device

commit f60b0787537a21ed3c4cea5101fcddc50f3bc55a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 13:12:13 2023 +0100

    Initialise all docker events chart dimensions at startup

commit 5d873d3439abaf3768530cb5b72c6b4ef6565353
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 00:53:35 2023 +0100

    WIP: Add Docker events logs

commit 2cc3d6d98f58fc3ab67a8da3014210b14d0926a1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 9 18:52:27 2023 +0100

    Use macros for num_of_logs_charts and custom_charts functions

commit fbd48ad3c9af674601238990d74192427475f2e3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 9 18:26:17 2023 +0100

    Refactor custom charts code for clarity and speed

commit a31d80b5dc91161c0d74b10d00bc4fd1e6da7965
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 23:58:27 2023 +0100

    Add first working iteration of custom charts

commit b1e4ab8a460f4b4c3e2804e2f775787d21fbee45
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 23:57:27 2023 +0100

    Add more custom charts for Netdata error.log

commit f1b7605e564da3e297942f073593cdd4c21f88e1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 20:39:40 2023 +0100

    Convert collected_logs_* chart updates to macros

commit 1459bc2b8bcd5ba21e024b10a8a5101048938f71
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 19:11:54 2023 +0100

    Use rrdset_timed_done() instead of duration_since_last_update for correct chart timings

commit 876854c6ee7586a3eb9fdbf795bcc17a5fd1e6ad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 3 21:53:14 2023 +0100

    Fix some bugs in chart updates

commit ae87508485499984bcb9b72bbc7d249c4168b380
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 3 21:32:55 2023 +0100

    Functioning generic_chart_init() and generic_chart_update()

commit 982a9c4108dbea9571c785b5ff8a9d1e5472066c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 23:32:52 2023 +0100

    Add support for multiple .conf files. Add stock examples.

commit 8e8abd0731227eb3fb3c6bcd811349575160799e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 17:38:30 2023 +0100

    Add support for logsmanagement.d/default.conf

commit 1bf0732217b1d9e9959e1507ea96fc2c92ffb2ff
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 14:31:03 2023 +0100

    Add capabilities. Fix paths in logsmanagement.d.conf

commit a849d5b405bb4e5d770726fe99413a4efa7df274
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 26 23:06:31 2023 +0100

    Change logs_manag_config_load()

commit b0d1783b996286cd87e0832bfb74c29a845d61fc
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 26 15:35:30 2023 +0100

    Working unit tests and argument parsing

commit 6da1b4267a4d58d3a7cbcca9507afe8158a2e324
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Sep 22 00:32:47 2023 +0300

    Build logs-management.plugin successfully

commit 9e30efe0422e4941f99cc66998d9f42e00a24676
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 21 16:13:21 2023 +0300

    Fix print format specifier in web_client_api_request_v1_logsmanagement()

* Modify CODEOWNERS

* Update README.md

Fix indentation

* Change snprintfz() to stncpyz() in circ_buff_search(). Change remaining 'chart_name' to 'chartname'.

* Replace SQLite version function with macro

* Fix some codacy warnings

* Update README.md

* Update Fluent Bit to v2.1.10

* Remove some comments

* Fix Fluent Bit shared library linking for armv7l and FreeBSD

* Remove compression source files

* Add prefix to rrd_api.h functions

* Add more unit tests

* Fix kmsg capabilities

* Separate kmsg and systemd default paths

* Fix some memory leaks and better termination of DB

* Add iterative queries if quota is exceeded

* Fix centos7 builds

* Fix issue where SYSTEMD timestamps are not parsed

* Fix logs management packaging.

* Fix typo in DEB control file.

* Fix indentation and missing new line at EOF

* Clean up functions and update help

* Fix 400 error when no queryable sources are available

* Fix if_modified_since. Add FACET_MAX_VALUE_LENGTH

* Add delta parameter and use anchor points in queries

* Fix CodeQL  warning

* Fix packaging issues.

* Fix postinstall script for DEB packages.

* Improve plugin shutdown speed

* Fix docker events chart grouping

* Fix functions evloop threads not terminating upon shutdown

* Fix coverity issues

* Fix logging

* Replace 'Netdata error.log' with 'Netdata daemon.log' in 'default.conf'

* Remove 'enabled = yes/no' config in logsmanagement.d.conf

* Remove 'enabled = X' unused config from logsmanagement.d.conf

---------

Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud>
2023-11-27 16:55:14 +00:00

2013 lines
74 KiB
Bash
Executable file

#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-or-later
# Next unused error code: I0012
export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
uniquepath() {
path=""
tmp="$(mktemp)"
(echo "${PATH}" | tr ":" "\n") > "$tmp"
while read -r REPLY;
do
if echo "${path}" | grep -v "(^|:)${REPLY}(:|$)"; then
[ -n "${path}" ] && path="${path}:"
path="${path}${REPLY}"
fi
done < "$tmp"
rm "$tmp"
[ -n "${path}" ]
export PATH="${path%:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin}"
} > /dev/null
uniquepath
PROGRAM="$0"
NETDATA_SOURCE_DIR="$(pwd)"
INSTALLER_DIR="$(dirname "${PROGRAM}")"
if [ "${NETDATA_SOURCE_DIR}" != "${INSTALLER_DIR}" ] && [ "${INSTALLER_DIR}" != "." ]; then
echo >&2 "Warning: you are currently in '${NETDATA_SOURCE_DIR}' but the installer is in '${INSTALLER_DIR}'."
fi
# -----------------------------------------------------------------------------
# reload the user profile
# shellcheck source=/dev/null
[ -f /etc/profile ] && . /etc/profile
# make sure /etc/profile does not change our current directory
cd "${NETDATA_SOURCE_DIR}" || exit 1
# -----------------------------------------------------------------------------
# load the required functions
if [ -f "${INSTALLER_DIR}/packaging/installer/functions.sh" ]; then
# shellcheck source=packaging/installer/functions.sh
. "${INSTALLER_DIR}/packaging/installer/functions.sh" || exit 1
else
# shellcheck source=packaging/installer/functions.sh
. "${NETDATA_SOURCE_DIR}/packaging/installer/functions.sh" || exit 1
fi
# Used to enable saved warnings support in functions.sh
# shellcheck disable=SC2034
NETDATA_SAVE_WARNINGS=1
# -----------------------------------------------------------------------------
# figure out an appropriate temporary directory
_cannot_use_tmpdir() {
testfile="$(TMPDIR="${1}" mktemp -q -t netdata-test.XXXXXXXXXX)"
ret=0
if [ -z "${testfile}" ]; then
return "${ret}"
fi
if printf '#!/bin/sh\necho SUCCESS\n' > "${testfile}"; then
if chmod +x "${testfile}"; then
if [ "$("${testfile}")" = "SUCCESS" ]; then
ret=1
fi
fi
fi
rm -f "${testfile}"
return "${ret}"
}
if [ -z "${TMPDIR}" ] || _cannot_use_tmpdir "${TMPDIR}"; then
if _cannot_use_tmpdir /tmp; then
if _cannot_use_tmpdir "${PWD}"; then
fatal "Unable to find a usable temporary directory. Please set \$TMPDIR to a path that is both writable and allows execution of files and try again." I0000
else
TMPDIR="${PWD}"
fi
else
TMPDIR="/tmp"
fi
fi
# -----------------------------------------------------------------------------
# set up handling for deferred error messages
#
# This leverages the saved warnings functionality shared with some functions from functions.sh
print_deferred_errors() {
if [ -n "${SAVED_WARNINGS}" ]; then
printf >&2 "\n"
printf >&2 "%b\n" "The following warnings and non-fatal errors were encountered during the installation process:"
printf >&2 "%b\n" "${SAVED_WARNINGS}"
printf >&2 "\n"
fi
}
download_go() {
download_file "${1}" "${2}" "go.d plugin" "go"
}
# make sure we save all commands we run
# Variable is used by code in the packaging/installer/functions.sh
# shellcheck disable=SC2034
run_logfile="netdata-installer.log"
# -----------------------------------------------------------------------------
# fix PKG_CHECK_MODULES error
if [ -d /usr/share/aclocal ]; then
ACLOCAL_PATH=${ACLOCAL_PATH-/usr/share/aclocal}
export ACLOCAL_PATH
fi
export LC_ALL=C
umask 002
# Be nice on production environments
renice 19 $$ > /dev/null 2> /dev/null
# you can set CFLAGS before running installer
# shellcheck disable=SC2269
LDFLAGS="${LDFLAGS}"
CFLAGS="${CFLAGS-"-O2 -pipe"}"
[ "z${CFLAGS}" = "z-O3" ] && CFLAGS="-O2"
# shellcheck disable=SC2269
ACLK="${ACLK}"
# keep a log of this command
{
printf "\n# "
date
printf 'CFLAGS="%s" ' "${CFLAGS}"
printf 'LDFLAGS="%s" ' "${LDFLAGS}"
printf "%s" "${PROGRAM}" "${@}"
printf "\n"
} >> netdata-installer.log
REINSTALL_OPTIONS="$(
printf "%s" "${*}"
printf "\n"
)"
# remove options that shown not be inherited by netdata-updater.sh
REINSTALL_OPTIONS="$(echo "${REINSTALL_OPTIONS}" | sed 's/--dont-wait//g' | sed 's/--dont-start-it//g')"
banner_nonroot_install() {
cat << NONROOTNOPREFIX
${TPUT_RED}${TPUT_BOLD}Sorry! This will fail!${TPUT_RESET}
You are attempting to install netdata as a non-root user, but you plan
to install it in system paths.
Please set an installation prefix, like this:
$PROGRAM ${@} --install-prefix /tmp
or, run the installer as root:
sudo $PROGRAM ${@}
We suggest to install it as root, or certain data collectors will
not be able to work. Netdata drops root privileges when running.
So, if you plan to keep it, install it as root to get the full
functionality.
NONROOTNOPREFIX
}
banner_root_notify() {
cat << NONROOT
${TPUT_RED}${TPUT_BOLD}IMPORTANT${TPUT_RESET}:
You are about to install netdata as a non-root user.
Netdata will work, but a few data collection modules that
require root access will fail.
If you are installing netdata permanently on your system, run
the installer like this:
${TPUT_YELLOW}${TPUT_BOLD}sudo $PROGRAM ${@}${TPUT_RESET}
NONROOT
}
usage() {
netdata_banner
progress "installer command line options"
cat << HEREDOC
USAGE: ${PROGRAM} [options]
where options include:
--install-prefix <path> Install netdata in <path>. Ex. --install-prefix /opt will put netdata in /opt/netdata.
--dont-start-it Do not (re)start netdata after installation.
--dont-wait Run installation in non-interactive mode.
--stable-channel Use packages from GitHub release pages instead of nightly updates.
This results in less frequent updates.
--nightly-channel Use most recent nightly updates instead of GitHub releases.
This results in more frequent updates.
--disable-go Disable installation of go.d.plugin.
--disable-ebpf Disable eBPF Kernel plugin. Default: enabled.
--disable-cloud Disable all Netdata Cloud functionality.
--require-cloud Fail the install if it can't build Netdata Cloud support.
--enable-plugin-freeipmi Enable the FreeIPMI plugin. Default: enable it when libipmimonitoring is available.
--disable-plugin-freeipmi Explicitly disable the FreeIPMI plugin.
--disable-https Explicitly disable TLS support.
--disable-dbengine Explicitly disable DB engine support.
--enable-plugin-nfacct Enable nfacct plugin. Default: enable it when libmnl and libnetfilter_acct are available.
--disable-plugin-nfacct Explicitly disable the nfacct plugin.
--enable-plugin-xenstat Enable the xenstat plugin. Default: enable it when libxenstat and libyajl are available.
--disable-plugin-xenstat Explicitly disable the xenstat plugin.
--enable-exporting-kinesis Enable AWS Kinesis exporting connector. Default: enable it when libaws_cpp_sdk_kinesis
and its dependencies are available.
--disable-exporting-kinesis Explicitly disable AWS Kinesis exporting connector.
--enable-exporting-prometheus-remote-write Enable Prometheus remote write exporting connector. Default: enable it
when libprotobuf and libsnappy are available.
--disable-exporting-prometheus-remote-write Explicitly disable Prometheus remote write exporting connector.
--enable-exporting-mongodb Enable MongoDB exporting connector. Default: enable it when libmongoc is available.
--disable-exporting-mongodb Explicitly disable MongoDB exporting connector.
--enable-exporting-pubsub Enable Google Cloud PubSub exporting connector. Default: enable it when
libgoogle_cloud_cpp_pubsub_protos and its dependencies are available.
--disable-exporting-pubsub Explicitly disable Google Cloud PubSub exporting connector.
--enable-lto Enable link-time optimization. Default: disabled.
--disable-lto Explicitly disable link-time optimization.
--enable-ml Enable anomaly detection with machine learning. Default: autodetect.
--disable-ml Explicitly disable anomaly detection with machine learning.
--disable-x86-sse Disable SSE instructions & optimizations. Default: enabled.
--use-system-protobuf Use a system copy of libprotobuf instead of bundled copy. Default: bundled.
--zlib-is-really-here
--libs-are-really-here If you see errors about missing zlib or libuuid but you know it is available, you might
have a broken pkg-config. Use this option to proceed without checking pkg-config.
--disable-telemetry Opt-out from our anonymous telemetry program. (DISABLE_TELEMETRY=1)
--skip-available-ram-check Skip checking the amount of RAM the system has and pretend it has enough to build safely.
--disable-logsmanagement Disable the logs management plugin. Default: autodetect.
--enable-logsmanagement-tests Enable the logs management tests. Default: disabled.
Netdata will by default be compiled with gcc optimization -O2
If you need to pass different CFLAGS, use something like this:
CFLAGS="<gcc options>" ${PROGRAM} [options]
If you also need to provide different LDFLAGS, use something like this:
LDFLAGS="<extra ldflag options>" ${PROGRAM} [options]
or use the following if both LDFLAGS and CFLAGS need to be overridden:
CFLAGS="<gcc options>" LDFLAGS="<extra ld options>" ${PROGRAM} [options]
For the installer to complete successfully, you will need these packages installed:
gcc
make
autoconf
automake
pkg-config
zlib1g-dev (or zlib-devel)
uuid-dev (or libuuid-devel)
For the plugins, you will at least need:
curl
bash (v4+)
python (v2 or v3)
node.js
HEREDOC
}
DONOTSTART=0
DONOTWAIT=0
NETDATA_PREFIX=
LIBS_ARE_HERE=0
NETDATA_ENABLE_ML=""
NETDATA_ENABLE_GTESTS=0
NETDATA_CONFIGURE_OPTIONS="${NETDATA_CONFIGURE_OPTIONS-}"
RELEASE_CHANNEL="nightly" # valid values are 'nightly' and 'stable'
IS_NETDATA_STATIC_BINARY="${IS_NETDATA_STATIC_BINARY:-"no"}"
while [ -n "${1}" ]; do
case "${1}" in
"--zlib-is-really-here") LIBS_ARE_HERE=1 ;;
"--libs-are-really-here") LIBS_ARE_HERE=1 ;;
"--use-system-protobuf")
USE_SYSTEM_PROTOBUF=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--without-bundled-protobuf}" | sed 's/$/ --without-bundled-protobuf/g')"
;;
"--dont-scrub-cflags-even-though-it-may-break-things") DONT_SCRUB_CFLAGS_EVEN_THOUGH_IT_MAY_BREAK_THINGS=1 ;;
"--dont-start-it") DONOTSTART=1 ;;
"--dont-wait") DONOTWAIT=1 ;;
"--auto-update" | "-u") ;;
"--auto-update-type") ;;
"--stable-channel") RELEASE_CHANNEL="stable" ;;
"--nightly-channel") RELEASE_CHANNEL="nightly" ;;
"--enable-plugin-freeipmi") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-plugin-freeipmi)}" | sed 's/$/ --enable-plugin-freeipmi/g')" ;;
"--disable-plugin-freeipmi") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-plugin-freeipmi)}" | sed 's/$/ --disable-plugin-freeipmi/g')" ;;
"--disable-https")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-openssl)}" | sed 's/$/ --disable-openssl/g')"
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-dbengine)}" | sed 's/$/ --disable-dbengine/g')"
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-exporting-kinesis)}" | sed 's/$/ --disable-exporting-kinesis/g')"
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-h2o)}" | sed 's/$/ --disable-h2o/g')"
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-cloud)}" | sed 's/$/ --disable-cloud/g')" ;;
"--disable-dbengine")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-dbengine)}" | sed 's/$/ --disable-dbengine/g')"
;;
"--enable-plugin-nfacct") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-plugin-nfacct)}" | sed 's/$/ --enable-plugin-nfacct/g')" ;;
"--disable-plugin-nfacct") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-plugin-nfacct)}" | sed 's/$/ --disable-plugin-nfacct/g')" ;;
"--enable-plugin-xenstat") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-plugin-xenstat)}" | sed 's/$/ --enable-plugin-xenstat/g')" ;;
"--disable-plugin-xenstat") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-plugin-xenstat)}" | sed 's/$/ --disable-plugin-xenstat/g')" ;;
"--enable-exporting-kinesis" | "--enable-backend-kinesis")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-exporting-kinesis)}" | sed 's/$/ --enable-exporting-kinesis/g')" ;;
"--disable-exporting-kinesis" | "--disable-backend-kinesis")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-exporting-kinesis)}" | sed 's/$/ --disable-exporting-kinesis/g')" ;;
"--enable-exporting-prometheus-remote-write" | "--enable-backend-prometheus-remote-write")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-exporting-prometheus-remote-write)}" | sed 's/$/ --enable-exporting-prometheus-remote-write/g')" ;;
"--disable-exporting-prometheus-remote-write" | "--disable-backend-prometheus-remote-write")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-exporting-prometheus-remote-write)}" | sed 's/$/ --disable-exporting-prometheus-remote-write/g')"
NETDATA_DISABLE_PROMETHEUS=1
;;
"--enable-exporting-mongodb" | "--enable-backend-mongodb")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-exporting-mongodb)}" | sed 's/$/ --enable-exporting-mongodb/g')" ;;
"--disable-exporting-mongodb" | "--disable-backend-mongodb")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-exporting-mongodb)}" | sed 's/$/ --disable-exporting-mongodb/g')" ;;
"--enable-exporting-pubsub") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-exporting-pubsub)}" | sed 's/$/ --enable-exporting-pubsub/g')" ;;
"--disable-exporting-pubsub") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-exporting-pubsub)}" | sed 's/$/ --disable-exporting-pubsub/g')" ;;
"--enable-lto") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-lto)}" | sed 's/$/ --enable-lto/g')" ;;
"--enable-ml")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-ml)}" | sed 's/$/ --enable-ml/g')"
NETDATA_ENABLE_ML=1
;;
"--disable-ml")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ml)}" | sed 's/$/ --disable-ml/g')"
NETDATA_ENABLE_ML=0
;;
"--disable-logsmanagement")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-logsmanagement)}" | sed 's/$/ --disable-logsmanagement/g')"
NETDATA_DISABLE_LOGS_MANAGEMENT=1
;;
"--enable-logsmanagement-tests") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-logsmanagement-tests)}" | sed 's/$/ --enable-logsmanagement-tests/g')" ;;
"--enable-gtests")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-gtests)}" | sed 's/$/ --enable-gtests/g')"
NETDATA_ENABLE_GTESTS=1
;;
"--disable-gtests")
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-gtests)}" | sed 's/$/ --disable-gtests/g')"
NETDATA_ENABLE_GTESTS=0
;;
"--disable-lto") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-lto)}" | sed 's/$/ --disable-lto/g')" ;;
"--disable-x86-sse") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-x86-sse)}" | sed 's/$/ --disable-x86-sse/g')" ;;
"--disable-telemetry") NETDATA_DISABLE_TELEMETRY=1 ;;
"--disable-go") NETDATA_DISABLE_GO=1 ;;
"--enable-ebpf") NETDATA_DISABLE_EBPF=0 ;;
"--disable-ebpf") NETDATA_DISABLE_EBPF=1 NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ebpf)}" | sed 's/$/ --disable-ebpf/g')" ;;
"--skip-available-ram-check") SKIP_RAM_CHECK=1 ;;
"--one-time-build") NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-dependency-tracking)}" | sed 's/$/ --disable-dependency-tracking/g')" ;;
"--disable-cloud")
if [ -n "${NETDATA_REQUIRE_CLOUD}" ]; then
warning "Cloud explicitly enabled, ignoring --disable-cloud."
else
NETDATA_DISABLE_CLOUD=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-cloud)}" | sed 's/$/ --disable-cloud/g')"
fi
;;
"--require-cloud")
if [ -n "${NETDATA_DISABLE_CLOUD}" ]; then
warning "Cloud explicitly disabled, ignoring --require-cloud."
else
NETDATA_REQUIRE_CLOUD=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--enable-cloud)}" | sed 's/$/ --enable-cloud/g')"
fi
;;
"--build-json-c")
NETDATA_BUILD_JSON_C=1
;;
"--install-prefix")
NETDATA_PREFIX="${2}/netdata"
shift 1
;;
"--install-no-prefix")
NETDATA_PREFIX="${2}"
shift 1
;;
"--help" | "-h")
usage
exit 1
;;
*)
echo >&2 "Unrecognized option '${1}'."
exit_reason "Unrecognized option '${1}'." I000E
usage
exit 1
;;
esac
shift 1
done
if [ ! "${DISABLE_TELEMETRY:-0}" -eq 0 ] ||
[ -n "$DISABLE_TELEMETRY" ] ||
[ ! "${DO_NOT_TRACK:-0}" -eq 0 ] ||
[ -n "$DO_NOT_TRACK" ]; then
NETDATA_DISABLE_TELEMETRY=1
fi
make="make"
# See: https://github.com/netdata/netdata/issues/9163
if [ "$(uname -s)" = "FreeBSD" ]; then
make="gmake"
NETDATA_CONFIGURE_OPTIONS="$NETDATA_CONFIGURE_OPTIONS --disable-dependency-tracking"
fi
if [ "$(uname -s)" = "Linux" ] && [ -f /proc/meminfo ]; then
mega="$((1024 * 1024))"
base=1024
scale=256
# shellcheck disable=SC2086
if [ -n "${MAKEOPTS}" ]; then
proc_count="$(echo ${MAKEOPTS} | grep -oE '\-j *[[:digit:]]+' | tr -d '\-j ')"
else
proc_count="$(find_processors)"
fi
target_ram="$((base * mega + (scale * mega * (proc_count - 1))))"
total_ram="$(grep MemTotal /proc/meminfo | cut -d ':' -f 2 | tr -d ' kB')"
total_ram="$((total_ram * 1024))"
if [ "${total_ram}" -le "$((base * mega))" ] && [ -z "${NETDATA_ENABLE_ML}" ]; then
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ml)}" | sed 's/$/ --disable-ml/g')"
NETDATA_ENABLE_ML=0
fi
if [ -z "${MAKEOPTS}" ]; then
MAKEOPTS="-j${proc_count}"
while [ "${target_ram}" -gt "${total_ram}" ] && [ "${proc_count}" -gt 1 ]; do
proc_count="$((proc_count - 1))"
target_ram="$((base * mega + (scale * mega * (proc_count - 1))))"
MAKEOPTS="-j${proc_count}"
done
else
if [ "${target_ram}" -gt "${total_ram}" ] && [ "${proc_count}" -gt 1 ] && [ -z "${SKIP_RAM_CHECK}" ]; then
target_ram="$(echo "${target_ram}" | awk '{$1/=1024*1024*1024;printf "%.2fGiB\n",$1}')"
total_ram="$(echo "${total_ram}" | awk '{$1/=1024*1024*1024;printf "%.2fGiB\n",$1}')"
run_failed "Netdata needs ${target_ram} of RAM to safely install, but this system only has ${total_ram}. Try reducing the number of processes used for the install using the \$MAKEOPTS variable."
exit_reason "Insufficient RAM to safely install." I000F
exit 2
fi
fi
fi
# set default make options
if [ -z "${MAKEOPTS}" ]; then
MAKEOPTS="-j$(find_processors)"
elif echo "${MAKEOPTS}" | grep -vqF -e "-j"; then
MAKEOPTS="${MAKEOPTS} -j$(find_processors)"
fi
if [ "$(id -u)" -ne 0 ]; then
if [ -z "${NETDATA_PREFIX}" ]; then
netdata_banner
banner_nonroot_install "${@}"
exit_reason "Attempted install as non-root user to /." I0010
exit 1
else
banner_root_notify "${@}"
fi
fi
netdata_banner
progress "real-time performance monitoring, done right!"
cat << BANNER1
You are about to build and install netdata to your system.
The build process will use ${TPUT_CYAN}${TMPDIR}${TPUT_RESET} for
any temporary files. You can override this by setting \$TMPDIR to a
writable directory where you can execute files.
It will be installed at these locations:
- the daemon at ${TPUT_CYAN}${NETDATA_PREFIX}/usr/sbin/netdata${TPUT_RESET}
- config files in ${TPUT_CYAN}${NETDATA_PREFIX}/etc/netdata${TPUT_RESET}
- web files in ${TPUT_CYAN}${NETDATA_PREFIX}/usr/share/netdata${TPUT_RESET}
- plugins in ${TPUT_CYAN}${NETDATA_PREFIX}/usr/libexec/netdata${TPUT_RESET}
- cache files in ${TPUT_CYAN}${NETDATA_PREFIX}/var/cache/netdata${TPUT_RESET}
- db files in ${TPUT_CYAN}${NETDATA_PREFIX}/var/lib/netdata${TPUT_RESET}
- log files in ${TPUT_CYAN}${NETDATA_PREFIX}/var/log/netdata${TPUT_RESET}
BANNER1
[ "$(id -u)" -eq 0 ] && cat << BANNER2
- pid file at ${TPUT_CYAN}${NETDATA_PREFIX}/var/run/netdata.pid${TPUT_RESET}
- logrotate file at ${TPUT_CYAN}/etc/logrotate.d/netdata${TPUT_RESET}
BANNER2
cat << BANNER3
This installer allows you to change the installation path.
Press Control-C and run the same command with --help for help.
BANNER3
if [ -z "$NETDATA_DISABLE_TELEMETRY" ]; then
cat << BANNER4
${TPUT_YELLOW}${TPUT_BOLD}NOTE${TPUT_RESET}:
Anonymous usage stats will be collected and sent to Netdata.
To opt-out, pass --disable-telemetry option to the installer or export
the environment variable DISABLE_TELEMETRY to a non-zero or non-empty value
(e.g: export DISABLE_TELEMETRY=1).
BANNER4
fi
have_autotools=
if [ "$(type autoreconf 2> /dev/null)" ]; then
autoconf_maj_min() {
OLDIFS=$IFS
IFS=.-
maj=$1
min=$2
# shellcheck disable=SC2046
set -- $(autoreconf -V | sed -ne '1s/.* \([^ ]*\)$/\1/p')
# shellcheck disable=SC2086
eval $maj=\$1 $min=\$2
IFS=$OLDIFS
}
autoconf_maj_min AMAJ AMIN
if [ "$AMAJ" -gt 2 ]; then
have_autotools=Y
elif [ "$AMAJ" -eq 2 ] && [ "$AMIN" -ge 60 ]; then
have_autotools=Y
else
echo "Found autotools $AMAJ.$AMIN"
fi
else
echo "No autotools found"
fi
if [ ! "$have_autotools" ]; then
if [ -f configure ]; then
echo "Will skip autoreconf step"
else
fatal "Could not find a usable version of GNU autotools, which is required for building Netdata. Version 2.60 or later of GNU autotools is required." I0001
fi
fi
if [ ${DONOTWAIT} -eq 0 ]; then
if [ -n "${NETDATA_PREFIX}" ]; then
printf '%s' "${TPUT_BOLD}${TPUT_GREEN}Press ENTER to build and install netdata to '${TPUT_CYAN}${NETDATA_PREFIX}${TPUT_YELLOW}'${TPUT_RESET} > "
else
printf '%s' "${TPUT_BOLD}${TPUT_GREEN}Press ENTER to build and install netdata to your system${TPUT_RESET} > "
fi
read -r REPLY
if [ "$REPLY" != '' ]; then
exit_reason "User did not accept install attempt." I0011
exit 1
fi
fi
build_error() {
netdata_banner
trap - EXIT
fatal "Netdata failed to build for an unknown reason." I0002
}
if [ ${LIBS_ARE_HERE} -eq 1 ]; then
shift
echo >&2 "ok, assuming libs are really installed."
export ZLIB_CFLAGS=" "
export ZLIB_LIBS="-lz"
export UUID_CFLAGS=" "
export UUID_LIBS="-luuid"
fi
trap build_error EXIT
# -----------------------------------------------------------------------------
build_protobuf() {
env_cmd=''
if [ -z "${DONT_SCRUB_CFLAGS_EVEN_THOUGH_IT_MAY_BREAK_THINGS}" ]; then
env_cmd="env CFLAGS='-fPIC -pipe' CXXFLAGS='-fPIC -pipe' LDFLAGS="
fi
cd "${1}" > /dev/null || return 1
if ! run eval "${env_cmd} ./configure --disable-shared --without-zlib --disable-dependency-tracking --with-pic"; then
cd - > /dev/null || return 1
return 1
fi
if ! run eval "${env_cmd} ${make} ${MAKEOPTS}"; then
cd - > /dev/null || return 1
return 1
fi
cd - > /dev/null || return 1
}
copy_protobuf() {
target_dir="${PWD}/externaldeps/protobuf"
run mkdir -p "${target_dir}" || return 1
run cp -a "${1}/src" "${target_dir}" || return 1
}
bundle_protobuf() {
if [ -n "${NETDATA_DISABLE_CLOUD}" ] && [ -n "${NETDATA_DISABLE_PROMETHEUS}" ]; then
echo "Skipping protobuf"
return 0
fi
if [ -n "${USE_SYSTEM_PROTOBUF}" ]; then
echo "Skipping protobuf"
warning "You have requested use of a system copy of protobuf. This should work, but it is not recommended as it's very likely to break if you upgrade the currently installed version of protobuf."
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling protobuf."
PROTOBUF_PACKAGE_VERSION="$(cat packaging/protobuf.version)"
if [ -f "${PWD}/externaldeps/protobuf/.version" ] && [ "${PROTOBUF_PACKAGE_VERSION}" = "$(cat "${PWD}/externaldeps/protobuf/.version")" ]
then
echo >&2 "Found compiled protobuf, same version, not compiling it again. Remove file '${PWD}/externaldeps/protobuf/.version' to recompile."
NETDATA_CONFIGURE_OPTIONS="${NETDATA_CONFIGURE_OPTIONS} --with-bundled-protobuf"
return 0
fi
tmp="$(mktemp -d -t netdata-protobuf-XXXXXX)"
PROTOBUF_PACKAGE_BASENAME="protobuf-cpp-${PROTOBUF_PACKAGE_VERSION}.tar.gz"
if fetch_and_verify "protobuf" \
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_PACKAGE_VERSION}/${PROTOBUF_PACKAGE_BASENAME}" \
"${PROTOBUF_PACKAGE_BASENAME}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_VERRIDE_PROTOBUF}"; then
if run tar --no-same-owner -xf "${tmp}/${PROTOBUF_PACKAGE_BASENAME}" -C "${tmp}" &&
build_protobuf "${tmp}/protobuf-${PROTOBUF_PACKAGE_VERSION}" &&
copy_protobuf "${tmp}/protobuf-${PROTOBUF_PACKAGE_VERSION}" &&
echo "${PROTOBUF_PACKAGE_VERSION}" >"${PWD}/externaldeps/protobuf/.version" &&
rm -rf "${tmp}"; then
run_ok "protobuf built and prepared."
NETDATA_CONFIGURE_OPTIONS="${NETDATA_CONFIGURE_OPTIONS} --with-bundled-protobuf"
else
run_failed "Failed to build protobuf. Netdata Cloud support will not be available in this build."
fi
else
run_failed "Unable to fetch sources for protobuf. Netdata Cloud support will not be available in this build."
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_protobuf
# -----------------------------------------------------------------------------
build_jsonc() {
env_cmd=''
if [ -z "${DONT_SCRUB_CFLAGS_EVEN_THOUGH_IT_MAY_BREAK_THINGS}" ]; then
env_cmd="env CFLAGS='-fPIC -pipe' CXXFLAGS='-fPIC -pipe' LDFLAGS="
fi
cd "${1}" > /dev/null || exit 1
run eval "${env_cmd} cmake -DBUILD_SHARED_LIBS=OFF ."
run eval "${env_cmd} ${make} ${MAKEOPTS}"
cd - > /dev/null || return 1
}
copy_jsonc() {
target_dir="${PWD}/externaldeps/jsonc"
run mkdir -p "${target_dir}" "${target_dir}/json-c" || return 1
run cp "${1}/libjson-c.a" "${target_dir}/libjson-c.a" || return 1
# shellcheck disable=SC2086
run cp ${1}/*.h "${target_dir}/json-c" || return 1
}
bundle_jsonc() {
# If --build-json-c flag or not json-c on system, then bundle our own json-c
if [ -z "${NETDATA_BUILD_JSON_C}" ] && pkg-config json-c; then
return 0
fi
if [ -z "$(command -v cmake)" ]; then
run_failed "Could not find cmake, which is required to build JSON-C. The install process will continue, but Netdata Cloud support will be disabled."
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling JSON-C."
progress "Prepare JSON-C"
JSONC_PACKAGE_VERSION="$(cat packaging/jsonc.version)"
tmp="$(mktemp -d -t netdata-jsonc-XXXXXX)"
JSONC_PACKAGE_BASENAME="json-c-${JSONC_PACKAGE_VERSION}.tar.gz"
if fetch_and_verify "jsonc" \
"https://github.com/json-c/json-c/archive/${JSONC_PACKAGE_BASENAME}" \
"${JSONC_PACKAGE_BASENAME}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_OVERRIDE_JSONC}"; then
if run tar --no-same-owner -xf "${tmp}/${JSONC_PACKAGE_BASENAME}" -C "${tmp}" &&
build_jsonc "${tmp}/json-c-json-c-${JSONC_PACKAGE_VERSION}" &&
copy_jsonc "${tmp}/json-c-json-c-${JSONC_PACKAGE_VERSION}" &&
rm -rf "${tmp}"; then
run_ok "JSON-C built and prepared."
else
run_failed "Failed to build JSON-C, Netdata Cloud support will be disabled in this build."
fi
else
run_failed "Unable to fetch sources for JSON-C, Netdata Cloud support will be disabled in this build."
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_jsonc
# -----------------------------------------------------------------------------
build_yaml() {
env_cmd=''
if [ -z "${DONT_SCRUB_CFLAGS_EVEN_THOUGH_IT_MAY_BREAK_THINGS}" ]; then
env_cmd="env CFLAGS='-fPIC -pipe -Wno-unused-value' CXXFLAGS='-fPIC -pipe' LDFLAGS="
fi
cd "${1}" > /dev/null || return 1
run eval "${env_cmd} ./configure --disable-shared --disable-dependency-tracking --with-pic"
run eval "${env_cmd} ${make} ${MAKEOPTS}"
cd - > /dev/null || return 1
}
copy_yaml() {
target_dir="${PWD}/externaldeps/libyaml"
run mkdir -p "${target_dir}" || return 1
run cp "${1}/src/.libs/libyaml.a" "${target_dir}/libyaml.a" || return 1
run cp "${1}/include/yaml.h" "${target_dir}/" || return 1
}
bundle_yaml() {
if pkg-config yaml-0.1; then
return 0
fi
if [ -z "$(command -v cmake)" ]; then
run_failed "Could not find cmake, which is required to build YAML. Critical error."
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling YAML."
progress "Prepare YAML"
YAML_PACKAGE_VERSION="$(cat packaging/yaml.version)"
tmp="$(mktemp -d -t netdata-yaml-XXXXXX)"
YAML_PACKAGE_BASENAME="yaml-${YAML_PACKAGE_VERSION}.tar.gz"
if fetch_and_verify "yaml" \
"https://github.com/yaml/libyaml/releases/download/${YAML_PACKAGE_VERSION}/${YAML_PACKAGE_BASENAME}" \
"${YAML_PACKAGE_BASENAME}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_OVERRIDE_YAML}"; then
if run tar --no-same-owner -xf "${tmp}/${YAML_PACKAGE_BASENAME}" -C "${tmp}" &&
build_yaml "${tmp}/yaml-${YAML_PACKAGE_VERSION}" &&
copy_yaml "${tmp}/yaml-${YAML_PACKAGE_VERSION}" &&
rm -rf "${tmp}"; then
run_ok "YAML built and prepared."
else
run_failed "Failed to build YAML, critical error."
fi
else
run_failed "Unable to fetch sources for YAML, critical error."
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_yaml
# -----------------------------------------------------------------------------
get_kernel_version() {
r="$(uname -r | cut -f 1 -d '-')"
tmpfile="$(mktemp)"
echo "${r}" | tr '.' ' ' > "${tmpfile}"
read -r maj min patch _ < "${tmpfile}"
rm -f "${tmpfile}"
printf "%03d%03d%03d" "${maj}" "${min}" "${patch}"
}
detect_libc() {
libc=
if ldd --version 2>&1 | grep -q -i glibc; then
echo >&2 " Detected GLIBC"
libc="glibc"
elif ldd --version 2>&1 | grep -q -i 'gnu libc'; then
echo >&2 " Detected GLIBC"
libc="glibc"
elif ldd --version 2>&1 | grep -q -i musl; then
echo >&2 " Detected musl"
libc="musl"
else
cmd=$(ldd /bin/sh | grep -w libc | cut -d" " -f 3)
if bash -c "${cmd}" 2>&1 | grep -q -i "GNU C Library"; then
echo >&2 " Detected GLIBC"
libc="glibc"
fi
fi
if [ -z "$libc" ]; then
warning "Cannot detect a supported libc on your system, eBPF support will be disabled."
return 1
fi
echo "${libc}"
return 0
}
build_libbpf() {
cd "${1}/src" > /dev/null || return 1
mkdir root build
# shellcheck disable=SC2086
run env CFLAGS='-fPIC -pipe' CXXFLAGS='-fPIC -pipe' LDFLAGS= BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=.. ${make} ${MAKEOPTS} install
cd - > /dev/null || return 1
}
copy_libbpf() {
target_dir="${PWD}/externaldeps/libbpf"
if [ "$(uname -m)" = x86_64 ]; then
lib_subdir="lib64"
else
lib_subdir="lib"
fi
run mkdir -p "${target_dir}" || return 1
run cp "${1}/usr/${lib_subdir}/libbpf.a" "${target_dir}/libbpf.a" || return 1
run cp -r "${1}/usr/include" "${target_dir}" || return 1
run cp -r "${1}/include/uapi" "${target_dir}/include" || return 1
}
bundle_libbpf() {
if { [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 1 ]; } || [ "$(uname -s)" != Linux ]; then
return 0
fi
# When libc is not detected, we do not have necessity to compile libbpf and we should not do download of eBPF programs
libc="${EBPF_LIBC:-"$(detect_libc)"}"
if [ -z "$libc" ]; then
NETDATA_DISABLE_EBPF=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ebpf)}" | sed 's/$/ --disable-ebpf/g')"
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling libbpf."
progress "Prepare libbpf"
if [ "$(get_kernel_version)" -ge "004014000" ]; then
LIBBPF_PACKAGE_VERSION="$(cat packaging/current_libbpf.version)"
LIBBPF_PACKAGE_COMPONENT="current_libbpf"
else
LIBBPF_PACKAGE_VERSION="$(cat packaging/libbpf_0_0_9.version)"
LIBBPF_PACKAGE_COMPONENT="libbpf_0_0_9"
fi
tmp="$(mktemp -d -t netdata-libbpf-XXXXXX)"
LIBBPF_PACKAGE_BASENAME="v${LIBBPF_PACKAGE_VERSION}.tar.gz"
if fetch_and_verify "${LIBBPF_PACKAGE_COMPONENT}" \
"https://github.com/netdata/libbpf/archive/${LIBBPF_PACKAGE_BASENAME}" \
"${LIBBPF_PACKAGE_BASENAME}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_OVERRIDE_LIBBPF}"; then
if run tar --no-same-owner -xf "${tmp}/${LIBBPF_PACKAGE_BASENAME}" -C "${tmp}" &&
build_libbpf "${tmp}/libbpf-${LIBBPF_PACKAGE_VERSION}" &&
copy_libbpf "${tmp}/libbpf-${LIBBPF_PACKAGE_VERSION}" &&
rm -rf "${tmp}"; then
run_ok "libbpf built and prepared."
else
if [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 0 ]; then
fatal "failed to build libbpf." I0005
else
run_failed "Failed to build libbpf. eBPF support will be disabled"
fi
fi
else
if [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 0 ]; then
fatal "Failed to fetch sources for libbpf." I0006
else
run_failed "Unable to fetch sources for libbpf. eBPF support will be disabled"
fi
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_libbpf
copy_co_re() {
cp -R "${1}/includes" "collectors/ebpf.plugin/"
}
bundle_ebpf_co_re() {
if { [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 1 ]; } || [ "$(uname -s)" != Linux ]; then
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling libbpf."
progress "eBPF CO-RE"
CORE_PACKAGE_VERSION="$(cat packaging/ebpf-co-re.version)"
tmp="$(mktemp -d -t netdata-ebpf-co-re-XXXXXX)"
CORE_PACKAGE_BASENAME="netdata-ebpf-co-re-glibc-${CORE_PACKAGE_VERSION}.tar.xz"
if fetch_and_verify "ebpf-co-re" \
"https://github.com/netdata/ebpf-co-re/releases/download/${CORE_PACKAGE_VERSION}/${CORE_PACKAGE_BASENAME}" \
"${CORE_PACKAGE_BASENAME}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_OVERRIDE_CORE}"; then
if run tar --no-same-owner -xf "${tmp}/${CORE_PACKAGE_BASENAME}" -C "${tmp}" &&
copy_co_re "${tmp}" &&
rm -rf "${tmp}"; then
run_ok "libbpf built and prepared."
else
if [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 0 ]; then
fatal "Failed to get eBPF CO-RE files." I0007
else
run_failed "Failed to get eBPF CO-RE files. eBPF support will be disabled"
NETDATA_DISABLE_EBPF=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ebpf)}" | sed 's/$/ --disable-ebpf/g')"
fi
fi
else
if [ -n "${NETDATA_DISABLE_EBPF}" ] && [ "${NETDATA_DISABLE_EBPF}" = 0 ]; then
fatal "Failed to fetch eBPF CO-RE files." I0008
else
run_failed "Failed to fetch eBPF CO-RE files. eBPF support will be disabled"
NETDATA_DISABLE_EBPF=1
NETDATA_CONFIGURE_OPTIONS="$(echo "${NETDATA_CONFIGURE_OPTIONS%--disable-ebpf)}" | sed 's/$/ --disable-ebpf/g')"
fi
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_ebpf_co_re
# -----------------------------------------------------------------------------
build_fluentbit() {
env_cmd="env CFLAGS='-w' CXXFLAGS='-w' LDFLAGS="
if [ -z "${DONT_SCRUB_CFLAGS_EVEN_THOUGH_IT_MAY_BREAK_THINGS}" ]; then
env_cmd="env CFLAGS='-fPIC -pipe -w' CXXFLAGS='-fPIC -pipe -w' LDFLAGS="
fi
mkdir -p fluent-bit/build || return 1
cd fluent-bit/build > /dev/null || return 1
rm CMakeCache.txt > /dev/null 2>&1
if ! run eval "${env_cmd} $1 -C ../../logsmanagement/fluent_bit_build/config.cmake -B./ -S../"; then
cd - > /dev/null || return 1
rm -rf fluent-bit/build > /dev/null 2>&1
return 1
fi
if ! run eval "${env_cmd} ${make} ${MAKEOPTS}"; then
cd - > /dev/null || return 1
rm -rf fluent-bit/build > /dev/null 2>&1
return 1
fi
cd - > /dev/null || return 1
}
bundle_fluentbit() {
progress "Prepare Fluent-Bit"
if [ -n "${NETDATA_DISABLE_LOGS_MANAGEMENT}" ]; then
warning "You have explicitly requested to disable Netdata Logs Management support, Fluent-Bit build is skipped."
return 0
fi
if [ ! -d "fluent-bit" ]; then
run_failed "Missing submodule Fluent-Bit. The install process will continue, but Netdata Logs Management support will be disabled."
return 0
fi
if [ "$(command -v cmake)" ] && [ "$(cmake --version | head -1 | cut -d ' ' -f 3 | cut -c-1)" -ge 3 ]; then
cmake="cmake"
elif [ "$(command -v cmake3)" ]; then
cmake="cmake3"
else
run_failed "Could not find a compatible CMake version (>= 3.0), which is required to build Fluent-Bit. The install process will continue, but Netdata Logs Management support will be disabled."
return 0
fi
patch -N -p1 fluent-bit/CMakeLists.txt -i logsmanagement/fluent_bit_build/CMakeLists.patch
patch -N -p1 fluent-bit/src/flb_log.c -i logsmanagement/fluent_bit_build/flb-log-fmt.patch
# If musl is used, we need to patch chunkio, providing fts has been previously installed.
libc="$(detect_libc)"
if [ "${libc}" = "musl" ]; then
patch -N -p1 fluent-bit/lib/chunkio/src/CMakeLists.txt -i logsmanagement/fluent_bit_build/chunkio-static-lib-fts.patch
patch -N -p1 fluent-bit/cmake/luajit.cmake -i logsmanagement/fluent_bit_build/exclude-luajit.patch
patch -N -p1 fluent-bit/src/flb_network.c -i logsmanagement/fluent_bit_build/xsi-strerror.patch
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Bundling Fluent-Bit."
if build_fluentbit "$cmake"; then
# If Fluent-Bit built with inotify support, use it.
if [ "$(grep -o '^FLB_HAVE_INOTIFY:INTERNAL=.*' fluent-bit/build/CMakeCache.txt | cut -d '=' -f 2)" ]; then
CFLAGS="${CFLAGS} -DFLB_HAVE_INOTIFY"
fi
FLUENT_BIT_BUILD_SUCCESS=1
run_ok "Fluent-Bit built successfully."
else
run_failed "Failed to build Fluent-Bit, Netdata Logs Management support will be disabled in this build."
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
bundle_fluentbit
# -----------------------------------------------------------------------------
# If we have the dashboard switching logic, make sure we're on the classic
# dashboard during the install (updates don't work correctly otherwise).
if [ -x "${NETDATA_PREFIX}/usr/libexec/netdata-switch-dashboard.sh" ]; then
"${NETDATA_PREFIX}/usr/libexec/netdata-switch-dashboard.sh" classic
fi
# -----------------------------------------------------------------------------
# By default, `git` does not update local tags based on remotes. Because
# we use the most recent tag as part of our version determination in
# our build, this can lead to strange versions that look ancient but are
# actually really recent. To avoid this, try and fetch tags if we're
# working in a git checkout.
if [ -d ./.git ] ; then
echo >&2
progress "Updating tags in git to ensure a consistent version number"
run git fetch -t || true
fi
# -----------------------------------------------------------------------------
echo >&2
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Configuring Netdata."
progress "Run autotools to configure the build environment"
if [ "$have_autotools" ]; then
if ! run autoreconf -ivf; then
fatal "Failed to prepare Netdata sources." I0009
fi
fi
# function to extract values from the config file
config_option() {
section="${1}"
key="${2}"
value="${3}"
if [ -x "${NETDATA_PREFIX}/usr/sbin/netdata" ] && [ -r "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ]; then
"${NETDATA_PREFIX}/usr/sbin/netdata" \
-c "${NETDATA_PREFIX}/etc/netdata/netdata.conf" \
-W get "${section}" "${key}" "${value}" ||
echo "${value}"
else
echo "${value}"
fi
}
# the user netdata will run as
if [ "$(id -u)" = "0" ]; then
NETDATA_USER="$(config_option "global" "run as user" "netdata")"
ROOT_USER="root"
else
NETDATA_USER="${USER}"
ROOT_USER="${USER}"
fi
NETDATA_GROUP="$(id -g -n "${NETDATA_USER}")"
[ -z "${NETDATA_GROUP}" ] && NETDATA_GROUP="${NETDATA_USER}"
echo >&2 "Netdata user and group set to: ${NETDATA_USER}/${NETDATA_GROUP}"
# shellcheck disable=SC2086
if ! run ./configure \
--prefix="${NETDATA_PREFIX}/usr" \
--sysconfdir="${NETDATA_PREFIX}/etc" \
--localstatedir="${NETDATA_PREFIX}/var" \
--libexecdir="${NETDATA_PREFIX}/usr/libexec" \
--libdir="${NETDATA_PREFIX}/usr/lib" \
--with-math \
--with-user="${NETDATA_USER}" \
${NETDATA_CONFIGURE_OPTIONS} \
CFLAGS="${CFLAGS}" LDFLAGS="${LDFLAGS}"; then
fatal "Failed to configure Netdata sources." I000A
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
# remove the build_error hook
trap - EXIT
# -----------------------------------------------------------------------------
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Building Netdata."
progress "Cleanup compilation directory"
run $make clean
# -----------------------------------------------------------------------------
progress "Compile netdata"
# shellcheck disable=SC2086
if ! run $make ${MAKEOPTS}; then
fatal "Failed to build Netdata." I000B
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
# -----------------------------------------------------------------------------
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Installing Netdata."
# -----------------------------------------------------------------------------
progress "Install netdata"
if ! run $make install; then
fatal "Failed to install Netdata." I000C
fi
# -----------------------------------------------------------------------------
progress "Fix generated files permissions"
run chmod 755 ./system/*/init.d/netdata ./system/*/rc.d/netdata ./system/runit/run ./system/install-service.sh
# -----------------------------------------------------------------------------
progress "Creating standard user and groups for netdata"
NETDATA_WANTED_GROUPS="docker nginx varnish haproxy adm nsd proxy squid ceph nobody"
NETDATA_ADDED_TO_GROUPS=""
if [ "$(id -u)" -eq 0 ]; then
progress "Adding group 'netdata'"
portable_add_group netdata || :
progress "Adding user 'netdata'"
portable_add_user netdata "${NETDATA_PREFIX}/var/lib/netdata" || :
progress "Assign user 'netdata' to required groups"
for g in ${NETDATA_WANTED_GROUPS}; do
# shellcheck disable=SC2086
portable_add_user_to_group ${g} netdata && NETDATA_ADDED_TO_GROUPS="${NETDATA_ADDED_TO_GROUPS} ${g}"
done
# Netdata must be able to read /etc/pve/qemu-server/* and /etc/pve/lxc/*
# for reading VMs/containers names, CPU and memory limits on Proxmox.
if [ -d "/etc/pve" ]; then
portable_add_user_to_group "www-data" netdata && NETDATA_ADDED_TO_GROUPS="${NETDATA_ADDED_TO_GROUPS} www-data"
fi
else
run_failed "The installer does not run as root. Nothing to do for user and groups"
fi
# -----------------------------------------------------------------------------
progress "Install logrotate configuration for netdata"
install_netdata_logrotate
# -----------------------------------------------------------------------------
progress "Read installation options from netdata.conf"
# create an empty config if it does not exist
[ ! -f "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] &&
touch "${NETDATA_PREFIX}/etc/netdata/netdata.conf"
# port
defport=19999
NETDATA_PORT="$(config_option "web" "default port" ${defport})"
# directories
NETDATA_LIB_DIR="$(config_option "global" "lib directory" "${NETDATA_PREFIX}/var/lib/netdata")"
NETDATA_CACHE_DIR="$(config_option "global" "cache directory" "${NETDATA_PREFIX}/var/cache/netdata")"
NETDATA_WEB_DIR="$(config_option "global" "web files directory" "${NETDATA_PREFIX}/usr/share/netdata/web")"
NETDATA_LOG_DIR="$(config_option "global" "log directory" "${NETDATA_PREFIX}/var/log/netdata")"
NETDATA_USER_CONFIG_DIR="$(config_option "global" "config directory" "${NETDATA_PREFIX}/etc/netdata")"
NETDATA_STOCK_CONFIG_DIR="$(config_option "global" "stock config directory" "${NETDATA_PREFIX}/usr/lib/netdata/conf.d")"
NETDATA_RUN_DIR="${NETDATA_PREFIX}/var/run"
NETDATA_CLAIMING_DIR="${NETDATA_LIB_DIR}/cloud.d"
cat << OPTIONSEOF
Permissions
- netdata user : ${NETDATA_USER}
- netdata group : ${NETDATA_GROUP}
- root user : ${ROOT_USER}
Directories
- netdata user config dir : ${NETDATA_USER_CONFIG_DIR}
- netdata stock config dir : ${NETDATA_STOCK_CONFIG_DIR}
- netdata log dir : ${NETDATA_LOG_DIR}
- netdata run dir : ${NETDATA_RUN_DIR}
- netdata lib dir : ${NETDATA_LIB_DIR}
- netdata web dir : ${NETDATA_WEB_DIR}
- netdata cache dir : ${NETDATA_CACHE_DIR}
Other
- netdata port : ${NETDATA_PORT}
OPTIONSEOF
# -----------------------------------------------------------------------------
progress "Fix permissions of netdata directories (using user '${NETDATA_USER}')"
if [ ! -d "${NETDATA_RUN_DIR}" ]; then
# this is needed if NETDATA_PREFIX is not empty
if ! run mkdir -p "${NETDATA_RUN_DIR}"; then
warning "Failed to create ${NETDATA_RUN_DIR}, it must becreated by hand or the Netdata Agent will not be able to be started."
fi
fi
# --- stock conf dir ----
[ ! -d "${NETDATA_STOCK_CONFIG_DIR}" ] && mkdir -p "${NETDATA_STOCK_CONFIG_DIR}"
[ -L "${NETDATA_USER_CONFIG_DIR}/orig" ] && run rm -f "${NETDATA_USER_CONFIG_DIR}/orig"
run ln -s "${NETDATA_STOCK_CONFIG_DIR}" "${NETDATA_USER_CONFIG_DIR}/orig"
# --- web dir ----
if [ ! -d "${NETDATA_WEB_DIR}" ]; then
echo >&2 "Creating directory '${NETDATA_WEB_DIR}'"
run mkdir -p "${NETDATA_WEB_DIR}" || exit 1
fi
run find "${NETDATA_WEB_DIR}" -type f -exec chmod 0664 {} \;
run find "${NETDATA_WEB_DIR}" -type d -exec chmod 0775 {} \;
# --- data dirs ----
for x in "${NETDATA_LIB_DIR}" "${NETDATA_CACHE_DIR}" "${NETDATA_LOG_DIR}"; do
if [ ! -d "${x}" ]; then
echo >&2 "Creating directory '${x}'"
if ! run mkdir -p "${x}"; then
warning "Failed to create ${x}, it must be created by hand or the Netdata Agent will not be able to be started."
fi
fi
run chown -R "${NETDATA_USER}:${NETDATA_GROUP}" "${x}"
#run find "${x}" -type f -exec chmod 0660 {} \;
#run find "${x}" -type d -exec chmod 0770 {} \;
done
run chmod 755 "${NETDATA_LOG_DIR}"
# --- claiming dir ----
if [ ! -d "${NETDATA_CLAIMING_DIR}" ]; then
echo >&2 "Creating directory '${NETDATA_CLAIMING_DIR}'"
if ! run mkdir -p "${NETDATA_CLAIMING_DIR}"; then
warning "failed to create ${NETDATA_CLAIMING_DIR}, it will need to be created manually."
fi
fi
run chown -R "${NETDATA_USER}:${NETDATA_GROUP}" "${NETDATA_CLAIMING_DIR}"
run chmod 770 "${NETDATA_CLAIMING_DIR}"
# --- plugins ----
if [ "$(id -u)" -eq 0 ]; then
# find the admin group
admin_group=
test -z "${admin_group}" && get_group root > /dev/null 2>&1 && admin_group="root"
test -z "${admin_group}" && get_group daemon > /dev/null 2>&1 && admin_group="daemon"
test -z "${admin_group}" && admin_group="${NETDATA_GROUP}"
run chown "${NETDATA_USER}:${admin_group}" "${NETDATA_LOG_DIR}"
run chown -R "root:${admin_group}" "${NETDATA_PREFIX}/usr/libexec/netdata"
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type d -exec chmod 0755 {} \;
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type f -exec chmod 0644 {} \;
# shellcheck disable=SC2086
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type f -a -name \*.plugin -exec chown :${NETDATA_GROUP} {} \;
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type f -a -name \*.plugin -exec chmod 0750 {} \;
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type f -a -name \*.sh -exec chmod 0755 {} \;
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1> /dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"
if run setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"; then
# if we managed to setcap, but we fail to execute apps.plugin setuid to root
"${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" -t > /dev/null 2>&1 && capabilities=1 || capabilities=0
fi
fi
if [ $capabilities -eq 0 ]; then
# fix apps.plugin to be setuid to root
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1> /dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin"
if run setcap cap_dac_read_search+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin"; then
# if we managed to setcap, but we fail to execute debugfs.plugin setuid to root
"${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin" -t > /dev/null 2>&1 && capabilities=1 || capabilities=0
fi
fi
if [ $capabilities -eq 0 ]; then
# fix debugfs.plugin to be setuid to root
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/debugfs.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1> /dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin"
if run setcap cap_dac_read_search+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin"; then
capabilities=1
fi
fi
if [ $capabilities -eq 0 ]; then
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/logs-management.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/logs-management.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1> /dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/logs-management.plugin"
if run setcap cap_dac_read_search,cap_syslog+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/logs-management.plugin"; then
capabilities=1
fi
fi
if [ $capabilities -eq 0 ]; then
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/logs-management.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1>/dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin"
if run sh -c "setcap cap_perfmon+ep \"${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin\" || setcap cap_sys_admin+ep \"${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin\""; then
capabilities=1
fi
fi
if [ $capabilities -eq 0 ]; then
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/perf.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/slabinfo.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/slabinfo.plugin"
capabilities=0
if ! iscontainer && command -v setcap 1>/dev/null 2>&1; then
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/slabinfo.plugin"
if run setcap cap_dac_read_search+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/slabinfo.plugin"; then
capabilities=1
fi
fi
if [ $capabilities -eq 0 ]; then
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/slabinfo.plugin"
fi
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/freeipmi.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/freeipmi.plugin"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/freeipmi.plugin"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/nfacct.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/nfacct.plugin"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/nfacct.plugin"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/xenstat.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/xenstat.plugin"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/xenstat.plugin"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ioping" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ioping"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ioping"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf.plugin" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf.plugin"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf.plugin"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh"
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh"
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/local-listeners" ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/local-listeners"
run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/local-listeners"
fi
else
# non-privileged user installation
run chown "${NETDATA_USER}:${NETDATA_GROUP}" "${NETDATA_LOG_DIR}"
run chown -R "${NETDATA_USER}:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata"
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type f -exec chmod 0755 {} \;
run find "${NETDATA_PREFIX}/usr/libexec/netdata" -type d -exec chmod 0755 {} \;
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
# -----------------------------------------------------------------------------
# govercomp compares go.d.plugin versions. Exit codes:
# 0 - version1 == version2
# 1 - version1 > version2
# 2 - version2 > version1
# 3 - error
# shellcheck disable=SC2086
govercomp() {
# version in file:
# - v0.14.0
#
# 'go.d.plugin -v' output variants:
# - go.d.plugin, version: unknown
# - go.d.plugin, version: v0.14.1
# - go.d.plugin, version: v0.14.1-dirty
# - go.d.plugin, version: v0.14.1-1-g4c5f98c
# - go.d.plugin, version: v0.14.1-1-g4c5f98c-dirty
# we need to compare only MAJOR.MINOR.PATCH part
ver1=$(echo "$1" | grep -E -o "[0-9]+\.[0-9]+\.[0-9]+")
ver2=$(echo "$2" | grep -E -o "[0-9]+\.[0-9]+\.[0-9]+")
if [ ${#ver1} -eq 0 ] || [ ${#ver2} -eq 0 ]; then
return 3
fi
num1=$(echo $ver1 | grep -o -E '\.' | wc -l)
num2=$(echo $ver2 | grep -o -E '\.' | wc -l)
if [ ${num1} -ne ${num2} ]; then
return 3
fi
for i in $(seq 1 $((num1+1))); do
x=$(echo $ver1 | cut -d'.' -f$i)
y=$(echo $ver2 | cut -d'.' -f$i)
if [ "${x}" -gt "${y}" ]; then
return 1
elif [ "${y}" -gt "${x}" ]; then
return 2
fi
done
return 0
}
should_install_go() {
if [ -n "${NETDATA_DISABLE_GO+x}" ]; then
return 1
fi
version_in_file="$(cat packaging/go.d.version 2> /dev/null)"
binary_version=$("${NETDATA_PREFIX}"/usr/libexec/netdata/plugins.d/go.d.plugin -v 2> /dev/null)
govercomp "$version_in_file" "$binary_version"
case $? in
0) return 1 ;; # =
2) return 1 ;; # <
*) return 0 ;; # >, error
esac
}
install_go() {
if ! should_install_go; then
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Installing go.d.plugin."
# When updating this value, ensure correct checksums in packaging/go.d.checksums
GO_PACKAGE_VERSION="$(cat packaging/go.d.version)"
ARCH_MAP='
i386::386
i686::386
x86_64::amd64
aarch64::arm64
armv64::arm64
armv6l::arm
armv7l::arm
armv5tel::arm
'
progress "Install go.d.plugin"
ARCH=$(uname -m)
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
for index in ${ARCH_MAP}; do
KEY="${index%%::*}"
VALUE="${index##*::}"
if [ "$KEY" = "$ARCH" ]; then
ARCH="${VALUE}"
break
fi
done
tmp="$(mktemp -d -t netdata-go-XXXXXX)"
GO_PACKAGE_BASENAME="go.d.plugin-${GO_PACKAGE_VERSION}.${OS}-${ARCH}.tar.gz"
if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}" ]; then
download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"
else
progress "Using provided go.d tarball ${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}"
run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}" "${tmp}/${GO_PACKAGE_BASENAME}"
fi
if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}" ]; then
download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"
else
progress "Using provided config file for go.d ${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}"
run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}" "${tmp}/config.tar.gz"
fi
if [ ! -f "${tmp}/${GO_PACKAGE_BASENAME}" ] || [ ! -f "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/${GO_PACKAGE_BASENAME}" ]; then
run_failed "go.d plugin download failed, go.d plugin will not be available"
echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
echo >&2
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
return 0
fi
grep "${GO_PACKAGE_BASENAME}\$" "${INSTALLER_DIR}/packaging/go.d.checksums" > "${tmp}/sha256sums.txt" 2> /dev/null
grep "config.tar.gz" "${INSTALLER_DIR}/packaging/go.d.checksums" >> "${tmp}/sha256sums.txt" 2> /dev/null
# Checksum validation
if ! (cd "${tmp}" && safe_sha256sum -c "sha256sums.txt"); then
echo >&2 "go.d plugin checksum validation failure."
echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
echo >&2
run_failed "go.d.plugin package files checksum validation failed. go.d.plugin will not be available."
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
return 0
fi
# Install new files
run rm -rf "${NETDATA_STOCK_CONFIG_DIR}/go.d"
run rm -rf "${NETDATA_STOCK_CONFIG_DIR}/go.d.conf"
run tar --no-same-owner -xf "${tmp}/config.tar.gz" -C "${NETDATA_STOCK_CONFIG_DIR}/"
run chown -R "${ROOT_USER}:${ROOT_GROUP}" "${NETDATA_STOCK_CONFIG_DIR}"
run tar --no-same-owner -xf "${tmp}/${GO_PACKAGE_BASENAME}"
run mv "${GO_PACKAGE_BASENAME%.tar.gz}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/go.d.plugin"
if [ "$(id -u)" -eq 0 ]; then
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/go.d.plugin"
fi
run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/go.d.plugin"
rm -rf "${tmp}"
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
install_go
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/go.d.plugin" ]; then
if command -v setcap 1>/dev/null 2>&1; then
run setcap "cap_net_admin+epi cap_net_raw=eip" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/go.d.plugin"
fi
fi
should_install_ebpf() {
if [ "${NETDATA_DISABLE_EBPF:=0}" -eq 1 ]; then
run_failed "eBPF has been explicitly disabled, it will not be available in this install."
return 1
fi
if [ "$(uname -s)" != "Linux" ] || [ "$(uname -m)" != "x86_64" ]; then
if [ "${NETDATA_DISABLE_EBPF:=1}" -eq 0 ]; then
run_failed "Currently eBPF is only supported on Linux on X86_64."
fi
return 1
fi
# Check Kernel Config
if ! run "${INSTALLER_DIR}"/packaging/check-kernel-config.sh; then
warning "Kernel unsupported or missing required config (eBPF may not work on your system)"
fi
return 0
}
remove_old_ebpf() {
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf_process.plugin" ]; then
echo >&2 "Removing alpha eBPF collector."
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf_process.plugin"
fi
if [ -f "${NETDATA_PREFIX}/usr/lib/netdata/conf.d/ebpf_process.conf" ]; then
echo >&2 "Removing alpha eBPF stock file"
rm -f "${NETDATA_PREFIX}/usr/lib/netdata/conf.d/ebpf_process.conf"
fi
if [ -f "${NETDATA_PREFIX}/etc/netdata/ebpf_process.conf" ]; then
echo >&2 "Renaming eBPF configuration file."
mv "${NETDATA_PREFIX}/etc/netdata/ebpf_process.conf" "${NETDATA_PREFIX}/etc/netdata/ebpf.d.conf"
fi
# Added to remove eBPF programs with name pattern: NAME_VERSION.SUBVERSION.PATCH
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/pnetdata_ebpf_process.3.10.0.o" ]; then
echo >&2 "Removing old eBPF programs with patch."
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/rnetdata_ebpf"*.?.*.*.o
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/pnetdata_ebpf"*.?.*.*.o
fi
# Remove old eBPF program to store new eBPF program inside subdirectory
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/pnetdata_ebpf_process.3.10.o" ]; then
echo >&2 "Removing old eBPF programs installed in old directory."
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/rnetdata_ebpf"*.?.*.o
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/pnetdata_ebpf"*.?.*.o
fi
# Remove old eBPF programs that did not have "rhf" suffix
if [ ! -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_process.3.10.rhf.o" ]; then
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf.d/"*.o
fi
# Remove old reject list from previous directory
if [ -f "${NETDATA_PREFIX}/usr/lib/netdata/conf.d/ebpf_kernel_reject_list.txt" ]; then
echo >&2 "Removing old ebpf_kernel_reject_list.txt."
rm -f "${NETDATA_PREFIX}/usr/lib/netdata/conf.d/ebpf_kernel_reject_list.txt"
fi
# Remove old reset script
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/reset_netdata_trace.sh" ]; then
echo >&2 "Removing old reset_netdata_trace.sh."
rm -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/reset_netdata_trace.sh"
fi
}
install_ebpf() {
if ! should_install_ebpf; then
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Installing eBPF code."
remove_old_ebpf
progress "Installing eBPF plugin"
# Detect libc
libc="${EBPF_LIBC:-"$(detect_libc)"}"
EBPF_VERSION="$(cat packaging/ebpf.version)"
EBPF_TARBALL="netdata-kernel-collector-${libc}-${EBPF_VERSION}.tar.xz"
tmp="$(mktemp -d -t netdata-ebpf-XXXXXX)"
if ! fetch_and_verify "ebpf" \
"https://github.com/netdata/kernel-collector/releases/download/${EBPF_VERSION}/${EBPF_TARBALL}" \
"${EBPF_TARBALL}" \
"${tmp}" \
"${NETDATA_LOCAL_TARBALL_OVERRIDE_EBPF}"; then
run_failed "Failed to download eBPF collector package"
echo 2>&" Removing temporary directory ${tmp} ..."
rm -rf "${tmp}"
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
return 1
fi
echo >&2 " Extracting ${EBPF_TARBALL} ..."
tar --no-same-owner -xf "${tmp}/${EBPF_TARBALL}" -C "${tmp}"
# chown everything to root:netdata before we start copying out of our package
run chown -R root:netdata "${tmp}"
if [ ! -d "${NETDATA_PREFIX}"/usr/libexec/netdata/plugins.d/ebpf.d ]; then
mkdir "${NETDATA_PREFIX}"/usr/libexec/netdata/plugins.d/ebpf.d
RET=$?
if [ "${RET}" != "0" ]; then
rm -rf "${tmp}"
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
return 1
fi
fi
run cp -a -v "${tmp}"/*netdata_ebpf_*.o "${NETDATA_PREFIX}"/usr/libexec/netdata/plugins.d/ebpf.d
rm -rf "${tmp}"
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
progress "eBPF Kernel Collector"
install_ebpf
should_install_fluentbit() {
if [ -n "${NETDATA_DISABLE_LOGS_MANAGEMENT}" ]; then
warning "netdata-installer.sh run with --disable-logsmanagement, Fluent-Bit installation is skipped."
return 1
elif [ "${FLUENT_BIT_BUILD_SUCCESS:=0}" -eq 0 ]; then
run_failed "Fluent-Bit was not built successfully, Netdata Logs Management support will be disabled in this build."
return 1
elif [ ! -f fluent-bit/build/lib/libfluent-bit.so ]; then
run_failed "libfluent-bit.so is missing, Netdata Logs Management support will be disabled in this build."
return 1
fi
return 0
}
install_fluentbit() {
if ! should_install_fluentbit; then
return 0
fi
[ -n "${GITHUB_ACTIONS}" ] && echo "::group::Installing Fluent-Bit."
run chown "root:${NETDATA_GROUP}" fluent-bit/build/lib
run chmod 0644 fluent-bit/build/lib/libfluent-bit.so
run cp -a -v fluent-bit/build/lib/libfluent-bit.so "${NETDATA_PREFIX}"/usr/lib/netdata
# Fix paths in logsmanagement.d.conf
run sed -i -e "s|# db dir =.*|db dir = ${NETDATA_CACHE_DIR}\/logs_management_db|g" "${NETDATA_STOCK_CONFIG_DIR}"/logsmanagement.d.conf
run sed -i -e "s|# log file =.*|log file = ${NETDATA_LOG_DIR}\/fluentbit.log|g" "${NETDATA_STOCK_CONFIG_DIR}"/logsmanagement.d.conf
[ -n "${GITHUB_ACTIONS}" ] && echo "::endgroup::"
}
progress "Installing Fluent-Bit plugin"
install_fluentbit
# -----------------------------------------------------------------------------
progress "Telemetry configuration"
# Opt-out from telemetry program
if [ -n "${NETDATA_DISABLE_TELEMETRY+x}" ]; then
run touch "${NETDATA_USER_CONFIG_DIR}/.opt-out-from-anonymous-statistics"
else
printf "You can opt out from anonymous statistics via the --disable-telemetry option, or by creating an empty file %s \n\n" "${NETDATA_USER_CONFIG_DIR}/.opt-out-from-anonymous-statistics"
fi
# -----------------------------------------------------------------------------
progress "Install netdata at system init"
# By default we assume the shutdown/startup of the Netdata Agent are effectively
# without any system supervisor/init like SystemD or SysV. So we assume the most
# basic startup/shutdown commands...
NETDATA_STOP_CMD="${NETDATA_PREFIX}/usr/sbin/netdatacli shutdown-agent"
NETDATA_START_CMD="${NETDATA_PREFIX}/usr/sbin/netdata"
if grep -q docker /proc/1/cgroup > /dev/null 2>&1; then
# If docker runs systemd for some weird reason, let the install proceed
is_systemd_running="NO"
if command -v pidof > /dev/null 2>&1; then
is_systemd_running="$(pidof /usr/sbin/init || pidof systemd || echo "NO")"
else
is_systemd_running="$( (pgrep -q -f systemd && echo "1") || echo "NO")"
fi
if [ "${is_systemd_running}" = "1" ]; then
echo >&2 "Found systemd within the docker container, running install_netdata_service() method"
install_netdata_service || run_failed "Cannot install netdata init service."
else
echo >&2 "We are running within a docker container, will not be installing netdata service"
fi
echo >&2
else
install_netdata_service || run_failed "Cannot install netdata init service."
fi
# -----------------------------------------------------------------------------
# check if we can re-start netdata
# TODO(paulfantom): Creation of configuration file should be handled by a build system. Additionally we shouldn't touch configuration files in /etc/netdata/...
started=0
if [ ${DONOTSTART} -eq 1 ]; then
create_netdata_conf "${NETDATA_PREFIX}/etc/netdata/netdata.conf"
else
if ! restart_netdata "${NETDATA_PREFIX}/usr/sbin/netdata" "${@}"; then
fatal "Cannot start netdata!" I000D
fi
started=1
run_ok "netdata started!"
create_netdata_conf "${NETDATA_PREFIX}/etc/netdata/netdata.conf" "http://localhost:${NETDATA_PORT}/netdata.conf"
fi
run chmod 0644 "${NETDATA_PREFIX}/etc/netdata/netdata.conf"
if [ "$(uname)" = "Linux" ]; then
# -------------------------------------------------------------------------
progress "Check KSM (kernel memory deduper)"
ksm_is_available_but_disabled() {
cat << KSM1
${TPUT_BOLD}Memory de-duplication instructions${TPUT_RESET}
You have kernel memory de-duper (called Kernel Same-page Merging,
or KSM) available, but it is not currently enabled.
To enable it run:
${TPUT_YELLOW}${TPUT_BOLD}echo 1 >/sys/kernel/mm/ksm/run${TPUT_RESET}
${TPUT_YELLOW}${TPUT_BOLD}echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs${TPUT_RESET}
If you enable it, you will save 40-60% of netdata memory.
KSM1
}
ksm_is_not_available() {
cat << KSM2
${TPUT_BOLD}Memory de-duplication not present in your kernel${TPUT_RESET}
It seems you do not have kernel memory de-duper (called Kernel Same-page
Merging, or KSM) available.
To enable it, you need a kernel built with CONFIG_KSM=y
If you can have it, you will save 40-60% of netdata memory.
KSM2
}
if [ -f "/sys/kernel/mm/ksm/run" ]; then
if [ "$(cat "/sys/kernel/mm/ksm/run")" != "1" ]; then
ksm_is_available_but_disabled
fi
else
ksm_is_not_available
fi
fi
# -----------------------------------------------------------------------------
progress "Check version.txt"
if [ ! -s web/gui/version.txt ]; then
cat << VERMSG
${TPUT_BOLD}Version update check warning${TPUT_RESET}
The way you downloaded netdata, we cannot find its version. This means the
Update check on the dashboard, will not work.
If you want to have version update check, please re-install it
following the procedure in:
https://docs.netdata.cloud/packaging/installer/
VERMSG
fi
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" ]; then
# -----------------------------------------------------------------------------
progress "Check apps.plugin"
if [ "$(id -u)" -ne 0 ]; then
cat << SETUID_WARNING
${TPUT_BOLD}apps.plugin needs privileges${TPUT_RESET}
Since you have installed netdata as a normal user, to have apps.plugin collect
all the needed data, you have to give it the access rights it needs, by running
either of the following sets of commands:
To run apps.plugin with escalated capabilities:
${TPUT_YELLOW}${TPUT_BOLD}sudo chown root:${NETDATA_GROUP} "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"${TPUT_RESET}
${TPUT_YELLOW}${TPUT_BOLD}sudo chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"${TPUT_RESET}
${TPUT_YELLOW}${TPUT_BOLD}sudo setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"${TPUT_RESET}
or, to run apps.plugin as root:
${TPUT_YELLOW}${TPUT_BOLD}sudo chown root:${NETDATA_GROUP} "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"${TPUT_RESET}
${TPUT_YELLOW}${TPUT_BOLD}sudo chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin"${TPUT_RESET}
apps.plugin is performing a hard-coded function of data collection for all
running processes. It cannot be instructed from the netdata daemon to perform
any task, so it is pretty safe to do this.
SETUID_WARNING
fi
fi
# -----------------------------------------------------------------------------
progress "Copy uninstaller"
if [ -f "${NETDATA_PREFIX}"/usr/libexec/netdata-uninstaller.sh ]; then
echo >&2 "Removing uninstaller from old location"
rm -f "${NETDATA_PREFIX}"/usr/libexec/netdata-uninstaller.sh
fi
sed "s|ENVIRONMENT_FILE=\"/etc/netdata/.environment\"|ENVIRONMENT_FILE=\"${NETDATA_PREFIX}/etc/netdata/.environment\"|" packaging/installer/netdata-uninstaller.sh > "${NETDATA_PREFIX}/usr/libexec/netdata/netdata-uninstaller.sh"
chmod 750 "${NETDATA_PREFIX}/usr/libexec/netdata/netdata-uninstaller.sh"
# -----------------------------------------------------------------------------
progress "Basic netdata instructions"
cat << END
netdata by default listens on all IPs on port ${NETDATA_PORT},
so you can access it with:
${TPUT_CYAN}${TPUT_BOLD}http://this.machine.ip:${NETDATA_PORT}/${TPUT_RESET}
To stop netdata run:
${TPUT_YELLOW}${TPUT_BOLD}${NETDATA_STOP_CMD}${TPUT_RESET}
To start netdata run:
${TPUT_YELLOW}${TPUT_BOLD}${NETDATA_START_CMD}${TPUT_RESET}
END
echo >&2 "Uninstall script copied to: ${TPUT_RED}${TPUT_BOLD}${NETDATA_PREFIX}/usr/libexec/netdata/netdata-uninstaller.sh${TPUT_RESET}"
echo >&2
# -----------------------------------------------------------------------------
progress "Installing (but not enabling) the netdata updater tool"
install_netdata_updater || run_failed "Cannot install netdata updater tool."
# -----------------------------------------------------------------------------
progress "Wrap up environment set up"
# Save environment variables
echo >&2 "Preparing .environment file"
cat << EOF > "${NETDATA_USER_CONFIG_DIR}/.environment"
# Created by installer
PATH="${PATH}"
CFLAGS="${CFLAGS}"
LDFLAGS="${LDFLAGS}"
MAKEOPTS="${MAKEOPTS}"
NETDATA_TMPDIR="${TMPDIR}"
NETDATA_PREFIX="${NETDATA_PREFIX}"
NETDATA_CONFIGURE_OPTIONS="${NETDATA_CONFIGURE_OPTIONS}"
NETDATA_ADDED_TO_GROUPS="${NETDATA_ADDED_TO_GROUPS}"
INSTALL_UID="$(id -u)"
NETDATA_GROUP="${NETDATA_GROUP}"
REINSTALL_OPTIONS="${REINSTALL_OPTIONS}"
RELEASE_CHANNEL="${RELEASE_CHANNEL}"
IS_NETDATA_STATIC_BINARY="${IS_NETDATA_STATIC_BINARY}"
NETDATA_LIB_DIR="${NETDATA_LIB_DIR}"
EOF
run chmod 0644 "${NETDATA_USER_CONFIG_DIR}/.environment"
echo >&2 "Setting netdata.tarball.checksum to 'new_installation'"
cat << EOF > "${NETDATA_LIB_DIR}/netdata.tarball.checksum"
new_installation
EOF
print_deferred_errors
# -----------------------------------------------------------------------------
echo >&2
progress "We are done!"
if [ ${started} -eq 1 ]; then
netdata_banner
progress "is installed and running now!"
else
netdata_banner
progress "is installed now!"
fi
echo >&2 " enjoy real-time performance and health monitoring..."
echo >&2
exit 0