0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-16 10:31:07 +00:00
Commit graph

16796 commits

Author SHA1 Message Date
Austin S. Hemmelgarn
d8e2aff0ae
Switch to using Debian as a base for our Docker images. ()
* Update Dockerfile to properly work with Debian base images.

* add missing setuid plugins

* add systemd-journal

* disable building ebpf in Docker

* Fix docker group handling.

* fix "no such user"/" No such file or directory" warning

* fix netdata group and rm nut from docs

* fix missing systemd-cat-native

---------

Co-authored-by: ilyam8 <ilya@netdata.cloud>
Co-authored-by: Tasos Katsoulas <12612986+tkatsoulas@users.noreply.github.com>
2023-11-28 14:28:54 +02:00
Mateusz Bularz
16926d80fc
On-Prem documentation full and light () 2023-11-28 11:14:09 +02:00
Costa Tsaousis
81f4c36401
log2journal: added missing yaml elements ()
added missing yaml elements
2023-11-28 00:23:19 +00:00
netdatabot
2f5b51164d [ci skip] Update changelog and version for nightly build: v1.43.0-284-nightly. 2023-11-28 00:18:13 +00:00
Costa Tsaousis
a64c8cdb43
log2journal moved to collectors ()
* log2journal moved to collectors

* split log2journal into multiple files

* update the path xxh headers

* json support for log2journal

* logfmt support

* fix warning

* fix logfmt prefix

* added support for UTF-8 escape sequences in json values
2023-11-28 00:05:34 +00:00
Austin S. Hemmelgarn
c8c25a35cd
Fix incorrect DEB package build dep. () 2023-11-27 20:38:30 +02:00
Tasos Katsoulas
15dc8abb55
Bump new version to cov-analysis tool ()
* Bump new version to cov-analysis tool
* full version
2023-11-27 20:37:24 +02:00
Emmanuel Vasilakis
e3d9ed2629
Disable netdata monitoring section by default () 2023-11-27 19:00:13 +02:00
Dimitris P
4e512411ec
Logs Management ()
This PR adds the logs-management external plugin. 
See the included README for an extensive list of features. 
-------------------------------------------------------------------------------------

* Add proper status return in JSON response of functions

* Add column info to functions

* Escape special characters when returning JSON response

* Add proper functions help and defaults. Fix help not working

* Add 'logs_management_meta' object in functions results

* Fix compiler warnings

* Replace tabs with 3 spaces in web_client_api_request_v1_logsmanagement_sources()

* Add 'sources' in functions to display list of log sources

* Update functions column values for logs

* Update chart titles and remove '/s' from units

* Add support for compound queries in circular buffers

* Refactor circ_buff_search() to get rid of circ_buff_search_compound()

* Fix incorrect docker events nano timestamp padding

* Fixed botched rebasing

* Replace get_unix_time_ms() with now_realtime_msec()

* Remove binary generation from Fluent-Bit lib build

* Fix compiler warnings due to new timestamp type

* Remove STDIN and STDOUT support from Fluent-Bit library

* Initial support for FLB_KMSG kernel logs collection

* Add kernel logs charts

* Add kernel logs subsystem and device charts

* Skip collection of pre-existing logs in kmsg ring buffer

* Add example of custom kmsg charts

* Add extra initialization error logs

* Fix bug of Docker Events collector failure disabling whole logs management engine

* Remove reduntant FLB output code

* Remove some obsolete TODO comments

* Remove some commented out error/debug prints

* Disable some Fluent-Bit config options not required

* Make circular buffer spare items option configurable

* Add DB mode configuration option

* Replace p_file_infos_arr->data[i] with p_file_info in db_api.c

* Remove db_loop due to all function calls being synchronous

* Add initial README.md

* Add DB mode = none changes

* Add a simple webpage to visualize log query results

* Add support for source selection to logs_query.html

* Add option to query multiple log sources

* Mark non-queryable sources as such in logs_query.html

* Add option to use either GET or functions request in logs_query.html

* Install logs_query.html when running stress tests

* Update README.md requirements

* Change installer behavior to build logs management by default

* Disable logs management at runtime by default

* Add global db mode configuration in 'logs management' config section

* Split logsmanagement.conf into required & optional sections

* Remove --enable-logsmanagement from stress test script

* Add global config option for 'circular buffer max size MiB'

* Add global config option for 'circular buffer drop logs if full'

* Update 'General Configuration' in README.md

* Add global config option for remaining optional settings

* Add systemd collector requirements to TOC

* README: Convert general configuration to table

* README: Fix previous botched commit

* Enable logs management by default when building for stress testing

* Move logging to collector.log from error.log

* Fix contenttype compilation errors

* Move logging to collector.log in plugin_logsmanagement.c

* Rename 'rows' to 'records' in charts

* Add Netdata error.log parsing

* Add more dashboard descriptions

* Sanitize chart ids

* Attempt to fix failing CI

* Update README.md

* Update README.md

* Another attempt to fix CI failures

* Fix undefined reference to 'uv_sleep' on certain platforms

* Support FLB forward input and FLB output plugins.

Squashed commit of the following:

commit 55e2bf4fb34a2e02ffd0b280790197310a5299f3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 16:41:09 2023 +0300

    Remove error.log from stock config

commit bbdc62c2c9727359bc3c8ef8c33ee734d0039be7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 16:37:48 2023 +0300

    Add cleanup of Fluent Bit outputs in p_file_info_destroy()

commit 09b0aa4268ec1ccef160c99c5d5f31b6388edd28
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 14:34:17 2023 +0300

    Some code and config cleanup

commit 030d074667d5ee2cad10f85cd836ca90e29346ad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 13:04:08 2023 +0300

    Enable additional Fluent Bit output plugins for shared library

commit 490aa5d44caa38042521d24c6b886b8b4a59a73c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Apr 13 01:33:19 2023 +0300

    Add initialization of Fluent Bit user-configured outputs

commit c96e9fe9cea96549aa5eae09d0deeb130da02793
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Apr 4 23:13:16 2023 +0100

    Complete read of parameters for FLB outputs config

commit 00988897f9b86d1ecc5c141b19df7ad7d74f7e96
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 19:43:31 2023 +0100

    Update README.md

commit 6deea5399c2707942aeaa51408f999ca45dfd351
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 16:02:28 2023 +0100

    Refactor Syslog_parser_config_t and add Flb_socket_config_t

commit 7bf998a4c298bbd489ef735c56a6e85a137772c9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Apr 3 14:19:57 2023 +0100

    Update README.md

commit c353d194b12c54f134936072ebaded0424d73cc0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Mar 31 14:52:57 2023 +0100

    Update README.md

commit 6be726eaff3738ba7884de799aa52949833af65a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Mar 31 13:06:29 2023 +0100

    Update README. Fix docker_events streaming

commit 6aabfb0f1ef0529a7a0ecbaf940bc0952bf42518
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 21:27:45 2023 +0100

    Fix stuck in infinite loop bug for FLB_GENERIC, FLB_WEB_LOG and FLB_SERIAL remote log sources

commit eea6346b708cc7a5ce6e2249366870f4924eabae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 21:04:12 2023 +0100

    Remove callback that searches for streamed p_file_info match

commit bc9c5a523b0b0ab5588adbff391a43ba8d9a0cdf
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 30 15:51:39 2023 +0100

    Basic streaming works

commit 4c80f59f0214bc07895f0b2edca47cb02bc06420
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Mar 28 22:05:22 2023 +0100

    WIP

commit eeb37a71b602fb0738fe8077ccddc0a8ce632304
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Mar 27 22:52:09 2023 +0100

    Add generic forward streaming input

commit 1459b91847c80c4d97de96b75b00771039458ad6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Mar 23 18:50:14 2023 +0000

    FLB_FORWARD: WIP

* Add number of logs per item in DB and in queries response

* Fix wrong number of lines stored in DB for web logs

* Refactor number of logs parsers and charts code

* Add option to toggle number of collected logs metrics and charts

* Disable kmsg log collector by default

* Fix logs_query.html to work with any server ip

* Fix regressed wrong number of web log lines bug

* Change query quota type from size_t to long long

* Update alpine version when searching for fts-dev requirements

* Update query results to return both requested and actual quota

* Fix bug of circ buffs not being read if head == read but not empty

* Squashed commit of the following:

commit 34edb316a737f3edcffcf8fa88a3801599011495
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu May 4 20:02:36 2023 +0100

    Comment out some debug prints

commit 51b9b87a88516186530f5b4b65f785b543fefe8c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 28 19:21:54 2023 +0100

    Fix wrong filenames in BLOBS_TABLE after rotation

commit 6055fc2893b48661af324f20ee61511a40abbc02
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 28 12:22:04 2023 +0100

    Add chart showing number of circular buffer items

commit 0bb5210b0847f4b7596f633ec96fc10aa8ebc791
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Apr 25 16:47:29 2023 +0300

    Various fixes.

    Fix num_lines calculation.
    Add debug prints for circ buffers.
    Remove circ buff spare items option.
    Fix calculation of circ buff memory consumption.
    Add buff_realloc_rwlock for db_mode = none case.
    Fix circ buff read to be done correctly when buff is full.

commit f494af8c95be84404c7d854494d26da3bcbd3ad7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 21 16:03:50 2023 +0300

    Fix freez() on non-malloced address

commit cce6d09e9cf9b847aface7309643e2c0a6041390
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Apr 21 15:41:25 2023 +0300

    Add option to dynamically expand circ buffs when full

* Use log timestamps when possible, instead of collection timestamps.
Also, add config options for Fluent Bit engine and remove tail_plugin.

Squashed commit of the following:

commit b16a02eb6e3a90565c90e0a274b87b123e7b18e5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 16 19:38:57 2023 +0100

    Add Fluent Bit service config options to netdata.conf. Add monitoring of new log file fluentbit.log

commit ab77c286294548ea62a3879ac0f8b8bbfe6a0687
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 21:25:17 2023 +0100

    Remove some debug prints

commit 46d64ad2434e69b1d20720297aec1ddb869e1f84
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 21:19:32 2023 +0100

    Fix null values in charts

commit 8ec96821d6a882f28cbd19244ebdfc86c807d2f4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 17:43:04 2023 +0100

    Update README.md to reflect log timestamp changes

commit 079a91858cf9db2f74711581235bc17eb97c7dad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:23:14 2023 +0100

    Add configurable option for 'update timeout'

commit 72b5e2505d4657fcbb5ccb6eeee00c45eb0b51ff
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:05:08 2023 +0100

    Revert logsmanagement.conf to logs-manag-master one

commit 70d0ea6f8d272fff318aa3095d90a78dcc3411a7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:02:00 2023 +0100

    Fix bug of circ buff items not marked as done

commit 5716420838771edb7842be4669bf96235b15cf71
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 15 16:01:41 2023 +0100

    Fix do_custom_charts_update() to work for all log sources

commit a8def8f53fd25c3efa56ef27e267df3261913a8e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 18:20:20 2023 +0100

    Remove GENERIC and WEB_LOG cases. Remove tail_plugin.c/h. Remove generic_parser().

commit 1cf05966e33491dbeb9b877f18d1ea8643aabeba
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 16:54:59 2023 +0100

    Fix FLB_GENERIC and FLB_SERIAL to work with new timestamp logic

commit df3266810531f1af5f99b666fbf44c503b304a39
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 14:55:04 2023 +0100

    Get rid of *_collect() functions and restructure plugin_logsmanagement workers

commit 3eee069842f3257fffe60dacfc274363bc43491c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 12 14:28:33 2023 +0100

    Fix wrong order of #define _XOPEN_SOURCE 700 in parser.c

commit 941aa80cb55d5a7d6fe8926da930d9803be52312
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu May 11 22:27:39 2023 +0100

    Update plugin_logsmanagement_web_log to use new timestamp logic and to support delayed logs. Refactor req_method metrics code.

commit 427a7d0e2366d43cb5eab7daa1ed82dfc3bc8bc8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 20:26:08 2023 +0100

    Update plugin_logsmanagement_kernel to use new timestamp logic and to support delayed charts

commit a7e95a6d3e5c8b62531b671fd3ec7b8a3196b5bb
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 15:22:14 2023 +0100

    Update plugin_logsmanagement_systemd to use new timestamp logic and support delayed charts

commit 48237ac2ce49c82abdf2783952fd9f0ef05d72e1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 9 13:29:44 2023 +0100

    Refactor number of collected logs chart update code

commit a933c8fcae61c23fa0ec6d0074526ac5d243cf16
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 8 22:11:19 2023 +0100

    Update plugin_logsmanagement_docker_ev to use new timestamp logic and support delayed charts

commit 5d8db057155affd5cb721399a639d75a81801b7f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 5 15:18:06 2023 +0100

    Change some Fluent Bit collectors to use log timestamps instead of collection timestamps

* Remove some unused defines and typedefs

* Improve flb_init()

* Update file-level doxygen. Add SPDX license declaration.

* Better handling of termination of Fluent Bit

* Better handling of DB errors. Various fixes.

Squashed commit of the following:

commit f55feea1274c3857eda1e9d899743db6e3eb5bf5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Jun 6 13:28:00 2023 +0100

    Fix web log parsing in case of lines terminated by \r

commit 9e05758a4ecfac57a0db14757cff9536deda51d8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 20:42:05 2023 +0100

    Fix warnings due to -Wformat-truncation=2

commit 63477666fa42446d74693aae542580d4e1e81f03
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 16:48:45 2023 +0100

    Autodiscovery of Netdata error.log based on netdata_configured_log_dir

commit cab5e6d6061f4259172bbf72666e8b4a3a35dd66
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 16:24:39 2023 +0100

    Replace Forward config default string literals with macros

commit 4213398031dbb53afbc943d76bf7df202d12bf6f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 15:56:29 2023 +0100

    Proper cleanup of flb_lib_out_cb *callback in case of error

commit f76fd7cc7bc2d0241e4d3517f61ae192d4246300
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Jun 5 15:36:07 2023 +0100

    Proper termination of Forward input and respective log sources in case of error

commit 3739fd96c29e13298eb3a6e943a63172cdf39d5f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 21:19:56 2023 +0100

    Merge db_search() and db_search_compound()

commit fcface90cb0a6df3c3a2de5e1908b1b3467dd579
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 19:17:26 2023 +0100

    Proper error handling in db_search() and db_search_compound(). Refactor the code too.

commit c10667ebee2510a1af77114b3a7e18a0054b5dae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 14:23:34 2023 +0100

    Update DB mode and dir when switching to db_mode_none

commit d37d4c3d79333bb9fa430650c13ad625458620e8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Jun 1 12:56:13 2023 +0100

    Fix flb_stop() SIGSEGV

commit 892e231c68775ff1a1f052d292d26384f1ef54b1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue May 30 21:14:58 2023 +0100

    Switch to db_writer_db_mode_none if db_writer_db_mode_full encounters error

commit f7a0c2135ff61d3a5b0460ec5964eb6bce164bd6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 29 21:41:21 2023 +0100

    Complete error handling changes to db_init(). Add some const type qualifiers. Refactor some code for readability

commit 13dbeac936d22958394cb1aaec394384f5a93fdd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon May 29 17:14:17 2023 +0100

    More error handling changes in db_init(). Change some global default settings if stress testing.

commit eb0691c269cd09054190bf0ee9c4e9247b4a2548
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri May 26 23:29:12 2023 +0100

    Better handling of db writer threads errors. Add db timings charts

* Fix mystrsep() replaced by strsep_skip_consecutive_separators()

* Fix older GCC failure due to label before declaration

* Fix failed builds when using libuv <= v1.19

* Fix some Codacy warnings

* Fix warning: implicit declaration of function ‘strsep’

* Use USEC_PER_SEC instead of 1000000ULL

* Use UUID_STR_LEN instead of GUID_LEN + 1

* Combine multiple 'ln -sf' Docker instructions to one

* Update README with systemd development libraries requirement

* Comment out mallocz() success checkes in parser_csv()

* Fix shellcheck warnings

* Remove asserts for empty SYSLOG_IDENTIFIER or PID

* Fix FreeBSD failing builds

* Fix some more shellcheck warnings

* Update Alpine fts-dev required packages

* First changes to use web log timestamp for correct metrics timings

* Initial work to add test_parse_web_log_line() unit test

* Complete test_parse_web_log_line() tests

* Improve parse_web_log_line() for better handling of \n, \r, double quotes etc.

* Fix 'Invalid TIME error when timezone sign is negative

* Add more logs to compression unit test case

* Misc read_last_line() improvements

* Fix failing test_auto_detect_web_log_parser_config() when test case terminated without '\n'

* Remove unused preprocessor macro

* Factor out setup of parse_config_expected_num_fields

* Add test for count_fields()

* Add unit test for read_last_line()

* Fix a read_last_line() bug

* Remove PLUGIN[logsmanagement] static thread and update charts synchronously, right before data buffering

* Fix web log parser potential SIGSEGV

* Fix web log metrics bug where they could show delayed by 1 collection interval

* WIP: Add multiline support to kmsg logs and fix metric timings

* Fix kmsg subsystem and device parsing and metrics

* Add option 'use log timestamp' to select between log timestamps or collection timestamps

* Add 'Getting Started' docs section

* Move logs management functions code to separate source files

* Add 'Nginx access.log' chart description

* Remove logsmanagement.plugin source files

* Fix some memory leaks

* Improve cleanup of logsmanagement_main()

* Fix a potential memory leak of fwd_input_out_cb

* Better termination and cleanup of main_loop and its handles

* Fix main_db_dir access() check bug

* Avoid uv_walk() SIGSEGV

* Remove main_db_dir access() check

* Better termination and cleanup of DB code

* Remove flb_socket_config_destroy() that could cause a segmentation fault

* Disable unique client IPs - all-time chart by default

* Update README.md

* Fix debug() -> netdata_log_debug()

* Fix read_last_line()

* Fix timestamp sign adjustment and wrong unit tests

* Change WEB_CLIENT_ACL_DASHBOARD to WEB_CLIENT_ACL_DASHBOARD_ACLK_WEBRTC

* Do not parse web log timestamps if 'use_log_timestamp = no'

* Add Logs Management back into buildinfo.c

* Update README.md

* Do not build Fluent Bit executable binary

* Change logs rate chart to RRDSET_TYPE_LINE

* Add kludge to prevent metrics breaking due to out of order logs

* Fix wrong flb_tmp_buff_cpy_timer expiration

* Refactor initialization of input plugin for local log sources.

* Rename FLB_GENERIC collector to FLB_TAIL.

* Switch 'Netdata fluentbit.log' to disabled by default

* Add 'use inotify' configuration option

* Update  in README.md

* Add docker event actions metrics

* Update README.md to include event action chart

* Remove commented out PLUGIN[logsmanagement] code block

* Fix some warnings

* Add documentation for outgoing log streaming and exporting

* Fix some code block formatting in README.md

* Refactor code related to error status of log query results and add new invalid timestamp case

* Reduce query mem allocs and fix end timestamp == 0 bug

* Add support for duplicate timestamps in db_search()

* Add support for duplicate timestamps in circ_buff_search()

* Fix docker events contexts

* Various query fixes prior to reverse order search.

- Add reverse qsort() function in circ buffers.
- Fix issues to properly support of duplicate timestamps.
- Separate requested from actual timestamps in query parameters.
- Rename results buffer variable name to be consistent between DB and
  buffers.
- Remove default start and end timestamp from functions.
- Improve handling of invalid quotas provided by users.
- Rename 'until' timestamp name to 'to'.
- Increase default quota to 10MB from 1MB.
- Allow start timestamp to be > than end timestamp.

* Complete descending timestamp search for circular buffers

* Complete descending timestamp search for DB

* Remove MEASURE_QUERY_TIME code block

* Complete descending timestamp search when data resides in both DB and circular buffers

* Use pointer instead of copying res_hdr in query results

* Refactor web log timezone parsing to use static memory allocation

* Add stats for CPU user & system time per MiB of query results

* Micro-optimization to slightly speed up queries

* More micro-optimizations and some code cleanup

* Remove LOGS_QUERY_DATA_FORMAT_NEW_LINE option

* Escape iscntrl() chars at collection rather at query

* Reduce number of buffer_strcat() calls

* Complete descending timestamp order queries for web_api_v1

* Complete descending timestamp order queries for functions

* Fix functions query timings to match web_api_v1 ones

* Add MQTT message collector

Squashed commit of the following:

commit dbe515372ee04880b1841ef7800abe9385b12e1c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 15:18:46 2023 +0100

    Update README.md with MQTT information

commit c0b5dbcb7cdef8c6fbd5e72e7bdd08957a0fd3de
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 14:59:36 2023 +0100

    Tidy up before merge

commit 9a69c4f17eac858532918a8f850a770b12710f80
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Aug 21 12:54:33 2023 +0100

    Fix issue with duplicate Log_Source_Path in DB, introduced in commit e417af3

commit 48213e9713216d62fca8a5bc1bbc41a3883fdc14
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sat Aug 19 05:05:36 2023 +0100

    WIP

commit e417af3b947f11bd61e3255306bc95953863998d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 17 18:03:39 2023 +0100

    Update functions logsmanagement help output

* Inhibit Fluent Bit build warnings

* Fix missing allow_subpaths value in api_commands_v1[].

* Fix missing HTTP_RESP_BACKEND_FETCH_FAILED error

* Fix an enum print warning

* Remove systemd-devel requirement from README and fix codacy warnings

* Update Alpine versions for musl-fts-dev

* Update Fluent Bit to v2.1.8

Squashed commit of the following:

commit faf6fc4b7919cc2611124acc67cb1973ce705530
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Aug 25 17:13:30 2023 +0100

    Fix wrong default CORE_STACK_SIZE on Alpine

commit a810238fe7830ce626f6d57245d68035b29723f7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Aug 25 00:40:02 2023 +0100

    Update Fluent Bit patches for musl

commit 8bed3b611dba94a053e22c2b4aa1d46f7787d9b4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 24 21:54:38 2023 +0100

    Fix an edge case crash when web log method is '-'

commit b29b48ea230363142697f9749508cd926e18ee19
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Aug 24 16:26:13 2023 +0100

    Disable FLB_OUT_CALYPTIA to fix Alpine dlsym() error

commit eabe0d0523ffe98ff881675c21b0763a49c05f16
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 21:25:54 2023 +0100

    Add 'use inotify = no' troubleshooting Q&A in README

commit 7f7ae85bdb0def63b4fc05ab88f6572db948e0e7
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 18:06:36 2023 +0100

    Update README.md links to latest version

commit 610c5ac7b920d4a1dfe364ad48f1ca14a0acc346
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 16:23:30 2023 +0100

    Update flb_parser_create() definition

commit f99608ff524b6f3462264e626a1073f9c2fdfdf5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 16:23:04 2023 +0100

    Add new config.cmake options

commit 446b0d564626055a0a125f525d0bd3754184b830
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Aug 22 12:21:25 2023 +0100

    Update Fluent Bit submodule to v2.1.8

* Add logs_management_unittest() to CI 'unittest'

* Remove obsolete query testing files

* Patch Fluent Bit log format to match netdata's format

* Update README with instructions on how to monitor Podman events logs

* Fix core dump in case of flb_lib_path dlopen()

* Fix some potential compiler warnings

* Fix queries crash if logs manag engine not running

* Much faster termination of LOGS MANAGEMENT

* Add facets support and other minor fixes.

logsmanagement_function_execute_cb() is replaced by
logsmanagement_function_facets() which adds facets support to logs
management queries.

Internal query results header now includes additional fields
(log_source, log_type, basename, filename, chartname), that are used as facets.

Queries now support timeout as a query parameter.

A web log timestamp bug is fixed (by using timegm() instead of mktime().

web_api_v1 logsmanagement API is only available in debugging now.

Squashed commit of the following:

commit 32cf0381283029d793ec3af30d96e6cd77ee9149
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 19 16:21:32 2023 +0300

    Tidy up

commit f956b5846451c6b955a150b5d071947037e935f0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 19 13:30:54 2023 +0300

    Add more accepted params. Add data_only option. Add if_modified_since option.

commit 588c2425c60dcdd14349b7b346467dba32fda4e9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 18 18:39:50 2023 +0300

    Add timeout to queries

commit da0f055fc47a36d9af4b7cc4cefb8eb6630e36d9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 19:17:16 2023 +0300

    Fix histogram

commit 7149890974e0d26420ec1c5cfe1023801dc973fa
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 17:58:52 2023 +0300

    Add keyword query using simple patterns and fix descending timestamp values

commit 0bd068c5a76e694b876027e9fa5af6f333ab825b
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 13:54:05 2023 +0300

    Add basename, filename, chartname as facets

commit 023c2b5f758b2479a0e48da575cd59500a1373b6
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 13:26:06 2023 +0300

    Add info and sources functions options

commit ab4d555b7d445f7291af474847bd9177d3726a76
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 12:54:37 2023 +0300

    Fix facet id filter

commit a69c9e2732f5a6da1764bb57d1c06d8d65979225
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 14 12:07:13 2023 +0300

    WIP: Add facet id filters

commit 3c02b5de81fa8a20c712863c347539a52936ddd8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:19:17 2023 +0300

    Add log source and log type to circ buff query results header

commit 8ca98672c4911c126e50f3cbdd69ac363abdb33d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:18:13 2023 +0300

    Fix logsmanagement facet function after master rebasing

commit 3f1517ad56cda2473a279a8d130bec869fc2cbb8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 18:14:25 2023 +0300

    Restrict /logsmanagement to ACL_DEV_OPEN_ACCESS only

commit 8ca98d69b08d006c682997268d5d2523ddde6be0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 12 14:40:22 2023 +0300

    Fix incorrectly parsed timestamps due to DST

commit f9b0848037b29c7fcc46da951ca5cd9eb129066f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 11 13:42:18 2023 +0300

    Add logs_management_meta object to facet query results

commit babc978f6c97107aaf8b337d8d31735d61761b6a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Sep 11 13:03:52 2023 +0300

    Query all sources if no arguments provided

commit 486d56de87af56aae6c0dc5d165341418222ce8b
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 7 18:38:04 2023 +0300

    Add log_source and log_type (only for DB logs) as facets. Add relative time support

commit b564c12843d355c4da6436af358d5f352cb58bfe
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 7 13:47:20 2023 +0300

    Working facet with descending timestamps

commit 68c6a5c64e8425cf28ec16adfb0c50289caa82a9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Wed Sep 6 01:55:51 2023 +0300

    WIP

* Fix linking errors

* Convert logs management to external plugin.

Squashed commit of the following:

commit 16da6ba70ebde0859aed734087f04af497ce3a77
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:44:12 2023 +0100

    Use higher value of update every from netdata.conf or logsmanagement.d.conf

commit 88cc3497c403e07686e9fc0876ebb0c610a1404c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:43:02 2023 +0100

    Tidy up

commit c3fca57aac169842637d210269519612b1a91e28
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 18:02:04 2023 +0100

    Use external update_every from agent, if available

commit f7470708ba82495b03297cdf8962a09b16617ddd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 17:40:46 2023 +0100

    Re-enable debug logs

commit b34f5ac6a2228361ab41df7d7e5e713f724368c0
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:49:20 2023 +0100

    Remove old API calls from web_api_v1.c/h

commit 7fbc1e699a7785ec837233b9562199ee6c7684da
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:32:04 2023 +0100

    Add proper termination of stats charts thread

commit 4c0fc05c8b14593bd7a0aa68f75a8a1205e04db4
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 24 15:31:36 2023 +0100

    Add tests for logsmanag_config functions

commit 4dfdacb55707ab46ed6c2d5ce538ac012574b27e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 22:01:19 2023 +0100

    Remove unused headers from logsmanagement.c

commit b324ef396207c5c32e40ea9ad462bf374470b230
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 21:56:26 2023 +0100

    Remove inline from get_X_dir() functions

commit e9656e8121b66cd7ef8b5daaa5d27a134427aa35
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 21:50:32 2023 +0100

    Proper termination when a signal is received

commit b09eec147bdeffae7b268b6335f6ba89f084e050
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 20:12:13 2023 +0100

    Refactor logs management config code in separate source files

commit 014b46a5008fd296f7d25854079c518d018abdec
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 14:54:47 2023 +0100

    Fix p_file_info_destroy() crash

commit e0bdfd182513bb8d5d4b4b5b8a4cc248ccf2d64e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 23 14:18:27 2023 +0100

    Code refactoring and cleanup

commit 6a61cb6e2fd3a535db150b01d9450f44b3e27b30
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Oct 20 14:08:43 2023 +0100

    Fix 'source:all' queries

commit 45b516aaf819ac142353e323209b7d01e487393f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 19 21:51:05 2023 +0100

    Working 'source:...' queries and regular data queries (but not 'source:all')

commit 8064b0ee71c63da9803f79424802f860e96326e5
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 19 15:34:23 2023 +0100

    Fix issue due to p_file_info_destroy()

commit a0aacc9cd00cea60218c9bfd2b9f164918a1e3de
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 17 22:06:34 2023 +0100

    Work on facet API changes

commit 480584ff9040c07e996b14efb4d21970a347633f
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 16 21:43:06 2023 +0100

    Add stats charts, running as separate thread

commit 34d582dbe4bf2d8d048afab41681e337705bc611
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 16 16:24:02 2023 +0100

    Add SSL cipher charts

commit ced27ee4e2c981d291f498244f2eef2556a074fb
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sun Oct 15 21:33:29 2023 +0100

    Add Response code family, Response code, Response code type, SSL protocol charts

commit 40c4a1d91892d49b1e4e18a1c3c43258ded4014d
Author: Dim-P <dimitris1703@gmail.com>
Date:   Sat Oct 14 00:48:48 2023 +0100

    Add more web log charts

commit 890ed3ff97153dd18d15df2d1b57a181bc498ca8
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Oct 13 22:14:11 2023 +0100

    Add web log vhosts and ports charts

commit 84733b6b1d353aff70687603019443610a8500c3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 12 21:40:16 2023 +0100

    Add systemd charts

commit 14673501e8f48560956f53d5b670bbe801b8f2ae
Author: Dim-P <dimitris1703@gmail.com>
Date:   Wed Oct 11 00:28:43 2023 +0100

    Add MQTT charts

commit 366eb63b0a27dde6f0f8ba65120f34c18c1b21fd
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 21:46:19 2023 +0100

    Complete kmsg changes. Reduce mem usage. Fix a dictionary key size bug

commit 3d0216365a526ffbc9ce13a20c45447bfccb47d9
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 19:18:41 2023 +0100

    Add kmsg Subsystem charts

commit e61af4bb130a5cf5a5a78133f1e44b2b4c457b24
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 16:21:29 2023 +0100

    Fix bug of wrong kmsg timestamps in case of use_log_timestamp == 0

commit 03d22e0b26bddf249aab431a4f977bbd5cde98ca
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 16:20:47 2023 +0100

    Add kmsg charts, except for Subsystem and Device

commit f60b0787537a21ed3c4cea5101fcddc50f3bc55a
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 13:12:13 2023 +0100

    Initialise all docker events chart dimensions at startup

commit 5d873d3439abaf3768530cb5b72c6b4ef6565353
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 10 00:53:35 2023 +0100

    WIP: Add Docker events logs

commit 2cc3d6d98f58fc3ab67a8da3014210b14d0926a1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 9 18:52:27 2023 +0100

    Use macros for num_of_logs_charts and custom_charts functions

commit fbd48ad3c9af674601238990d74192427475f2e3
Author: Dim-P <dimitris1703@gmail.com>
Date:   Mon Oct 9 18:26:17 2023 +0100

    Refactor custom charts code for clarity and speed

commit a31d80b5dc91161c0d74b10d00bc4fd1e6da7965
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 23:58:27 2023 +0100

    Add first working iteration of custom charts

commit b1e4ab8a460f4b4c3e2804e2f775787d21fbee45
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 23:57:27 2023 +0100

    Add more custom charts for Netdata error.log

commit f1b7605e564da3e297942f073593cdd4c21f88e1
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 20:39:40 2023 +0100

    Convert collected_logs_* chart updates to macros

commit 1459bc2b8bcd5ba21e024b10a8a5101048938f71
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Oct 5 19:11:54 2023 +0100

    Use rrdset_timed_done() instead of duration_since_last_update for correct chart timings

commit 876854c6ee7586a3eb9fdbf795bcc17a5fd1e6ad
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 3 21:53:14 2023 +0100

    Fix some bugs in chart updates

commit ae87508485499984bcb9b72bbc7d249c4168b380
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Oct 3 21:32:55 2023 +0100

    Functioning generic_chart_init() and generic_chart_update()

commit 982a9c4108dbea9571c785b5ff8a9d1e5472066c
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 23:32:52 2023 +0100

    Add support for multiple .conf files. Add stock examples.

commit 8e8abd0731227eb3fb3c6bcd811349575160799e
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 17:38:30 2023 +0100

    Add support for logsmanagement.d/default.conf

commit 1bf0732217b1d9e9959e1507ea96fc2c92ffb2ff
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 28 14:31:03 2023 +0100

    Add capabilities. Fix paths in logsmanagement.d.conf

commit a849d5b405bb4e5d770726fe99413a4efa7df274
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 26 23:06:31 2023 +0100

    Change logs_manag_config_load()

commit b0d1783b996286cd87e0832bfb74c29a845d61fc
Author: Dim-P <dimitris1703@gmail.com>
Date:   Tue Sep 26 15:35:30 2023 +0100

    Working unit tests and argument parsing

commit 6da1b4267a4d58d3a7cbcca9507afe8158a2e324
Author: Dim-P <dimitris1703@gmail.com>
Date:   Fri Sep 22 00:32:47 2023 +0300

    Build logs-management.plugin successfully

commit 9e30efe0422e4941f99cc66998d9f42e00a24676
Author: Dim-P <dimitris1703@gmail.com>
Date:   Thu Sep 21 16:13:21 2023 +0300

    Fix print format specifier in web_client_api_request_v1_logsmanagement()

* Modify CODEOWNERS

* Update README.md

Fix indentation

* Change snprintfz() to stncpyz() in circ_buff_search(). Change remaining 'chart_name' to 'chartname'.

* Replace SQLite version function with macro

* Fix some codacy warnings

* Update README.md

* Update Fluent Bit to v2.1.10

* Remove some comments

* Fix Fluent Bit shared library linking for armv7l and FreeBSD

* Remove compression source files

* Add prefix to rrd_api.h functions

* Add more unit tests

* Fix kmsg capabilities

* Separate kmsg and systemd default paths

* Fix some memory leaks and better termination of DB

* Add iterative queries if quota is exceeded

* Fix centos7 builds

* Fix issue where SYSTEMD timestamps are not parsed

* Fix logs management packaging.

* Fix typo in DEB control file.

* Fix indentation and missing new line at EOF

* Clean up functions and update help

* Fix 400 error when no queryable sources are available

* Fix if_modified_since. Add FACET_MAX_VALUE_LENGTH

* Add delta parameter and use anchor points in queries

* Fix CodeQL  warning

* Fix packaging issues.

* Fix postinstall script for DEB packages.

* Improve plugin shutdown speed

* Fix docker events chart grouping

* Fix functions evloop threads not terminating upon shutdown

* Fix coverity issues

* Fix logging

* Replace 'Netdata error.log' with 'Netdata daemon.log' in 'default.conf'

* Remove 'enabled = yes/no' config in logsmanagement.d.conf

* Remove 'enabled = X' unused config from logsmanagement.d.conf

---------

Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud>
2023-11-27 16:55:14 +00:00
Costa Tsaousis
1f0164ede4
Log2journal yaml configuration support ()
* allow command line parameters to be specified with = or as separate arguments without =; memory errors handling

* add yaml support to log2journal

* updated log2journal docs

* fix command line parsing

* fix codeql warnings
2023-11-27 13:05:53 +02:00
Netdata bot
81f3679d3a
Regenerate integrations.js ()
Co-authored-by: ilyam8 <ilyam8@users.noreply.github.com>
2023-11-27 10:11:22 +02:00
netdatabot
c3f2e47107 [ci skip] Update changelog and version for nightly build: v1.43.0-276-nightly. 2023-11-27 00:18:25 +00:00
Costa Tsaousis
a59097ca50
journals management improvements ()
* different approach to distribute estimatons evenly

* log estimations calculation to spot the issue

* better log

* more logs

* more logs

* more logs

* more logs

* commented logs

* process journal files and directories in reverse order, so that the system will immediately be available after start

* put the right filename in the dictionary

* fix scan dir recursion

* add missing fields when sending logs to systemd-journal-remote

* the standard datetime format is now switched to rfc3339
2023-11-26 02:22:47 +02:00
netdatabot
e08ce9a587 [ci skip] Update changelog and version for nightly build: v1.43.0-274-nightly. 2023-11-25 00:17:34 +00:00
Stelios Fragkakis
e7801f7a72
Fix error limit to respect the log every ()
Set last_logged time
2023-11-24 19:31:09 +02:00
Costa Tsaousis
a8ed7d2d03
Journal better estimations and watcher ()
* get the number of lines per journal file

* accurate estimates when using systemd 254+

* scan the journal files in reverse order, the newest first

* disable running journalctl; use the file last modification timestamp if the messages cannot be retrieved

* attempt to fix order of files scanned

* find the timestamps of all boot_ids during header updates

* added inotify watcher for detecting journal file changes

* handle all cases in watcher

* completed watcher

* remove debug info

* watcher now restarts when it gets inotify errors

* removed log

* do not remain the main thread of netdata
2023-11-24 18:48:42 +02:00
Satyadeep Ashwathnarayana
0f466bba47
SEO changes for Collector names () 2023-11-24 16:12:15 +02:00
Ilya Mashchenko
df8a577c95
fix charts.d plugin loading configuration () 2023-11-24 12:44:58 +02:00
netdatabot
0225b8d01b [ci skip] Update changelog and version for nightly build: v1.43.0-269-nightly. 2023-11-24 00:17:35 +00:00
Stelios Fragkakis
85f359fc26
Handle ephemeral hosts ()
* Handle ephemeral hosts

* Node empheral removal timeout 86400 seconds (1 day)

* Move config from health to global section

* Set a node to queryable false when it is ephemeral and is removed

* Log queryable. Send queryable=0 only when forcing host deletion (the node is ephemeral)

* Switch to "is ephemeral node"
Document stream.conf

* Unregister node id
2023-11-23 23:56:34 +02:00
Ilya Mashchenko
a721fcb61f
update go.d plugin version to v0.57.1 () 2023-11-23 15:32:57 +02:00
vkalintiris
880c8d78f3
Update non-zero uuid key + child conf. () 2023-11-23 13:50:46 +02:00
vkalintiris
77b081a59e
Add option to disable ML. () 2023-11-23 13:17:45 +02:00
netdatabot
673c0e334e [ci skip] Update changelog and version for nightly build: v1.43.0-264-nightly. 2023-11-23 00:18:07 +00:00
Costa Tsaousis
90d58688d8
fix analytics logs () 2023-11-23 00:54:53 +02:00
Costa Tsaousis
e1b95a1db8
fix logs bashism ()
fix logs of ioping
2023-11-23 00:36:48 +02:00
Costa Tsaousis
d5143c1573
fix log2journal incorrect log () 2023-11-22 23:56:05 +02:00
Ilya Mashchenko
6302005bd6
add LogNamespace to systemd units () 2023-11-22 23:48:07 +02:00
Costa Tsaousis
5230b15f59
fixes for logging ()
* fixes for logging

* added log environment variables to cgroup-network

* fix wrong condition

* rename variable

* fix leftovers

* fix log2journal docs and logs
2023-11-22 23:47:55 +02:00
Ilya Mashchenko
ae733dd494
add sbindir_POST to PATH of bash scripts that use systemd-cat-native ()
* use sbindir_POST in charts.d and alarm-notify

* convert cgroup-name.sh to in

* convert cgroup-network-helper.sh to in

* simplify cgroup-network-helper
2023-11-22 22:14:00 +02:00
Ilya Mashchenko
2a58bd96ff
log flood protection to 1000 log lines / 1 minute () 2023-11-22 22:12:41 +02:00
Costa Tsaousis
53d4c224ea
when the namespace socket does not work, continue trying () 2023-11-22 22:03:02 +02:00
Costa Tsaousis
a5c8d88d78
set journal path for logging () 2023-11-22 21:41:23 +02:00
vkalintiris
7cbdc17d14
Add missing argument. () 2023-11-22 18:44:02 +02:00
Costa Tsaousis
9b83b2b2f5
journal estimations ()
* preliminary work to enable estimations

* working estimations

* clear estimated counter when switching files

* use the actual query timeframe for estimations

* properly estimate the remaining lines in the files

* fix the query duration according to file limits

* require at least 2 seconds of data per file

* estimation starts at 5% of each file

* work on estimation

* distribute estimations accurately on the histogram

* stop query permaturely if it is going to timeout

* more aggressive estimation of next file duration

* added missing dimension names in histogram.chart.summary.dimension; reordered estimated and unsampled; added histogram.chart.view.dimensions.colors to estimated and unsampled; added message

* include message always

* message is sent on non-data-only queries; empty message object when there is nothing to tell

* use syslog priorities

* use standard netdata log priorities
2023-11-22 17:50:36 +02:00
Ilya Mashchenko
7b04af1f3c
fix: link daemon.log to stderr in docker () 2023-11-22 15:13:11 +02:00
Costa Tsaousis
3e508c8f95
New logging layer ()
* cleanup of logging - wip

* first working iteration

* add errno annotator

* replace old logging functions with netdata_logger()

* cleanup

* update error_limit

* fix remanining error_limit references

* work on fatal()

* started working on structured logs

* full cleanup

* default logging to files; fix all plugins initialization

* fix formatting of numbers

* cleanup and reorg

* fix coverity issues

* cleanup obsolete code

* fix formatting of numbers

* fix log rotation

* fix for older systems

* add detection of systemd journal via stderr

* finished on access.log

* remove left-over transport

* do not add empty fields to the logs

* journal get compact uuids; X-Transaction-ID header is added in web responses

* allow compiling on systems without memfd sealing

* added libnetdata/uuid directory

* move datetime formatters to libnetdata

* add missing files

* link the makefiles in libnetdata

* added uuid_parse_flexi() to parse UUIDs with and without hyphens; the web server now read X-Transaction-ID and uses it for functions and web responses

* added stream receiver, sender, proc plugin and pluginsd log stack

* iso8601 advanced usage; line_splitter module in libnetdata; code cleanup

* add message ids to streaming inbound and outbound connections

* cleanup line_splitter between lines to avoid logging garbage; when killing children, kill them with SIGABRT if internal checks is enabled

* send SIGABRT to external plugins only if we are not shutting down

* fix cross cleanup in pluginsd parser

* fatal when there is a stack error in logs

* compile netdata with -fexceptions

* do not kill external plugins with SIGABRT

* metasync info logs to debug level

* added severity to logs

* added json output; added options per log output; added documentation; fixed issues mentioned

* allow memfd only on linux

* moved journal low level functions to journal.c/h

* move health logs to daemon.log with proper priorities

* fixed a couple of bugs; health log in journal

* updated docs

* systemd-cat-native command to push structured logs to journal from the command line

* fix makefiles

* restored NETDATA_LOG_SEVERITY_LEVEL

* fix makefiles

* systemd-cat-native can also work as the logger of Netdata scripts

* do not require a socket to systemd-journal to log-as-netdata

* alarm notify logs in native format

* properly compare log ids

* fatals log alerts; alarm-notify.sh working

* fix overflow warning

* alarm-notify.sh now logs the request (command line)

* anotate external plugins logs with the function cmd they run

* added context, component and type to alarm-notify.sh; shell sanitization removes control character and characters that may be expanded by bash

* reformatted alarm-notify logs

* unify cgroup-network-helper.sh

* added quotes around params

* charts.d.plugin switched logging to journal native

* quotes for logfmt

* unify the status codes of streaming receivers and senders

* alarm-notify: dont log anything, if there is nothing to do

* all external plugins log to stderr when running outside netdata; alarm-notify now shows an error when notifications menthod are needed but are not available

* migrate cgroup-name.sh to new logging

* systemd-cat-native now supports messages with newlines

* socket.c logs use priority

* cleanup log field types

* inherit the systemd set INVOCATION_ID if found

* allow systemd-cat-native to send messages to a systemd-journal-remote URL

* log2journal command that can convert structured logs to journal export format

* various fixes and documentation of log2journal

* updated log2journal docs

* updated log2journal docs

* updated documentation of fields

* allow compiling without libcurl

* do not use socket as format string

* added version information to newly added tools

* updated documentation and help messages

* fix the namespace socket path

* print errno with error

* do not timeout

* updated docs

* updated docs

* updated docs

* log2journal updated docs and params

* when talking to a remote journal, systemd-cat-native batches the messages

* enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote

* Revert "enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote"

This reverts commit b079d53c11.

* note about uncompressed traffic

* log2journal: code reorg and cleanup to make modular

* finished rewriting log2journal

* more comments

* rewriting rules support

* increased limits

* updated docs

* updated docs

* fix old log call

* use journal only when stderr is connected to journal

* update netdata.spec for libcurl, libpcre2 and log2journal

* pcre2-devel

* do not require pcre2 in centos < 8, amazonlinux < 2023, open suse

* log2journal only on systems pcre2 is available

* ignore log2journal in .gitignore

* avoid log2journal on centos 7, amazonlinux 2 and opensuse

* add pcre2-8 to static build

* undo last commit

* Bundle to static

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add build deps for deb packages

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add dependencies; build from source

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Test build for amazon linux and centos expect to fail for suse

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* fix minor oversight

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Reorg code

* Add the install from source (deps) as a TODO
* Not enable the build on suse ecosystem

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

---------

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>
2023-11-22 10:27:25 +02:00
netdatabot
8f31356a0c [ci skip] Update changelog and version for nightly build: v1.43.0-250-nightly. 2023-11-22 00:18:55 +00:00
Luis Johnstone
3acbb6b5ef
Doc change: Curl no longer supports spaces in the URL. ()
Replaced spaces with: %20
2023-11-22 01:29:02 +02:00
vkalintiris
b515c74228
Add support for gorilla pages for tier 0. ()
---------

Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
2023-11-21 21:42:59 +02:00
Costa Tsaousis
7521dfae3a
journal startup ()
* change the main loop of systemd-journal.plugin so that it registers functions before scanning files, pings netdata while scanning files and allows only 1 scan at a time

* do not lock the output on startup

* properly register functions on startup
2023-11-21 16:44:22 +02:00
Shyam Sreevalsan
f21b7ff077
Fix icon filename () 2023-11-21 14:45:19 +02:00
Stelios Fragkakis
85d4369435
Remove queue limit from ACLK sync event loop ()
Code cleanup
2023-11-21 12:00:51 +02:00
Emmanuel Vasilakis
c0feaec456
Minor: Small health docs typo fix ()
small typo fix
2023-11-21 11:03:52 +02:00
netdatabot
c9e081ef8d [ci skip] Update changelog and version for nightly build: v1.43.0-243-nightly. 2023-11-21 00:19:01 +00:00
Aliki92
a177e22447
Removes Observabilitycon banner README.md () 2023-11-20 13:44:29 +02:00
Costa Tsaousis
8d83b5897b
Journal sampling ()
* initial work to support sampling in journal logs

* always sample and entry that is useful as an item to return

* add [unsampled] to the dashboard

* facets sampling updates only the histogram

* add unsampled to facets items

* updated default query
2023-11-20 12:13:23 +02:00
netdatabot
d5d3cfa227 [ci skip] Update changelog and version for nightly build: v1.43.0-240-nightly. 2023-11-18 00:18:04 +00:00
Shyam Sreevalsan
adbee25bc3
Update netdata-functions.md ()
Co-authored-by: ilyam8 <ilya@netdata.cloud>
2023-11-17 19:57:35 +02:00
Netdata bot
4787184e7d
Regenerate integrations.js ()
Co-authored-by: tkatsoulas <tkatsoulas@users.noreply.github.com>
2023-11-17 19:03:31 +02:00