apps.plugin improvements (#18652)

* apps.plugin now supports simple patterns when an asterisk is in the middle of a match; expanded kernel threads matching to group them into meaningful entities

* removed cli tools

* systemd merged

* apps.plugin now has the option to print the tree with the target assignment

* apps.plugin now extracts the full comm name from the cmdline

* optimizations

* updated windows comm handling

* get the full command line on windows

* extract service names for svchost.exe processes

* get service names from SCM

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* Update src/collectors/apps.plugin/README.md

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

* fix compilation on freebsd and macos

* windows priveleges

* add missing opening quote on windows spawn server

* fix alerts notifications infinite loop when alarm-notify.sh cannot be executed

---------

Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>

CMakeLists.txt

@@ -1895,12 +1895,14 @@ if(ENABLE_PLUGIN_APPS)
             src/collectors/apps.plugin/apps_os_macos.c
             src/collectors/apps.plugin/apps_os_windows.c
             src/collectors/apps.plugin/apps_incremental_collection.c
+            src/collectors/apps.plugin/apps_os_windows_nt.c
     )
 
     add_executable(apps.plugin ${APPS_PLUGIN_FILES})
 
     target_link_libraries(apps.plugin libnetdata ${CAP_LIBRARIES}
-            "$<$<BOOL:${OS_WINDOWS}>:Version>")
+            "$<$<BOOL:${OS_WINDOWS}>:Version>"
+            "$<$<BOOL:${OS_WINDOWS}>:ntdll>")
 
     target_include_directories(apps.plugin PRIVATE ${CAP_INCLUDE_DIRS})
     target_compile_options(apps.plugin PRIVATE ${CAP_CFLAGS_OTHER})

src/collectors/apps.plugin/README.md

@@ -132,6 +132,26 @@ its CPU resources will be cut in half, and data collection will be once every 2
 
 The configuration file is `/etc/netdata/apps_groups.conf`. You can edit this file using our [`edit-config`](docs/netdata-agent/configuration/README.md) script.
 
+### Configuring process managers
+
+`apps.plugin` needs to know the common process managers, meaning the names of the processes
+which spawn other processes. Process managers are used so that `apps.plugin` will automatically
+consider all their sub-processes important to monitor.
+
+Process managers are configured in `apps_groups.conf` with the prefix `managers:`, like this:
+
+```
+managers: process1 process2 process3
+```
+
+Multiple lines may exist, all starting with `managers:`.
+
+The process names given here should be exactly as the operating system sets them. In Linux these
+process names are limited to 15 characters. Usually the command `ps -e` or `cat /proc/{PID}/stat`
+states the names needed here.
+
+### Configuring process groups and renaming processes
+
 The configuration file works accepts multiple lines, each having this format:
 
 ```txt
@@ -140,48 +160,39 @@ group: process1 process2 ...
 
 Each group can be given multiple times, to add more processes to it.
 
-For the **Applications** section, only groups configured in this file are reported.
+For each process given, all of its sub-processes will be grouped, not just the matched process.
-All other processes will be reported as `other`.
-
-For each process given, its whole process tree will be grouped, not just the process matched.
-The plugin will include both parents and children. If including the parents into the group is
-undesirable, the line `other: *` should be appended to the `apps_groups.conf`.
 
 The process names are the ones returned by:
 
--   `ps -e` or `cat /proc/PID/stat`
+-   **comm**: `ps -e` or `cat /proc/{PID}/stat`
--   in case of substring mode (see below): `/proc/PID/cmdline`
+-   **cmdline**: in case of substring mode (see below): `/proc/{PID}/cmdline`
+
+On Linux **comm** is limited to just a few characters. `apps.plugin` attempts to find the entire
+**comm** name by looking for it at the **cmdline**. When this is successful, the entire process name
+is available, otherwise the shortened one is used.
 
 To add process names with spaces, enclose them in quotes (single or double)
 example: `'Plex Media Serv'` or `"my other process"`.
 
-You can add an asterisk `*` at the beginning and/or the end of a process:
+You can add asterisks (`*`) to provide a pattern:
 
--   `*name` _suffix_ mode: will search for processes ending with `name` (at `/proc/PID/stat`)
+-   `*name` _suffix_ mode: will match a **comm** ending with `name`.
--   `name*` _prefix_ mode: will search for processes beginning with `name` (at `/proc/PID/stat`)
+-   `name*` _prefix_ mode: will match a **comm** beginning with `name`.
--   `*name*` _substring_ mode: will search for `name` in the whole command line (at `/proc/PID/cmdline`)
+-   `*name*` _substring_ mode: will search for `name` in **cmdline**.
 
-If you enter even just one _name_ (substring), `apps.plugin` will process
+Asterisks may appear in the middle of `name` (like `na*me`), without affecting what is being
-`/proc/PID/cmdline` for all processes (of course only once per process: when they are first seen).
+matched (**comm** or **cmdline**). 
 
 To add processes with single quotes, enclose them in double quotes: `"process with this ' single quote"`
 
 To add processes with double quotes, enclose them in single quotes: `'process with this " double quote'`
 
-If a group or process name starts with a `-`, the dimension will be hidden from the chart (cpu chart only).
+The order of the entries in this list is important: the first one that matches a process is used, so follow a top-down hierarchy.
+Processes not matched by any row, will inherit it from their parents.
 
-If a process starts with a `+`, debugging will be enabled for it (debugging produces a lot of output - do not enable it in production systems).
+There are a few command line options you can pass to `apps.plugin`. The list of available
-
+options can be acquired with the `--help` flag. The options can be set in the `netdata.conf` using the [`edit-config` script](/docs/netdata-agent/configuration/README.md).
-You can add any number of groups. Only the ones found running will affect the charts generated.
+For example, to disable user and user group charts you would set:
-However, producing charts with hundreds of dimensions may slow down your web browser.
-
-The order of the entries in this list is important: the first that matches a process is used, so put important
-ones at the top. Processes not matched by any row, will inherit it from their parents or children.
-
-The order also controls the order of the dimensions on the generated charts (although applications started
-after apps.plugin is started, will be appended to the existing list of dimensions the `netdata` daemon maintains).
-
-There are a few command line options you can pass to `apps.plugin`. The list of available options can be acquired with the `--help` flag. The options can be set in the `netdata.conf` file. For example, to disable user and user group charts you should set
 
 ```
 [plugin:apps]

src/collectors/apps.plugin/apps_aggregations.c

@@ -74,7 +74,7 @@ static inline void aggregate_pid_on_target(struct target *w, struct pid_stat *p,
     if(!w->uptime_min || p->values[PDF_UPTIME] < w->uptime_min) w->uptime_min = p->values[PDF_UPTIME];
     if(!w->uptime_max || w->uptime_max < p->values[PDF_UPTIME]) w->uptime_max = p->values[PDF_UPTIME];
 
-    if(unlikely(debug_enabled || w->debug_enabled)) {
+    if(unlikely(debug_enabled)) {
         struct pid_on_target *pid_on_target = mallocz(sizeof(struct pid_on_target));
         pid_on_target->pid = p->pid;
         pid_on_target->next = w->root_pid;
@@ -110,27 +110,61 @@ static inline void cleanup_exited_pids(void) {
     }
 }
 
-static struct target *get_app_group_target_for_pid(struct pid_stat *p) {
+static struct target *matched_apps_groups_target(struct pid_stat *p, struct target *w) {
+    if(is_process_manager(p))
+        return NULL;
+
+    p->matched_by_config = true;
+    return w->target ? w->target : w;
+}
+
+static struct target *get_apps_groups_target_for_pid(struct pid_stat *p) {
     targets_assignment_counter++;
 
     for(struct target *w = apps_groups_root_target; w ; w = w->next) {
         if(w->type != TARGET_TYPE_APP_GROUP) continue;
 
-        // find it - 4 cases:
+        if(!w->starts_with && !w->ends_with) {
-        // 1. the target is not a pattern
+            if(w->ag.pattern) {
-        // 2. the target has the prefix
+                if(simple_pattern_matches_string(w->ag.pattern, p->comm))
-        // 3. the target has the suffix
+                    return matched_apps_groups_target(p, w);
-        // 4. the target is something inside cmdline
+            }
-
+            else {
-        if(unlikely(( (!w->starts_with && !w->ends_with && w->compare == p->comm)
+                if(w->ag.compare == p->comm || w->ag.compare == p->comm_orig)
-                      || (w->starts_with && !w->ends_with && string_starts_with_string(p->comm, w->compare))
+                    return matched_apps_groups_target(p, w);
-                      || (!w->starts_with && w->ends_with && string_ends_with_string(p->comm, w->compare))
+            }
-                      || (proc_pid_cmdline_is_needed && w->starts_with && w->ends_with && strstr(pid_stat_cmdline(p), string2str(w->compare)))
+        }
-                          ))) {
+        else if(w->starts_with && !w->ends_with) {
-
+            if(w->ag.pattern) {
-            p->matched_by_config = true;
+                if(simple_pattern_matches_string(w->ag.pattern, p->comm))
-            if(w->target) return w->target;
+                    return matched_apps_groups_target(p, w);
-            else return w;
+            }
+            else {
+                if(string_starts_with_string(p->comm, w->ag.compare) ||
+                    (p->comm != p->comm_orig && string_starts_with_string(p->comm, w->ag.compare)))
+                    return matched_apps_groups_target(p, w);
+            }
+        }
+        else if(!w->starts_with && w->ends_with) {
+            if(w->ag.pattern) {
+                if(simple_pattern_matches_string(w->ag.pattern, p->comm))
+                    return matched_apps_groups_target(p, w);
+            }
+            else {
+                if(string_ends_with_string(p->comm, w->ag.compare) ||
+                    (p->comm != p->comm_orig && string_ends_with_string(p->comm, w->ag.compare)))
+                    return matched_apps_groups_target(p, w);
+            }
+        }
+        else if(w->starts_with && w->ends_with && p->cmdline) {
+            if(w->ag.pattern) {
+                if(simple_pattern_matches_string(w->ag.pattern, p->cmdline))
+                    return matched_apps_groups_target(p, w);
+            }
+            else {
+                if(strstr(string2str(p->cmdline), string2str(w->ag.compare)))
+                    return matched_apps_groups_target(p, w);
+            }
         }
     }
 
@@ -141,19 +175,23 @@ static void assign_a_target_to_all_processes(void) {
     // assign targets from app_groups.conf
     for(struct pid_stat *p = root_of_pids(); p ; p = p->next) {
         if(!p->target)
-            p->target = get_app_group_target_for_pid(p);
+            p->target = get_apps_groups_target_for_pid(p);
     }
 
     // assign targets from their parents, if they have
     for(struct pid_stat *p = root_of_pids(); p ; p = p->next) {
         if(!p->target) {
-            for(struct pid_stat *pp = p->parent ; pp ; pp = pp->parent) {
+            if(!p->is_manager) {
-                if(pp->target) {
+                for (struct pid_stat *pp = p->parent; pp; pp = pp->parent) {
-                    if(pp->matched_by_config) {
+                    if(pp->is_manager) break;
-                        // we are only interested about app_groups.conf matches
+
-                        p->target = pp->target;
+                    if (pp->target) {
+                        if (pp->matched_by_config) {
+                            // we are only interested about app_groups.conf matches
+                            p->target = pp->target;
+                        }
+                        break;
                     }
-                    break;
                 }
             }
 
@@ -180,6 +218,7 @@ void aggregate_processes_to_targets(void) {
 
     // this has to be done, before the cleanup
     struct target *w = NULL, *o = NULL;
+    (void)w; (void)o;
 
     // concentrate everything on the targets
     for(struct pid_stat *p = root_of_pids(); p ; p = p->next) {

src/collectors/apps.plugin/apps_groups.conf

@@ -4,19 +4,21 @@
 ## Documentation at:
 ## https://github.com/netdata/netdata/blob/master/src/collectors/apps.plugin/README.md
 ##
-## The list of process managers can be configured here (uncomment and edit):
+## Subprocesses of process managers are monitored.
+## (uncomment to edit - the default is also hardcoded into the plugin)
 
-## Linux
+## Linux process managers
-#managers: init systemd containerd-shim dumb-init gnome-shell docker-init
+#managers: init systemd containerd-shim-runc-v2 dumb-init gnome-shell docker-init
+#managers: openrc-run.sh crond plasmashell xfwm4
 
-## FreeBSD
+## FreeBSD process managers
 #managers: init
 
-## MacOS
+## MacOS process managers
 #managers: launchd
 
-## Windows
+## Windows process managers
-#managers: System services wininit
+#managers: wininit services explorer System
 
 ## -----------------------------------------------------------------------------
 ## Processes of interest
@@ -26,23 +28,23 @@ netdata: netdata
 ## netdata known plugins
 ## plugins not defined here will be accumulated into netdata, above
 apps.plugin: *apps.plugin*
-freeipmi.plugin: *freeipmi.plugin*
+go.d.plugin: *go.d.plugin*
-nfacct.plugin: *nfacct.plugin*
-cups.plugin: *cups.plugin*
-xenstat.plugin: *xenstat.plugin*
-perf.plugin: *perf.plugin*
-charts.d.plugin: *charts.d.plugin*
-python.d.plugin: *python.d.plugin*
 systemd-journal.plugin: *systemd-journal.plugin*
 network-viewer.plugin: *network-viewer.plugin*
 windows-events.plugin: *windows-events.plugin*
-tc-qos-helper: *tc-qos-helper.sh*
+cups.plugin: *cups.plugin*
-fping: fping
+perf.plugin: *perf.plugin*
-ioping: ioping
+nfacct.plugin: *nfacct.plugin*
-go.d.plugin: *go.d.plugin*
+xenstat.plugin: *xenstat.plugin*
+freeipmi.plugin: *freeipmi.plugin*
+charts.d.plugin: *charts.d.plugin*
+python.d.plugin: *python.d.plugin*
 slabinfo.plugin: *slabinfo.plugin*
 ebpf.plugin: *ebpf.plugin*
 debugfs.plugin: *debugfs.plugin*
+tc-qos-helper: *tc-qos-helper.sh*
+fping: fping
+ioping: ioping
 
 ## agent-service-discovery
 agent_sd: agent_sd
@@ -65,32 +67,26 @@ azure: mdsd *waagent* *omiserver* *omiagent* hv_kvp_daemon hv_vss_daemon *auoms*
 datadog: *datadog*
 newrelic: newrelic*
 google-agent: *google_guest_agent* *google_osconfig_agent*
-ceph: ceph-* ceph_* radosgw* rbd-* cephfs-* osdmaptool crushtool
-samba: smbd nmbd winbindd ctdbd ctdb-* ctdb_*
-nfs: rpcbind rpc.* nfs*
-zfs: spl_* z_* txg_* zil_* arc_* l2arc*
-iscsi: iscsid iscsi_eh
-afp: netatalk afpd cnid_dbd cnid_metad
 aws-s3: '*aws s3*' s3cmd s5cmd
 proxmox-ve: pve* spiceproxy
 libvirt: virtlogd virtqemud virtstoraged virtnetworkd virtlockd virtinterfaced
 libvirt: virtnodedevd virtproxyd virtsecretd libvirtd
 guest-agent: qemu-ga spice-vdagent cloud-init*
-dhcp: *dhcp* dhclient
+dhcp: dhcp* dhclient
 
 build: cc1 cc1plus as gcc* cppcheck ld make cmake automake autoconf autoreconf
 build: cargo rustc bazel buck git gdb valgrind* rpmbuild dpkg-buildpackage
-packagemanager: apt* dpkg* dselect dnf yum rpm zypp* yast* pacman xbps* swupd* emerge*
+packagemanager: apt* dpkg* dselect dnf yum rpm zypp* yast* pacman xbps* swupd*
-packagemanager: packagekitd pkgin pkg apk snapd slackpkg slapt-get
+packagemanager: packagekitd pkgin pkg apk snapd slackpkg slapt-get emerge*
 clam: clam* *clam
 backup: rsync lsyncd bacula* borg rclone
 cron: cron* atd anacron *systemd-cron* incrond
 ups: upsmon upsd */nut/* apcupsd
-audio: pulse* pipewire wireplumber jack*
 
 rabbitmq: *rabbitmq*
 sidekiq: *sidekiq*
 erlang: beam.smp
+postfix: *postfix*
 
 ## -----------------------------------------------------------------------------
 ## java applications
@@ -117,12 +113,134 @@ kafka: *kafka.Kafka*
 
 ## -----------------------------------------------------------------------------
 ## Kernel / System
+## The following are interesting kernel threads and related processes to
+## monitor individually, mainly for their CPU utilization.
 
+## These kernel threads switch tasks all the time, so they should never be
+## categorized as anything specific.
+kernel: kworker/*
+
+## Kernel Samepage Merging (KSM) daemon that looks for identical memory pages
+## across processes and merges them to save memory.
 ksmd: ksmd
-khugepaged: khugepaged
+
+## Handles migration of processes between CPU cores to balance load.
+kmigration: migration/*
+
+## Manages memory compaction, moving memory pages around to reduce
+## fragmentation.
+kcompactd: kcompactd*
+
+## Responsible for freeing up memory by swapping pages to disk when needed.
+kswapd: kswapd*
+
+## DAMON is a mechanism designed to efficiently monitor the memory access
+## patterns of running processes or the system itself.
 kdamond: kdamond
-kswapd: kswapd
+
-zswap: zswap
+## Manages ballooning in virtualized environments.
-kcompactd: kcompactd
+vballoon: vballoon*
-ipvs: ipvs_*
+
+## virtio - Handles or I/O (storage and network) on virtual machines.
+kvirtio: virtio-* vhost-*
+
+## Layer 4 (transport layer) load balancing
+ipvs: ipvsd ipvs_* ip_vs_*
+
+## Hugepages
+## Scans memory regions and tries to promote regular-sized pages (4KB) into
+## hugepages (2MB) where possible. Merge smaller contiguous 4KB pages into 2MB
+## pages. Hugepages also use: kswapd, kcompactd, and migration.
+khugepaged: khugepaged
+
+## Note about zswap:
+## zswap does not introduce its own dedicated kernel threads. Instead, it
+## operates within the existing memory management and swapping framework of the
+## kernel:
+## - kswapd: swaps pages in/out of memory, using compression in the process.
+## - kcompactd: compacts memory when pages are compressed or moved around.
+
+## -----------------------------------------------------------------------------
+## Block Devices
+
+## Handles deferred block I/O operations for block devices.
+kblockd: kblockd
+
+## Device Mapper (DM)
+device-mapper: kcopyd/* kcryptd/* kdmflush/* dm_bufio_cache
+device-mapper: raid1/* raid5/* raid10/* multipathd bioset/*
+
+## Software RAID (MD)
+md-raid: md*_raid* md*_resync md*_reshape md*_recovery md_thread
+md-raid: flush_md* raid*_sync
+
+## iSCSI
+iscsi: iscsid iscsiadm iscsi_eh/* iscsi_xmit/* iscsi_ttx/* iscsi_rx/* iscsi_trx/*
+
+## SCSI
+scsi: scsi_eh/* scsi_tmf/* scsi_wq/*
+
+## BCACHE
+bcache: bcache* bch_btree_io bch_journal
+
+## SAS
+sas: sas_task/* mpt*
+
+## Fibre Channel (FC)
+fc: fc_transport qla2xxx*
+
+## loop devices
+loop: loop* flush-loop*
+
+## -----------------------------------------------------------------------------
+## Filesystems
+
+## Ext4
+ext4: ext4-* jbd2/*
+
+## XFS
+xfs: xfs*
+
+## BTRFS
 btrfs: btrfs*
+
+## NFS
+nfs: rpcbind rpc.* nfs* rpciod
+
+## ZFS
+zfs: spl_* z_* txg_* zil_* arc_* l2arc* zfs* zed zdb zpool*
+
+## CEPH
+ceph: ceph-* ceph_* radosgw* rbd-* cephfs-*
+ceph: ceph cephadm osdmaptool crushtool rados rbd
+
+## CIFS & Samba
+cifs: smbd nmbd winbindd ctdbd ctdb-* ctdb_*
+cifs: cifsd cifscreds cifs.upcall
+
+## Apple Filling Protocol (AFP)
+afp: netatalk afpd cnid_dbd cnid_metad
+
+## -----------------------------------------------------------------------------
+## Desktops
+
+systemd-journald: *systemd-journal*
+systemd: systemd systemd-*
+
+## GNOME
+desktop: gnome-* gsd-* gjs goa-* gcr-* gvfs-* *xdg-*-gnome* passimd gvfsd*
+desktop: at-spi-* at-spi2-* dconf-service gcr-*
+
+## KDE
+desktop: plasmashell kwin-* kde* *-kde-* klauncher kactivitymanagerd krunner
+desktop: kdeconnectd ksmserver kglobalaccel5 plasma-* *org.kde.*
+desktop: sddm* kwalletd5 knotify5 kmix kscreen kwayland-*
+
+## XFCE4
+desktop: xfce4-* xfwm4 xfdesktop xfce4-panel xfsettingsd xfconfd
+desktop: lightdm lightdm-*
+
+## Generic tools related to desktop
+desktop: gdm gdm-* dbus-* xdg-* ibus-* evolution-* accounts-daemon colord
+desktop: geoclue pulse* pipewire* wireplumber jack* touchegg pulseaudio
+desktop: Xwayland Xorg

src/collectors/apps.plugin/apps_os_freebsd.c

@@ -291,7 +291,7 @@ bool apps_os_read_pid_stat_freebsd(struct pid_stat *p, void *ptr) {
     usec_t started_ut = timeval_usec(&proc_info->ki_start);
     p->values[PDF_UPTIME] = (system_current_time_ut > started_ut) ? (system_current_time_ut - started_ut) / USEC_PER_SEC : 0;
 
-    if(unlikely(debug_enabled || (p->target && p->target->debug_enabled)))
+    if(unlikely(debug_enabled || p->target))
         debug_log_int("READ PROC/PID/STAT: %s/proc/%d/stat, process: '%s' on target '%s' (dt=%llu) VALUES: utime=" KERNEL_UINT_FORMAT ", stime=" KERNEL_UINT_FORMAT ", cutime=" KERNEL_UINT_FORMAT ", cstime=" KERNEL_UINT_FORMAT ", minflt=" KERNEL_UINT_FORMAT ", majflt=" KERNEL_UINT_FORMAT ", cminflt=" KERNEL_UINT_FORMAT ", cmajflt=" KERNEL_UINT_FORMAT ", threads=%d",
                       netdata_configured_host_prefix, p->pid, pid_stat_comm(p), (p->target)?string2str(p->target->name):"UNSET",
                       p->stat_collected_usec - p->last_stat_collected_usec,

src/collectors/apps.plugin/apps_os_linux.c

@@ -93,7 +93,7 @@ bool apps_os_read_pid_fds_linux(struct pid_stat *p, void *ptr __maybe_unused) {
         if(unlikely(l == -1)) {
             // cannot read the link
 
-            if(debug_enabled || (p->target && p->target->debug_enabled))
+            if(debug_enabled)
                 netdata_log_error("Cannot read link %s", p->fds[fdid].filename);
 
             if(unlikely(p->fds[fdid].fd < 0)) {
@@ -689,7 +689,7 @@ bool apps_os_read_pid_stat_linux(struct pid_stat *p, void *ptr __maybe_unused) {
         }
     }
 
-    if(unlikely(debug_enabled || (p->target && p->target->debug_enabled)))
+    if(unlikely(debug_enabled))
         debug_log_int("READ PROC/PID/STAT: %s/proc/%d/stat, process: '%s' on target '%s' (dt=%llu) VALUES: utime=" KERNEL_UINT_FORMAT ", stime=" KERNEL_UINT_FORMAT ", cutime=" KERNEL_UINT_FORMAT ", cstime=" KERNEL_UINT_FORMAT ", minflt=" KERNEL_UINT_FORMAT ", majflt=" KERNEL_UINT_FORMAT ", cminflt=" KERNEL_UINT_FORMAT ", cmajflt=" KERNEL_UINT_FORMAT ", threads=" KERNEL_UINT_FORMAT,
                       netdata_configured_host_prefix, p->pid, pid_stat_comm(p), (p->target)?string2str(p->target->name):"UNSET", p->stat_collected_usec - p->last_stat_collected_usec,
                       p->values[PDF_UTIME],

src/collectors/apps.plugin/apps_os_macos.c

@@ -242,7 +242,7 @@ bool apps_os_read_pid_stat_macos(struct pid_stat *p, void *ptr) {
     // Note: Some values such as guest time, cutime, cstime, etc., are not directly available in MacOS.
     // You might need to approximate or leave them unset depending on your needs.
 
-    if(unlikely(debug_enabled || (p->target && p->target->debug_enabled))) {
+    if(unlikely(debug_enabled || p->target)) {
         debug_log_int("READ PROC/PID/STAT for MacOS: process: '%s' on target '%s' VALUES: utime=" KERNEL_UINT_FORMAT ", stime=" KERNEL_UINT_FORMAT ", minflt=" KERNEL_UINT_FORMAT ", majflt=" KERNEL_UINT_FORMAT ", threads=%d",
                       pid_stat_comm(p), (p->target) ? string2str(p->target->name) : "UNSET",
                       p->values[PDF_UTIME],

src/collectors/apps.plugin/apps_os_windows.c

@@ -451,6 +451,8 @@
 #include <tchar.h>
 #include <strsafe.h>
 
+WCHAR* GetProcessCommandLine(HANDLE hProcess);
+
 struct perflib_data {
     PERF_DATA_BLOCK *pDataBlock;
     PERF_OBJECT_TYPE *pObjectType;
@@ -458,34 +460,17 @@ struct perflib_data {
     DWORD pid;
 };
 
-BOOL EnableDebugPrivilege() {
-    HANDLE hToken;
-    LUID luid;
-    TOKEN_PRIVILEGES tkp;
-
-    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken))
-        return FALSE;
-
-    if (!LookupPrivilegeValue(NULL, SE_DEBUG_NAME, &luid))
-        return FALSE;
-
-    tkp.PrivilegeCount = 1;
-    tkp.Privileges[0].Luid = luid;
-    tkp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
-
-    if (!AdjustTokenPrivileges(hToken, FALSE, &tkp, sizeof(tkp), NULL, NULL))
-        return FALSE;
-
-    CloseHandle(hToken);
-
-    return TRUE;
-}
-
 void apps_os_init_windows(void) {
     PerflibNamesRegistryInitialize();
 
-    if(!EnableDebugPrivilege())
+    if(!EnableWindowsPrivilege(SE_DEBUG_NAME))
-        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable debug privilege");
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_DEBUG_NAME);
+
+    if(!EnableWindowsPrivilege(SE_SYSTEM_PROFILE_NAME))
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_SYSTEM_PROFILE_NAME);
+
+    if(!EnableWindowsPrivilege(SE_PROF_SINGLE_PROCESS_NAME))
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_PROF_SINGLE_PROCESS_NAME);
 }
 
 uint64_t apps_os_get_total_memory_windows(void) {
@@ -500,10 +485,31 @@ uint64_t apps_os_get_total_memory_windows(void) {
     return memStat.ullTotalPhys;
 }
 
-static __thread wchar_t unicode[PATH_MAX];
+// remove the PID suffix and .exe suffix, if any
+static void fix_windows_comm(struct pid_stat *p, char *comm) {
+    char pid[UINT64_MAX_LENGTH + 1]; // +1 for the underscore
+    pid[0] = '_';
+    print_uint64(&pid[1], p->pid);
+    size_t pid_len = strlen(pid);
+    size_t comm_len = strlen(comm);
+    if (pid_len < comm_len) {
+        char *compare = &comm[comm_len - pid_len];
+        if (strcmp(pid, compare) == 0)
+            *compare = '\0';
+    }
+
+    // remove the .exe suffix, if any
+    comm_len = strlen(comm);
+    size_t exe_len = strlen(".exe");
+    if(exe_len < comm_len) {
+        char *compare = &comm[comm_len - exe_len];
+        if (strcmp(".exe", compare) == 0)
+            *compare = '\0';
+    }
+}
 
 // Convert wide string to UTF-8
-static STRING *wchar_to_string(WCHAR *s) {
+static char *wchar_to_utf8(WCHAR *s) {
     static __thread char utf8[PATH_MAX];
     static __thread int utf8_size = sizeof(utf8);
 
@@ -512,33 +518,152 @@ static STRING *wchar_to_string(WCHAR *s) {
         return NULL;
 
     WideCharToMultiByte(CP_UTF8, 0, s, -1, utf8, utf8_size, NULL, NULL);
-    return string_strdupz(utf8);
+    return utf8;
 }
 
-STRING *GetProcessFriendlyName(WCHAR *path) {
+// Convert wide string to UTF-8
+static STRING *wchar_to_string(WCHAR *s) {
+    return string_strdupz(wchar_to_utf8(s));
+}
+
+// --------------------------------------------------------------------------------------------------------------------
+
+// return a sanitized name for the process
+STRING *GetProcessFriendlyNameSanitized(WCHAR *path) {
     static __thread uint8_t void_buf[1024 * 1024];
+    static __thread DWORD void_buf_size = sizeof(void_buf);
+    static __thread wchar_t unicode[PATH_MAX];
+    static __thread DWORD unicode_size = sizeof(unicode) / sizeof(*unicode);
 
     DWORD handle;
     DWORD size = GetFileVersionInfoSizeW(path, &handle);
-    if (size == 0 || size > sizeof(void_buf))
+    if (size == 0 || size > void_buf_size)
         return FALSE;
 
     if (GetFileVersionInfoW(path, handle, size, void_buf)) {
         LPWSTR value = NULL;
         UINT len = 0;
-        DWORD unicode_size = sizeof(unicode) / sizeof(*unicode);
         if (VerQueryValueW(void_buf, L"\\StringFileInfo\\040904B0\\FileDescription", (LPVOID*)&value, &len) &&
             len > 0 && len < unicode_size) {
             wcsncpy(unicode, value, unicode_size - 1);
             unicode[unicode_size - 1] = L'\0';
-            return wchar_to_string(unicode);
+            char *name = wchar_to_utf8(unicode);
+            sanitize_chart_meta(name);
+            return string_strdupz(name);
         }
     }
 
     return NULL;
 }
 
+#define SERVICE_PREFIX "Service "
+// return a sanitized name for the process
+static STRING *GetNameFromCmdlineSanitized(struct pid_stat *p) {
+    if(!p->cmdline) return NULL;
+
+    char buf[string_strlen(p->cmdline) + 1];
+    memcpy(buf, string2str(p->cmdline), sizeof(buf));
+    char *words[100];
+    size_t num_words = quoted_strings_splitter(buf, words, 100, isspace_map_pluginsd);
+
+    if(string_strcmp(p->comm, "svchost") == 0) {
+        // find -s SERVICE in the command line
+        for(size_t i = 0; i < num_words ;i++) {
+            if(strcmp(words[i], "-s") == 0 && i + 1 < num_words) {
+                char service[strlen(words[i + 1]) + sizeof(SERVICE_PREFIX)]; // sizeof() includes a null
+                strcpy(service, SERVICE_PREFIX);
+                strcpy(&service[sizeof(SERVICE_PREFIX) - 1], words[i + 1]);
+                sanitize_chart_meta(service);
+                return string_strdupz(service);
+            }
+        }
+    }
+
+    return NULL;
+}
+
+static void GetServiceNames(void) {
+    SC_HANDLE hSCManager = OpenSCManager(NULL, NULL, SC_MANAGER_ENUMERATE_SERVICE);
+    if (hSCManager == NULL) return;
+
+    DWORD dwBytesNeeded = 0, dwServicesReturned = 0, dwResumeHandle = 0;
+    ENUM_SERVICE_STATUS_PROCESS *pServiceStatus = NULL;
+
+    // First, query the required buffer size
+    EnumServicesStatusEx(
+            hSCManager, SC_ENUM_PROCESS_INFO, SERVICE_WIN32, SERVICE_STATE_ALL,
+            NULL, 0, &dwBytesNeeded, &dwServicesReturned, &dwResumeHandle, NULL);
+
+    if (dwBytesNeeded == 0) {
+        CloseServiceHandle(hSCManager);
+        return;
+    }
+
+    // Allocate memory to hold the services
+    pServiceStatus = mallocz(dwBytesNeeded);
+
+    // Now, retrieve the list of services
+    if (!EnumServicesStatusEx(
+            hSCManager, SC_ENUM_PROCESS_INFO, SERVICE_WIN32, SERVICE_STATE_ALL,
+            (LPBYTE)pServiceStatus, dwBytesNeeded, &dwBytesNeeded, &dwServicesReturned,
+            &dwResumeHandle, NULL)) {
+        freez(pServiceStatus);
+        CloseServiceHandle(hSCManager);
+        return;
+    }
+
+    // Loop through the services
+    for (DWORD i = 0; i < dwServicesReturned; i++) {
+        if(!pServiceStatus[i].lpDisplayName || !*pServiceStatus[i].lpDisplayName)
+            continue;
+
+        struct pid_stat *p = find_pid_entry((pid_t)pServiceStatus[i].ServiceStatusProcess.dwProcessId);
+        if(p && !p->got_service) {
+            p->got_service = true;
+
+            size_t len = strlen(pServiceStatus[i].lpDisplayName);
+            char buf[len + 1];
+            memcpy(buf, pServiceStatus[i].lpDisplayName, sizeof(buf));
+            sanitize_chart_meta(buf);
+
+            string_freez(p->name);
+            p->name = string_strdupz(buf);
+        }
+    }
+
+    free(pServiceStatus);
+    CloseServiceHandle(hSCManager);
+}
+
+static WCHAR *executable_path_from_cmdline(WCHAR *cmdline) {
+    if (!cmdline || !*cmdline) return NULL;
+
+    WCHAR *exe_path_start = cmdline;
+    WCHAR *exe_path_end = NULL;
+
+    if (cmdline[0] == L'"') {
+        // Command line starts with a double quote
+        exe_path_start++;  // Move past the first double quote
+        exe_path_end = wcschr(exe_path_start, L'"');  // Find the next quote
+    }
+    else {
+        // Command line does not start with a double quote
+        exe_path_end = wcschr(exe_path_start, L' ');  // Find the first space
+    }
+
+    if (exe_path_end) {
+        // Null-terminate the string at the end of the executable path
+        *exe_path_end = L'\0';
+        return exe_path_start;
+    }
+
+    return NULL;
+}
+
 void GetAllProcessesInfo(void) {
+    static __thread wchar_t unicode[PATH_MAX];
+    static __thread DWORD unicode_size = sizeof(unicode) / sizeof(*unicode);
+
     calls_counter++;
 
     HANDLE hSnapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
@@ -552,45 +677,70 @@ void GetAllProcessesInfo(void) {
         return;
     }
 
+    bool need_service_names = false;
+
     do {
+        if(!pe32.th32ProcessID) continue;
+
         struct pid_stat *p = get_or_allocate_pid_entry((pid_t)pe32.th32ProcessID);
         p->ppid = (pid_t)pe32.th32ParentProcessID;
         if(p->got_info) continue;
         p->got_info = true;
 
-        if(!p->initialized) {
+        HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, p->pid);
-            string_freez(p->comm);
+        if (hProcess == NULL)
-            p->comm = wchar_to_string(pe32.szExeFile);
+            continue;
-            p->assigned_to_target = false;
+
+        // Get the full command line, if possible
+        {
+            WCHAR *cmdline = GetProcessCommandLine(hProcess); // returns malloc'd buffer
+            if (cmdline) {
+                string_freez(p->cmdline);
+                p->cmdline = wchar_to_string(cmdline);
+
+                // extract the process full path from the command line
+                WCHAR *path = executable_path_from_cmdline(cmdline);
+                if(path) {
+                    string_freez(p->name);
+                    p->name = GetProcessFriendlyNameSanitized(path);
+                }
+
+                free(cmdline); // free(), not freez()
+            }
         }
 
-        HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, p->pid);
+        if(!p->cmdline || !p->name) {
-        if (hProcess == NULL) continue;
+            if (QueryFullProcessImageNameW(hProcess, 0, unicode, &unicode_size)) {
+                // put the full path name to the command into cmdline
+                if(!p->cmdline)
+                    p->cmdline = wchar_to_string(unicode);
 
-        STRING *full_path = NULL;
+                if(!p->name)
-        STRING *friendly_name = NULL;
+                    p->name = GetProcessFriendlyNameSanitized(unicode);
-
+            }
-        DWORD unicode_size = sizeof(unicode) / sizeof(*unicode);
-        if(QueryFullProcessImageNameW(hProcess, 0, unicode, &unicode_size)) {
-            full_path = wchar_to_string(unicode);
-            friendly_name = GetProcessFriendlyName(unicode);
         }
 
         CloseHandle(hProcess);
 
-        if(full_path) {
+        char *comm = wchar_to_utf8(pe32.szExeFile);
-            string_freez(p->cmdline);
+        fix_windows_comm(p, comm);
-            p->cmdline = full_path;
+        update_pid_comm(p, comm); // will sanitize p->comm
+
+        if(!need_service_names && string_strcmp(p->comm, "svchost") == 0)
+            need_service_names = true;
+
+        STRING *better_name = GetNameFromCmdlineSanitized(p);
+        if(better_name) {
+            string_freez(p->name);
+            p->name = better_name;
         }
 
-        if(friendly_name) {
-            string_freez(p->name);
-            p->name = friendly_name;
-            p->assigned_to_target = false;
-        }
     } while (Process32NextW(hSnapshot, &pe32));
 
     CloseHandle(hSnapshot);
+
+    if(need_service_names)
+        GetServiceNames();
 }
 
 static inline kernel_uint_t perflib_cpu_utilization(COUNTER_DATA *d) {
@@ -692,40 +842,17 @@ bool apps_os_collect_all_pids_windows(void) {
             // a new pid
             p->initialized = true;
 
-            static __thread char name[MAX_PATH];
+            static __thread char comm[MAX_PATH];
 
-            if (getInstanceName(d.pDataBlock, d.pObjectType, d.pi, name, sizeof(name))) {
+            if (getInstanceName(d.pDataBlock, d.pObjectType, d.pi, comm, sizeof(comm)))
-                // remove the PID suffix, if any
+                fix_windows_comm(p, comm);
-                char pid[UINT64_MAX_LENGTH + 1]; // +1 for the underscore
-                pid[0] = '_';
-                print_uint64(&pid[1], p->pid);
-                size_t pid_len = strlen(pid);
-                size_t name_len = strlen(name);
-                if (pid_len < name_len) {
-                    char *compare = &name[name_len - pid_len];
-                    if (strcmp(pid, compare) == 0)
-                        *compare = '\0';
-                }
-
-                // remove the .exe suffix, if any
-                name_len = strlen(name);
-                size_t exe_len = strlen(".exe");
-                if(exe_len < name_len) {
-                    char *compare = &name[name_len - exe_len];
-                    if (strcmp(".exe", compare) == 0)
-                        *compare = '\0';
-                }
-            }
             else
-                strncpyz(name, "unknown", sizeof(name) - 1);
+                strncpyz(comm, "unknown", sizeof(comm) - 1);
 
-            if(strcmp(name, "wininit") == 0)
+            if(strcmp(comm, "wininit") == 0)
                 INIT_PID = p->pid;
 
-            string_freez(p->comm); // it may be detected in a previous run via GetAllProcessesInfo()
+            update_pid_comm(p, comm); // will sanitize p->comm
-            p->comm = string_strdupz(name);
-            p->got_info = false;
-            p->assigned_to_target = false;
             added++;
 
             COUNTER_DATA ppid = {.key = "Creating Process ID"};

src/collectors/apps.plugin/apps_os_windows_nt.c

@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+// this must not include libnetdata.h because STRING is defined in winternl.h
+
+#include "config.h"
+#if defined(OS_WINDOWS)
+
+#include <windows.h>
+#include <winternl.h>
+#include <psapi.h>
+#include <stdint.h>
+
+// --------------------------------------------------------------------------------------------------------------------
+// Get the full windows command line
+
+WCHAR* GetProcessCommandLine(HANDLE hProcess) {
+    PROCESS_BASIC_INFORMATION pbi;
+    ULONG len;
+    NTSTATUS status = NtQueryInformationProcess(hProcess, 0, &pbi, sizeof(pbi), &len);
+    if (status != 0)
+        return NULL;
+
+    // The rest of the function remains the same as before
+    PEB peb;
+    if (!ReadProcessMemory(hProcess, pbi.PebBaseAddress, &peb, sizeof(peb), NULL))
+        return NULL;
+
+    RTL_USER_PROCESS_PARAMETERS procParams;
+    if (!ReadProcessMemory(hProcess, peb.ProcessParameters, &procParams, sizeof(procParams), NULL))
+        return NULL;
+
+    WCHAR* commandLine = (WCHAR*)malloc(procParams.CommandLine.MaximumLength);
+    if (!commandLine)
+        return NULL;
+
+    if (!ReadProcessMemory(hProcess, procParams.CommandLine.Buffer, commandLine, procParams.CommandLine.MaximumLength, NULL)) {
+        free(commandLine);
+        return NULL;
+    }
+
+    return commandLine;
+}
+
+#endif

src/collectors/apps.plugin/apps_pid.c

@@ -134,6 +134,7 @@ void del_pid_entry(pid_t pid) {
     freez(p->fds);
 #endif
 
+    string_freez(p->comm_orig);
     string_freez(p->comm);
     string_freez(p->cmdline);
     aral_freez(pids.all_pids.aral, p);
@@ -316,7 +317,49 @@ static inline void link_all_processes_to_their_parents(void) {
 
 // --------------------------------------------------------------------------------------------------------------------
 
+static inline STRING *comm_from_cmdline_sanitized(char *comm, STRING *cmdline) {
+    if(!cmdline) {
+        sanitize_chart_meta(comm);
+        return string_strdupz(comm);
+    }
+
+    const char *cl = string2str(cmdline);
+    size_t len = string_strlen(cmdline);
+
+    char buf_cmd[len + 1];
+    // if it is enclosed in (), remove the parenthesis
+    if(cl[0] == '(' && cl[len - 1] == ')') {
+        memcpy(buf_cmd, &cl[1], len - 2);
+        buf_cmd[len - 2] = '\0';
+    }
+    else
+        memcpy(buf_cmd, cl, sizeof(buf_cmd));
+
+    size_t comm_len = strlen(comm);
+    char *start = strstr(buf_cmd, comm);
+    if(start) {
+        char *end = start + comm_len;
+        while(*end && !isspace((uint8_t)*end) && *end != '/' && *end != '\\' && *end != '"') end++;
+        *end = '\0';
+
+        sanitize_chart_meta(start);
+        return string_strdupz(start);
+    }
+
+    sanitize_chart_meta(comm);
+    return string_strdupz(comm);
+}
+
 void update_pid_comm(struct pid_stat *p, const char *comm) {
+    if(p->comm_orig && string_strcmp(p->comm_orig, comm) == 0)
+        // no change
+        return;
+
+#if (PROCESSES_HAVE_CMDLINE == 1)
+    if(likely(proc_pid_cmdline_is_needed && !p->cmdline))
+        managed_log(p, PID_LOG_CMDLINE, read_proc_pid_cmdline(p));
+#endif
+
     // some process names have ( and ), remove the parenthesis
     size_t len = strlen(comm);
     char buf[len + 1];
@@ -327,22 +370,18 @@ void update_pid_comm(struct pid_stat *p, const char *comm) {
     else
         memcpy(buf, comm, sizeof(buf));
 
-    // check if the comm is changed
+    string_freez(p->comm_orig);
-    if(!p->comm || strcmp(pid_stat_comm(p), buf) != 0) {
+    p->comm_orig = string_strdupz(comm);
-        // it is changed
 
-        string_freez(p->comm);
+    string_freez(p->comm);
-        p->comm = string_strdupz(buf);
+    p->comm = comm_from_cmdline_sanitized(buf, p->cmdline);
 
-#if (PROCESSES_HAVE_CMDLINE == 1)
+    p->is_manager = is_process_manager(p);
-        if(likely(proc_pid_cmdline_is_needed))
+    p->is_aggregator = is_process_aggregator(p);
-            managed_log(p, PID_LOG_CMDLINE, read_proc_pid_cmdline(p));
-#endif
 
-        // the process changes comm, we may have to reassign it to
+    // the process changed comm, we may have to reassign it to
-        // an apps_groups.conf target.
+    // an apps_groups.conf target.
-        p->target = NULL;
+    p->target = NULL;
-    }
 }
 
 // --------------------------------------------------------------------------------------------------------------------

src/collectors/apps.plugin/apps_plugin.c

@@ -121,6 +121,84 @@ size_t pagesize;
 // ----------------------------------------------------------------------------
 // update chart dimensions
 
+// Helper function to count the number of processes in the linked list
+int count_processes(struct pid_stat *root) {
+    int count = 0;
+
+    for(struct pid_stat *p = root; p ; p = p->next)
+        if(p->updated) count++;
+
+    return count;
+}
+
+// Comparator function to sort by pid
+int compare_by_pid(const void *a, const void *b) {
+    struct pid_stat *pa = *(struct pid_stat **)a;
+    struct pid_stat *pb = *(struct pid_stat **)b;
+    return ((int)pa->pid - (int)pb->pid);
+}
+
+// Function to print a process and its children recursively
+void print_process_tree(struct pid_stat *root, struct pid_stat *parent, int depth, int total_processes) {
+    // Allocate an array of pointers for processes with the given parent
+    struct pid_stat **children = (struct pid_stat **)malloc(total_processes * sizeof(struct pid_stat *));
+    int children_count = 0;
+
+    // Populate the array with processes that have the given parent
+    struct pid_stat *p = root;
+    while (p != NULL) {
+        if (p->updated && p->parent == parent) {
+            children[children_count++] = p;
+        }
+        p = p->next;
+    }
+
+    // Sort the children array by pid
+    qsort(children, children_count, sizeof(struct pid_stat *), compare_by_pid);
+
+    // Print each child and recurse
+    for (int i = 0; i < children_count; i++) {
+        // Print the current process with indentation based on depth
+        if (depth > 0) {
+            for (int j = 0; j < (depth - 1) * 4; j++) {
+                printf(" ");
+            }
+            printf(" \\_ ");
+        }
+
+#if (PROCESSES_HAVE_COMM_AND_NAME == 1)
+        printf("[%d] %s (name: %s) [%s]: %s\n", children[i]->pid,
+               string2str(children[i]->comm),
+               string2str(children[i]->name),
+               string2str(children[i]->target->name),
+               string2str(children[i]->cmdline));
+#else
+        printf("[%d] %s [%s]: %s\n", children[i]->pid,
+               string2str(children[i]->comm),
+               string2str(children[i]->target->name),
+               string2str(children[i]->cmdline));
+#endif
+
+        // Recurse to print this child's children
+        print_process_tree(root, children[i], depth + 1, total_processes);
+    }
+
+    // Free the allocated array
+    free(children);
+}
+
+// Function to print the full hierarchy
+void print_hierarchy(struct pid_stat *root) {
+    // Count the total number of processes
+    int total_processes = count_processes(root);
+
+    // Start printing from processes with parent = NULL (i.e., root processes)
+    print_process_tree(root, NULL, 0, total_processes);
+}
+
+// ----------------------------------------------------------------------------
+// update chart dimensions
+
 #if (ALL_PIDS_ARE_READ_INSTANTLY == 0)
 static void normalize_utilization(struct target *root) {
     struct target *w;
@@ -297,6 +375,7 @@ cleanup:
 }
 
 static bool profile_speed = false;
+static bool print_tree_and_exit = false;
 
 static void parse_args(int argc, char **argv)
 {
@@ -316,6 +395,11 @@ static void parse_args(int argc, char **argv)
             exit(0);
         }
 
+        if(strcmp("print", argv[i]) == 0 || strcmp("-print", argv[i]) == 0 || strcmp("--print", argv[i]) == 0) {
+            print_tree_and_exit = true;
+            continue;
+        }
+
 #if defined(OS_LINUX)
         if(strcmp("test-permissions", argv[i]) == 0 || strcmp("-t", argv[i]) == 0) {
             if(!check_proc_1_io()) {
@@ -618,7 +702,7 @@ int main(int argc, char **argv) {
     procfile_adaptive_initial_allocation = 1;
     os_get_system_HZ();
     os_get_system_cpus_uncached();
-    apps_orchestrators_and_aggregators_init(); // before parsing args!
+    apps_managers_and_aggregators_init(); // before parsing args!
     parse_args(argc, argv);
 
 #if !defined(OS_WINDOWS)
@@ -702,6 +786,11 @@ int main(int argc, char **argv) {
         normalize_utilization(apps_groups_root_target);
 #endif
 
+        if(unlikely(print_tree_and_exit)) {
+            print_hierarchy(root_of_pids());
+            exit(0);
+        }
+
         if(send_resource_usage)
             send_resource_usage_to_netdata(dt);
 

src/collectors/apps.plugin/apps_plugin.h

@@ -371,7 +371,10 @@ struct target {
 
     TARGET_TYPE type;
     union {
-        STRING *compare;
+        struct {
+            SIMPLE_PATTERN *pattern;
+            STRING *compare;
+        } ag;
 #if (PROCESSES_HAVE_UID == 1)
         uid_t uid;
 #endif
@@ -393,11 +396,8 @@ struct target {
 #endif
 
     bool exposed:1;         // if set, we have sent this to netdata
-    bool hidden:1;          // if set, we set the hidden flag on the dimension
+    bool ends_with:1;       // if set, the compare string matches the end of the command
-    bool debug_enabled:1;
+    bool starts_with:1;     // if set, the compare string matches the start of the command
-    bool ends_with:1;
-    bool starts_with:1;     // if set, the compare string matches only the
-                            // beginning of the command
 
     struct pid_on_target *root_pid; // list of aggregated pids for target debugging
 
@@ -476,7 +476,7 @@ struct pid_stat {
     struct pid_stat *next;
     struct pid_stat *prev;
 
-    struct target *target;          // app_groups.conf targets
+    struct target *target;          // app_groups.conf/tree targets
 
 #if (PROCESSES_HAVE_UID == 1)
     struct target *uid_target;      // uid based targets
@@ -485,9 +485,10 @@ struct pid_stat {
     struct target *gid_target;      // gid based targets
 #endif
 
-    STRING *comm;                   // the command name (short version)
+    STRING *comm_orig;              // the command, as-collected
-    STRING *name;                   // a better name, or NULL
+    STRING *comm;                   // the command, sanitized
-    STRING *cmdline;                // the full command line (or on windows, the full pathname of the program)
+    STRING *name;                   // the command name if any, sanitized
+    STRING *cmdline;                // the full command line of the program
 
 #if defined(OS_WINDOWS)
     COUNTER_DATA perflib[PDF_MAX];
@@ -531,6 +532,8 @@ struct pid_stat {
     bool updated:1;                 // true when the process is currently running
     bool merged:1;                  // true when it has been merged to its parent
     bool keep:1;                    // true when we need to keep this process in memory even after it exited
+    bool is_manager:1;              // true when this pid is a process manager
+    bool is_aggregator:1;           // true when this pid is a process aggregator
 
     bool matched_by_config:1;
 
@@ -540,7 +543,7 @@ struct pid_stat {
 
 #if defined(OS_WINDOWS)
     bool got_info:1;
-    bool assigned_to_target:1;
+    bool got_service:1;
     bool initialized:1;
 #endif
 
@@ -631,7 +634,7 @@ bool managed_log(struct pid_stat *p, PID_LOG log, bool status);
 #define pid_incremental_cpu(type, idx, value) \
     incremental_rate(p->values[idx], p->raw[idx], value, p->type##_collected_usec, p->last_##type##_collected_usec, CPU_TO_NANOSECONDCORES)
 
-void apps_orchestrators_and_aggregators_init(void);
+void apps_managers_and_aggregators_init(void);
 void apps_users_and_groups_init(void);
 void apps_pids_init(void);
 
@@ -675,6 +678,8 @@ struct pid_stat *find_pid_entry(pid_t pid);
 void del_pid_entry(pid_t pid);
 void update_pid_comm(struct pid_stat *p, const char *comm);
 
+bool is_process_manager(struct pid_stat *p);
+bool is_process_aggregator(struct pid_stat *p);
 
 // --------------------------------------------------------------------------------------------------------------------
 // targets management

src/collectors/apps.plugin/apps_targets.c

@@ -31,35 +31,7 @@ struct target *find_target_by_name(struct target *base, const char *name) {
 }
 
 // --------------------------------------------------------------------------------------------------------------------
-// Tree
+// Process managers and aggregators
-
-static inline STRING *comm_from_cmdline(STRING *comm, STRING *cmdline) {
-    if(!cmdline) return sanitize_chart_meta_string(comm);
-
-    const char *cl = string2str(cmdline);
-    size_t len = string_strlen(cmdline);
-
-    char buf_cmd[len + 1];
-    // if it is enclosed in (), remove the parenthesis
-    if(cl[0] == '(' && cl[len - 1] == ')') {
-        memcpy(buf_cmd, &cl[1], len - 2);
-        buf_cmd[len - 2] = '\0';
-    }
-    else
-        memcpy(buf_cmd, cl, sizeof(buf_cmd));
-
-    char *start = strstr(buf_cmd, string2str(comm));
-    if(start) {
-        char *end = start + string_strlen(comm);
-        while(*end && !isspace((uint8_t)*end) && *end != '/' && *end != '\\') end++;
-        *end = '\0';
-
-        sanitize_chart_meta(start);
-        return string_strdupz(start);
-    }
-
-    return sanitize_chart_meta_string(comm);
-}
 
 struct comm_list {
     STRING *comm;
@@ -111,21 +83,26 @@ static void managed_list_add(struct managed_list *list, const char *s) {
 
 static STRING *KernelAggregator = NULL;
 
-void apps_orchestrators_and_aggregators_init(void) {
+void apps_managers_and_aggregators_init(void) {
     KernelAggregator = string_strdupz("kernel");
 
     managed_list_clear(&tree.managers);
 #if defined(OS_LINUX)
-    managed_list_add(&tree.managers, "init");               // linux systems
+    managed_list_add(&tree.managers, "init");                       // linux systems
-    managed_list_add(&tree.managers, "systemd");            // lxc containers and host systems (this also catches "systemd --user")
+    managed_list_add(&tree.managers, "systemd");                    // lxc containers and host systems (this also catches "systemd --user")
-    managed_list_add(&tree.managers, "containerd-shim");    // docker containers
+    managed_list_add(&tree.managers, "containerd-shim-runc-v2");    // docker containers
-    managed_list_add(&tree.managers, "docker-init");        // docker containers
+    managed_list_add(&tree.managers, "docker-init");                // docker containers
-    managed_list_add(&tree.managers, "dumb-init");          // some docker containers use this
+    managed_list_add(&tree.managers, "dumb-init");                  // some docker containers use this
-    managed_list_add(&tree.managers, "gnome-shell");        // gnome user applications
+    managed_list_add(&tree.managers, "openrc-run.sh");              // openrc
+    managed_list_add(&tree.managers, "crond");                      // linux crond
+    managed_list_add(&tree.managers, "gnome-shell");                // gnome user applications
+    managed_list_add(&tree.managers, "plasmashell");                // kde user applications
+    managed_list_add(&tree.managers, "xfwm4");                      // xfce4 user applications
 #elif defined(OS_WINDOWS)
-    managed_list_add(&tree.managers, "System");
-    managed_list_add(&tree.managers, "services");
     managed_list_add(&tree.managers, "wininit");
+    managed_list_add(&tree.managers, "services");
+    managed_list_add(&tree.managers, "explorer");
+    managed_list_add(&tree.managers, "System");
 #elif defined(OS_FREEBSD)
     managed_list_add(&tree.managers, "init");
 #elif defined(OS_MACOS)
@@ -142,49 +119,52 @@ void apps_orchestrators_and_aggregators_init(void) {
 #endif
 }
 
-static inline bool is_orchestrator(struct pid_stat *p) {
+bool is_process_manager(struct pid_stat *p) {
     for(size_t c = 0; c < tree.managers.used ; c++) {
-        if(p->comm == tree.managers.array[c].comm)
+        if(p->comm == tree.managers.array[c].comm ||
+            p->comm_orig == tree.managers.array[c].comm)
             return true;
     }
 
     return false;
 }
 
-static inline bool is_aggregator(struct pid_stat *p) {
+bool is_process_aggregator(struct pid_stat *p) {
     for(size_t c = 0; c < tree.aggregators.used ; c++) {
-        if(p->comm == tree.aggregators.array[c].comm)
+        if(p->comm == tree.aggregators.array[c].comm ||
+            p->comm_orig == tree.aggregators.array[c].comm)
             return true;
     }
 
     return false;
 }
 
+// --------------------------------------------------------------------------------------------------------------------
+// Tree
+
 struct target *get_tree_target(struct pid_stat *p) {
 //    // skip fast all the children that are more than 3 levels down
 //    while(p->parent && p->parent->pid != INIT_PID && p->parent->parent && p->parent->parent->parent)
 //        p = p->parent;
 
     // keep the children of INIT_PID, and process orchestrators
-    while(p->parent && p->parent->pid != INIT_PID && p->parent->pid != 0 && !is_orchestrator(p->parent))
+    while(p->parent && p->parent->pid != INIT_PID && p->parent->pid != 0 && !p->parent->is_manager)
         p = p->parent;
 
     // merge all processes into process aggregators
-    STRING *search_for = string_dup(p->comm);
+    STRING *search_for = NULL;
-    bool aggregator = false;
+    if((p->ppid == 0 && p->pid != INIT_PID) || (p->parent && p->parent->is_aggregator)) {
-    if((p->ppid == 0 && p->pid != INIT_PID) || (p->parent && is_aggregator(p->parent))) {
-        aggregator = true;
         search_for = string_dup(KernelAggregator);
     }
-
+    else {
-    if(!aggregator) {
 #if (PROCESSES_HAVE_COMM_AND_NAME == 1)
-        search_for = sanitize_chart_meta_string(p->name ? p->name : p->comm);
+        search_for = string_dup(p->name ? p->name : p->comm);
 #else
-        search_for = comm_from_cmdline(p->comm, p->cmdline);
+        search_for = string_dup(p->comm);
 #endif
     }
 
+    // find an existing target with the required name
     struct target *w;
     for(w = apps_groups_root_target; w ; w = w->next) {
         if (w->name == search_for) {
@@ -196,7 +176,7 @@ struct target *get_tree_target(struct pid_stat *p) {
     w = callocz(sizeof(struct target), 1);
     w->type = TARGET_TYPE_TREE;
     w->starts_with = w->ends_with = false;
-    w->compare = string_dup(p->comm);
+    w->ag.compare = string_dup(search_for);
     w->id = search_for;
     w->name = string_dup(search_for);
     w->clean_name = get_clean_name(w->name);
@@ -302,17 +282,17 @@ struct target *apps_groups_root_target = NULL;
 
 // find or create a new target
 // there are targets that are just aggregated to other target (the second argument)
-static struct target *get_apps_groups_target(const char *id, struct target *target, const char *name) {
+static struct target *get_apps_groups_target(const char *comm, struct target *target, const char *name) {
-    bool tdebug = false, thidden = target ? target->hidden : false, ends_with = false, starts_with = false;
+    bool ends_with = false, starts_with = false, has_asterisk_inside = false;
 
-    STRING *id_lookup = NULL;
+    STRING *comm_lookup = NULL;
     STRING *name_lookup = NULL;
 
     // extract the options from the id
     {
-        size_t len = strlen(id);
+        size_t len = strlen(comm);
         char buf[len + 1];
-        memcpy(buf, id, sizeof(buf));
+        memcpy(buf, comm, sizeof(buf));
 
         if(buf[len - 1] == '*') {
             buf[--len] = '\0';
@@ -320,37 +300,25 @@ static struct target *get_apps_groups_target(const char *id, struct target *targ
         }
 
         const char *nid = buf;
-        while (nid[0] == '-' || nid[0] == '+' || nid[0] == '*') {
+        if (nid[0] == '*') {
-            if (nid[0] == '-') thidden = true;
+            ends_with = true;
-            if (nid[0] == '+') tdebug = true;
-            if (nid[0] == '*') ends_with = true;
             nid++;
         }
 
-        id_lookup = string_strdupz(nid);
+        if(strchr(nid, '*'))
+            has_asterisk_inside = true;
+
+        comm_lookup = string_strdupz(nid);
     }
 
     // extract the options from the name
-    {
+    name_lookup = string_strdupz(name);
-        size_t len = strlen(name);
-        char buf[len + 1];
-        memcpy(buf, name, sizeof(buf));
-
-        const char *nn = buf;
-        while (nn[0] == '-' || nn[0] == '+') {
-            if (nn[0] == '-') thidden = true;
-            if (nn[0] == '+') tdebug = true;
-            nn++;
-        }
-
-        name_lookup = string_strdupz(nn);
-    }
 
     // find if it already exists
     struct target *w, *last = apps_groups_root_target;
     for(w = apps_groups_root_target ; w ; w = w->next) {
-        if(w->id == id_lookup) {
+        if(w->id == comm_lookup) {
-            string_freez(id_lookup);
+            string_freez(comm_lookup);
             string_freez(name_lookup);
             return w;
         }
@@ -368,19 +336,22 @@ static struct target *get_apps_groups_target(const char *id, struct target *targ
 
     if(target && target->target)
         fatal("Internal Error: request to link process '%s' to target '%s' which is linked to target '%s'",
-              id, string2str(target->id), string2str(target->target->id));
+            comm, string2str(target->id), string2str(target->target->id));
 
     w = callocz(sizeof(struct target), 1);
     w->type = TARGET_TYPE_APP_GROUP;
-    w->compare = string_dup(id_lookup);
+    w->ag.compare = string_dup(comm_lookup);
     w->starts_with = starts_with;
     w->ends_with = ends_with;
-    w->id = string_dup(id_lookup);
+    w->id = string_dup(comm_lookup);
+
+    if(has_asterisk_inside)
+        w->ag.pattern = simple_pattern_create(comm, " ", SIMPLE_PATTERN_EXACT, true);
 
     if(unlikely(!target))
         w->name = string_dup(name_lookup); // copy the name
     else
-        w->name = string_dup(id_lookup); // copy the id
+        w->name = string_dup(comm_lookup); // copy the id
 
     // dots are used to distinguish chart type and id in streaming, so we should replace them
     w->clean_name = get_clean_name(w->name);
@@ -388,29 +359,20 @@ static struct target *get_apps_groups_target(const char *id, struct target *targ
     if(w->starts_with && w->ends_with)
         proc_pid_cmdline_is_needed = true;
 
-    w->hidden = thidden;
-#ifdef NETDATA_INTERNAL_CHECKS
-    w->debug_enabled = tdebug;
-#else
-    if(tdebug)
-        fprintf(stderr, "apps.plugin has been compiled without debugging\n");
-#endif
     w->target = target;
 
     // append it, to maintain the order in apps_groups.conf
     if(last) last->next = w;
     else apps_groups_root_target = w;
 
-    debug_log("ADDING TARGET ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s"
+    debug_log("ADDING TARGET ID '%s', process name '%s' (%s), aggregated on target '%s'"
               , string2str(w->id)
-              , string2str(w->compare)
+              , string2str(w->ag.compare)
               , (w->starts_with && w->ends_with)?"substring":((w->starts_with)?"prefix":((w->ends_with)?"suffix":"exact"))
               , w->target?w->target->name:w->name
-              , (w->hidden)?"hidden":"-"
-              , (w->debug_enabled)?"debug":"-"
     );
 
-    string_freez(id_lookup);
+    string_freez(comm_lookup);
     string_freez(name_lookup);
 
     return w;

src/collectors/windows-events.plugin/windows-events.c

@@ -1139,6 +1139,15 @@ int main(int argc __maybe_unused, char **argv __maybe_unused) {
     sid_cache_init();
     field_cache_init();
 
+    if(!EnableWindowsPrivilege(SE_SECURITY_NAME))
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_SECURITY_NAME);
+
+    if(!EnableWindowsPrivilege(SE_BACKUP_NAME))
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_BACKUP_NAME);
+
+    if(!EnableWindowsPrivilege(SE_AUDIT_NAME))
+        nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to enable %s privilege", SE_AUDIT_NAME);
+
     // ------------------------------------------------------------------------
     // debug
 

src/health/health_notifications.c

@@ -20,17 +20,27 @@ struct health_raised_summary {
 };
 
 void health_alarm_wait_for_execution(ALARM_ENTRY *ae) {
-    if (!(ae->flags & HEALTH_ENTRY_FLAG_EXEC_IN_PROGRESS))
+    // this has to ALWAYS remove the given alarm entry from the queue
-        return;
 
-    if(!ae->popen_instance) {
+    int code = 0;
-        // nd_log(NDLS_DAEMON, NDLP_ERR, "attempted to wait for the execution of alert that has not spawn a notification");
+
-        return;
+    if (!(ae->flags & HEALTH_ENTRY_FLAG_EXEC_IN_PROGRESS)) {
+        nd_log(NDLS_DAEMON, NDLP_ERR, "attempted to wait for the execution of alert that has not an execution in progress");
+        code = 128;
+        goto cleanup;
     }
 
-    ae->exec_code = spawn_popen_wait(ae->popen_instance);
+    if(!ae->popen_instance) {
+        nd_log(NDLS_DAEMON, NDLP_ERR, "attempted to wait for the execution of alert that has not spawn a notification");
+        code = 128;
+        goto cleanup;
+    }
 
+    code = spawn_popen_wait(ae->popen_instance);
     netdata_log_debug(D_HEALTH, "done executing command - returned with code %d", ae->exec_code);
+
+cleanup:
+    ae->exec_code = code;
     ae->flags &= ~HEALTH_ENTRY_FLAG_EXEC_IN_PROGRESS;
 
     if(ae->exec_code != 0)
@@ -466,13 +476,18 @@ void health_send_notification(RRDHOST *host, ALARM_ENTRY *ae, struct health_rais
         ae->exec_run_timestamp = now_realtime_sec(); /* will be updated by real time after spawning */
 
         netdata_log_debug(D_HEALTH, "executing command '%s'", command_to_run);
-        ae->flags |= HEALTH_ENTRY_FLAG_EXEC_IN_PROGRESS;
         ae->popen_instance = spawn_popen_run(command_to_run);
-        enqueue_alarm_notify_in_progress(ae);
+        if(ae->popen_instance) {
+            ae->flags |= HEALTH_ENTRY_FLAG_EXEC_IN_PROGRESS;
+            enqueue_alarm_notify_in_progress(ae);
+        }
+        else
+            netdata_log_error("Failed to execute alarm notification");
+
         health_alarm_log_save(host, ae);
-    } else {
-        netdata_log_error("Failed to format command arguments");
     }
+    else
+        netdata_log_error("Failed to format command arguments");
 
     buffer_free(warn_alarms);
     buffer_free(crit_alarms);

src/libnetdata/os/os-windows-wrappers.c

@@ -58,4 +58,42 @@ bool netdata_registry_get_string(char *out, unsigned int length, void *hKey, cha
     return status;
 }
 
+bool EnableWindowsPrivilege(const char *privilegeName) {
+    HANDLE hToken;
+    LUID luid;
+    TOKEN_PRIVILEGES tkp;
+
+    // Open the process token with appropriate access rights
+    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken))
+        return false;
+
+    // Lookup the LUID for the specified privilege
+    if (!LookupPrivilegeValue(NULL, privilegeName, &luid)) {
+        CloseHandle(hToken);  // Close the token handle before returning
+        return false;
+    }
+
+    // Set up the TOKEN_PRIVILEGES structure
+    tkp.PrivilegeCount = 1;
+    tkp.Privileges[0].Luid = luid;
+    tkp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+
+    // Adjust the token's privileges
+    if (!AdjustTokenPrivileges(hToken, FALSE, &tkp, sizeof(tkp), NULL, NULL)) {
+        CloseHandle(hToken);  // Close the token handle before returning
+        return false;
+    }
+
+    // Check if AdjustTokenPrivileges succeeded
+    if (GetLastError() == ERROR_NOT_ALL_ASSIGNED) {
+        CloseHandle(hToken);  // Close the token handle before returning
+        return false;
+    }
+
+    // Close the handle to the token after success
+    CloseHandle(hToken);
+
+    return true;
+}
+
 #endif

src/libnetdata/os/os-windows-wrappers.h

@@ -14,5 +14,7 @@ bool netdata_registry_get_dword(unsigned int *out, void *hKey, char *subKey, cha
 long netdata_registry_get_string_from_open_key(char *out, unsigned int length, void *lKey, char *name);
 bool netdata_registry_get_string(char *out, unsigned int length, void *hKey, char *subKey, char *name);
 
+bool EnableWindowsPrivilege(const char *privilegeName);
+
 #endif // OS_WINDOWS
 #endif //NETDATA_OS_WINDOWS_WRAPPERS_H

src/libnetdata/spawn_server/spawn_server_windows.c

@@ -54,7 +54,7 @@ static BUFFER *argv_to_windows(const char **argv) {
     BUFFER *wb = buffer_create(0, NULL);
 
     // argv[0] is the path
-    char b[strlen(argv[0]) * 2 + 1024];
+    char b[strlen(argv[0]) * 2 + FILENAME_MAX];
     cygwin_conv_path(CCP_POSIX_TO_WIN_A | CCP_ABSOLUTE, argv[0], b, sizeof(b));
 
     for(size_t i = 0; argv[i] ;i++) {
@@ -84,6 +84,8 @@ static BUFFER *argv_to_windows(const char **argv) {
             else
                 buffer_putc(wb, ' ');
         }
+        else if (needs_quotes)
+            buffer_putc(wb, '"');
 
         for(const char *c = s; *c ; c++) {
             switch(*c) {