Measuring accounting metrics of the job¶
ARC CE has built-in capabilities to collect information about per-job resource consumption. This includes both ARC CE resources (e.g. data transfers, software environments) and worker nodes resources (e.g. CPU and memory usage). The full list of attributes stored in the A-REX Accounting Records (AAR) can be found in this document.
A-REX can use different methods (descibed below) to measure memory and CPU usage on the worked nodes, depending on their availability in the particular deployment case.
Measuring memory and CPU usage on the WN with cgroups¶
New in version 6.2.
When recent versions of GNU/Linux OS are used on the worker nodes the most precise and transparent way to measure all job workload
is to rely on
cgroups kernel subsystem. Any
systemd-based Linux distribution relies on
cgroups heaviely and they are already used.
Some older operating systems may require to mount cgroups tree explicitely.
For example in RHEL6 it can be easiely done with
[root ~]# yum install libcgroup [root ~]# service cgconfig start
The benefit of using
cgroups is that everyting will be accounted.
Even if several payloads are executed (e.g. in pilot mode) or extra helper processes are spawned - the resource accounting will be accurate for the all workload done.
Enabling cgroups usage¶
To be able to use
cgroups for accounting ARC needs an extra tool installed on the worker nodes – the
Based on the tool availability job script will or will NOT use
cgroups for measuring accounting metrics automatically.
arc-job-cgroup tool is available for majority of OSes as a packaged binary build as a part of ARC distribution (
So the easiest way to install it is to use your package manager on the worker nodes, e.g.:
[root ~]# yum install nordugrid-arc-wn
If it is not possible to install the packaged version for some reason, it is easy to compile the pure C source code with standard C library calls only.
[root ~]# wget https://source.coderefinery.org/nordugrid/arc/raw/master/src/wn/arc-job-cgroup.c [root ~]# cc -o arc-job-cgroup arc-job-cgroup.c [root ~]# mv arc-job-cgroup /usr/local/bin/ [root ~]# chmod 4755 /usr/local/bin/arc-job-cgroup
How ARC operates cgroups¶
- The idea behind LRMS-independent
cgroup-based resource usage measurements in ARC is to:
- create a child cgroups for
- put the jobscript process into created cgroups (this will automatically catch all child processes)
- collect the accounting data at the end of jobscipt
- remove the child cgroup created at the beginning (moving all processes to parent cgroup)
- create a child cgroups for
cgroups are used in Kernel, the process already belongs to some
cgroup. It can be either a root
cgroup (used for all processes) or some dedicated
cgroup created by LRMS with
cgroups support, container management system, etc.
All resources used by child
cgroup are accounted in parent
cgroup. Moreover all parent-defined limits are inherited and enforced as well.
So creating another child
cgroup in hierarchy is safe from all points of view.
Creating a child cgroup and put a task into it requires
root priveledges. This is the reason behind the SUID bit for
However the code itself is as simple as the
mkdir. You can review this 333 lines to reason out of possible fears.
arc-job-cgroup tool is not avaiable,
cgroups tree is not mounted or any other issues with
cgroups creation, the job script code fall back to the GNU time measurement method.
Measuring memory and CPU usage on the WN with GNU time¶
time utility is capable of measuring and displaing information about the resources used by the executable it runs.
It is used as a part of ARC-generated job script if found on the worker node.
Changed in version 6.2.
In case of successfull
cgroups usage, GNU time will NOT be used by job script.
time is a separate binary typically installed by dedicated package. Do not mix it up with built-in version of
time in your shell (e.g. bash, zsh).
Typically you can install it with e.g.
yum install time or similar package management command.
For non-standard location of GNU Time the gnu_time configuration option can be used to define it.
If the GNU
time utility is not available the job will run as it is and only LRMS-provided metrics will be accounted.
Using LRMS-provided metrics¶
After the job had finised execution in LRMS, the batch system backend scan-script extract accounting information about the job from LRMS, either executing command line clients, parsing logs or using API.
The exact data measurements and the method of these data collection is completely depends on LRMS backend implementation and differs from one backed to another.
The common metrics includes
LRMSEndTime. There are also typically some memory and CPU usage metrics available.