Tuning tips for ARC and FAQ

Note

Work-in-progress!

CA certificates

The ENV/PROXY RTE by default makes a copy of CA certificates in each job’s session directory. Copying many small files to a shared file system is potentially expensive so it is preferred to have CA certificates locally installed on each worker node, and turn off the copying in ENV/PROXY with arcctl rte params-set ENV/PROXY COPY_CACERT_DIR No.

The CA certificate packages can be found in the repositories for ARC (http://www.nordugrid.org/documents/arc6/common/repos/repository.html)

You should also install the fetch-crl package and enable the services as you would on the ARC-CE.

File-system tuning

If you have your sessiondirectories and/or cache directories on NFS make sure you have increased the number of nfs kernel threads. The default is 8 which is too low. The RPCNFSDCOUNT variable should be increased to 32 or maybe even 64 or even larger depending on your system (number of worker-nodes, number of cores on the nfs servers).

How to extract backtrace of core dump file

How to extract backtrace of ARC core dump file using gdb and save the output to a file.

gdb  <path-to-arc-binary> <path-to-core-file>
(gdb) set logging file gdb.log
(gdb) set logging on
(gdb) thread apply all bt

Example: Assuming ARC is installed in default location: .. code-block:: console

gdb /usr/sbin/arched /var/log/arc/arccore/core.30547 (gdb) set logging file gdb.log (gdb) set logging on (gdb) thread apply all bt

How to extract backtraces using a-rex-backtrace-collect

a-rex-backtrace-collect processes core file(s) collected in ARC_LOGS_DIR/arccore folder and produces their backtraces. The backtrace(s) are stored in files <core_name>.backtrace. The ARC instalation location can be adjusted using ARC_LOCATION environment variable. The location of configuration file can be specified using ARC_CONFIG environment variable.

… code-block::console
a-rex-backtrace-collect