[CIP-Security] [CR2.10] Response to audit processing failure 

Revision History
Revision No	Date	Change description	Author	Reviewed by
001	2022-02-21	Draft IEC-62443-4-2 CR-2.10 guideline	George Hsiao<br>Jared Wu	To be reviewed by CIP Security WG members

1. Objective 

The primary objective of this document is to create a guideline for protection of critical system functions in case of audit processing failiure. Without appropriate response to audit processing failure, an attacker’s activities can go unnoticed, and evidence of whether or not the attack led to a breach can be inconclusive.

2. Common Approach for Response to audit processing failure 

The audit logs are stored with-in a pre-allocated storage capability. The CIP user should consider the types of audit logging to be performed and the audit log processing requirements when allocating audit log storage capacity. Allocating sufficient audit log storage capacity reduces the likelihood of capacity exceeded and resulting in the potential loss or reduction of audit logging capability.

The CIP platform should consider to response for the audit processing failure

2.1. Alert the allocated audit log storage volume is nearly full 

The CIP user shall consider to monitor the audit log storage volume usage. Alert to the organization-defined personnel or roles in organization-defined time period.

2.2. Take the actions to response to audit log processing failure 

Audit logging process failures include software and hardware errors, failures in audit log capturing mechanisms, and reaching or exceeding audit log storage capacity. Organization-defined actions include overwriting oldest audit records, shutting down the system, and stopping the generation of audit records.

3. CIP Features for Response to Audit Processing Failure 

This requirement is supported by the CIP platform. However, this requirement cannot be met by the CIP platform alone. This will be done by the application developers using the CIP platform. For example, building a GUI in application to configure auditd configuration.

3.1. auditd 

3.1.1. Space Left Action for auditd

auditd provided in CIP can be used to detect and set the appropriate action when the free space in the filesystem containing log files drops drop to a specified threshold

CIP user can configure space_left and space_left_action parameters in /etc/audit/auditd.conf to spcify the remaining space (in megabytes) for low disk alert and the what action (ignore, syslog, rotate, email, exec, suspend, single, halt) to take.

In example below, warning email will be sent to email account specified in action_mail_acct parameter when the free space in the filesystem containing log files drop below 75 megabytes

space_left = 75
space_left_action = email

3.1.2. Disk Error and Disk Full Detection for auditd

Configure disk_full_action and disk_error_action in /etc/audit/auditd.conf to suspend when disk got error or full. The actions are ignore, syslog, rotate (for disk full only), exec, suspend, single, and halt. For example, action “syslog” means that it will issue a warning to syslog.

disk_full_action = syslog
disk_error_action = syslog

3.1.3. The max log file action for auditd

Configure the max_log_file, max_log_file_action and num_logs in auditd.conf. The actions areignore, syslog, suspend, rotate and *keep_logs**.*

In example below, the max_log_file is 30 M and number of log files to keep is 6 as “rotate” action is given. Their multiplied value is equal to 180 MB.

max_log_file = 30
num_logs = 6
max_log_file_action = ROTATE

3.2. The log daemon not support the space left, error detection or max log file features 

3.2.1. The space left action for the log daemon that does not support space left detection

Use the cron daemon on CIP platform to monitor the disk usage periodic and send a mail to alert the organization-defined personnel or roles.

3.2.1.1. Use df to report the storage usage

$ df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }'
28% /dev/sda1`

Save this result in a variable

$ disk_usage = `df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }'`

3.2.1.2. Use awk and cut to get the storage usage number

$ echo $disk_usage | awk '{print $1}'|cut -d '%' -f1
28

Keep this result in a variable

$ usage = `df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }'`

3.2.1.3. Compare the disk usage and send a mail to alert

if [ "$usage" -ge "80" ]; then
      logger -t "system.storage" -p syslog.warn "Running out of space \"$partition ($usage%)\""
      mail -s "Alert: Almost out of disk space $usage%" your@email.tld
fi

Use the above disk usage code in a script for checking disk usage and alert by e-mail if over the threashold. Setup cron job to monitor the disk usage hourly.

$ cp diskAlert.sh /etc/cron.hourly/
$ chmod +x /etc/cron.hourly/diskAlert.sh

3.2.2. For the log daemon that does not support disk error detection

Alert when within the predefined time period to the predefined person when the audit processing failure occurs

3.2.2.1. Use smart tool to report the disk health

$ smartctl -H /dev/sda
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.9.0-16-686] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Keep the SMART Health Status in a variable.

$ SMART_status=`smartctl -H /dev/sda|grep 'SMART Health Status'| awk '{ print $4 }'`

3.2.2.2. Check for disk bad blocks or bad sector using badblocks.

$ badblocks -v /dev/sda1
Checking blocks 0 to 16777215
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Keep the badblocks scan result in a variable.

$ BADBLOCK_status=$?

3.2.2.3. Alert if one of the above disk health status is not OK.

if [ "$SMART_status" != "OK" -o "$BADBLOCK_status" != "0" ]; then
      logger -t "system.storage" -p syslog.warn "Disk error detected"
      mail -s "Alert: Disk error detected" your@email.tld
fi

Use the above disk error code in a diskError.sh script and setup cron job to monitor the disk error periodically.

$ cp diskError.sh /etc/cron.hourly/
$ chmod +x /etc/cron.hourly/diskError.sh

3.2.3. For the log daemon that does not support log rotation

Use logrotate to limit the disk space usage. In /etc/logrotate.config and all the files in /etc/logrotate.d/* to rotate the log file.

This example we configure /etc/logrotate.d/rsyslog to rotate /var/log/syslog while it overs the size 2M with only 3 rotation.

/var/log/syslog

{
  {
    rotate 3
    maxsize 2M
    ...
  }
}

Reference 

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf#page=95 https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf#page=96 https://manpages.debian.org/testing/auditd/auditd.conf.5.en.html https://linux.die.net/man/8/logrotate https://linux.die.net/man/8/syslog-ng https://linux.die.net/man/1/df https://man7.org/linux/man-pages/man5/crontab.5.html https://www.simplified.guide/linux/disk-error-check