Event Driven Compliance with SaltStack

This article is a walk through of the talk we gave at the 2020 SaltConf called ‘Event Driven Compliance’. The video of this talk is available at the end of the article.

The reader needs to have some familiarity with SaltStack. 

This is a 12 minute read.

Saltstack and Compliance

SaltStack has many exciting features for automating the state control of servers, applications and infrastructure devices such as network switches and loadbalancers. It is unique as an automation framework, as it’s entirely built around an event bus. 

Lets explore this a bit further.

One of SaltStacks features is called a Beacon. Beacons take advantage of SaltStack’s built-in event system. Beacons can be used to create events on all minions which will be sent to the Salt Master.

To compliment Beacons, another SaltStack system called Reactors are able to filter events and subscribe salt commands to those events.

Through Beacons and Reactors SaltStack can be used to form a closed-loop compliance automation tool. If a server deviates from a defined state which is part of your compliance framework an event can be sent to the Salt Master. The Salt Master can then be configured to produce notifications and automatically re-apply the state to the server.

SaltStack will also produce a difference between the what the state was changed to and the compliant server state. 

This gives all the building blocks for Event Driven Compliance.

Lets have a look at how this works. 

Beacons

Let’s have a look at Beacons first. SaltStack Beacons are plugins that monitor aspects of a server and generate events when things change. They’re written in Python and SaltStack has a bunch of them available by default but you can also write your own, custom Beacon.

Have a look a the code for a built-in Beacon in order to get a better handle on writing custom Beacons.

In order to activate beacons, we merely need to include options in the Salt Minion’s configuration. For example. if we wish to use the btmp and wtmp Beacons on the Salt Minion to generate events associated with user logon’s we can add the following minimal configuration:

  beacons:
    btmp: []
    wtmp: []

Using blank configuration options for these two Beacons means that the Salt Minion will generate an event for all logons, failed or otherwise.

Reactors

When an event reaches the Salt Master, it can be used to trigger a job on the Salt Master. The Reactors are configured in the Salt Master’s configuration file and can be used to drive Orchestrations.

The following Salt Master configuration registers a reactor btmp.sls with all events for the btmp Beacon.

  reactor:
    - 'salt/beacon/*/btmp/':
      - salt://reactor/btmp.sls

The 'salt/beacon/*/btmp/' string is the event matching string. The astrisk here is where the Salt Minion ID is placed in the event string, and an asterisk matches any value so we react to all minion generated events.

salt://reactor/btmp.sls is the reactor state file to run when an event matches the matching string above. The reactor state is available on the Salt file server.

The reactor could be something as simple as a notification for the Beacon event to show when someone has attempted, but failed to logon to a server.

report-btmp:
  runner.salt.cmd:
    - args:
      - fun: slack.post_message
      - channel: failed-logins
      - from_name: BeaconBot
      - message: "Failed login from `{{ data.get('user', '') or 'unknown user' }}` on `{{ data['id'] }}`"

NOTE: Pillar and Grain data is not available in reactors. The data available in a reactor is just the data included in the event.

Also note that the reactor is run on the Salt Master, not on the Salt Minion that generated the event. That just needs to be thought through when you’re writing the reactor.

data['id'] is the Salt Minion ID.

data.get('user', ... is data that comes from the btmp Beacon and represents the username that failed to log in successfully.

Event Data

The data in an event generated by a built-in Beacon is not usually documented. However, events can be monitored on the Salt Master with a simple command:

$ salt-run state.event pretty=true

Events contain some information that is common to all events such as the Salt Minion ID and other data that is specific to the event.

"salt/beacon/vault01-pvlx-fle.int.wanless.systems/wtmp/" : {
  "PID": 15232,
  "_stamp": "2020-10-28T22:28:53.180088",
  "action": "logout",
  "addr": 1603924132,
  "exit_status": 0,
  "hostname": "",
  "id": "vault01-pvlx-fle.int.wanless.systems",
  "inittab": "",
  "line": "pts/2",
  "session": 0,
  "time": 0,
  "type": 8,
  "user": "bjs"
}

Enforcing Compliance with Salt States

Beacons and Reactors can detect unwanted changes to server state and enforce the state to ensure compliance to a standard.

Let’s take the example of SELinux. For some compliant environments, SELinux must be set to enforcing. If someone or something disables SELinux, it is essential that this is detected, logged and acted upon.

On RHEL7 SELinux=Enforcing is a requirement of NIST 800-53 for controls AC-3,AC-3(3)(a),AU-9,SC-7(21) and CCE-27334-2

SELinux must be configured to run in Enforcing mode in /etc/selinux/config.

We have a standard state that enforces SELinux configuration and mode:

ensure_selinux_is_configured:
  file.managed:
    - name: /etc/selinux/config
    - source: salt://compliance_controls/selinux_config.j2
    - template: jinja

ensure_selinux_is_enforcing:
  selinux.mode:
    - name: enforcing

In the above, example Salt states we manage the selinux configuration file to make sure the configured mode is Enforcing and we also manage the running SELinux state to ensure it’s currently running in Enforcing mode.

When we run this state we know that the server will be running in Enforcing mode. We also know that if the server reboots it will start again in Enforcing mode.

Configuration File Beacon

We want to know when anyone changes the controlled file /etc/selinux/config as it is a file that Salt is managing as part of our compliance framework.

For this, we can use Salt’s built-in beacon called inotify which is a Linux kernel filesystem monitoring event system. We configure the following Beacon on the Salt Minion.

beacons:
  inotify:
    - files:
      /etc/selinux/config:
        mask:
          - modify
    - disable_during_state_run: True

The inotify beacon, configured as above will generate events for this Salt Minion whenever the selinux configuration file is modified.

SELinux Running Mode Beacon

The running state of SELinux is not monitored by the inotify Beacon above because it’s possible for someone to change the running state of SELinux using the command setenforce 0.

As mentioned, we can write our own custom Beacons for Salt:

def beacon(config):

    global LAST_STATUS

    _events = []
    _state = { "mode": __salt__["selinux.getenforce"]() }
    _details = {}

    if "mode" not in LAST_STATUS:
        LAST_STATUS["mode"] = "unknown"

    if _state["mode"] != LAST_STATUS["mode"] and _state["mode"].lower() != "enforcing":
        _details = {"mode": _state["mode"]}

        try:
            import subprocess
            _output = subprocess.run(["w"], stdout=subprocess.PIPE)
            _details["who"] = _output.stdout
        except:
            pass

    if 'mode' in _details:
        _events.append({"changes": [_details], "tag": "mode"})

    LAST_STATUS = _state
    return _events

Let’s go through that code a little to understand how we’ve understood a Beacon.

The beacon function is run periodically by the Salt Minion in order to generate any required events. The beacon function is responsible for detecting changes in order to generate the events as necessary.

The current state of SELinux is calculated by running the salt execution module function selinux.getenforce. It is best practice here to run salt execution modules in preference to trying to write your own functions to do the same functionality.

The result of this execution module is returned in the _state variable and we can interrogate this to see if the mode has changed and the mode is currently not Enforcing. In this case we will include some details for the new event.

As well as knowing that the mode has changed, we can also include further information which may help the company. Here, we use the Linux who command to include details about who was logged on to the server at the time the SELinux mode changed from the desired state.

The events generated by this Beacon will look similar to the following:

"salt/beacon/minion.propellent.int/selinux/mode": {
  "_stamp": "2020-10-28T11:49:27.177666",
  "changes": [{
    "mode": "Permissive",
    "who": "..."
  }],
  "id": "minion.propellent.int"
}

We’ll see which accounts were logged into the server at this time and what the SELinux mode was set to.

Event Driven Compliance and Reporting

We can use the Reactor tied to the selinux event to run a simple orchestration which will again enforce selinux state on the minion where the event was produced.

We can get the salt minion id from the data that is passed to the orchestration. We can pass the event data to the orchestration as custom pillar data to help target the correct minion:

{% set host = pillar.data['id'] %}

CM1_reenabled_selinux_on_selinux_disable_{{ host }}:
  salt.state:
    - tgt: {{ host }}
    - sls:
      - compliance_controls.selinux.mode

With Slack based reporting in the orchestration as well. You will end up seeing the compliance states in Slack like this:

Event Driven Compliance

This is a fully automatic system for detecting changes to the running and configured selinux mode on all servers. Evidence of compliant activity is available to audit in Slack.

Tada! We now have Event Driven Compliance! 

Instead of Slack or a messaging system, this could immediately open and process a ticket in your Security Incident and Event Management (SIEM) tool as necessary.

SaltConf20 Video

At SaltConf20 we talked about and demonstrated event-driven compliance. As well as covering the selinux mode, we also cover priviledged commands and a live demo of Beacons and Reactors. So check out the SaltConf talk for more information.

Play Video

Why do this?

The ultimate question covering this approach is why do this when there are a multitude of tools that can be combined to do a similar job?

There’s a few reason why this approach is valid:

  1. The tool that knows how things ‘should’ be configured and working is the right tool to measure if they aren’t.
  2. See above for the same but with remedial activities 
  3. Everything happens in the same tool, so one agent, and one engineering skill set. 
  4. SaltStack allows full customisation, so this can be extended to specific use cases not covered by other tools, bespoke business applications etc.
  5. Because it’s all code and config file driven, the entire configuration can be source controlled. Compare this to a multi tooled approach where generally the only approach is to rely on change management for a full picture of changes…

If this is something that would benefit your organisation, Event Driven Compliance is a service we offer via our SaltOps service. Please get in contact if you’t like to learn more.

Let us know what you think in the comments section.

Leave a Reply

Your email address will not be published. Required fields are marked *