Yet another IT blog...

Monitoring CISCO ACLs with ELK stack 5.4 (part3)

Alejandro Olivan Alvarez -

Hi again world!

So, let's continue this series of articles about setting up a little, sigle server, all-in-one, ELK environment to draw nice dashboards about our CISCO labs.

In this third article I'm going to dump my notes, and configuration for Logstash.
I'll briefly describe the installation and focus on the configuration, which will include the filtering patterns I tailored to get my CISCO 2801 ACLs and PIX515e (running ASA 8.x) ACLs log lines processed by logstash.

Find previous articles here:

  • part1 talks about overall ELK, environment, repositories and so...
  • part2 covers setting up the Elasticsearch part of the stack.

Same as on the second article, in this one I just focus on the matter, asuming that environment is ready, and further, that elasticsearch is up and running.

Installing with apt-get

Again, once we got the repo ready, installing ELK software components is pretty straightforward, in this case:

apt-get install logstash

 
Pretty simple uh? :-)

Prepare the environment:

First, at the very system level, Logstash may have permission problems when accesing the log files present in the filesystem. Permissions are something we, Linux admins, do have to always keep in mind...
Logstash process runs as the logstash user, and an easy way to get access to most log files at Debian's default log folder is to add logstash user to the adm group (probably this is not the best or secure way to do this).

usermod -a -G adm logstash

 
I have to say that I have dealt with problems in the past, regarding logstash and permissions, with services storing log files on folders whith certain sets of ownerships and permissions (I remember Icecast2 now, for example...).
So keep an eye on logstash log at /var/log/logstash folder and so, because log file permissions and folder permissions had been among the top root causes of the problems I faced setting up logstash.

Now, Regarding logstash itself, I will first install/enable a plugin.
It is the translate plugin, which without it some things on some of my filters do not work, and make even logstash to refuse to start.
I guess it is not necessary here, I'm not sure, but it doesn't hurt anyways if it is not strictly needed by our setup.
As you see, installing plugins is also very easy:

/usr/share/logstash/bin/logstash-plugin install logstash-filter-translate

 
Second, since I want logstash to add geolocation information based on IP addresses on the ACLs, I will prepare the free MaxMind's GeoLite database environment for the logstash installation.
Basically I'll download the database file, uncompress it, put it in a proper location and set the necessary permissions:

mkdir /usr/share/logstash/vendor/geoip
wget -N http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
tar -xvf GeoLite2-City.tar.gz
cd GeoLite2-City_*
mv GeoLite2-City.mmdb /usr/share/logstash/vendor/geoip
chown -R logstash:logstash /usr/share/logstash/vendor/geoip
rm -rf GeoLite2-City*

 

Logstash 'patterns'

The actual filtering of every log line that logstash process performs includes, very often (although not necessarely only) a process of matching the log line against a candidate pattern, and more specifically, a grok pattern.
This is both, because the grok regular expression language is a very handy way to match a log line identifying its pieces of information substrings, and because the grok-filter plugin is included and enabled by default in logstash install.
Upon match, the grok syntax, and the logstash grok-filter plugin, enables logstash to decompose the original log line into all of its components, as independent (yet still related to a same event) pieces of information, that can be classified, converted (from text to an integer or float, so statistics can be performed, for instance), expanded (gettin aditional geolocation data from an IP address), and so.

Since grok match patterns could be very very long and complex, the way to mitigate this is the usage of grok's nested patterns capability.
For example, if you know several of your services, although generating different log-line structures as a whole, do share a common partial structure (a common timestamp, for example), you could write a grok-pattern separatelly for just that common part, and call it from the other patterns.
Moreover, a big library of common, useful or often-used library of patterns is available and ready to use in your logstash install, from simple examples such as %{MONTH}, %{MONTHDAY}, %{TIME} to a pre-cooked %{COMBINEDAPACHELOG} that, you guess, will process Apache log lines with just a one-word grok pattern!!!

Take a look here (it is probably outdated) to see a common, ready to use, set of patterns available in logstash, and how they're all build from very simple to more complex ones, by reusing the code.

You can look and find CISCO logstash patterns around, but the ones I tried didn't work to me, were very heavy, or for models I didn't own, so, as often, I ended up writing my own patterns.
There are several tools that help writng grok patterns, but I would like to mention this web application: The Grok Constructor

So, I wrote two patterns, one that matches log lines generated by my 2801, and another for my PIX515e, and put them in an appropriate location with correct permissions:

mkdir /usr/share/logstash/patterns

 
Using a text editor (nano, vi, etc...) I created the following file /usr/share/logstash/patterns/cisco-ios-acl whith the following content, for the 2801 logs:

CISCOIOSTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
CISCOIOSLOG %{WORD:log_facility}-%{INT:log_severity}-%{WORD:log_mnemonic}:
CISCOIOSACL %{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{POSINT:syslog_sequence_number}: \*%{CISCOIOSTIMESTAMP:acl_timestamp}: \%%{CISCOIOSLOG} list %{DATA:ACL_name} %{WORD:ACL_action} %{WORD:proto} %{IP:src_ip}\(%{INT:src_port}\) -> %{IP:dst_ip}\(%{INT:dst_port}\)\, %{INT:packets} packet

 
And also the following file /usr/share/logstash/patterns/cisco-pixasa-acl whit the following content for the PIX/ASA logs:

CISCOPIXASALOG %{WORD:log_facility}-%{INT:log_severity}-%{INT:log_sequence_number}:
CISCOPIXASAACL %{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} : \%%{CISCOPIXASALOG} %{WORD:proto} access denied by ACL from %{IP:src_ip}/%{INT:src_port} to %{WORD:interface}:%{IP:dst_ip}/%{INT:dst_port}

 
Then, setup the permissions:

chown -R logstash:logstash /usr/share/logstash/patterns

 
Basically, what i have done is to prepare a pair of single-word patterns, %{CISCOIOSACL} and %{CISCOPIXASAACL}, that I will use in the actual logstash filter configuration files, when calling grok filtering, instead of having to paste there all that grok stuff.
So, configuration files remain just configuration, not matching, clean and homogeneous in format.

 

Configuration files

Logstash configuration can, like many other services, read its configuration from either a single file, or from a set of separate ones.
The difference is that here, in logstash, order matters, since the configuration not only defines how does logstash reads incoming logs (inputs) and where does it sends them at the end of the processing (outputs), but also defines a set of filters, that conditionally apply to the currently processed line or not. In other words, the configuration has (or can have) somehow some programmatical logic.
For instance, an early filter may be use for early classification and detection of the kind of log line, adding appropriate tags, so only selected filters are executed afterwards if those tags are present.

Logstash configuration will be build-up from the content of files present in the folder /etc/logstash/conf.d. The reading of the files will be done alfabetically.
So, a common practice is to name the files there with starting numbers to control the order.
I also like the practice of putting actual configuration files in a separate folder and 'enable' them as needed by linking them from the config folder.
So, I put my files at /etc/logstash/logstash.d and I link them as needed from /etc/logstash/conf.d

I also like the practice of using the first file, 01-inputs.conf to configure just the inputs, a 10-filter.conf file to put there common early clasification tasks, and have as last file 99-outputs.conf to configure the output configuration.
Then, I use a different file for every service this logstash instance is processing, from 11 to 98, XX-someservice.conf.

Here I show a tree example of logstash config files in a server at the logstash.d folder...

/etc/logstash/logstash.d
├── 01-inputs.conf
├── 10-filter.conf
├── 20-ciscoiosacl.conf
├── 21-ciscopixasaacl.conf
├── 22-pfsense2.3.4.conf
├── 23-suricata3.2.1.conf
├── 31-apache.conf
└── 99-outputs.conf

 
... that is processing logs only from pfSense firewalls running Suricata IDS/IPS, so there is a link in the conf.d folder for just those services:

/etc/logstash/conf.d
├── 01-inputs.conf -> ../logstash.d/01-inputs.conf
├── 10-filter.conf -> ../logstash.d/10-filter.conf
├── 22-pfsense2.3.4.conf -> ../logstash.d/22-pfsense2.3.4.conf
├── 23-suricata3.2.1.conf -> ../logstash.d/23-suricata3.2.1.conf
└── 99-outputs.conf -> ../logstash.d/99-outputs.conf

 
So, in the following sections I'm going to show the contents of the 5 files that will involve this setup, one by one.
Here's the tree list for reference, as would appear in /etc/logstash/conf.d (either being the actual files or links to them, this is up to you):

├── 01-inputs.conf
├── 10-filter.conf
├── 20-ciscoiosacl.conf
├── 21-ciscopixasaacl.conf
└── 99-outputs.conf

 

Config file: 01-inputs.conf

Note that here I'm using log files that have been captured by rsyslogd, in the same Debian lab server that runs ELK, as they're sent by my 2801 and PIX515e to remote syslog server. So, by logstash, the input is local log files.
It is possible for logstash itself to capture logs, as a kind of syslog server, by making it listening on TCP/UDP ports as input, it is really a very powerful piece of software!

input {
  file {
    type => "cisco-ios-acl"
    path => [ "/var/log/CISCO/yourIOSlogs.log" ]
  }

  file {
    type => "cisco-pixasa-acl"
    path => [ "/var/log/CISCO/yourPIXASAlogs.log" ]
  }
}

 

Config file: 02-filter.conf

I use this file to add aditional information, classify things, or things like that.
Here's a very basic example I use as a starting point.

filter {
  if [type] == "cisco-ios-acl" {
    mutate {
      add_tag => "cisco-ios-acl"
      add_field => {
        "syslog_server_name" => "syslogserverX"
        "syslog_server_type" => "Debian"
        "syslog_server_domain" => "example.com"
      }
    }
  }

  if [type] == "cisco-pixasa-acl" {
    mutate {
      add_tag => "cisco-pixasa-acl"
      add_field => {
        "syslog_server_name" => "syslogserverY"
        "syslog_server_type" => "CentOS"
        "syslog_server_domain" => "example.net"
      }
    }
  }

}

 

Config file: 20-ciscoiosacl.conf

Here's the config part responsible of processing IOS logs from my 2801.

  • Note that a very early conditional if, that looks for a certain tag in the current log line (which I added in the early filtering), ensures that the processing only occurs if the line does match the criteria.

  • Also note in the grok section how do I tell logstash where to look for custom patern files, and how I call my previously created grok pattern in the match stanza, instead of pasting there an actual big grok pattern.

  • Finally note that here, upon IP addresses being identifiable on the log line, logstash will add Geolocation data to the log line information.

    filter {
    if "cisco-ios-acl" in [tags] {

      grok {
    patterns_dir => [ "/usr/share/logstash/patterns" ]
    match => { "message" => "%{CISCOIOSACL}" }
  }

  date {
    match => [ "acl_timestamp", "MMM dd HH:mm:ss" ]
    locale => "en"
  }

  if [src_ip]  {
    geoip {
      source => "src_ip"
      target => "geoip"
      database => "/usr/share/logstash/vendor/geoip/GeoLite2-City.mmdb"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }

  if "_grokparsefailure" not in [tags] {
    mutate {
      remove_field => "message"
      add_tag => "ciscoiosacl-match"
    }
  }

  if "_grokparsefailure" in [tags] {
    mutate {
      add_tag => "ciscoiosacl-mismatch"
    }
    #drop { }
  }

}

    }

 

Config file: 21-ciscopixasaacl.conf

This line has the same structure and working logic as the other one, but here it will only apply when a log line from my Pix515e 'passes' through the 'config logic' after have being read from the file and classified...

filter {
  if "cisco-pixasa-acl" in [tags] {

    grok {
      patterns_dir => [ "/usr/share/logstash/patterns" ]
      match => { "message" => "%{CISCOPIXASAACL}" }
    }

    date {
      match => [ "syslog_timestamp", "ISO8601" ]
      locale => "en"
    }

    if [src_ip]  {
      geoip {
        source => "src_ip"
        target => "geoip"
        database => "/usr/share/logstash/vendor/geoip/GeoLite2-City.mmdb"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
      mutate {
        convert => [ "[geoip][coordinates]", "float" ]
      }
    }

    if "_grokparsefailure" not in [tags] {
      mutate {
        remove_field => "message"
        add_tag => "ciscopixasaacl-match"
      }
    }

    if "_grokparsefailure" in [tags] {
      mutate {
        add_tag => "ciscopixasaacl-mismatch"
      }
      #drop { }
    }

  }
}

 

Config file: 99-outputs.conf

This file defines the final destination of every processed log line.
You may send them to several possible destinations, from another logstash instance through TCP/UDP for further processing, to an intermediate cache server such as redis, or, typically, to an Elasticsearch database (just to mention the ones I have used, but there's more!).

  • Note so, that IP address shown here is 127.0.0.1, since Elasticsearch is running on the same machine, but in production you may inidicate several IP addresses from ingest nodes from an Elasticsearch cluster.
    Also of interest here is that at this file, depending on the type of log line, I'll clasify information on dedicated Elasticsearch Indices, so information is grouped at hte database.

  • Also note that any non-identified information eventually arriving here will be dumped to Elasticsearch Indexed as bulk.

    output {

    #Enable stdout
#Only selected logs are routed to stdout
#Repalce "DEBUGTAG" tag to whatever filter you're debugging
#if "DEBUGTAG" in [tags]{
#  stdout {
#    codec => rubydebug
#  }
#}

if "_grokparsefailure" not in [tags] {
  if "ciscoiosacl-match" in [tags] {
     elasticsearch {
       hosts => ["127.0.0.1:9200"]
       index => "logstash-ciscoiosacl-%{+YYYY.MM.dd}"
    }
  } else if "ciscopixasaacl-match" in [tags] {
    elasticsearch {
      hosts => ["127.0.0.1:9200"]
      index => "logstash-ciscopixasaacl-%{+YYYY.MM.dd}"
    }
 } else {
    elasticsearch {
      hosts => ["127.0.0.1:9200"]
      index => "logstash-bulk-%{+YYYY.MM.dd}"
    }
  }
}

    }

 

Enabling the config files

If you have your files directly at /etc/logstash/conf.dyour're done.
If, like me, you like to have them separated, you may find useful this copy/paste (I have created my config files at /etc/logstash/logstash.d)

cd /etc/logstash/conf.d
ln -s ../logstash.d/01-inputs.conf
ln -s ../logstash.d/10-filter.conf
ln -s ../logstash.d/20-ciscoiosacl.conf
ln -s ../logstash.d/20-ciscopixasaacl.conf
ln -s ../logstash.d/99-outputs.conf

 
We're done with configuration!

 

Running Logstash...

This is tha las step... here we go:

systemctl restart logstash.service

 
Check it out:

root@server:/etc/logstash/conf.d# service logstash status
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled)
   Active: active (running) since Sun 2017-06-04 21:53:55 CEST; 59s ago
 Main PID: 32658 (java)
   CGroup: /system.slice/logstash.service
           └─32658 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly ...

Jun 04 21:53:55 server01 systemd[1]: Started logstash.

 
We are done!!!
Now, if everything goes well, you may query your Elasticsearch database looking for indices and browsing their documents.
In the next article I'll talk a little bit about Kibana... but for now...

Have an elastic day!!!