In my last post I explained how do I manage settiing munin in pfSense systems, so, you may guess, I'm one of those monitoring fans around...

So, that we have plenty of information on how to get our pfSense boxes monitoried by ELK (Elasticsearch Logstash Kibana) stack, I'm going to post how do I get my boxes monitored in an incredible monitoring and graphing tool stack: Nagios + Check_mk + PNP4Nagios.

If you regard the list of abailable packages at pfSense 2.2.x you may notice we DO have a Check_mk package ready to install, whereas this package has fallen from 2.3 release. (this makes me still stick to 2.2.6).
The reason to drop the package is logical: it is unmantained and, although it provides some GUI about Check_mk and auto-configures necessary firewall rules transparently (like any pfSense package) the provided agent is obsolete, and does not work out of the box.

Probably, the original package included Check_MK FreeBSD 8.x agent, and, upon BSD 10 migration, it ceased to work.
Anyhow, you can have it working by just manually overwrite the agent.... so here we go!

Installing nano

You will need to ssh to your pfSense appliance, select option 8 in the menu to access CLI, and overwrite the check_mk agent file at /usr/local/bin/check_mk_agent .
If you feel no comfortable with tools like vi, FreeBSD does include the nano editor:

First, consider wether your install is a nanobsd setup, if so, it is read-only, so you need to remount it read-write mode (do not forget to do the contrary once we're finished!)

/etc/rc.conf_mount_rw

 
If it is the first time you access the CLI or you still have never used the package manager, install it by issuing the following command (remember, if you're in nanobsd all this can be veeeery slow, be patient):

pkg

 
Then we install nano like this:

pkg update
pkg install nano
rehash

Pasting the new agent script

We are ready. We will do a backup of the currently installed file and we will create a new, working one:

mv /usr/local/bin/check_mk_agent /usr/local/bin/check_mk_agent.bak
nano /usr/local/bin/check_mk_agent

 
Now, you simply paste this modified shell script into it:

#!/bin/sh
# +------------------------------------------------------------------+
# |                  _               _        __  __ _  __           |
# |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
# |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
# |           | |___| | | |  __/ (__|   <    | |  | | . \            |
# |            \    |_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
# |                                                                  |
# | Copyright Mathias Kettner 2014             mk@mathias-kettner.de |
# +------------------------------------------------------------------+
#
# This file is part of Check_MK.
# The official homepage is at http://mathias-kettner.de/check_mk.
#
# check_mk is free software;  you can redistribute it and/or modify it
# under the  terms of the  GNU General Public License  as published by
# the Free Software Foundation in version 2.  check_mk is  distributed
# in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
# out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
# PARTICULAR PURPOSE. See the  GNU General Public License for more de-
# ails.  You should have  received  a copy of the  GNU  General Public
# License along with GNU Make; see the file  COPYING.  If  not,  write
# to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
# Boston, MA 02110-1301 USA.

# Author: Lars Michelsen <lm@mathias-kettner.de>
#         Florian Heigl <florian.heigl@gmail.com>
#           (Added sections: df mount mem netctr ipmitool)

# NOTE: This agent has beed adapted from the Check_MK linux agent.
#       The most sections are commented out at the moment because
#       they have not been ported yet. We will try to adapt most
#       sections to print out the same output as the linux agent so
#       that the current checks can be used.

# This might be a good source as description of sysctl output:
# http://people.freebsd.org/~hmp/utilities/satbl/_sysctl.html

# Remove locale settings to eliminate localized outputs where possible
export LC_ALL=C
unset LANG

export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_TMPDIR="/var/run/check_mk"


# Make sure, locally installed binaries are found
PATH=$PATH:/usr/local/bin

# All executables in PLUGINSDIR will simply be executed and their
# ouput appended to the output of the agent. Plugins define their own
# sections and must output headers with '<<<' and '>>>'
PLUGINSDIR=$MK_LIBDIR/plugins

# All executables in LOCALDIR will by executabled and their
# output inserted into the section <<<local>>>. Please refer
# to online documentation for details.
LOCALDIR=$MK_LIBDIR/local


# close standard input (for security reasons) and stderr
#if [ "$1" = -d ]
#then
#    set -xv
#else
#    exec </dev/null 2>/dev/null
#fi

# Runs a command asynchronous by use of a cache file

echo '<<<check_mk>>>'
echo Version: 1.2.7i1
echo AgentOS: freebsd



osver="$(uname -r)"
is_jailed="$(sysctl -n security.jail.jailed)"


# Partitionen (-P verhindert Zeilenumbruch bei langen Mountpunkten)
# Achtung: NFS-Mounts werden grundsaetzlich ausgeblendet, um
# Haenger zu vermeiden. Diese sollten ohnehin besser auf dem
# Server, als auf dem Client ueberwacht werden.

echo '<<<df>>>'
# no special zfs handling so far, the ZFS.pools plugin has been tested to
# work on FreeBSD
if df -T > /dev/null ; then
    df -kTP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)'
else
    df -kP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)' | awk '{ print $1,"ufs",$2,$3,$4,$5,$6 }'
fi

# Check NFS mounts by accessing them with stat -f (System
# call statfs()). If this lasts more then 2 seconds we
# consider it as hanging. We need waitmax.
#if type waitmax >/dev/null
#then
#    STAT_VERSION=$(stat --version | head -1 | cut -d" " -f4)
#    STAT_BROKE="5.3.0"
#
#    echo '<<<nfsmounts>>>'
#    sed -n '/ nfs /s/[^ ]* \([^ ]*\) .*/\1/p' < /proc/mounts |
#        while read MP
#  do
#   if [ $STAT_VERSION != $STAT_BROKE ]; then
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" || \
#    echo "$MP hanging 0 0 0 0"
#   else
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" && \
#      printf '\n'|| echo "$MP hanging 0 0 0 0"
#   fi
#  done
#fi

# Check mount options.
# FreeBSD doesn't do remount-ro on errors, but the users might consider
# security related mount options more important.
echo '<<<mounts>>>'
mount -p -t ufs

# processes including username, without kernel processes
echo '<<<ps>>>'
COLUMNS=10000
if [ is_jailed = 0 ]; then
    ps ax -o state,user,vsz,rss,pcpu,command | sed -e 1d  -e '/\([^ ]*J\) */d' -e 's/*\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\2,\3,\4,\5) /'
else
    ps ax -o user,vsz,rss,pcpu,command | sed -e 1d -e 's/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /'
fi


# Produce compatible load/cpu output to linux agent. Not so easy here.
echo '<<<cpu>>>'
echo `sysctl -n vm.loadavg | tr -d '{}'` `top -b -n 1 | grep -E '^[0-9]+ processes' | awk '{print $3"/"$1}'` `sysctl -n kern.lastpid` `sysctl -n hw.ncpu`

# Calculate the uptime in seconds since epoch compatible to /proc/uptime in linux
echo '<<<uptime>>>'
  up_seconds=$(( `date +%s` - `sysctl -n kern.boottime  | cut -f1 -d\, | awk '{print $4}'`))
idle_seconds=$(ps axw | grep idle | grep -v grep | awk '{print $4}' | cut -f1 -d\: )

# second value can be grabbed from "idle" process cpu time / num_cores
echo "$idle_seconds $up_seconds"


# Platten- und RAID-Status von LSI-Controlleren, falls vorhanden
#if which cfggen > /dev/null ; then
#   echo '<<<lsi>>>'
#   cfggen 0 DISPLAY | egrep '(Target ID|State|Volume ID|Status of volume)[[:space:]]*:' | sed -e 's/ *//g' -e 's/:/ /'
#fi


# Multipathing is supported in FreeBSD by now
# http://www.mywushublog.com/2010/06/freebsd-and-multipath/
if kldstat -v | grep g_multipath > /dev/null ; then
    echo '<<<freebsd_multipath>>>'
    gmultipath status | grep -v ^Name
fi


# Soft-RAID
echo '<<<freebsd_geom_mirrors>>>'
gmirror status | grep -v ^Name

# Performancecounter Kernel
echo "<<<kernel>>>"
date +%s
forks=`sysctl -n vm.stats.vm.v_forks`
vforks=`sysctl -n vm.stats.vm.v_vforks`
rforks=`sysctl -n vm.stats.vm.v_rforks`
kthreads=`sysctl -n vm.stats.vm.v_kthreads`
echo "cpu" `sysctl -n kern.cp_time | awk ' { print $1" "$2" "$3" "$5" "$4 } '`
echo "ctxt" `sysctl -n vm.stats.sys.v_swtch`
echo "processes" `expr $forks + $vforks + $rforks + $kthreads`

# Network device statistics (Packets, Collisions, etc)
# only the "Link/Num" interface has all counters.
echo '<<<lnx_if:sep(58)>>>'
#date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    netstat -inb | grep ovpn | grep Link | awk '{print"\t"$1":\t"$7"\t"$4"\t"$5"\t"$6"\t0\t0\t0\t0\t"$10"\t"$8"\t"$9"\t0\t0\t0\t0\t0"}'
else
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
fi


# State of LSI MegaRAID controller via MegaCli.
# To install: pkg install megacli
if which MegaCli >/dev/null ; then
    echo '<<<megaraid_pdisks>>>'
    MegaCli -PDList -aALL -NoLog < /dev/null | egrep 'Enclosure|Raw Size|Slot Number|Device Id|Firmware state|Inquiry'
    echo '<<<megaraid_ldisks>>>'
    MegaCli -LDInfo -Lall -aALL -NoLog < /dev/null | egrep 'Size|State|Number|Adapter|Virtual'
    echo '<<<megaraid_bbu>>>'
    MegaCli -AdpBbuCmd -GetBbuStatus -aALL -NoLog < /dev/null | grep -v Exit
fi


# OpenVPN Clients. 
# Correct log location unknown, sed call might also be broken
if [ -e /var/log/openvpn/openvpn-status.log ] ; then
    echo '<<<openvpn_clients:sep(44)>>>'
    sed -n -e '/CLIENT LIST/,/ROUTING TABLE/p' < /var/log/openvpn/openvpn-status.log  | sed -e 1,3d -e '$d' 
fi


if which ntpq > /dev/null 2>&1 ; then
   echo '<<<ntp>>>'
   # remote heading, make first column space separated
   ntpq -np | sed -e 1,2d -e 's/^\(.\)/\1 /' -e 's/^ /%/'
fi


# Checks for cups monitoring
#if which lpstat > /dev/null 2>&1; then
#  echo '<<<cups_queues>>>'
#  lpstat -p
#  echo '---'
#  for i in $(lpstat -p | grep -E "^(printer|Drucker)" | awk '{print $2}' | grep -v "@"); do
#    lpstat -o "$i"
#  done
#fi

# Heartbeat monitoring
#if which cl_status > /dev/null 2>&1; then
#  # Different handling for heartbeat clusters with and without CRM
#  # for the resource state
#  if [ -S /var/run/heartbeat/crm/cib_ro ]; then
#    echo '<<<heartbeat_crm>>>'
#    crm_mon -1 -r | grep -v ^$ | sed 's/^\s/_/g'
#  else
#    echo '<<<heartbeat_rscstatus>>>'
#    cl_status rscstatus
#  fi
#
#  echo '<<<heartbeat_nodes>>>'
#  for NODE in $(cl_status listnodes); do
#    if [ $NODE != $HOSTNAME ]; then
#      STATUS=$(cl_status nodestatus $NODE)
#      echo -n "$NODE $STATUS"
#      for LINK in $(cl_status listhblinks $NODE 2>/dev/null); do
#        echo -n " $LINK $(cl_status hblinkstatus $NODE $LINK)"
#      done
#      echo
#    fi
#  done
#fi

# Number of TCP connections in the various states
echo '<<<tcp_conn_stats>>>'
netstat -na | awk ' /^tcp/ { c[$6]++; } END { for (x in c) { print x, c[x]; } }'


# Postfix mailqueue monitoring
#
# Only handle mailq when postfix user is present. The mailq command is also
# available when postfix is not installed. But it produces different outputs
# which are not handled by the check at the moment. So try to filter out the
# systems not using postfix by searching for the postfix user.
#
# Cannot take the whole outout. This could produce several MB of agent output
# on blocking queues.
# Only handle the last 6 lines (includes the summary line at the bottom and
# the last message in the queue. The last message is not used at the moment
# but it could be used to get the timestamp of the last message.
#if which mailq >/dev/null 2>&1 && getent passwd postfix >/dev/null 2>&1; then
#  echo '<<<postfix_mailq>>>'
#  mailq | tail -n 6
#fi

#Check status of qmail mailqueue
#if type qmail-qstat >/dev/null
#then
#   echo "<<<qmail_stats>>>"
#   qmail-qstat
#fi

# check zpool status
#if [ -x /sbin/zpool ]; then
#   echo "<<<zpool_status>>>"
#   /sbin/zpool status -x | grep -v "errors: No known data errors"
#fi

# Memory Usage
# currently we'll need sysutils/muse for this.
if [ -x /usr/local/bin/muse ]
then
echo '<<<mem>>>'
# yes, i don't know sed well.
muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'

# comment it out in normal (not nanobsd) installs
swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'
# comment it out in nanobsd or swapless installs, otherwise inventorization will fail
# swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "0" kB\nSwapFree: "0" kB" }'

fi



# Fileinfo-Check: put patterns for files into /etc/check_mk/fileinfo.cfg
if [ -r "$MK_CONFDIR/fileinfo.cfg" ] ; then
    echo '<<<fileinfo:sep(124)>>>'
    date +%s
    stat -f "%N|%z|%m" $(cat "$MK_CONFDIR/fileinfo.cfg")
fi

 
almost ready, make the new script executable:

chmod +x /usr/local/bin/check_mk_agent

 
if your pfSense is nanobsd, turn it back read-only:

/etc/rc.conf_mount_ro

 
You're done! declare the firewall as a new host in WATO and enjoy!