Building an IDS Sensor with Suricata & Zeek with Logs to ELK

Building an IDS Sensor with Suricata & Zeek with Logs to ELK
Over the past several years I have used multiple pre-built sensors using readily available ISO images (rockNSM, SO, OPNSense, etc) but what I was really looking for was just a sensor to parse traffic (i.e Zeek) and IDS alerts (Suricata) to ELK. To speed up the deployment of each sensors, I created a basic CentOS7 server VM where I copied all the scripts and files, I need to get Suricata & Zeek up and running. Since my ELK cluster is the recipient of these logs, it includes Elastic filebeat. I saved all the important scripts and changes into two tarballs (installation and sensor). The sensor tarball has a copy of the softflowd (netflow binary) that can be use to capture netflow data. Using this document as a template, to build the sensor, it is time to download and extract the installation tarball on the sensor to install Suricata & Zeek as well as the Elasticsearch applications filebeate, metricbeat and packetbeat if using ELK to analyze the traffic. Refer to the document to configure each of the ELK applications. There are two tarball, the first installation.tgz is to setup all the scripts listed below to install the software and the second tarball is to preconfigure some of the sensor configuration files (Suricata, Zeek, softflowd, Filebeat, Metricbeat & Packetbeat). $ wget https://handlers.sans.edu/gbruneau/scripts/installation.tgz $ wget https://handlers.sans.edu/gbruneau/scripts/sensor.tgz Extract the tarball with the scripts as follow: $ sudo tar zxvf installation.tgz -C / Install Suricata: $ sudo yum -y install suricata Install Zeek: $ sudo yum -y install zeek After Suricata & Zeek have been installed, if you plan to send the logs to Elasticsearch, install filebeat (metricbeat & packetbeat are optional). Install Filebeat: $ sudo yum -y install filebeat (metricbeat and packetbeat) The sensor.tgz tarball has Zeek configured to save the logs in JSON format which has support by most commercial products like ELK, RSA NetWitness, Splunk, etc. Extract this tarball after installing all the packages: $ sudo tar zxvf sensor.tgz -C / If the packet capture interface is other than ens160 (ifconfig), update the following files: /opt/zeek/etc/node.cfg /etc/suricata/suricata.yaml If using packetbeat: /etc/packetbeat/packetbeat.yml If using softflowd (make script executable: chmod 755 /etc/rc.local): /etc/rc.local Enable Suricata & Zeek to start on reboot: $ sudo systemctl enable suricata $ sudo systemctl enable zeek Update Suricata's rules: $ sudo /usr/bin/suricata-update update --reload-command "/usr/bin/systemctl kill -s USR2 suricata" Lets start some services: $ sudo systemctl start suricata $ sudo systemctl status suricata $ sudo systemctl start zeek $ sudo systemctl status zeek Last, configure filebeat (metricbeat & packetbeat are optional) Elasticsearch server section to send the logs to the server. To make sure nothing is missed to configure Elasticsearch applications, review this document Logging Data to Elasticsearch which contains all the steps to configure these Elastic Beats. /etc/filebeat/filebeat.yml /etc/metricbeat/metricbeat.yml /etc/packetbeat/packetbeat.yml If using any of the Beats, enable them to start on reboot: $ sudo systemctl enable filebeat $ sudo systemctl enable metricbeat $ sudo systemctl enable packetbeat Let's start Filebeat: $ sudo systemctl start filebeat $ sudo systemctl status filebeat Note: Because Suricata logs are sent to ELK with filebeat, there is an hourly cronjob that delete the previous hour logs from the /nsm/suricata directory to keep it clean and in the end requires a minimal /nsm/suricata partition documented in [4]. Since I use VMs as sensors, I exported this sensor template as an OVA, which requires minimum configuration changes for the next deployment. [1] https://github.com/irino/softflowd [2] https://handlers.sans.edu/gbruneau/elastic.htm [3] https://handlers.sans.edu/gbruneau/elk/TLS_elasticsearch_configuration.pdf [4] https://handlers.sans.edu/gbruneau/elk/Building_Custom_IDS_Sensor.pdf [5] https://handlers.sans.edu/gbruneau/scripts/installation.tgz [6] https://handlers.sans.edu/gbruneau/scripts/sensor.tgz ----------- Guy Bruneau IPSS Inc. My Handler Page Twitter: GuyBruneau gbruneau at isc dot sans dot edu	Guy 522 Posts ISC Handler Apr 10th 2021
Thread locked Subscribe	Apr 10th 2021 1 year ago
Great article! What's missing, and this is common out there, is the detailed part for configuring dashboards. Most of the time you see guide how to export to ELK, but only a few stuff, or just predefined dashboards, on what to do with this data. Thx! :)	Anonymous
Quote	Apr 12th 2021 1 year ago
Agree that is usually the difficult part to have good dashboards and good actionable information. The current version has 2 Suricata dashboards and Zeek one. However, with the introduction of the Threat module in filebeat, the current version combined with detection rules Indicator match rules, Elasticsearch is on its way to provide more accurate information. The Enrich processor is another option I use for adding meta I can use to create custom dashboards. elastic.co/guide/en/beats/filebeat/master/… elastic.co/guide/en/security/current/… elastic.co/guide/en/elasticsearch/reference/master/…	Guy 522 Posts ISC Handler
Quote	Apr 12th 2021 1 year ago

Over the past several years I have used multiple pre-built sensors using readily available ISO images (rockNSM, SO, OPNSense, etc) but what I was really looking for was just a sensor to parse traffic (i.e Zeek) and IDS alerts (Suricata) to ELK. To speed up the deployment of each sensors, I created a basic CentOS7 server VM where I copied all the scripts and files, I need to get Suricata & Zeek up and running. Since my ELK cluster is the recipient of these logs, it includes Elastic filebeat. I saved all the important scripts and changes into two tarballs (installation and sensor). The sensor tarball has a copy of the softflowd (netflow binary) that can be use to capture netflow data.

Using this document as a template, to build the sensor, it is time to download and extract the installation tarball on the sensor to install Suricata & Zeek as well as the Elasticsearch applications filebeate, metricbeat and packetbeat if using ELK to analyze the traffic. Refer to the document to configure each of the ELK applications.

There are two tarball, the first installation.tgz is to setup all the scripts listed below to install the software and the second tarball is to preconfigure some of the sensor configuration files (Suricata, Zeek, softflowd, Filebeat, Metricbeat & Packetbeat).

$ wget https://handlers.sans.edu/gbruneau/scripts/installation.tgz
$ wget https://handlers.sans.edu/gbruneau/scripts/sensor.tgz
Extract the tarball with the scripts as follow: $ sudo tar zxvf installation.tgz -C /
Install Suricata: $ sudo yum -y install suricata
Install Zeek: $ sudo yum -y install zeek

After Suricata & Zeek have been installed, if you plan to send the logs to Elasticsearch, install filebeat (metricbeat & packetbeat are optional).

Install Filebeat: $ sudo yum -y install filebeat (metricbeat and packetbeat)

The sensor.tgz tarball has Zeek configured to save the logs in JSON format which has support by most commercial products like ELK, RSA NetWitness, Splunk, etc.

Extract this tarball after installing all the packages: $ sudo tar zxvf sensor.tgz -C /

If the packet capture interface is other than ens160 (ifconfig), update the following files:

/opt/zeek/etc/node.cfg
/etc/suricata/suricata.yaml

If using packetbeat:

/etc/packetbeat/packetbeat.yml

If using softflowd (make script executable: chmod 755 /etc/rc.local):

/etc/rc.local

Enable Suricata & Zeek to start on reboot:

$ sudo systemctl enable suricata
$ sudo systemctl enable zeek

Update Suricata's rules:

$ sudo /usr/bin/suricata-update update --reload-command "/usr/bin/systemctl kill -s USR2 suricata"

Lets start some services:

$ sudo systemctl start suricata
$ sudo systemctl status suricata
$ sudo systemctl start zeek
$ sudo systemctl status zeek

Last, configure filebeat (metricbeat & packetbeat are optional) Elasticsearch server section to send the logs to the server. To make sure nothing is missed to configure Elasticsearch applications, review this document Logging Data to Elasticsearch which contains all the steps to configure these Elastic Beats.

/etc/filebeat/filebeat.yml
/etc/metricbeat/metricbeat.yml
/etc/packetbeat/packetbeat.yml

If using any of the Beats, enable them to start on reboot:

$ sudo systemctl enable filebeat
$ sudo systemctl enable metricbeat
$ sudo systemctl enable packetbeat

Let's start Filebeat:

$ sudo systemctl start filebeat
$ sudo systemctl status filebeat

Note: Because Suricata logs are sent to ELK with filebeat, there is an hourly cronjob that delete the previous hour logs from the /nsm/suricata directory to keep it clean and in the end requires a minimal /nsm/suricata partition documented in [4].

Since I use VMs as sensors, I exported this sensor template as an OVA, which requires minimum configuration changes for the next deployment.

[1] https://github.com/irino/softflowd
[2] https://handlers.sans.edu/gbruneau/elastic.htm
[3] https://handlers.sans.edu/gbruneau/elk/TLS_elasticsearch_configuration.pdf
[4] https://handlers.sans.edu/gbruneau/elk/Building_Custom_IDS_Sensor.pdf
[5] https://handlers.sans.edu/gbruneau/scripts/installation.tgz
[6] https://handlers.sans.edu/gbruneau/scripts/sensor.tgz

-----------
Guy Bruneau IPSS Inc.
My Handler Page
Twitter: GuyBruneau
gbruneau at isc dot sans dot edu

Guy

522 Posts
ISC Handler

Apr 10th 2021

Great article! What's missing, and this is common out there, is the detailed part for configuring dashboards.
Most of the time you see guide how to export to ELK, but only a few stuff, or just predefined dashboards, on what to do with this data.
Thx! :)

Anonymous

Agree that is usually the difficult part to have good dashboards and good actionable information. The current version has 2 Suricata dashboards and Zeek one. However, with the introduction of the Threat module in filebeat, the current version combined with detection rules Indicator match rules, Elasticsearch is on its way to provide more accurate information. The Enrich processor is another option I use for adding meta I can use to create custom dashboards.

elastic.co/guide/en/beats/filebeat/master/…
elastic.co/guide/en/security/current/…
elastic.co/guide/en/elasticsearch/reference/master/…

Guy

522 Posts
ISC Handler