Using Syslog-NG to send Fortinet Logs to Splunk

Introduction

As a network security professional, we are constantly tasked with continuous monitoring of different types of network equipment. This could be things like next-generation firewalls, web-application firewalls, identity management, secure email gateways, etc. Expanding beyond the network, we can incorporate logging from our host endpoints to help provide a complete picture of what is occurring in our environment during a snapshot of time. Having a centralized location where these logs can be ingested and insight can be drawn from is pivotal to our success. This is typically handled by a Security Incident and Event Management platform (SIEM).

Splunk for Enterprise is one of these solutions that gives you single pane of glass access to all of your machine data generated in your environment. This article will focus on the beginning steps to ship data from your Fortinet equipment into the Splunk platform for processing.

What is Syslog?

It is typical that many enterprise level network devices support a standard method for sending logs from that device across the network. The most common term referring to this “syslog”. Here is the definition of syslog from Wikipedia below:

In computing, syslog /ˈsɪslɒɡ/ is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, indicating the software type generating the message, and assigned a severity level.

https://en.wikipedia.org/wiki/Syslog

Having the logs shipped off the device generating them is very useful for a multitude of reasons:

1. You do not have to access the device directly to see the local events occurring on it.
2. If the device becomes unavailable, you have a record of what led up to that event potentially providing insight as to the cause prior to it becoming unavailable.
3. The logs can be analyzed and manipulated off of the platform reserving the resources of that platform to be dedicated to its purpose.
4. These logs can be shipped to a device that has more storage capacity which will allow for the logs to be retained a lot longer than if they were limited to only residing on the host that generated them.

Shipping the Logs

In many cases, these enterprise network devices send their logs directly to the platform that processes their logs over the network. It is so common that syslog has a standard port defined to it. It uses “User Datagram Protocol (UDP) port 514” as its well known port. This is fine in some cases but it is not robust because the transmission of these logs is not considered “reliable”. To increase the reliability in sending these logs, we can leveraging syslog broker software like syslog-ng.

A Syslog Broker

A syslog broker (i.e. syslog-ng) is a piece of software that can serve as an intermediary between a network device (i.e. FortiGate) and its information platform (i.e. Splunk) that ingests its logs. The broker can receive the logs being sent from the network device and forward that log onto the information platform. The key component to this is that the broker allows you to do processing on the syslog after it is received from the network device before it is relayed to the information platform. Here are some of benefits using a syslog broker provides:

1. Placing syslog broker close to the sources of syslog reducing the risk of the logs being loss during transmission over an unreliable network.
2. Ability to write the syslog to a file (persistent buffer) in case the network connectivity goes down.
3. Easier defining access policies to allow communication between the broker and the information platform instead of requiring access between each source of syslog and the information platform.
4. Ability to filter out unnecessary logs from the syslog source when licensing and store costs need to be minimized.

There’s a ton of benefits with leveraging a syslog broker and to make it even more clear, here is an example of a real-world scenario where these pieces came together to create a robust logging solution.

Real-World Scenario

I have a colocation center where I host different services and applications that support my home lab. Those services and applications are protected by Fortinet virtual machines. Each of those devices generate their own set of logs but as usage in those environments increased, constantly monitoring those logs became overwhelming. In order to improve my efficiency in monitoring that environment, I am sending all of my logs to Splunk.

The Topology

The topology for this colocation environment looks similar to the following:

Within the colocation datacenter, I have all of my Fortinet related hosts and webserver to send their logs to the syslog-ng server. Once the syslog-ng server receives those logs, it performs its processing and then forwards them on to Splunk via a VPN tunnel between my two locations. To put this concept into practice, I had to configure the following components:

1. Configure syslog-NG (running on Ubuntu 18.03.3)
2. Configure the Fortinet devices to point their syslog to the syslog-NG
3. Configure the Splunk Universal forwarder on the syslog-NG server

It is assumed that the syslogNG, Fortinet VMs, web server and Splunk Enterprise hosts were already set up. Also, it is already assumed that network connectivity is established between all hosts that need to communicate. It is outside of the scope of this article to cover the set up and installation of those components, however a link will be provided to those well documented tasks.

syslogNG

Syslog-NG is a well known open source syslog broker used by many platforms. Since it is so well supported, many repositories already include a packaged binary that can be downloaded and installed in the most popular linux distributions. To configure the Ubuntu host for syslog-NG, follow the steps in the section below:

Disable rsyslog

Before configuring the syslog-NG client, it is best to remove the included rsyslog client to prevent contention for resources. To purge this application from the host and reboot, execute the following commands from the command line:

1. Purge rsyslog from the machine

sudo dpkg --purge rsyslog

2. Restart the computer

sudo shutdown -r now

Install syslog-NG

Once the old syslog utility has been removed from the host, you can proceed with installing syslog-NG.

1. Update the repositories on the host with the following command.

apt update

2. Once the repositories have been updated, perform an upgrade of all out of date software.

apt upgrade

3. Perform the install of syslog-NG

apt install syslog-ng

Configure syslog-NG

Once the machine restarts, the configuration of syslog-NG can commence. To accomplish this, perform the following steps below:

1. Move the existing syslog-ng configuration file to a backup in the same directory

mv /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.old

2. Open the syslog-ng configuration file in a text editor (i.e. nano)

nano /etc/syslog-ng/syslog-ng.conf

3. Add the following content and save the file

@version: 3.13
@include "scl.conf"

# Syslog-ng configuration file, compatible with default Debian syslogd
# installation.

# First, set some global options.
options { 
    chain_hostnames(off); 
    flush_lines(0); 
    use_dns(no); 
    use_fqdn(no);
    owner("root"); 
    group("adm"); 
    perm(0640); 
    stats_freq(0);
    bad_hostname("^gconfd$");
    create_dirs(yes);
    keep_hostname(yes);
    log_fifo_size(2048);
    log_msg_size(8192);
    time_reopen(10);
};
# Adding param to make syslog-ng listen on udp/514 for syslog
source s_net {
    udp(port(514));
};
# Adding destination for local file to receive FortiGate logs

destination d_fortinet_fortigate {
    file("/root/syslog/logs/fortinet/fortigate/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortigate.log" create_dirs(yes));
};

destination d_fortinet_fortiweb {
    file("/root/syslog/logs/fortinet/fortiweb/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiweb.log" create_dirs(yes));
};

destination d_fortinet_fortiauthenticator {
    file("/root/syslog/logs/fortinet/fortiauthenticator/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortiauthenticator.log" create_dirs(yes));
};

destination d_fortinet_fortimail {
    file("/root/syslog/logs/fortinet/fortimail/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortimail.log" create_dirs(yes));
};
# Filter to instruct syslog-ng how to identify FortiGate syslog
filter f_fortinet_fortigate {
    match("devid=\"FG[A-Z0-9]+\"" value("MESSAGE"));
};

filter f_fortinet_fortiweb {
    match("device_id=FV[A-Z0-9]+" value("MESSAGE"));
};

filter f_fortinet_fortiauthenticator {
    match("subcategory=\"Authentication\"" value("MESSAGE"));
};

filter f_fortinet_fortimail {
    match("device_id=FE[A-Z0-9]+" value("MESSAGE"));
};
# Creating the different parts together for logging

log {
    source(s_net);
    filter(f_fortinet_fortigate);
    destination(d_fortinet_fortigate);
};

log { 
    source(s_net);
    filter(f_fortinet_fortiweb);
    destination(d_fortinet_fortiweb);
};

log { 
    source(s_net);
    filter(f_fortinet_fortiauthenticator);
    destination(d_fortinet_fortiauthenticator);
};

log {
    source(s_net);
    filter(f_fortinet_fortimail);
    destination(d_fortinet_fortimail);
};
###
# Include all config files in /etc/syslog-ng/conf.d/
###
@include "/etc/syslog-ng/conf.d/*.conf"

4. Restart the syslog-NG service

service syslog-ng restart

At the conclusion of this section, the syslog-NG server should be listening on udp/port 514 ready to receive syslog.

Syslog sources

Configure FortiGate Syslog

1. Add the following CLI to the FortiGate to send syslog to syslog-NG

config log syslogd setting
    set status enable
    set server "<ip of syslog-NG server>"
end

Configure FortiWeb Syslog

1. Add the following CLI to the FortiWeb to send syslog to syslog-NG

config log syslog-policy
  edit "syslogNG"
    config  syslog-server-list
      edit 1
        set server <ip of syslog-NG server>
      next
    end
  next
end
config log syslogd
  set status enable
  set policy syslogNG
end

Configure FortiAuthenticator Syslog

The FortiAuthenticator does not support adding hosts to send syslog via the CLI. Instead, this must be accomplished via the WebGUI.

1. Log into the FortiAuthenticator as an administrative user

2. Navigate to the “Logging | Log Config | Syslog Servers”

3. Click “Create New”

4. Fill out the information in the syslog dialog box | Click “OK’

5. Navigate to “Logging | Log Config | Log Settings”

6. In the “Edit Log Setting” dialog box, enable “Send logs to remote Syslog servers” | Select the syslog server | Click OK”

Configure FortiMail Syslog

1. Add the following CLI to the FortiMail to send syslog to syslog-NG

config log setting remote
  edit syslogNG
    set status enable
    set server <ip of syslog-NG server>
    set event-log-status enable
    set event-log-category smtp 
    set sysevent-log-status enable
    set sysevent-log-category configuration admin system update 
    set virus-log-status enable
    set spam-log-status enable
    set history-log-status enable
    set encryption-log-status enable
  next
end

Splunk Universal Forwarder

At the conclusion of configuring the syslog from the sources, you should see the syslog from the remote machines written to files on the local disk. The files should follow the following path:

/root/syslog/logs/fortinet/<type of fortinet device>/$HOST/$YEAR-$MONTH-$DAY-fortinet-fortigate.log

An example log after this has been configured will look like the following:

/root/syslog/logs/fortinet/fortigate/192.168.1.1/2019-09-21-fortinet-fortigate.log

In order to get these files to Splunk, the Universal Forwarder can be leveraged to monitor the files in that directory and send over those log events to Splunk for processing.

Install the Splunk Universal Forwarder

The universal forwarder can be installed using the instructions at the following link:

Install the universal forwarder on Linux

Configure the Splunk Universal Forwarder

Once the Universal Forwarder has been installed, it needs to be set up and configured to monitor the files written via syslog. To complete this, follow the steps below:

1. Run the command to do the initial set up wizard for Splunk Forwarder

/opt/splunkforwarder/bin/splunk start

Follow the prompts and complete the set up

2. Run the command to set up the Universal Forwarder to automatically start upon boot

/opt/splunkforwarder/bin/splunk enable boot-start

3. Run the command to add an indexer the Universal Forwarder should send logs to.

/opt/splunkforwarder/bin/splunk add forward-server <ip addr of indexer>:9997

4. Create an inputs.conf file to put in the configuration to monitor the logs generated by syslog-NG.

nano /opt/splunkforwarder/etc/apps/search/local/inputs.conf

5. Add (paste) the following content to this file.

# FortiGate
[monitor:///root/syslog/logs/fortinet/fortigate/*/*.log]
sourcetype = fgt_log
index = fortigate
disabled = false
host_segment = 6

# FortiWeb
[monitor:///root/syslog/logs/fortinet/fortiweb/*/*.log]
sourcetype = fwb_log
index = fortiweb
disabled = false
host_segment = 6

# FortiMail
[monitor:///root/syslog/logs/fortinet/fortimail/*/*.log]
sourcetype = fml_log
index = fortimail
disabled = false
host_segment = 6

# FortiAuthenticator
[monitor:///root/syslog/logs/fortinet/fortiauthenticator/*/*.log]
sourcetype = fac_log
index = fortiauthenticator
disabled = false
host_segment = 6

6. Restart the splunk service to make the settings take effect

/opt/splunkforwarder/bin/splunk restart

It may take up to 5 minutes before the logs show up in Splunk once the Universal Forwarder has been configured.

Validation/Troubleshooting

Validate Data in Splunk

To validate the data that is coming from the universal forwarder in Splunk you can run the following Search Processing Language (SPL) query:

index=_internal host=<splunk hostname> sourcetype="splunkd" (fortigate OR fortimail OR fortiauthenticator OR fortiweb)

This query shows the file that is being indexed as well as the amount of bandwidth is being used to send this data from the Universal Forwarder to the indexer.

Troubleshooting in syslog-ng

If you want to confirm that the syslog are actually being received from the syslogNG server, you can use the following command to see the syslog as it is being received:

nc -u -l 514

If you encounter any errors related to the configuration file when attempting to restart the service or you have issues with syslog-NG not properly recording the syslog to disk, you can execute the following command to debug the process:

/usr/sbin/syslog-ng -Fvde

Make sure that the time zone is configured properly on this machine. To configure the time zone in Ubuntu use the following command

 dpkg-reconfigure tzdata

As always, I hope this is a useful blog article to help you get syslog data from your Fortinet devices into the Splunk Enteprise indexer via a Universal Forwarder. Please leave your feedback below and let me know if you have any questions.