Reading LZ4 log files

You may, or may not have, noticed in the What’s New for FortiOS 5.4.0, the introduction of the use the .lz4 compression format. Log files are compressed to save space on the disk and to increase performance when transmitting them between the FortiGate and other devices such as a FortiAnalyzer. The LZ4 format focuses on the time it takes to compress the files rather than how small a compression file can be made of the file. Log files are text files and will compress quite well regardless, but because this compression takes place in real time, the speed at which it compresses is a high priority.

The drawback to this use of compression is that if the files are being sent to something other than another Fortinet device, such as an FTP server for archival, the files cannot be read unaided.

LZ4 Reader

Seeing as how it would be cruel to point out a potential problem like this without providing a solution, there is a tool to read LZ4 files with the snazzy name of lz4_reader.  Provided that JDK is installed to run the script, the tool works on the following platforms:

• Windows
• Linux
• Mac

Tool availability

At the time of writing this article, the tool was not available for download from a publically accessible site. To get the tool contact TAC and they should be able to track it down for you.

If the technician is unfamiliar with the tool, you can impress them with your insider knowledge and tell them to check Mantis bug 0366327.

Installing the tool

Step #1 – Verify JDK is installed

How you determine if Java is already installed on your computer will depend on the platform that you are using, but if you haven’t got it, to get download the files you need, you can head over to http://www.oracle.com/technetwork/java/javase/downloads/index.html. 

Instructions for installing JDK are already on the Internet so we won’t go over them here.

If you skip this step and run the program, you could get an error like:

'java' is not recognized as an internal or external command, operable program or batch file.

This is a good indicator that you do not have JDK installed.

Another thing that you will want to be careful of is that it is not just Java that is installed but JDK specifically. The first time I tried to run the program on my Mac, I dutifully checked and made sure the latest version of Java was there and ready to go. When I ran the program I got:

talesian$ java -jar log_reader.jar tlog.FGT3HD3914800177.vd1.20160327162450 
Exception in thread "main" java.lang.UnsupportedClassVersionError: lz4_reader_main : Unsupported major.minor version 51.0
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:249)
  at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:56)

Once I downloaded and installed the correct JDK everything worked smoothly.

Step #2 – Get the file

This is actually the most difficult part of the process. The file that you are looking for is lz4_reader.tar.gz.  It is not currently available to be downloaded by the public. You will have to get it from some helpful support person. It’s just a little over 3 MB. As you will notice by the extension, it is in a compressed file format as well.

Step #3 -Extract the files

Once you have downloaded the .tar.gz file, extract the files. This can be done with most compression or archive applications.

Windows

For the purposes of testing it on Windows, I used 7-zip, but most compression utilities will work just as well. This particular application had to extract the files in steps; first the gz layer and then the tar layer. Once you are at the level where the .bat and jar files are, take that folder and place it where it is easily accessible. If feasible, the root of the C: drive is a simple option; as it is nice and easy to find through the command line.

Linux and Mac

For the Mac users of the GUI , the Archive Utility app will extract the files directly to the lz4_reader folder without going through the steps the the 7-zip program did. For Linux users and Mac users that prefer the command line, you can use the tar utility.

$ tar xvzf lz4_reader.tar.gz 
x lz4_reader/._.DS_Store 
x lz4_reader/.DS_Store 
x lz4_reader/run.bat 
x lz4_reader/log_reader.jar 
x lz4_reader/

This will create a folder called lz4_reader in the same folder that you ran the command, though you won’t see the files that start with a “.” unless you have it set up to be able to view hidden files.

Running the tool

The tool is run from the command line. This means using cmd.exe in Windows or the terminal emulator in Linux and Mac. To keep things nice and simple, you can put the log file that you want to read in the same folder as the program.

To run a java command you have to start with java, and in this case because the program that we are going to be running is a .jar file, the -jar option also needs to be used.

Running the program is simple, in the command terminal go to the directory and run the command:

java -jar log_reader.jar <path><name of the file>

Windows

Change the context of the session to the folder or directory holding the utility and then run the command.

C:\lz4_reader> java -jar log_reader.jar tlog.FGT3HD3914800177.vd1.20160327162450 
All readable contents are saved to C:\lz4_reader\tlog.FGT3HD3914800177.vd1.20160327162450_readable. C:\lz4_reader>

If the log file is not in the same folder as the lz4_reader files, in this case, a subfolder called test, include the path in the file name.

C:\lz4_reader> java -jar log_reader.jar C:\lz4_reader\test\DISK_alog_FGVM010000017392_root_20160614_042922

A folder called tlog.FGT3HD3914800177.vd1.20160327162450_readable is created in the same folder as the original file and within that folder, there is a file called tlog.65485_readable.txt

Linux and Mac

In Linux and Mac, the program is run the same way with one notable difference. In Windows, a backslash is used to separate directories and in Linux and Mac a slash is used. The command in *nix based platform would be:

java -jar log_reader.jar test/tlog.FGT3HD3914800177.vd1.20160327162450 
All readable contents are saved to /Fortinet/working/lz4_reader/test/tlog.FGT3HD3914800177.vd1.20160327162450_readable.

Reading the file

Once the file has been converted into readable text, you need to pick an application to read it. For easy reading, I would not advise using word processor applications such as Notepad or Word to read the file. These products are intended to put words to paper so they have a tendency to impose formatting styles on them that may not be appropriate for log files. You are probably not going to print out all of the logs, so a code editor or something along those lines might be a better choice for quickly going through the logs for the purposes of looking for something specific.

To give an idea of the differences, I’ve copied the output of the first 5 lines of a test log file below using two different type of text applications. The first is a word processor/editor; in this case, it was openned using Microsoft Word the next example was openned using a code editor; in this case, Atom but something like Notepad++ produces the same results:

Word output:

date=2016-03-27 time=16:24:32 logid=0001000014 type=traffic subtype=local level=notice vd=vd1 srcip=172.16.200.2 srcport=49984 srcintf=”vd1″ dstip=172.16.95.16 dstport=53 dstintf=”port1″ sessionid=3378 proto=17 action=accept policyid=0 policytype=policy dstcountry=”Reserved” srccountry=”Reserved” trandisp=noop service=”DNS” app=”DNS” duration=476 sentbyte=7568 rcvdbyte=37905 sentpkt=118 rcvdpkt=76 appcat=”unscanned”
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=144.20.202.235 srcport=55165 srcintf=”lo” dstip=112.250.20.205 dstport=53 dstintf=”lo” sessionid=1954188563 proto=17 action=close policyid=2 policytype=policy dstcountry=”China” srccountry=”Spain” trandisp=noop service=”DNS” appid=27457 app=”Windows.File.Sharing” appcat=”Network.Service” apprisk=elevated applist=”default” duration=0 sentbyte=1708 rcvdbyte=3717 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=64.114.19.214 srcport=9953 srcintf=”lo” dstip=32.98.1.172 dstport=21 dstintf=”lo” sessionid=1954188564 proto=6 action=close policyid=0 policytype=policy dstcountry=”United States” srccountry=”Canada” trandisp=noop service=”FTP” appid=27946 app=”Fortiguard.Search” appcat=”Cloud.IT” apprisk=medium applist=”default” duration=0 sentbyte=2508 rcvdbyte=2038 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=44.103.247.160 srcport=1390 srcintf=”dummy0″ dstip=168.125.107.178 dstport=25 dstintf=”lo” sessionid=1954188565 proto=17 action=close policyid=1 policytype=policy dstcountry=”United States” srccountry=”United States” trandisp=noop service=”udp/25″ appid=15895 app=”SSL” appcat=”Network.Service” apprisk=elevated applist=”default” duration=0 sentbyte=1084 rcvdbyte=3061 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=44.103.247.160 srcport=30592 srcintf=”lo” dstip=16.62.205.154 dstport=443 dstintf=”lo” sessionid=1954188566 proto=6 action=close policyid=2 policytype=policy dstcountry=”United States” srccountry=”United States” trandisp=noop service=”HTTPS” appid=34789 app=”SNMP_GetRequest” appcat=”Network.Service” apprisk=elevated applist=”default” duration=0 sentbyte=3101 rcvdbyte=618 sentpkt=0 rcvdpkt=0

Atom output:

date=2016-03-27 time=16:24:32 logid=0001000014 type=traffic subtype=local level=notice vd=vd1 srcip=172.16.200.2 srcport=49984 srcintf="vd1" dstip=172.16.95.16 dstport=53 dstintf="port1" sessionid=3378 proto=17 action=accept policyid=0 policytype=policy dstcountry="Reserved" srccountry="Reserved" trandisp=noop service="DNS" app="DNS" duration=476 sentbyte=7568 rcvdbyte=37905 sentpkt=118 rcvdpkt=76 appcat="unscanned"
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=144.20.202.235 srcport=55165 srcintf="lo" dstip=112.250.20.205 dstport=53 dstintf="lo" sessionid=1954188563 proto=17 action=close policyid=2 policytype=policy dstcountry="China" srccountry="Spain" trandisp=noop service="DNS" appid=27457 app="Windows.File.Sharing" appcat="Network.Service" apprisk=elevated applist="default" duration=0 sentbyte=1708 rcvdbyte=3717 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=64.114.19.214 srcport=9953 srcintf="lo" dstip=32.98.1.172 dstport=21 dstintf="lo" sessionid=1954188564 proto=6 action=close policyid=0 policytype=policy dstcountry="United States" srccountry="Canada" trandisp=noop service="FTP" appid=27946 app="Fortiguard.Search" appcat="Cloud.IT" apprisk=medium applist="default" duration=0 sentbyte=2508 rcvdbyte=2038 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=44.103.247.160 srcport=1390 srcintf="dummy0" dstip=168.125.107.178 dstport=25 dstintf="lo" sessionid=1954188565 proto=17 action=close policyid=1 policytype=policy dstcountry="United States" srccountry="United States" trandisp=noop service="udp/25" appid=15895 app="SSL" appcat="Network.Service" apprisk=elevated applist="default" duration=0 sentbyte=1084 rcvdbyte=3061 sentpkt=0 rcvdpkt=0
date=2016-03-27 time=16:24:39 logid=0000000013 type=traffic subtype=forward level=notice vd=vd1 srcip=44.103.247.160 srcport=30592 srcintf="lo" dstip=16.62.205.154 dstport=443 dstintf="lo" sessionid=1954188566 proto=6 action=close policyid=2 policytype=policy dstcountry="United States" srccountry="United States" trandisp=noop service="HTTPS" appid=34789 app="SNMP_GetRequest" appcat="Network.Service" apprisk=elevated applist="default" duration=0 sentbyte=3101 rcvdbyte=618 sentpkt=0 rcvdpkt=0

You can probably make the file even easier to sort through by converting it to a spreadsheet but I will leave that as an exercise for the reader.

The post Reading LZ4 log files appeared first on Fortinet Cookbook.