Nfsen: Traffic Classification

Nfsen is open source sensor: it accepts netflow data from multiple netflow probes (servers, routers, vpn concentrators etc) and then visualizes it into human readable form. So using Nfsen you can see traffic statistics of every network device in your network in one place (actually Nfsen provides much more features).

By default Nfsen makes it possible to see only inbound and outbound traffic statistics but no protocol breakdown or any traffic classification. In the meantime it’s always useful to know what network applications are eating the bandwidth to understand if that fits baseline or not and take necessary actions. For example, if you’re monitoring Linux server which primary task is to host some website but in Nfsen you see that it generates 90% of SSH traffic and only 10% of web traffic then it would be reasonable idea to check if somebody is trying to brute force SSH password and stop that activity. In other words it’s better to have traffic statistics classified. In this article I’ll tell you how to enable traffic classification in Nfsen.

If you’re not familiar with Nfdump and Nfsen just follow this article and/or read Nfsen documentation. As soon as Nfsen is workable and shows some traffic statistics it’s time to create new profile for traffic classification.

1. Go to Nfsen main page http://{your_nfsen_server}/nfsen/nfsen.php and choose Profile –> New Profile.

2. Choose the name for new profile, select its type as Shadow, set expiration and choose necessary source(s) of Netflow data.

3. As the next step you would have to create the two channels for every traffic pattern: www-in for inbound HTTP traffic, www-out for outbound HTTP traffic, pop3-in for inbound POP3 traffic and so on. Every channel should include corresponding filter to pick out traffic of particular kinds from general traffic, for example for inbound HTTP traffic it will be something like ‘((src port 80 and proto tcp) and (src port 443 and proto tcp))’ and so on.

Below is the script that makes it possible to save time and not to type in those channels and their filters by hands.

4. Create new folder for the script and its components:

mkdir -p /path/to/script/breakdown/
cd /path/to/script/breakdown/

5. Create file protocols.list that will include traffic classifications by port numbers and protocols. Syntax is pretty simple:

/ [/] [...]|

Port ranges should be specified as <port_start>-<port_end>. Here is my protocols.list file:

80/tcp 80/udp 443/tcp 443/udp|http
20-21/tcp 20-21/udp 989-990/tcp 989-990/udp|ftp
5190-5193/tcp 1863/tcp 531/tcp 531/udp 5050/tcp 5190/tcp 5222/tcp 5223/tcp 5269/tcp 5298/tcp 5298/udp 6660-6664/tcp 6665-6669/tcp 6679/tcp 6697/tcp 1247/udp 2940-3000/tcp|im
5060-5061/tcp 5060-5061/udp|sip
22/tcp 22/udp 911/tcp|ssh
23/tcp 23/udp 107/tcp 992/tcp 992/udp|telnet
110/tcp 995/tcp|pop3
143/tcp 993/tcp|imap
25/tcp 465/tcp|smtp
53/udp 53/tcp 953/tcp 953/udp 5353/udp|dns
161-162/tcp 161-162/udp|snmp
1494/tcp 2512/tcp 2513/tcp 1604/udp 8082/tcp 27000/tcp 2598/tcp 9001-9002/tcp 9005/tcp 2897/tcp|citrix
1194/tcp 1194/udp 5000/tcp 5000/udp 12972/tcp 32976/tcp|vpn
1645-1646/tcp 1645-1646/udp 1812-1823/tcp 1812-1813/udp 2083/tcp|radius

6. Create file colors.list (it will include colors for each traffic type), here is mine:

#FF0000
#FCFF00
#00FF2A
#FFA800
#613702
#133D7C
#00EAFF
#9600FF
#FF00C6
#B0F10F
#746FAE
#993366
#B8860B
#CCFFFF
#660066
#FF6666
#0066CC
#CCCCFF
#000066
#0000FF
#A7A3A3
#C4B782

7. Create main script add.nfsen.channels.sh (don’t forget to change path to nfsen binary in the beginning of this shell script):

#!/bin/bash

nfsen="/opt/nfsen/bin/nfsen"
remote=$1
ips=$2
group=$3

if [[ $1 == "" || $2 == "" || $3 == "" ]];then
	exit
fi

$nfsen -a $group/$remote shadow=1
dstfilter=""
for dstip in $2;do
	if ((${#dstfilter}>0));then
		dstfilter=$dstfilter" or dst net "$dstip
	else
		dstfilter="(dst net "$dstip
	fi
done
dstfilter=$dstfilter")"

srcfilter=""
for srcip in $2;do
        if ((${#srcfilter}>0));then
                srcfilter=$srcfilter" or src net "$srcip
        else
                srcfilter="(src net "$dstip
        fi
done
srcfilter=$srcfilter")"

colorcounter=0
cat ./protocols.list | while read line;do
	echo
	echo
	echo $line
	let colorcounter++
	currentcolor=$(head -n $colorcounter colors.list | tail -1)
	dstportfilter=""
	srcportfilter=""
	otherdstportfilter=""
	othersrcportfilter=""
	serviceport=$(echo $line | awk -F '|' '{print$1}')
	servicename=$(echo $line | awk -F '|' '{print$2}')
	for line1 in $serviceport;do
		if [[ -n $(echo $line1 | grep '-') ]];then
			#range
			port1=$(echo $line1 | awk -F '/' '{print$1}' | awk -F '-' '{print$1}')
			port2=$(echo $line1 | awk -F '/' '{print$1}' | awk -F '-' '{print$2}')
                        proto=$(echo $line1 | awk -F '/' '{print$2}')
			let "port1n=port1-1"
                        let "port2n=port2+1"
			if ((${#srcportfilter}>0));then
                                srcportfilter=$srcportfilter" or (src port > $port1n and src port  $port1n and src port 0));then
                                dstportfilter=$dstportfilter" or (dst port > $port1n and dst port  $port1n and dst port 0));then
                                othersrcportfilter=$othersrcportfilter" and not (src port > $port1n and src port  $port1n and src port 0));then
                                otherdstportfilter=$otherdstportfilter" and not (dst port > $port1n and dst port  $port1n and dst port 0));then
				srcportfilter=$srcportfilter" or (src port $port and proto $proto)"
			else
				srcportfilter="(src port $port and proto $proto)"
			fi
			if ((${#dstportfilter}>0));then
                                dstportfilter=$dstportfilter" or (dst port $port and proto $proto)"
                        else
                                dstportfilter="(dst port $port and proto $proto)"
                        fi
			#for other
			if ((${#othersrcportfilter}>0));then
                                othersrcportfilter=$othersrcportfilter" and not (src port $port and proto $proto)"
                        else
                                othersrcportfilter="not (src port $port and proto $proto)"
                        fi
                        if ((${#otherdstportfilter}>0));then
                                otherdstportfilter=$otherdstportfilter" and not (dst port $port and proto $proto)"
                        else
                                otherdstportfilter="not (dst port $port and proto $proto)"
                        fi
		fi
	done
	$nfsen --add-channel $group/$remote/$servicename-in  sourcelist="BComEdgeDE1-1" filter="$dstfilter and ($srcportfilter)" colour="$currentcolor" sign=+ order=$colorcounter
	$nfsen --add-channel $group/$remote/$servicename-out sourcelist="BComEdgeDE1-1" filter="$srcfilter and ($dstportfilter)" colour="$currentcolor" sign=- order=$colorcounter
	otherdstportfilterall=$otherdstportfilterall" and ("$otherdstportfilter")"
	othersrcportfilterall=$othersrcportfilterall" and ("$othersrcportfilter")"
	echo $otherdstportfilterall > /tmp/otherdstportfilterall
	echo $othersrcportfilterall > /tmp/othersrcportfilterall
	echo $colorcounter > /tmp/colorcounter
done

if [[ -s /tmp/otherdstportfilterall && -s /tmp/othersrcportfilterall && -s /tmp/colorcounter ]];then
	otherdstportfilterall=$(cat /tmp/otherdstportfilterall)
	othersrcportfilterall=$(cat /tmp/othersrcportfilterall)
	colorcounter=$(cat /tmp/colorcounter)
	let colorcounter++
	$nfsen --add-channel $group/$remote/other-in  sourcelist="BComEdgeDE1-1" filter="$dstfilter $othersrcportfilterall" colour="#000000" sign=+ order=$colorcounter
	$nfsen --add-channel $group/$remote/other-out sourcelist="BComEdgeDE1-1" filter="$srcfilter $otherdstportfilterall" colour="#000000" sign=- order=$colorcounter
fi
rm -f /tmp/otherdstportfilterall
rm -f /tmp/othersrcportfilterall
rm -f /tmp/colorcounter

$nfsen --commit-profile $group/$remote

7. Lauch the script using the following command:

./add.nfsen.channels.sh Host1 "1.2.3.4/29 5.6.7.8/29" Breakdown

Where Host1 is the name for new profile, ‘1.2.3.4/29 5.6.7.8/29’ is IP addresses of the networks you wish to include into the graph and Breakdown is the name of the group for Profile. If do not plan to filter traffic in this profile by IPs of the networks just put ‘0.0.0.0/0’ instead of ‘1.2.3.4/29 5.6.7.8/29’.

8. Once the script is finished you’ll see new item in profile menu Breakdown –> Host1.