Nfsen is open source sensor: it accepts netflow data from multiple netflow probes (servers, routers, vpn concentrators etc) and then visualizes it into human readable form. So using Nfsen you can see traffic statistics of every network device in your network in one place (actually Nfsen provides much more features).
By default Nfsen makes it possible to see only inbound and outbound traffic statistics but no protocol breakdown or any traffic classification. In the meantime it’s always useful to know what network applications are eating the bandwidth to understand if that fits baseline or not and take necessary actions. For example, if you’re monitoring Linux server which primary task is to host some website but in Nfsen you see that it generates 90% of SSH traffic and only 10% of web traffic then it would be reasonable idea to check if somebody is trying to brute force SSH password and stop that activity. In other words it’s better to have traffic statistics classified. In this article I’ll tell you how to enable traffic classification in Nfsen.
If you’re not familiar with Nfdump and Nfsen just follow this article and/or read Nfsen documentation. As soon as Nfsen is workable and shows some traffic statistics it’s time to create new profile for traffic classification.
1. Go to Nfsen main page http://{your_nfsen_server}/nfsen/nfsen.php and choose Profile –> New Profile.
2. Choose the name for new profile, select its type as Shadow, set expiration and choose necessary source(s) of Netflow data.
3. As the next step you would have to create the two channels for every traffic pattern: www-in for inbound HTTP traffic, www-out for outbound HTTP traffic, pop3-in for inbound POP3 traffic and so on. Every channel should include corresponding filter to pick out traffic of particular kinds from general traffic, for example for inbound HTTP traffic it will be something like ‘((src port 80 and proto tcp) and (src port 443 and proto tcp))’ and so on.
Below is the script that makes it possible to save time and not to type in those channels and their filters by hands.
4. Create new folder for the script and its components:
mkdir -p /path/to/script/breakdown/ cd /path/to/script/breakdown/
5. Create file protocols.list that will include traffic classifications by port numbers and protocols. Syntax is pretty simple:
<port number>/<protocol> [<port number>/<protocol>] [...]|<traffic pattern name>
Port ranges should be specified as <port_start>-<port_end>. Here is my protocols.list file:
80/tcp 80/udp 443/tcp 443/udp|http 20-21/tcp 20-21/udp 989-990/tcp 989-990/udp|ftp 5190-5193/tcp 1863/tcp 531/tcp 531/udp 5050/tcp 5190/tcp 5222/tcp 5223/tcp 5269/tcp 5298/tcp 5298/udp 6660-6664/tcp 6665-6669/tcp 6679/tcp 6697/tcp 1247/udp 2940-3000/tcp|im 5060-5061/tcp 5060-5061/udp|sip 22/tcp 22/udp 911/tcp|ssh 23/tcp 23/udp 107/tcp 992/tcp 992/udp|telnet 110/tcp 995/tcp|pop3 143/tcp 993/tcp|imap 25/tcp 465/tcp|smtp 53/udp 53/tcp 953/tcp 953/udp 5353/udp|dns 161-162/tcp 161-162/udp|snmp 1494/tcp 2512/tcp 2513/tcp 1604/udp 8082/tcp 27000/tcp 2598/tcp 9001-9002/tcp 9005/tcp 2897/tcp|citrix 1194/tcp 1194/udp 5000/tcp 5000/udp 12972/tcp 32976/tcp|vpn 1645-1646/tcp 1645-1646/udp 1812-1823/tcp 1812-1813/udp 2083/tcp|radius
6. Create file colors.list (it will include colors for each traffic type), here is mine:
#FF0000 #FCFF00 #00FF2A #FFA800 #613702 #133D7C #00EAFF #9600FF #FF00C6 #B0F10F #746FAE #993366 #B8860B #CCFFFF #660066 #FF6666 #0066CC #CCCCFF #000066 #0000FF #A7A3A3 #C4B782
7. Create main script add.nfsen.channels.sh (don’t forget to change path to nfsen binary in the beginning of this shell script):
#!/bin/bash nfsen="/opt/nfsen/bin/nfsen" remote=$1 ips=$2 group=$3 if [[ $1 == "" || $2 == "" || $3 == "" ]];then exit fi $nfsen -a $group/$remote shadow=1 dstfilter="" for dstip in $2;do if ((${#dstfilter}>0));then dstfilter=$dstfilter" or dst net "$dstip else dstfilter="(dst net "$dstip fi done dstfilter=$dstfilter")" srcfilter="" for srcip in $2;do if ((${#srcfilter}>0));then srcfilter=$srcfilter" or src net "$srcip else srcfilter="(src net "$dstip fi done srcfilter=$srcfilter")" colorcounter=0 cat ./protocols.list | while read line;do echo echo echo $line let colorcounter++ currentcolor=$(head -n $colorcounter colors.list | tail -1) dstportfilter="" srcportfilter="" otherdstportfilter="" othersrcportfilter="" serviceport=$(echo $line | awk -F '|' '{print$1}') servicename=$(echo $line | awk -F '|' '{print$2}') for line1 in $serviceport;do if [[ -n $(echo $line1 | grep '-') ]];then #range port1=$(echo $line1 | awk -F '/' '{print$1}' | awk -F '-' '{print$1}') port2=$(echo $line1 | awk -F '/' '{print$1}' | awk -F '-' '{print$2}') proto=$(echo $line1 | awk -F '/' '{print$2}') let "port1n=port1-1" let "port2n=port2+1" if ((${#srcportfilter}>0));then srcportfilter=$srcportfilter" or (src port > $port1n and src port < $port2n and proto $proto)" else srcportfilter="(src port > $port1n and src port < $port2n and proto $proto)" fi if ((${#dstportfilter}>0));then dstportfilter=$dstportfilter" or (dst port > $port1n and dst port < $port2n and proto $proto)" else dstportfilter="(dst port > $port1n and dst port < $port2n and proto $proto)" fi #for other if ((${#othersrcportfilter}>0));then othersrcportfilter=$othersrcportfilter" and not (src port > $port1n and src port < $port2n and proto $proto)" else othersrcportfilter="not (src port > $port1n and src port < $port2n and proto $proto)" fi if ((${#otherdstportfilter}>0));then otherdstportfilter=$otherdstportfilter" and not (dst port > $port1n and dst port < $port2n and proto $proto)" else otherdstportfilter="not (dst port > $port1n and dst port < $port2n and proto $proto)" fi else #norange port=$(echo $line1 | awk -F '/' '{print$1}') proto=$(echo $line1 | awk -F '/' '{print$2}') if ((${#srcportfilter}>0));then srcportfilter=$srcportfilter" or (src port $port and proto $proto)" else srcportfilter="(src port $port and proto $proto)" fi if ((${#dstportfilter}>0));then dstportfilter=$dstportfilter" or (dst port $port and proto $proto)" else dstportfilter="(dst port $port and proto $proto)" fi #for other if ((${#othersrcportfilter}>0));then othersrcportfilter=$othersrcportfilter" and not (src port $port and proto $proto)" else othersrcportfilter="not (src port $port and proto $proto)" fi if ((${#otherdstportfilter}>0));then otherdstportfilter=$otherdstportfilter" and not (dst port $port and proto $proto)" else otherdstportfilter="not (dst port $port and proto $proto)" fi fi done $nfsen --add-channel $group/$remote/$servicename-in sourcelist="BComEdgeDE1-1" filter="$dstfilter and ($srcportfilter)" colour="$currentcolor" sign=+ order=$colorcounter $nfsen --add-channel $group/$remote/$servicename-out sourcelist="BComEdgeDE1-1" filter="$srcfilter and ($dstportfilter)" colour="$currentcolor" sign=- order=$colorcounter otherdstportfilterall=$otherdstportfilterall" and ("$otherdstportfilter")" othersrcportfilterall=$othersrcportfilterall" and ("$othersrcportfilter")" echo $otherdstportfilterall > /tmp/otherdstportfilterall echo $othersrcportfilterall > /tmp/othersrcportfilterall echo $colorcounter > /tmp/colorcounter done if [[ -s /tmp/otherdstportfilterall && -s /tmp/othersrcportfilterall && -s /tmp/colorcounter ]];then otherdstportfilterall=$(cat /tmp/otherdstportfilterall) othersrcportfilterall=$(cat /tmp/othersrcportfilterall) colorcounter=$(cat /tmp/colorcounter) let colorcounter++ $nfsen --add-channel $group/$remote/other-in sourcelist="BComEdgeDE1-1" filter="$dstfilter $othersrcportfilterall" colour="#000000" sign=+ order=$colorcounter $nfsen --add-channel $group/$remote/other-out sourcelist="BComEdgeDE1-1" filter="$srcfilter $otherdstportfilterall" colour="#000000" sign=- order=$colorcounter fi rm -f /tmp/otherdstportfilterall rm -f /tmp/othersrcportfilterall rm -f /tmp/colorcounter $nfsen --commit-profile $group/$remote
7. Lauch the script using the following command:
./add.nfsen.channels.sh Host1 "1.2.3.4/29 5.6.7.8/29" Breakdown
Where Host1 is the name for new profile, ’1.2.3.4/29 5.6.7.8/29′ is IP addresses of the networks you wish to include into the graph and Breakdown is the name of the group for Profile. If do not plan to filter traffic in this profile by IPs of the networks just put ’0.0.0.0/0′ instead of ’1.2.3.4/29 5.6.7.8/29′.
8. Once the script is finished you’ll see new item in profile menu Breakdown –> Host1.
Pingback : Top 5 Linux Monitoring Tools. Web Based. | TechFeeder
Hi Artem,
This is quite nice, in your script your have BComEdgeDE1-1 mentioned, what value should this be for us configuring this.
Regards,
Darius
Artem, thank you for this great tutorial and script. Just for the sake of completeness:
"BComEdgeDE1-1" needs to be replaced by the name of the source you want to analyze in detail (defined in the %source section of nfsen.conf).
Pingback : Linux Monitoring Tools. Web Based. | general linux, admin linux, admin network
Pingback : Top 5 Linux Monitoring Tools: Web Based « Ubuntu Arena..!