March 2005

Shell Corner: Finding Abusive Users with netstat

Hosted by Ed Schaefer

This month, Todd Neal explains (in bash script netstat.ss) how the netstat command can identify abusive users who are accessing your server from the Internet.

Finding Abusive Users with netstat
by Todd Neal

If you run any type of service accessible from the Internet, sooner or later someone will abuse it. You'll notice it by a slow response from your server, increased bandwidth usage, or a total denial of service. You can solve this problem manually each time it happens, or you can write a script that helps identify and block access to the abusers.

Whenever someone makes a connection to your server, the connection can be listed with the netstat command. I will demonstrate how to use this command to locate troublesome users.

Use the command netstat -n. The netstat MAN pages describes the -n option as, "Show numerical addresses instead of trying to determine symbolic host, port or user names." I am only interested in IP addresses for now, and it will be faster as it saves a DNS lookup for each IP address.

Example output of netstat -n:

internet@fw:~$ netstat -n 
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 192.168.0.21:110        192.168.0.222:3843      TIME_WAIT   
tcp        0      0 192.168.0.21:22         192.168.0.55:1945       ESTABLISHED 
tcp        4      0 192.168.0.21:3326       192.168.0.222:139       CLOSE       

Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  15     [ ]         DGRAM                    234      -                   /dev/log
unix  3      [ ]         STREAM     CONNECTED     830179   -                   /tmp/.X11-unix/X0

Translation

The first connections are the active TCP connections. They show every IP address that is connected to your server and the ports. For example, 192.168.0.222 is connected to the POP3 daemon, which runs on port 110. 192.168.0.55 is connected to port 22, which is SSH.

I don't really need the Active Unix Domain sockets, though. They are sockets that are used by processes on your machine to communicate with one another locally. If you are using Linux, more information on these sockets can be obtained by typing man 7 unix.

Because I only want the UDP and TCP connections listed, my first thought is to grep for UDP and TCP. However, this seems like a common choice, so I checked the netstat documentation. Sure enough, netstat provides the --tcp and --udp options, which are just what I need to display TCP and UDP connections.

Revised Command

internet@fw:~$ netstat -n --tcp --udp 
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 192.168.0.21:110        192.168.0.222:3843      TIME_WAIT   
tcp        0      0 192.168.0.21:22         192.168.0.55:1945       ESTABLISHED 
tcp        4      0 192.168.0.21:3326       192.168.0.222:139       CLOSE

This is much better; only connections I want display, and only one process is required instead of piping netstat to grep.

Let's see what someone hammering the Web server would look like. I will simulate this by using netcat to make a connection to Apache.

Netcat is a "networking Swiss army knife". You can use it to listen on ports for incoming connections or create outbound connections to remote servers.

I'll just repeat the netcat command nc 192.168.0.21 80 & a few times from a remote computer. This command connects to port 80 on 192.168.0.21 and then is dropped to the background.

Hammered

internet@fw:~$ netstat -n --tcp --udp 
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 192.168.0.21:110        192.168.0.222:3843      TIME_WAIT   
tcp        0      0 192.168.0.21:22         192.168.0.55:1945       ESTABLISHED 
tcp        4      0 192.168.0.21:3326       192.168.0.222:139       CLOSE       
tcp        0      0 192.168.0.21:80         192.168.0.55:1685       ESTABLISHED 
tcp        0      0 192.168.0.21:80         192.168.0.55:1684       ESTABLISHED 
...
tcp        0      0 192.168.0.21:80         192.168.0.55:1646       ESTABLISHED 
tcp        0      0 192.168.0.21:80         192.168.0.55:1645       ESTABLISHED

Above is the list of all connections. I saved this listing to a text file by typing netstat -n --tcp --udp > /tmp/text. Then I could test the rest of my script on this data and not worry about keeping the netcat connections open.

The fifth column of the data is the only important one. It shows the IP address that is connected to us. I use awk to print only this column:

netstat + awk

tolchz@spareknet:/tmp$ head text | awk '{print $5}'
servers)
Address
192.168.0.222:3843
192.168.0.55:1945
192.168.0.222:139
192.168.0.55:1685
192.168.0.55:1684
192.168.0.55:1683
192.168.0.55:1682
192.168.0.55:1681

This is close to what I want. Note that I use head to only display the first few lines of output. After the script works correctly, I will test it on all the data.

The only problem is the portions of column headings that are listed. I will use awk to also display the protocol field and the IP address:

tolchz@spareknet:/tmp$ head text | awk '{print $1" "$5}'
Active servers)
Proto Address
tcp 192.168.0.222:3843
tcp 192.168.0.55:1945
tcp 192.168.0.222:139
tcp 192.168.0.55:1685
tcp 192.168.0.55:1684
tcp 192.168.0.55:1683
tcp 192.168.0.55:1682
tcp 192.168.0.55:1681

I could grep for ( tcp | udp ) and then use awk again to print the second field. However, awk is fairly powerful by itself. I can use it to check for the tcp or udp text and only if it is present, print the line:

tolchz@spareknet:/tmp$ head text | awk '{if (/(tcp|udp)/) { print $5 }}'

192.168.0.222:3843
192.168.0.55:1945
192.168.0.222:139
192.168.0.55:1685
192.168.0.55:1684
192.168.0.55:1683
192.168.0.55:1682
192.168.0.55:1681
192.168.0.55:1680
192.168.0.55:1679

The awk portion is fairly simple. /(tcp|udp)/ is a regular expression that only matches lines that contain "tcp" or "udp". If the line contains this text, I print the fifth column.

I could also use awk to remove the local port number from the list of IP addresses, but sed is better suited for this task.

netstat + awk + sed

For each line, I use the regular expression s/:.*// to remove everything after the colon:

tolchz@spareknet:/tmp$ head text | awk '{if (/(tcp|udp)/) { print $5 }}' | sed 's/:.*//'

192.168.0.222
192.168.0.55
192.168.0.222
192.168.0.55
192.168.0.55
192.168.0.55
192.168.0.55
192.168.0.55
192.168.0.55
192.168.0.55

I now have a list of IP addresses that have open connections to the server. It is still not very useful, though. I really need a count of how many times each IP address repeats. The sort command of some Unices may be capable of doing this removal of duplicates and counting by itself. However under Linux, I will need to use the uniq command to do it. uniq requires that its data be sorted so I will first sort the IP addresses and then use the uniq -c command.

This command removes duplicate lines and the -c option causes the number of occurrences of each line to be printed:

netstat + awk + Sed + sort + uniq

tolchz@spareknet:/tmp$ cat text | awk '{if (/(tcp|udp)/) { print $5 }}' | sed 's/:.*//'  | sort | uniq -c

      2 192.168.0.222
     42 192.168.0.55

I'm not really worried about the 192.168.0.222 IP address. Two connections are nothing to worry about and are probably a legitimate user. I'll use awk again to check and see whether the first column is greater than 25:

tolchz@spareknet:/tmp$ cat text | awk '{if (/(tcp|udp)/) { print $5 }}' | \
sed 's/:.*//'  | sort | uniq -c | awk '{ if ($1 > 25) { print; }}'
     42 192.168.0.55

It would be nice if this script could tell us exactly what to type to block these connections. Under Linux, I can use the route command to block someone from connecting by typing "/sbin/route add -host ip.ad.dr.es gw 127.0.0.1" I'll add this to the command.

At this point, the one-liner is turning into a very long line, so I'll start using the built-in fc command of bash to edit the last typed command. I'll also use the \ character to split it across several lines; this has to be the last character on the line or it will not work correctly, so be sure no white space occurs after the \:

tolchz@spareknet:/tmp$ cat text | \
        awk '{if (/(tcp|udp)/) { print $5 }}' | \
        sed 's/:.*//'  | sort | uniq -c | \
        awk '{ if ($1 > 25) { print "/sbin/route add -host " $2 " gw 127.0.0.1"; }}'

/sbin/route add -host 192.168.0.55 gw 127.0.0.1

I'll also add a display of the number of connections and make it so that the command can still be run as printed by putting it in a comment.

tolchz@spareknet:/tmp$ cat text | \
        awk '{if (/(tcp|udp)/) { print $5 }}' | \
        sed 's/:.*//'  | sort | uniq -c | \
        awk '{ if ($1 > 25) { print "/sbin/route add -host " $2 " gw 127.0.0.1 #" $1 " open connections"; }}'

/sbin/route add -host 192.168.0.55 gw 127.0.0.1 #42 open connections

This is looking good. Now, I'll replace the cat textfile with the netstat command and place it into text file so I can execute it as a script:

#!/bin/bash
# netstat.ss
# author: Todd Neal

netstat -n --tcp --udp | \
awk '{if (/(tcp|udp)/) { print $5 }}' | \
sed 's/:.*//'  | sort | uniq -c | \
awk '{ if ($1 > 25) { print "/sbin/route add -host " $2 " gw 127.0.0.1 #" $1 " open connections"; }}'

Automation

I can also expand this script. I can change 25 to a higher number — such as 100 — and create a cron job to run this script every few minutes. Its output can be executed by piping it to a shell. Then, each time someone makes more than 100 simultaneous connections, their IP is null routed.

More Information

GNU Netcat— http://netcat.sourceforge.net/
grep Manual — http://www.gnu.org/software/grep/doc/
head,sort, and uniq Manuals — http://www.gnu.org/software/textutils/manual/textutils/textutils.html
sed Manual — http://www.gnu.org/software/sed/manual/sed.html