A UDP Flood Story

I recently suffered a UDP flood attack on my little virtual private server (VPS) and thought I'd describe the steps I went through to discover and fix the problem.

Symptoms

Periodically, my server would stall and become unresponsive. It was effectively dead, although not down. These 'stalling' events would last from 5-20 minutes, and then the server would come back up. Looking at my Munin  charts told me that my public ethernet interface (eth0) was being flooded. Here's a particularly bad day:

And this was after I had rate limited eth0 to 2mbits/sec using tc (more on tc in a bit). CPU usage and interrupts for eth0 also spiked. So something was flooding eth0, and stalling the server.

Discovery

To start with, I had no idea what was causing the big spikes in traffic, although it was outbound traffic, which should have been my first clue. I could see nothing in any of my log files; Apache, or anything else (well almost) - which should have been another clue. What I wanted was a way to start capturing packets during these peak events so that I could see exactly what was happening on eth0 when eth0 spiked. I found this very cool bash script here at Server Fault , and was able to start capturing network packets with tcpdump whenever the packet rate on the eth0 exceeded my defined rate of packets per second. Thanks Oliver  (Server Fault user) for the cool bash work. However, Oliver's script was monitoring received packets on '/proc/net/dev'. I was seeing interesting tcpdumps using the script this way, but when the next event occurred, there was nothing in the dump files. So I switched to monitoring transmitted packets instead. Here's the complete script:

#!/bin/bash

HOST=`hostname`
INTERFACE=eth0
DUMPDIR=~/dump/
SUBJECT="WARNING:Packet alert on $HOST"
EMAIL="youraddress@yoursite.com"
EMAILMESSAGE="~/dump/emailmessage.txt"

# print $2 for inbound packets, $10 for outbound

while /bin/true; do
  pkt_old=`grep $INTERFACE: /proc/net/dev | cut -d :  -f2 | awk '{ print $10 }'`
  sleep 1
  pkt_new=`grep $INTERFACE: /proc/net/dev | cut -d :  -f2 | awk '{ print $10 }'`

  pkt=$(( $pkt_new - $pkt_old ))
  echo -ne "\r$pkt outbound packets/s\033[0K"

  if [ $pkt -gt 250 ]; then
    echo -e "\n`date` Peak rate exceeded, dumping packets."
    tcpdump -n -s0 -c 2000 -w $DUMPDIR/dump.`date +"%Y%m%d-%H%M%S"`.cap
    echo "`date` Packets dumped, sleeping now."
    echo "Packet rate was $pkt packets/s at `date`"  > $EMAILMESSAGE
    /usr/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE
    sleep 150
  fi
done

And sure enough - it worked. Here's a Wirehsark screen capture of the traffic on eth0 just before the server stalled (click on the image for a larger version):

There was a very large volume of UDP packets coming from my server. The packet capture file is currently filtered for the destination IP address, and although you can only see one source port in the screen shot, source ports were random. The destination IP address is hmm... a dubious site, and the random source ports suggested my server was being used as a proxy. Given the rate of UDP packet generation (and the eventual effect it had on my server), this wasn't exactly nice proxy behavior. It was a UDP flood attack, coming from my server, directed at another site. At this point, I wandered off into looking for malware on my server, running ClamAV  and rkhunter  (a little late for rkhunter which is best run on a clean install), and I found nothing. I then did a little searching for UDP flooding, and found that there is a fairly simple PHP function that can be used to do exactly this. I'm hosting several WordPress blogs and so began to suspect a rouge PHP script. The php function for creating a socket (network connection) is fsockopen. UDP can be specified with a scheme prefix, and so I performed the following grep search across all of my hosted directories (all in seperate user accounts beneath the server's home directory)...

grep -ir 'fsockopen.*udp' /home/

This will search all files that have fsockopen followed by any character, for any number of times on the same line as udp. The search is case insensitive, and recursive. Sure enough, there was a little nastygram of a PHP script sitting in the theme directory for a site I'd been experimenting with a few months ago. The theme had a well publicized bug  in it that allowed an attacker to upload an executable script to the server. The script could then be called from a regular Web request, passing in the target server and duration for the UDP flood attack.

Fix

I removed the script, patched the theme, and turned off the experimental site (since I was no longer using it). I also did a little more reading on how to guard against UDP flooding, as well as SYN floods, and updated my iptables settings accordingly. Below is my iptables baseline script. I don't believe this is a comprehensive iptables script for securing a Linux server - but it's a start, and the UDP flood protection seemed to be working after a few tests.

#!/bin/sh
echo "Flushing iptables and allowing everything..."
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

echo "Applying iptables baseline rules..."
# Deny everything on the default INPUT chain
iptables -P INPUT DROP

# Allow connections that are already connected to the server.
iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT

# #########################
# Outbound UDP Flood protection in a user defined chain.
iptables -N udp-flood
iptables -A OUTPUT -p udp -j udp-flood
iptables -A udp-flood -p udp -m limit --limit 200/s -j RETURN
iptables -A udp-flood -j LOG --log-level 4 --log-prefix 'UDP-flood attempt: '
iptables -A udp-flood -j DROP

# #########################
# SYN-Flood protection in a user defined chain
iptables -N syn-flood
iptables -A INPUT -p tcp --syn -j syn-flood
iptables -A syn-flood -m limit --limit 30/s --limit-burst 60 -j RETURN
iptables -A syn-flood -j LOG --log-level 4 --log-prefix 'SYN-flood attempt: '
iptables -A syn-flood -j DROP

# #########################
# SSH
# Rate limit SSH on 57328
iptables -A INPUT -p tcp --dport 57328 -m state --state NEW -m recent --set --name SSH-LIMIT
iptables -A INPUT -p tcp --dport 57328 -m state --state NEW -m recent --update --rttl --seconds 60 --hitcount 20 -j REJECT --reject-with tcp-reset --name SSH-LIMIT
# Allow SSH on 57328 port
iptables -A INPUT -p tcp --dport 57328 -j ACCEPT 

# #########################
# HTTP
# Allow HTTP and HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# If we made it this far the packet will be dropped - so log it as denied.
iptables -A INPUT -j LOG --log-level 4 --log-prefix 'Denied: '

And lastly - I left my tcpdump peak packet rate script installed and running, so that if anything unusual happens on the server, I'll have a chance to look at it and decide if it's a normal event, or something else.

Lessons Learned

A few things: Firstly - traffic control and traffic shaping with tc is very cool. Here's a great introduction and howto  article on tc. If you're hosting a VPS on a fat pipe at one of the better hosting services, you need to keep tc and queuing disciplines up your sleeve - since a big increase in bandwith usage will probably cost you $$$. Secondly - iptables  are also very cool and the first mandatory step in securing your hosted Linux box. Thirdly - tools like grep, find, ps, netstat, lsof etc. are all part of a sysadmin's toolbox. I'm not claiming Fu-like sysadmin status by any means, but it's amazing what you can accomplish with the long list of small sharp tools that are part of the free GNU user space application collection, and free with every GNU/Linux installation. Fourthly - Linux is fun. That is all.