SolutionBase: Diagnose and eliminate packet storms

[Chalmers Ingersoll <chalmersi@xxxxxxxxx>]

SolutionBase: Diagnose and eliminate packet storms

by Guest Contributor | Jul 26, 2004 7:00:00 AM

Tags: Brien M. Posey MCSE

0 comment(s)

• Email
• Save
• Print
• Digg This
• 
  
• 0

Takeaway: Are users e-mailing vacation videos to friends, or has unauthorized software infiltrated your perimeter? If something is tying up your network, here are tips for diagnosing and repairing the cause.

Quickly solve PC problems with TechRepublic's Quick Reference: PC Troubleshooting Pak. The set of four durable laminated charts keeps proven solutions for troubleshooting PC hardware, networking, Windows desktop, and security issues at your fingertips.

Imagine that one afternoon you start getting phone calls from users complaining of extremely slow network response time. You aren't doing anything out of the ordinary that might cause things to run slowly, so you decide to get out the protocol analyzer and investigate the problem. When you do, you discover a flood of packets coming from one specific PC. The question now is, why?

On one hand, there are lots of different hardware problems that can cause a slowdown. I’ve seen malfunctioning network cards just spew out an endless stream of junk packets. I've also seen PCs connected to faulty network cables that generate excessive retry packets. On the other hand, the user of the PC in question might be doing something disruptive or may have even picked up some sort of Trojan that’s designed to flood the network with traffic. So how do you tell the difference?

One way is to pick up the phone and ask the user what he or she is doing right now. There’s a slim chance that the user will say something like, "I’m e-mailing a 50-MB video file to 500 friends." More than likely, though, the user won’t tell you anything useful. Even so, I still recommend making the phone call because, if the user is doing something disruptive, he or she might stop after you hang up the phone—for fear of being caught. This saves you from having to investigate the problem.

For this article, let’s assume that the user tells you that nothing unusual is going on, and that the strange behavior doesn’t go away after you hang up the phone. How do you figure out what’s really happening?

Check for loose connections

If you happen to be in close proximity to the machine in question, I recommend making a quick trip to check its network cable. Try pushing the RJ-45 connector onto the network cable more tightly, and see if there is play in the connector. When you think of a loose RJ-45 connector, it’s easy to think of a connection that doesn’t work at all. However, I've seen countless examples of a loose connector making partial contact with the cable, causing the machine to flood the network with retries.

One of the offices where I used to work was especially notorious for having this problem. Many of the employee’s jobs required them to store lots of data on paper. Since storage space was tight, several employees would box up forms and place them under their desks. Often the boxes would sit on top of the network cable. Over time, the action of kicking the boxes or stacking additional boxes would pull on the network cable, causing it to become loose from the RJ-45 connector. If you work in such an environment, this should be one of the first things you check.

Look for malformed packets

Take another look at the data you captured earlier from your protocol analyzer. You can learn a lot about the problem by examining the captured packets. For example, many protocol analyzers have a mechanism that allows you to count malformed packets. While it’s normal to have the occasional malformed packet, there's no way that a healthy machine should be spewing out large numbers of them. Generally, if a machine is generating lots of malformed packets, one of two things is occurring: Either the network card is malfunctioning, or there's a Trojan infecting the machine and it's trying to use malformed packets in a denial of service attack against your network.

Of these two scenarios, a malfunctioning network card is much more likely to be the culprit than a Trojan—assuming you have good network security in place. However, there are ways of finding out which is the culprit. One way is to make a Remote Installation Service (RIS) boot disk and boot the machine from it. When you boot your machine from a RIS boot disk, it will try to attach to a RIS server and allow you to remotely install an operating system.

For this particular test, you aren’t interested in remotely installing an operating system. Instead, the RIS boot disk just provides you with a way of booting a network requestor outside of the operating system. When the machine has been booted from the RIS boot disk, you can check the protocol analyzer again. If the machine is still sending out lots of malformed packets, you can be sure that the network card is either bad or needs to be reseated. If the malformed packets stop, then something in the operating system is causing the malformed packets. If this is the case, you should scan the machine with a pestware removal tool such as Lavasoft’s Ad-Aware.

Check the port number

What if the machine is flooding the network with legitimate packets rather than malformed packets? You can use information within the packet to help figure out what’s happening. For example, if you notice that most of the packets are being sent through ports ranging from 550 to 5503, there's a good chance that the user has installed Hot Line, a peer-to-peer file sharing program. If the data is flowing across port 5631, the user probably has a PC Anywhere session going on. If you go to the Internet Assigned Numbers Authority Web site, you can find a list of the various ports.

The port number isn’t your only clue to determining the source of the problem. Check out the destination address. Is it internal or external? If the address is internal, it might be fairly easy to figure out who the machine is trying to communicate with and what the user is trying to do. If the address is external, you may have to use the port number to track down the trouble spot.

Monitor activity as a last resort

If you've determined that the issue is definitely software-related, but you aren’t sure if the user is doing something or if a Trojan has made it into the system, the best advice I can give you is to keep an eye on the user. You might try using a remote control session to monitor the user’s activity. There are several good programs out there that will allow you to watch users without them knowing that they're being watched.

If the user doesn’t seem to be doing anything wrong, you'll probably have to scan the machine for viruses and other forms of malware to resolve the problem. If you suspect that a virus may be to blame, it’s important to take care of the problem quickly before it spreads to other PCs.