Saving the Precious Internets

As I'm sure some of my readers know, even the best networks are fragile things. The slightest thing from someone tripping over a cable in the dark to something as simple as a tornado can bring them down. I had a mysterious issue with the Fave network earlier this week that was infinitely perplexing.

I have been in the process of porting all our phones from one VoIP provider to another, and in the process have ported about 1/3rd of our phones. Very suddenly the other day all of the phones from the old provider went down simultaneously. My initial thought was that the T1, also from that same provider and through which all those phones worked, had gone down. I rushed down to the Haunted Basement to investigate, leaving a network tracer in the server room. When I connected to the T1 box downstairs the connection was good. Despite red lights on the T1 terminus itself the connection to the internet from there was good, too. I was mystified.

When I reascended to ground level things got weirder. The internet connection for all the desktop computers had gone slow, and the VoIP phones from the new provider were dropping out, breaking up, and generally sounding like cell phones circa 1998. Bad times, to be sure, but what was the cause? After conferring with the T1 provider to check if they had full connectivity to our router and firewall on that side of the network, then after speaking with our shiny new IT consultant, Joe, it became apparent that our big network switch may have been at fault. Power-cycling the switch would fix the problem for a few minutes but then seemingly random ports would lock up. Finally I went back to the server room to look at the switch to see what she was up to.

Pandaemonium. Utter and complete chaos. Every single collision light was flashing as fast as possible in unison, indicating that something somewhere was broadcasting packets like a messenger, hopped up on speed, who'd panicked and lost their way. It was exactly as Joe had predicted. So I began unplugging each of the 24 ports on my main binding post one by one. When I got to port 4, all the traffic calmed down considerably to the network equivalent of a dull roar. Looking at my port map, I saw that port 4 was in the old creative closet—a tiny room where all the Creative department used to be. It is now only used as a sound studio and occasionally for cold-calling potential clients. I rushed back there.

The room was empty save a couple of phones and one of our sales intern's laptops, which was shut down. I picked up the hub that was connected to port 4 and found that its collision lights were going crazy like those in the server room's main switch. Intriguing! On closer inspection, there were four ethernet cables plugged into it, and only an uplink, the laptop, and one phone to account for. On even closer inspection, the "PC" port and "WAN" port of the phone were both plugged into the hub! The phone was in a perpetual state of colliding packets with itself, constantly retransmitting its own requests to the network and to itself. Since these were keep-alive signals to the network at large they weren't stopped at the switch level and were getting passed along, thus completely and totally hosing the network. I unplugged the cable plugged into the "PC" port and saw the hub's lights immediately glow green. Going back into the server room I saw green lights all around with a slow rhythmic pulsing of amber collision lights.

When I returned to the main room of the office, there were green lights on every phone, two or three people noticed an immediate speed-up in the network, the next phone call I made on my desk phone was crystal clear, and all was right with the world of Fave.