My First Cybersecurity Incident

You never forget your first cybersecurity incident. It was the summer of 2001, and I was working as a field tech, supporting several research departments at a major research hospital. This was back in the days of pagers, so you didn’t know anything about the problem until you called the number showing on your beeper. That morning, a customer reported something about “weird printer behavior.”

I clearly remember visiting that printer: a big, heavy, cube-shaped HP networked LaserJet. It had printed through its entire tray of copy paper, just a few characters on every page and then spitting out another one. The characters were random nonsense. If you gave it more paper, it would keep printing random letters until it used up that paper too. If you rebooted it or modified any settings, it seemed to sort itself out… for about two minutes.

It didn’t take long to exhaust all the troubleshooting steps that had worked to straighten out any printer I’d previously encountered. While I was puzzling over that printer, I got three more pages, all who turned out to be callers reporting similar strange printer behavior. A fourth came in while I was returning those calls.

Things went downhill from there. By the afternoon, over 20 networked HP printers we supported were completely useless. None of the online resources I consulted had any solutions. HP’s customer support had no answers. We were on our own.

The Net Friends field team put our collective heads together. We knew the problem, whatever it was, only affected HP printers, and only printers connected to the hospital's network. We didn’t know what caused the problem, or even how widespread it might be. Was this major research hospital only affected, or was something crippling HP printers all over the world?

Our troubleshooting attempts got increasingly creative as hours, and then days, passed without a solution. Entire departments couldn’t print a single page (this was a lot more significant 20 years ago!). Finally, someone noticed that, among the dozens of HP printers brought down by the problem, we did have one that was perfectly fine. It printed normally, connected to the network normally, turned off and on normally, and was otherwise unremarkable – except that being normal that week was a very abnormal thing for a printer to do.

We analyzed diagnostic pages that had been printed by the “sick” printers and the “immune” printer, and one detail stood out – the immune printer had a more recent firmware version than any of the sick printers. (Firmware is the basic operating system embedded in the silicon chips of printers, switches, access points – anything that relies on a simple internal computer to function.) We wondered whether the solution was as simple as that – if we could update the firmware in the sick printers to the newer version, could it prevent whatever seemed to be infecting them? Could we create, in effect, a vaccine?‍

We decided to conduct a “clinical trial” of sorts. We isolated one of the sick printers from the network, connected a single computer to it, and uploaded the new HP firmware to its internal storage. Then we rebooted it, hooked up the network cable, and waited. Two minutes passed… five minutes… ten. Nothing happened. We sent it a print job… it printed. The patient was cured!

I spent the next few days going room by room, looking for every HP printer in the buildings I supported and manually updated the firmware on every single one. In my spare time, I scoured the internet for news on what might have caused the epidemic of random-character-spewing networked printers. A few weeks later, we found out: a novel virus, never seen before. The technology press labeled it Code Red, after a highly caffeinated soda that was popular among the security teams hunting the new threat.

It turned out that Code Red’s effect on HP laser printers was unintended. Its real target had been Microsoft’s Internet Information System (IIS) web hosting product. It used an unpatched vulnerability to infect IIS servers and force them to continuously scan local networks looking for other IIS servers. Thanks to a bug in HP’s code, the scanning activity caused networked laser printers to crash over and over, printing out nonsense until someone switched them off. The most recent firmware had patched that bug, which is why the newer printer wasn’t affected.

After the dust settled, I took steps to permanently immunize our stable of HP printers against future attacks – intentional or not. We set up a server that could monitor every printer under management and send updates to all of them simultaneously. We changed the administrative password from the default setting, so that unauthorized users couldn’t hijack them. And we created a standard deployment process for new printers, ensuring the same steps were taken on every device. These hardening procedures would prove vital for blocking later attacks, while departments that took no corrective action would be hit hard (sometimes repeatedly).

I think back to the Code Red incident often these days. Not only because of the obvious metaphorical parallels with the COVID pandemic, but because it prefigured the kind of proactive, managed, security-focused service that would eventually come to distinguish Net Friends from competitors who for many years retained a more “reactive” posture toward security. I’m proud of our team’s rapid response to Code Red, and of the many steps since then we’ve taken toward protecting our customers’ technology from security threats, whether it’s a million-dollar storage array or just a simple printer.

‍

‍WHAT TO READ NEXT:
- How I Was Hired At Net Friends
- A Tale of Net Friendly IT Support (How I Met My Wife)
- Celebrating Net Friends' 23rd Anniversary‍
- What A Teacher Taught Us