The FBI called and said I’d been hacked ...

It seemed like a pretty typical day until Agent Smith from the FBI called to tell me that our server had been hacked. Agent Smith’s name has been changed to protect his identity, but his real name was just as unbelievable.

He waited a moment to let me digest the news, and then added “your site is a target. It has either been compromised or very soon will be”.

I was skeptical.

Our site was running a small web server for an embedded software company Embedthis Software. We’d been pretty careful. It was running a customized embedded web server, and I knew that normal web-server exploits would not work. Furthermore, the server was fully patched, and running a minimal recent Linux distribution. It should be safe.

However, I was also aware that this server was hosting and distributing software that is used in hundreds of millions of embedded devices. Certainly, it would offer an attractive target in which to insert a few malicious lines of code.

But, come on! The guy said his name was Agent Smith. This sounded totally bogus! It had to be a social engineering ploy.

Still, I had no choice but to hear him out. Agent Smith repeated that he wanted to know if we’d been hacked recently. I asked if he could give me more details, but he did not want to disclose anything more over the phone. Agent Smith was being careful. I was being careful too. We had a standoff.

To resolve the situation, Agent Smith suggested I go to http://fbi.gov, call the listed phone number at his regional office, and ask to be put through to him. I did. To my surprise, Agent Smith answered the phone. I was gobsmacked.

It was clear this was a threat that we needed to take seriously. It was time for a full investigation of our site.

The O/S looked fine, none of the core utilities had been modified, log files were intact, and there were no suspicious root kits. Our software and the web server had not been touched. There were no hidden files, no recently modified files, or permissions, and no mysterious programs. In short, we could not detect a single speck of evidence of an attack. We had no indication that the server had been hacked, and yet we no longer had confidence in its integrity.

Out of an abundance of caution, we prepared a new server and decommissioned the old one. I went home and slept well, confident the threat had been erased.

Or had it?

We never found out for sure, and long after that implausible but very real call from the FBI, I would wonder, “had the server really been hacked or not, and is it safe now”?

Ignorance isn’t bliss

The problem is that it is immensely difficult to prove a negative. The server hasn’t been hacked, but can you know for sure?

You can do all the right things:

• Patch quickly and regularly
• Run with minimum privilege
• Lock down ports, firewall and use a minimal distribution
• Capture and actively monitor logs
• Install IDS tools
• etc, etc.

But can you know for sure?

This question would nag at me for years. I always felt I was playing catch-up. We did not have a dedicated security team, so I would patch, but only after zero-days were already active in the wild. I would update IDS signatures, but only after security vendors had seen the exploit and had taken their time to prepare signatures.

I knew there was no universal panacea. Security is an endless, vigilant race against threats. But I wanted to answer three simple questions:

1. Have I been hacked?
2. Am I being attacked?
3. Where am I vulnerable?

I want to end the endless manual checking. I want 24/7 peace of mind, and most of all, I want no more calls from Agent Smith.

Later I would learn that an excellent solution would be to use Immutable Infrastructure so that my server cannot be modified as easily in the first place.