One of the secret joys of being a techie is when you have a problem that baffles the experts in a field. It doesn't happen often, and it's frequently an aggravating experience, but in retrospect it can be shared as a war story and as a way of boosting your geek cred.
Earlier today I got an email from the web security experts at Sucuri, and it reminded me about a baffling security issue which I had never shared publicly. Since the problem has long since gone away (or at least I think it has) I thought it would be interesting to share a problem which I was unable to resolve even after I hired security experts.
In February of this year I discovered, quite by accident, that my blog had picked up a new ad. Anyone visiting my blog from an Android device got to see an ad for White Castles overlayed across the bottom of the screen. It looked like this:
I first saw that advert while I was on a trip while using the Wifi network at a conference center, so I initially thought that the center had injected the advert. But then I saw the ad in my hotel room, and after I got confirmation on twitter that others could see it I realized that I had a more serious problem.
That is not my advert, which very likely means that someone hacked my website somehow. (If you run a website you can probably understand the overwhelming sense of panic which accompanied that conclusion.)
I immediately set out to solve the problem. Here's what didn't help:
- To start, I checked Google Webmaster Tools. This isn't strictly a security tool but it does keep an eye out for malware and sometimes it will flag an issue. No such luck.
- Next, I went through my blog's theme and looked for any code which didn't belong. Nada.
- Then, I double-checked that none of my plugins were misbehaving or tied to reports of malware. Zip.
- Also, I checked to see if I could find a suspicious file uploaded to my site without my knowledge. Bupkus.
- Finally, I installed a couple different security scanning plugins (one at a time) and had them look for issues. Zilch.
That is all the security troubleshooting I could think of, so I decided to hire an expert and tell them to fix it.
On the advice of a friend, I signed up for a year's service with Sucuri. This company provides website monitoring and malware cleanup. They cost $100 a year (for a single site), a price I was willing to pay to get this fixed.
After Sucuri's regularly scheduled scans failed to find anything wrong, I filed a support ticket and pointed them at the problem. I gave them the screenshot, explained how to recreate the issue, and gave Sucuri FTP access to my blog.
They couldn't find it.
Oh, Sucuri found all sorts of non-issues, including code for inactive websites which I had never bothered to delete, but they couldn't find what was injecting the advert into my blog. On the plus side they also didn't find any evidence of malware which could threaten my visitors, but that is a small consolation.
I was never able to solve this problem, so what I should have done next was to publicly ask for help (or, if i wanted to throw money at the issue, perhaps hire a second expert). I honestly can't tell you why I didn't, but that's neither here nor there.
The problem has since gone away. I can't tell you how, but somewhere in the process of switching webhosts, changing my blog theme, and the general cleanup maintenance I performed in July, that advert went away.
Or at least I think it has.
If you see that advert, let me know so I can try to fix the issue - again. You might also forward it to any web security experts you know; I bet they would be interested.
And if you know what was causing the problem, please leave a comment. Even at this late of a date, I would love to know how this happened.