Can someone recommend an expert who can help me trouble shoot the problems with my blog?
Over the past 3 days I have been having serious issues with my blog using excess resources and hitting the limits imposed by my web host. (This is why the blog has been inaccessible.) I have tried everything I can think of to resolve this issue but nothing works. At this point I need an expert.
The problems started sometime on Tuesday. Without any increase in traffic, my blog suddenly started maxing out its resources limits. Since then I have run security checks, turned off plugins, replaced the blog theme, and tried everything else that I could find online (short of switching to a new webhost.)
I’m stumped. If anyone has any suggestions, or knows an expert that I can hire, please let me know.
Andres May 29, 2014 um 9:07 pm
Do you have access to the, well, access logs of your blog? That can help you diagnose the problem.
Of the top of my head, it sounds like a web crawler is stuck on your site. My best guess would be to add to your robots.txt the instructions to not crawl the site ( http://www.robotstxt.org/robotstxt.html ). A bit overdramatic, but it could help.
Nate Hoffelder May 29, 2014 um 9:32 pm
Just tried it. This doesn’t seem to have helped.
And I do have access to the traffic logs; I don’t see any issues than I can recognize.
DaveZ May 30, 2014 um 12:31 pm
Make sure you undo the robots block when you’re done testing lest you lose your search engine indexing.
Nate Hoffelder May 30, 2014 um 2:16 pm
Yep. Did that a few hours after I knew it wasn’t helping.
Sturmund Drang May 29, 2014 um 10:16 pm
Can you give us a bit more information? What messages are you receiving that lead you to believe there’s a resource problem? Which resources? You say you’re not experiencing increased traffic, what has increased? The more information you can give us the better we can diagnose your problem.
Won’t your host’s tech department assist you in ferreting out the problem?
Nate Hoffelder May 30, 2014 um 5:37 am
I remotely log in over SSH , run ps aux at the command prompt, and can see that I am running too many instances of the WordPress Index file. For example, last night I was showing 8, 10, or more instances when my site traffic should have only required one or two. This led me to repeatedly hitting the hard limits imposed by my webhost.
Michael May 30, 2014 um 2:39 pm
Do you have write access to your httpd.conf / apache2.conf file and the ability to restart Apache? You might want to check if your StartServers, MinSpareServers, MaxSpareServers settings changed. Depending on how you get Apache updates, that file can get overwritten with default values if not careful and you may inadvertently be running on settings too generous for your server setup. You may need to experiment to find the best settings, but generally 1 or 2 spare instances ought to be fine for reasonable response time for moderate traffic, with a max of 4 or 5 for busy times. I’ve done Max of 2 on really low RAM systems and things still ran fine, but if you can spare more then a couple extra spares can make things zippier. Apache will need to be restarted before changes will go into effect.
Howard Lee Harkness May 29, 2014 um 10:45 pm
I had something similar happen to one of my sites when a brute-force login attack was attempting several dozen login attempts per second, even after having been locked out by Limit Login Attempts. That does not appear in the visitors log.
The way I now handle that problem is to track down the attack origination IP and put a "deny from" entry in my .htaccess file. If you are not able to get to your .htaccess file (if you are have a resource exhaustion issue, that can be a challenge), call your ISP support, give them the IP, and ask them to verify that your site is under attack by a script kiddie, and ask them to block it in their IPTables.
The first time it happened, I was completely locked out of the account (even the cPanel), and I had my ISP do the IPTables thing. After that, I have been able to catch it quickly enough to use the .htaccess deny.
I use a dummy admin account that has no access to the site (and a very long random password plus google authentication, so that it appears to be a heavily-armored admin account). The vast majority of brute-force attacks are on the dummy account, and as soon as I get the message from Limit Login Attempts, I add the IP to the .htaccess deny list.
I’m looking for somebody who can write a plugin that detects when somebody tries to log in to that account (guaranteed to be a script-kiddie) and immediate add a deny in the .htaccess using the first three octets of the IP (lots of the script-kiddies will detect an .htaccess deny and switch to another IP address in the same block, so you might as well deny the whole block).
I would definitely give my hosting service my kudos for working with me to address this problem. (h2ha.com/d9hosting)
Nate Hoffelder May 30, 2014 um 5:40 am
Were you under attack from the bots that tried to access the site via xmlrpc.php or the bots that tried to register? I got slammed by both last year, and my solution was a lot like yours.
Those botnets don’t seem to be the trouble today.
Howard Lee Harkness June 4, 2014 um 12:43 pm
I think I had a bit of both. Just this morning, I went into StatPress Visitors, and saw that there was an unknown spambot hitting my site about once per second. A view of cPanel showed a 50% CPU load. Adding that spambot’s IP to my .htaccess reduced the CPU load to <8% — I probably still have a minor bot-problem, since I only have an average of about 1 legitimate visitor/minute on that site.
After a while, the .htaccess load process can slow things down a bit, which is why I block the *.*.* range to reduce the number of entries in the deny list.
Keeping track of the deny list for 35 sites is getting to be a PITA (although not as bad as having a site taken down with CPU overload). I'm about to write a utility to automate the process and reduce the amount of time editing the .htaccess file. And I really do need to figure out how to write that plugin to auto-insert malicious IP's into the deny list.
Tim Hare May 29, 2014 um 11:32 pm
Have you ever deleted old entries and/or requested reorganization of the database holding the posts? This kind of a long shot, and arcane technologically but some databases have architectures which go nuts after the number of records originally allocated to the database.
Nate Hoffelder May 30, 2014 um 6:48 am
Not recently, no.
Edit: This didn’t help, but thanks anyway for the reminder.
Michael May 30, 2014 um 2:44 am
In disabling plugins, did you disable all of them? In my experience, these types of plugins are usually the culprit in turning into resource hogs on WP sites:
* Related posts plugins. These almost universally hit a point as a site grows where they turn into monsters.
* Backup plugins. These typically fire long-running processes which may not always complete before your PHP execution time limit is reached, then instead of failing gracefully with a helpful error message to that effect, they try the backup again. And again and again.
* Caching plugins. Sometimes due to interference with dynamic content from other plugins, these can sometimes end up re-indexing stuff way too much, even in extreme situations on every page load, turning every visit to your blog into a disk-thrashing session as a new cached copy gets unnecessarily rebuilt.
* Events plugins, calendars, etc. Things that go beyond WP’s basic post scheduling and load up WP Cron with a bunch of stuff to happen at a later time. These often don’t play well with caching either.
* Various thumbnail plugins as well as thumbnailing scripts used by themes can turn into resource hogs at times.
* Some AJAX-based stuff can be too hard on one’s server.
Other stuff that can muck things up:
* PHP memory limit set too high or too low. Too high and some plugin or theme scripts will use too much RAM. Too low and something important fails. Similarly, PHP execution time limit set too short or too long. The right values for either can vary greatly from one site and server to another, so it can require experimentation.
* Incorrect permissions on an important folder. Eg., some themes or plugins make a folder somewhere under /wp-content for temp files or other things, and if they have trouble accessing this folder later will often just keep trying instead of giving you a helpful error, wasting CPU and IO time.
* Crawlers or hack attacks as others have mentioned. Sometimes these hit so fast and hard that your visitor logs can’t keep up and you won’t even realize it’s going on. If your URL scheme has changed any recently, causing a lot of redirects or 404s, that can hit your resources hard when a crawler visits without really showing up much in bandwidth usage.
* If you’re on a VPS or dedicated server where it’s up to you to handle upgrades, you may need to update something.
* Looks to me like you’re on 1&1. They’re one of the worst hosts I’ve ever dealt with. I haven’t used them since 2009 though, so they may be much better now, but when I was there it was frustrating because their customer service was set up with tiers you had to work through before getting to anyone truly helpful. Then when I cancelled, they continued to bill me for months after. Every time I’d call they’d swear it was fixed, refund me, and it’d happen again the next month.
If you have command line access, it might be helpful to run TOP and watch your Apache processes as well as keep an eye on total RAM available, CPU load, and IO. If your server allows swapping, you’ll see IO spike a lot when RAM gets low and the disk starts thrashing.
Michael May 30, 2014 um 3:01 am
Much simpler is what I do. Rather than blocking an ever-changing bunch of IP ranges via .htaccess, block all IPs from accessing the login except your own. The WP codex has some basic info on that here:
I need to be able to log in from anywhere though and my IP changes, so I set up a secret subfolder with a name the hackbots would never try, and in it a script that grabs my IP and updates .htaccess to allow access, then redirects to the login page. That way hackers never even see the login page once, and most bots give up immediately. Sometimes when the first bot fails they roll over to another, but even those give up quickly. One server I work with that was under almost constant attack from all over now only gets a couple of attempts per week. And none ever get to the login page to begin with.
Howard Lee Harkness June 4, 2014 um 12:52 pm
I would love to see your script! Thanks for the lead. It does sound simpler and faster than denying the ever-morphing scriptkiddie IPs. However, that might be a little trickier on the blogs that have multiple user (and author) accounts.
Howard Lee Harkness June 4, 2014 um 1:06 pm
I just implemented your solution for the site that I found under attack this morning, and it dropped the CPU load to <1%.
Yes, it is MUCH simpler (for that site). If I have to access it from another location, though, I'm going to have to implement something like your hidden-folder trick.
Michael June 4, 2014 um 2:28 pm
Glad to hear that helped! I’ll get in touch with you about my script. I think I’ll turn it into a proper plugin. Accomodating multiple connections isn’t a problem, as I can have it add to the list of allowed IPs rather than simply replacing, with an optional limit on how many to keep so it can cycle out old ones.
Howard Lee Harkness June 5, 2014 um 3:23 pm
Ran into one problem — I was using an htaccess editor pluging to access the WP .htaccess file. That worked fine — but it applies to *all* accesses, not just to the admin page. You have to put that .htaccess file into the wp-admin directory! Learned that the hard way when my traffic to one of my busier sites went to zero.
Michael June 5, 2014 um 3:29 pm
Howard, sorry I didn’t make that clear above! I’m going to try to get that plugin done this weekend so you and others can have an easier way to set this up. Since .htaccess is so important, I’ll have a friend of mine who specializes in server hardening take a look at it before I upload to WordPress.org to ensure it’s effective and also doesn’t introduce security weaknesses of its own.
Howard Lee Harkness June 8, 2014 um 8:21 pm
Please let me know the name of your plugin and when you get it done. I am very interested in using it — and studying it.
Howard Lee Harkness June 8, 2014 um 8:33 pm
BTW, I was able to substantially reduce the number of brute force attempts on one of my sites (with multiple authors) by simply removing the "meta" widget with its link to wp-login.php. That indicates that about half of the scriptkiddies are looking for that meta entry, and not even going after the wp-admin.
I was a more than a little surprised.
Nate Hoffelder June 8, 2014 um 8:37 pm
I found disabling open registration to be useful at keeping down spam.
Michael May 30, 2014 um 3:10 am
@Nate: I forgot to add, I’m no expert, but feel free to e-mail me if you want me to take a look at things for you. I can refer you to clients of mine with WP sites I manage if you need references.
Eduardo Melo May 30, 2014 um 4:23 am
Hi Nate. I’m very used to wordpress sites and might be able to help. Until a year ago, I was experiencing a very similar situation… outages, etc. My primary solution was to move to a realiable (yet costly…) host, wpengine.com. It’s by far the best support out there, and the best service. They have automatic backups, update things for you, etc…
Also, I’ll would be glad to help assisting you on a regular basis – it would be a fair retribution to you, one of my favorite sources for news.
Nate Hoffelder May 30, 2014 um 6:18 am
I should move to WPEngine, but I don’t have enough revenue to be able to afford it.
Eduardo Melo May 30, 2014 um 2:48 pm
Nate, write me on my email… I think I can fit you on my existing account with wpengine.
Howard Lee Harkness June 4, 2014 um 12:49 pm
1&1 is a member of the infamous EIG family. So is BlueHost, and now HostGator. Avoid EIG hosting if your site has any value at all. (do a search on EIG HOSTING)
Better to move to a smaller independent hosting such as D9 (http://h2ha/d9hosting). I have used D9 for about 6 months now, and have found them to be fast, reliable, and well-supported.
They have also been good about teaching me how to handle the more common problems myself, which saves a lot of time.
Dave Z May 30, 2014 um 8:10 am
1) Look at the traffic logs for excessive hits from specific locations and block them in .htaccess.
2) Did you install or update any plugins recently? Plugins can do bad things…
3) Related to #1, how’s your site security? Sucuri thinks you may have been hacked:
Nate Hoffelder May 30, 2014 um 8:39 am
I checked the logs (this is how I identified a hacking attempt last year). I can’t see any IP addresses with especially heavy traffic.
And I am already signed up with Sucuri. The problems they see are related to a couple dead sites which are sitting on my server space but aren’t connected to anything.
Dave Z May 30, 2014 um 8:14 am
Just saw a comment above. If you’re really on 1&1, you need to move. They’re the worst. There are other inexpensive providers that will treat you better – like Bluehost. For a site of your size, you should be on a managed WordPress solution tho. If WPEngine, Pressable, etc is out of reach, check out Media Temple’s new $29/mo plan. I can’t personally vouch for the new offering, but in general Media Temple provides excellent support (versus 1&1 who once held my site hostage) and most legit providers won’t cut you off when you sit some artificial threshold.
Nate Hoffelder May 30, 2014 um 8:41 am
I’ve gotten a number of comments about 1and1. I really need to move. I don’t know about Media Temple, though. It’s owned by GoDaddy now, and they don’t have a much better reputation.
Why and how did 1and1 hold your site hostage?
Dave Z May 30, 2014 um 8:56 am
Media Temple is an independent operation within GoDaddy, whose elephant-hunting CEO has since been replaced. Like I said I can’t vouch for the new plan or really the company for that matter. But I spent several years with them and their 24/7 tech support was top notch (even though I occasionally ran into performance problems) — including calling me at a hotel in Germany (since I couldn’t call out) to help get unhacked.
Regarding 1&1, on multiple occasions I received worthless and not real time tech support. I broke thru their BS thresholds a few times back when Digg was a thing and they prevented folks from browsing my site and cut off my FTP access until I upgraded into an unreasonably priced plan which I didn’t really need. Fortunately, ssh access remained and I was able to get my stuff via command line, host to host. (Nowadays, I got everything syncing nightly to VaultPress, so that’ll never happen again.)
Michael May 30, 2014 um 10:34 am
If you’re wary of Media Temple, a Digital Ocean or Linode VPS, plus the free tier of Incapsula or Cloudflare for CDN + basic security enhancements, would probably be a good step up from 1&1 and run you about $20/month. Not managed WordPress, but unlike 1&1 they won’t bug you about resource usage unless it’s impacting other customers, and they both have great people in support willing and able to help out when things go wrong.
Speaking of 1&1 holding things hostage, I’d almost forgotten that in addition to my other problems, I had a domain name with them that they wouldn’t let go. I made the mistake of doing one of those buy-hosting-get- a-domain-name-free deals, and when I left 1&1 they wouldn’t let me buy it off them. They stubbornly renewed the damn thing at their expense, pointing to a parking page, for 2 years after. When they finally let it expire I managed to grab it.
Howard Lee Harkness June 8, 2014 um 8:20 pm
I don’t have any problem with elephant hunting, but GoDaddy service has declined, and they were in favor of rules that would make the whole internet less secure, and less usable.
Valentine May 30, 2014 um 2:02 pm
I have nothing new to offer, in fact I’m learning lots of interesting things here!
And I love that we have so many nice and helpful people around!
Name May 31, 2014 um 9:00 am
The exact problem could be almost anything. Who knows? However, since I’m reading suggestions here about hosting that costs $20 and more per month for just blog hosting, that sounds absurdly high. For less than $35, you could already get a dedicated server with unmetered bandwidth. And sufficient virtual private servers can be had at about the same prices that are often seen for blog hosting only and such. You certainly shouldn’t pay more than $5 a month for simple blog hosting.
As an aside, I liked the alternative theme, which you tested.
Nate Hoffelder May 31, 2014 um 9:08 am
Thanks. That’s going to be the new theme, when it’s done.
Nate Hoffelder May 31, 2014 um 9:20 am
I like the idea of a managed WordPress site over coping with my own VPS. I know the latter is more capable but the former is a much smaller headache.
question June 16, 2014 um 12:48 pm
Is your web host Dreamhost?
After almost a decade of hosting very small, low traffic web sites there with no problem, they came at me with this excess resources schtick. They offered me very little help to solve the problem, and the methods were obviously meant to intimidate the average non-technical: use PuTTY command line, etc. I spent HOURS of unpaid time on research and site tinkering, and WEEKS of interacting with Dreamhost. In the end I had to reduce the functionality of my web sites, which were supposedly residing in "unlimited" packages to the bare minimum of 3-4 plugins.
I have concluded this is DREAMHOST’S PROBLEM. It is DREAMHOST that wants everyone to reduce resource usage for the sake of their profits. It is DREAMHOST that wants to push people with dinky low traffic personal wordpress web sites onto a high price VPS. It is DREAMHOST that regrets giving people good deals in the past and is now looking for a way to change those supposedly "forever" deals by tricking the non-technical users, the ones who always just wanted to slap up a web site and be done with it, with an incomprehensible storm of technicalese about "server resources".
If your web host isn’t Dreamhost, it’s someone who got a sniff of their UPSELL STRATEGY. These dweebs need to be outed.
Nate Hoffelder June 16, 2014 um 12:50 pm
My webhost was 1and1. I have since moved on to a better host.