Sorry for the outages

[kato13]

A server cluster in the Dominican Republic has been hammering the crap out of our box for several days. At times it was responsible for 95% of our hourly traffic. That is a 20 fold increase seen in short bursts. It was like free load testing. (I am still seeing a peak of 8 page requests per second hit the firewall)

This is a very light weight box (its heavier duty replacement is coming) but it has the same software as its bigger brother.

These outages have actually been a learning experience for me on the inefficiencies of the standard Apache/PHP architecture. Once we move I expect sheer power (and honestly RAM) to keep this from happening. Being on a smaller box exasperates the problem so it makes it easier to detect. Curiosity on the underlying problem led me to watch more than simply block the ip and reboot.

But the chirping of my phone (when we go down) has now gotten more annoying than the outages were interesting. I have blocked the cluster and tuned Apache a bit.

If anyone gets "Service Unavaible" or any other cryptic message or if the site seems consistently slower than it has been in the past month, let me know here.


Thanks
kato13

edit changed "resolve" to "simply block the ip and reboot". I was looking for a resolution but not to get us backup immediately.