Army.ca Status- The History of Outages & Dark Days

[armyvern]

Geez, pour souls...It's 1643 here...I was travelling home from work when it happened!!

 

[Mike Baker]

I don't find it bad at all.

 

[Klc]

  As well, PMs often freeze when sending.  Just wondering if anyone else has been having any of these problems...

I have seen this problem often as well, also when posting. The post/pm will still go thru, it just freezes before changing to the next page.

 

[Mike Bobbitt]

Sorry folks, the site has been having some serious problems today. The web server itself is overwhelming the system, and I'm not sure why.

 

[Mike Bobbitt]

Folks,

I seem to have found and neutralized the source of the problem, at least for now. Thanks for your patience.

 

[Nfld Sapper]

I hadn't noticed any problems with it.

 

[Mike Bobbitt]

Looks like I didn't find the source after all. Ugh.

 

[George Wallace]

Ah!  “A Tortoise in the Works!”  And I was wondering if it were my corrupting my system…..

So slow, I can't post without timing out.....

 

[Mike Baker]

I am on now, went off at about 5 an couldn't get back on. It is still very slow.

 

[MOOXE]

Just curious....

What are the server specs for army.ca? And what are the load averages during peak times?

 

[Mike Bobbitt]

2 Gb RAM, 2.8GHz CPU, 250 Gb SATA drive

Load average was well above 100 earlier today. (Geeks will whistle in awe at that)

I've narrowed it down to apache, the httpd processes are raking up all the CPU. Not sure why. strace shows a lot of '-1 EAGAIN (Resource temporarily unavailable)' messages, but since I didn't check before the problems, I don't know if that's normal or not. The problem seemed to come out of nowhere this afternoon, reboots don't help. When apache is shut down, the system goes quiet.

You know what I know.

 

[Journeyman]

You know what I know.

Not even remotely.

Since I was obviously intended to be one of the sheep within the Matrix, never offered the choice between the red or blue pill, I'm massively grateful for you guys that can keep this sort of thing running  ;D


...and yes, dammit, my VCR does flash  12:00  12:00  12:00

 

[Klc]

Had an outage here for at least one hour around 1800 CST, both the boards and the website were inaccessible.

 

[Mithras]

What version of Apache are you using? And have you optimized any of the options or are they stock?  How is mysql/php configured?

I have had similar issues with Apache and in my case it was the keepalives settings gobbling up all the memory.

 

[Mike Bobbitt]

Hi Mithras,

We're using:

php-5.1.6-1.1
httpd-2.2.2-1.2

KeepAliveTimeout was recently reduced to 5 from 15, but had not discernible effect. We're using the prefork MPM with these settings:

StartServers      8
MinSpareServers    10
MaxSpareServers  30
ServerLimit      256
MaxClients      256
MaxRequestsPerChild  0

I'm open to suggestions on those, I haven't adjusted them for over a year and in that time, have largely forgotten the nuances.


Thanks
Mike

 

[Mithras]

You may want to try giving "MaxRequestsPerChild" a value.  Apaches default is 10000 and worked well enough for the high traffic server I was administrating.

What you are doing by not giving it a value is allowing child servers to run indefinitely and potentially incurring memory leakages which of course leads to excess memory usage.  Giving it a value will cause the child server to "die" and a new one to start.

What is your setting for "MaxKeepAliveRequests"? If it is '0' you may want to give it a value.. default is 100 but a higher value is generally better.  From experience somewhere between 500-1000 works alright.  But each server is unique so you may have to tinker.

Aside from that, if you haven't optimized mysql you might want to try that too.  You can get discernable results by tweaking only a few settings.

 

[Mike Bobbitt]

Hi Mithras,

Thanks for the info. I've given MaxRequestsPerChild a value of 10000 and bumped MaxKeepAliveRequests up from 500 to 800. MySQL was "roughly" optimized not too long ago, but doesn't seem to be the issue... it generally doesn't break 10% CPU and 25% memory. httpd on the other hand spawns a lot of processes (which is not abnormal for Army.ca) but these take up all available CPU.

Anyway, thanks for the advice, we'll see where things go from here.


Cheers
Mike

 

[Mithras]

No problem.  Hopefully you find the sweet spot.. I know from experience how annoying it can be.

 

[MOOXE]

There have been reports that last month there was a strong surge in spam. All types of it, referrer spam being the one web admins have to deal with. Sometimes when I check my logs theres 1000's of lines of poker sites. On my site (in my profile) every now and then, people from China shut down the site. They use programs that are meant to download the site all at once. They open so many mysql threads that my load average goes above 50. The site is non accessible then. I tried fighting it, but theres just too many ways around an .htaccess file, and too many types of site downloader clients, some of which can change thier ids. I just live with it now, they always go away after an hour or so. For this one incident I wouldnt bother changing your httpd conf file.

When the server does overload netstat -p will tell you whos all connected.

 

[GUNS]

I am not saying this site is slow but I am seriously considering using snail mail to respond to topics. ;D