Army.ca Status- The History of Outages & Dark Days

[Mike Bobbitt]

Hi MOOXE,

netstat -p shows a whack of connections... I sorted them by source and nobody stood out as a real resource hog. I've also done some log analysis and I can't seem to find any particular addresses that are connecting too much or dragging down a lot of data (other than DND's public IPs, of course ).

Let's hope you're right and it just eases down on it's own.

GUNS: Signal fires work better, they keep you warmer this time of year.

 

[Trinity]

We just have another outage?  (10 minutes before this post?!)

 

[Mithras]

Personally I would try running this box with "keepalives" off and see if that improves things.  Seems like apache is running wild and gobbling up all the memory. 

BTW do you know how many requests/sec you are getting?

 

[Mike Bobbitt]

Right now it's 12.2 requests/second. Keepalives are now off, let's see what happens.

 

[GUNS]

Hi MOOXE,

netstat -p shows a whack of connections... I sorted them by source and nobody stood out as a real resource hog. I've also done some log analysis and I can't seem to find any particular addresses that are connecting too much or dragging down a lot of data (other than DND's public IPs, of course ).

Let's hope you're right and it just eases down on it's own.

GUNS: Signal fires work better, they keep you warmer this time of year.

Mike, signal fires don't work here on the Rock, too much wind.  ;D

 

[GUNS]

I tore-up my snail mail, site works perfect today. 8)

 

[Trinity]

I tore-up my snail mail, site works perfect today. 8)

By saying that.. you just jinxed the site

 

[George Wallace]

Egggsactly!

 

[Trinity]

Egggsactly!

I was only kidding....  we were down for what.... 30 minutes?!?!?!?

GUNS.....  it's your fault!

 

[Mithras]

Are you guys running any cron jobs during the morning hours? Specifically stats software?

 

[Mike Bobbitt]

The stats software runs hourly (so it doesn't have as much to process) and doesn't create a significant peak in the system load. There are a bunch of other jobs that run as well (backups, etc) but they don't *seem* to be the source of the problem. I've just updated our PHP and MySQL (last interruption today, I promise) so we'll see if that helps at all.

 

[Mithras]

Heh, was going to suggest maybe running the stats every couple of hours but you have that taken care of. Was the upgrade the cause of the hiccup this morning?

 

[Mike Bobbitt]

No, that was another CPU spike, this time so high it made the site effectively useless.

 

[navymich]

Board seems to be slowing down a bit.  But chat is really lagging and doing the fun thing of bouncing people out.

 

[Trinity]

So

Domestic Niners and I were singing the Time Warp from Rocky Horror Picture show in chat


Then the site crashed...

We're sorry.  We didn't know the system wasn't partial to Rocky Horror Picture Show songs.... we
won't sing again (in costume no less)

 

[Mike Bobbitt]

I think the internet link to the server is just veeeeery slow right now.

 

[niner domestic]

It's Trin's fault, I told him not to put on so much red lipgloss... but would he listen.......nooooooooooooo. Poor server's all stressed out now. 

 

[Nfld Sapper]

I think the internet link to the server is just veeeeery slow right now.

Looks like it as this is the only site that loads veeeeeeeeeeeeeeeeeeeeeeerrrrrrrrrrrrrrrrrrrrrrrrrrrrrrryyyyyyyyyyyyyyyyyyy slow. 

 

[Edward Campbell]

The site has been painfully slow or stalled since about 1500 (Central) today (2 Dec 06).  The rest of the Internet seems OK from here.


Edit: typos

 

[George Wallace]

You are not the only one.  It seems to have taken over 15 minutes to just load one page. 

Mike will also have to sort out some of the new Registrations that have occurred during this time.  Some people are duplicating their posts, along with other things, in their impatience to use the site.