Major Site Upgrade

[Mike Bobbitt]

In the recent past, the "big" problem is overwhelming the database with connections. Basically the database starts to fall behind when things get busy, and connections stack up, eventually hitting our max (700*) at which point new connections are flat out refused. That's wen you see the "could not connect to database" error, which I am trying to resolve.

Yesterday's conversion will hopefully allow the database to respond faster, preventing a large "lineup" of connections from forming. Watching the performance, I did see it approach 700 yesterday, then immediately drop to ~400 as the requests were processed. I won't be so daring as to say the problem is fixed, but I believe things are improved. We will find out how much as we move forward.

Note that this does not in any way solve the "high server load" issue... in fact it may make it worse. Before, when processes were languishing about waiting to be served by the database, they weren't causing load. If we improve the database response time, the server has to do more work, faster to keep up. See the attached technical diagram for a more detailed explanation.


Cheers
Mike

* 700 does not mean only 700 users can be visiting the site. It's significantly more complicated, but essentially the web server forks processes to handle requests (each process handles many simultaneous and sequential requests), and each process then generally creates one connection.

 

[kev994]

I wasn't following until I saw the diagram

 

[dapaterson]

Having big issues connecting with TapaTalk; the mobile site on Chrome works fine.  Mind you, I stopped updating TapaTalk several versions ago; it wants too much access and I do not have a good sandbox in system for Android.

 

[kev994]

Having big issues connecting with TapaTalk; the mobile site on Chrome works fine.  Mind you, I stopped updating TapaTalk several versions ago; it wants too much access and I do not have a good sandbox in system for Android.

Updated iPhone version is working for me.

 

[Mike Bobbitt]

Currently at 72 connections so "all is good" in that regard.

 

[Mike Bobbitt]

A quick update with good news... Immediately after the conversion, there was not much difference in the performance and number of connections (we even peaked at our max of 700 at least once). However we are now looking much better.

For those who care, the database uses I/O credits to allow bursts of activity, but limit prolonged high-utilization. (I.E. when you use all your credits, things slow to a crawl until the credits regenerate.) Yesterday's conversion consumed all available credits, as it was a pretty hefty operation. The credits regenerate over time, but the backlog took some time to burn off. Today, we are finally hitting our normal operating state, with I/O credits regenerated and the server running normally.

With over 700 active users, we hit a max of 60 connections, but right now are sitting at zero. As in no pending connections at all. When I first checked, I thought the site was down (the only other cause for seeing zero connections in the past.)

So this is great news, it means that yesterday's change was very effective.

Now we just need to put wheels on the donkey so it can keep up.


Cheers
Mike

 

[Mike Bobbitt]

A couple of weeks with only the regular amount of issues... I think we can put this one to bed and move on to the next one.

 

[BeyondTheNow]

A couple of weeks with only the regular amount of issues... I think we can put this one to bed and move on to the next one.

Thanks for the update Mike!