It seems we had our first unexpected downtime yesterday. :( We've had some offline times previously but most of them have been for a shorter period of time and on purpose due to maintenance and updates.

Access to our services were failing on Thursday from about 4:00 to 20:30 when I finally brought things back online. Those times are on my local EEST+3 timezone, which means from 1:00 to 17:30 UTC and from Wed 18:00 to 10:30 PDT. This is about 16 and half hours of lost time.

Looking through logs, this seems to have been caused by the server running out of memory and subsequently OOM-killing itself to death. I have few things I can do to prevent same problem bringing us down in the future. (Like moving the DB to a different server as the OOM-killer chose the poor DB to die on first round.)

I did notice the problems in the morning but due to unrelated complications (non-project ones) I didn't manage to get the server back online until that evening. I do apologize for this extending our downtime. :(

Everything should be back to normal now but due let me know if there are still problems around. Thanks and now let's crunch hard to make up for the lost time! ;)

The Cache took the strain :) Otherwise no dramas at this end, its kinda what the cache is for anyway.

First significant downtime, that wasnt really an issue due to cache, so the silence on the Boards is not remarkable, could even say it was a testiment of trust that it would get sorted as any downtime is so rare.

Queue flood of posts yelling "not for me it wasnt" rofl ...... Murphy and I can be close aquaintences at times :)

Personally I'd say take a bow, not a slap round the head :)

Regards
Zy

The Cache took the strain :) Otherwise no dramas at this end, its kinda what the cache is for anyway.

First significant downtime, that wasnt really an issue due to cache, so the silence on the Boards is not remarkable, could even say it was a testiment of trust that it would get sorted as any downtime is so rare.

Queue flood of posts yelling "not for me it wasnt" rofl ...... Murphy and I can be close aquaintences at times :)

Personally I'd say take a bow, not a slap round the head :)

Regards
Zy

+++1

And thx for an honest report on the why. Good job.

---

The Cache took the strain :) Otherwise no dramas at this end, its kinda what the cache is for anyway.

First significant downtime, that wasnt really an issue due to cache, so the silence on the Boards is not remarkable, could even say it was a testiment of trust that it would get sorted as any downtime is so rare.

Queue flood of posts yelling "not for me it wasnt" rofl ...... Murphy and I can be close aquaintences at times :)

Personally I'd say take a bow, not a slap round the head :)

Regards
Zy

+++1

And thx for an honest report on the why. Good job.

+1 ... :) Good job, Teemu.