Follow

Sorry about the Pine Store downtime (which never got mentioned by me on Mastodon -- oops!), we're back online now.

We had some data corruption issues that had to be fixed for the sales team's sake, but to be clear: NO ORDERS WERE LOST. However, a small handful of customers may be directly contacted by the sales team to clarify some missing information.

@PINE64 I guess the database didn't get to write some information to disk during the heavy orders of the pmOS PinePhones?

@caskd The technical reason is that the traffic spike was so massive and demanding that our Store server ran out of memory, and the Linux kernel had to do an oom-kill to recover itself.

By sheer bad luck, oom-killer chose mysqld to kill, while it was in the middle of a write. This crashed a database table which holds order metadata. We had to rebuild that metadata before the store could be reopened.

@PINE64 isn't mysqld supposed to have a oom score adjustment that would tell the kernel to reap the process in the worst case scenario where no other processes are left?

@caskd I honestly do not know. But what happened, happened. It's been recovered now, and we've done some hardware/software upgrades in the meantime.

@PINE64
I figured since the store was down. A few hours ago I was able to purchase my PinePhone convergence kit, I'm so happy, can't wait to out my hands on it!!

@PINE64
Good to know!
Order worked for me, but I did not get an order confirmation by mail, yet.
Hope everything is fine know.
Thanks for the update.

Sign in to participate in the conversation
Fosstodon

Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.