Due to a critical bug in the database server software that underpins Accelo, we've placed the Client Portal into maintenance mode until Friday the 15th of November.
For further information, please continue reading below or email [email protected]. We're very sorry for any inconvenience this causes.
[Update 1]�Wednesday 13th November: we've successfully deployed a copy of our live database�server infrastructure, tested it, and confirmed that the underlying bug is gone in the upgraded version. We've also tested and made sure the new version doesn't introduce any other bugs or problems.
[Update 2]: Friday 15th November: while the process of upgrading the database has worked flawlessly (and while systems are online) in our tests of the cloned environment, we don't want to take any chances that having systems live and under load could cause problems for such important infrastructure. As a result, we've decided to have our first period of mult-hour planned maintenance this Friday/Saturday (depending on where you are in the world). See our planned maintenance blog post for more information.
We launched the Accelo client portal last Thursday, and after many months of active development and a slew of awesome new features, we couldn't wait to share it with you. Unfortunately, however, we've been bitten by a very unusual bug in the MySQL database engine that provides the primary database storage for the Accelo platform and have taken the tough decision to turn off the Client Portal across the board until we can resolve the issue. We expect to have the issue resolved by the end of this week, and have it live again on Friday the 15th of November.
What is the bug and what does it mean?
The bug we're being affected by causes a segmentation fault when certain query types are processed by MySQL. With the enhanced permissions, access control and security used by the Client Portal, we're doing more and more advanced things under the hood - unfortunately, one of these issues is causing the underlying MySQL database that we use to have a segfault.
For those who haven't dealt with server software much, a segfault or "Segmentation Fault" is kind of equivalent to the Blue Screen of Death in Windows, except that is crashes a single program, not the whole operating system.
If this was happening to something less critical we'd work on an upgrade/fix in the background and let you know that there might be a few bugs or hiccups - the Client Portal is a beta release at this stage after all. However, because of the nature of this crash/bug being in a critical piece of our server infrastructure - the database server - and because any crashes are breaking more than just the Client Portal, we've taken the tough decision to disable external access and put the portal into maintenance mode temporarily.
What is the plan from here?
The good news is that, while rare, we're not the first people to come across this bug. It affects the version of MySQL we use, which is by no means the latest version that is available. In fact, we're using an older version because the database at the heart of Accelo is so critical that we don't want to be on the bleeding edge - older tends to be more stable. However, when bugs are found in something like MySQL, the vendor, Oracle, tends to only patch the recent versions. The exception to this rule relates to any security vulnerabilities - these are patched quickly and retrospectively (but the good news is that older versions tend to have had these problems mostly found and fixed).
Picking the right version of something as critical as a database server is very important, and because of the importance of this software there's a strong bias towards "if it ain't broke, don't fix it". Unfortunately, for the first time, we've come across a bug in the MySQL database server which is causing a crash that is really important for us to solve.
The plan is for us to spool up a copy of our live database infrastructure, perform the upgrade on the underlying software, and then run it in parallel with our live database server so we can test it and make sure it doesn't result in the introduction of any new/different bugs or problems. This process of testing and upgrade will remain invisible to all of our users, and we're expecting that if all goes according to plan we should be able to upgrade the version of MySQL, overcome this bug and reactivate the client portal later this week.
We'll continue to update this blog post with new information as it comes to hand over the coming days.
