As you probably noticed shortly after sending our regularly monthly newsletter ScubaBoard encountered a critical error and was taken down for several hours. For those of you who have been members for a long period you will remember that our last major hiccup was in December of last year. Like during that outage, serious data corruption occurred which threatened to take down the entire board. At this time it appears the error has been resolved and most if not all posts, user profiles and data has been recovered, however, further testing is necessary.
How did this happen? Oddly enough this happened while we were in the process of rebuilding our backup process. For several months we have been using an enhanced backup method to allow us to be able to easily recovery the board should the database fail. However, this process seemed to be causing more users than it was preventing and we were in the process of reverting to a slower but more stable import. During the backup process the script errored, which caused it to prematurely exit, something which our database software (mysql) did not like. Obviously several backups have been run since we recovered the database and our new backup script has been put back in place to run on a daily process. Issues like this should be few and far between but it is very important that we have frequent backups just incase.
This error should not have happened and I take full responsibility for the downtime that resulted.
Should you notice ANY glitches with the board its self please let us know immediately. Thank you for your understanding.
How did this happen? Oddly enough this happened while we were in the process of rebuilding our backup process. For several months we have been using an enhanced backup method to allow us to be able to easily recovery the board should the database fail. However, this process seemed to be causing more users than it was preventing and we were in the process of reverting to a slower but more stable import. During the backup process the script errored, which caused it to prematurely exit, something which our database software (mysql) did not like. Obviously several backups have been run since we recovered the database and our new backup script has been put back in place to run on a daily process. Issues like this should be few and far between but it is very important that we have frequent backups just incase.
This error should not have happened and I take full responsibility for the downtime that resulted.
Should you notice ANY glitches with the board its self please let us know immediately. Thank you for your understanding.
