Failure of Cloud Service

On July 17, 2015, at 8:45am EST, the provider who hosts the Cadent web site as well as syncing services experienced what they called a "catastrophic hardware failure”.  They were not able to recover the lost sites/systems from backups.  Their summary: "We are now classifying this as a complete data loss.”  Certainly not news anyone wants to hear.

I had occasionally taken snapshots of the entire system, but as it is several gigabytes of data, I did not do this very often.  I basically was trusting that the provider was doing daily backups which would work so I was very lax in these snapshots.

This weekend, I rebuilt the site from the last snapshot I had.  It was from Oct. 15, 2014!!!  The web site has not changed at all since then so there was no loss there.  It has taken awhile to get the security certificate (SSL certificate) properly re-installed.  As such, many customers have seen communication errors and other such problems.  As of this writing, the certificate seems to be working, but I have seen the occasional failure possibly due to residual caching.  

As for the syncing database, now there is a problem.  The data there has been reset back to Oct. 15, 2014.  This is going to cause grief to customers who have been syncing since then (most I would expect).

After looking at the problem from many angles, there is really only one sure way to get the sync database back up to date: a complete re-sync of all customer data.  It would be great if there was a way to re-trigger a complete re-sync automatically as each customer attempts their next sync, but the code was not written to expect such a thing to be needed.

So, I am advising all customers to do the following to get the syncing database back up to date with their devices:

  1. Select the device which has the most up to date collection on it.  If you own only one device running wineCellar, than use that one.
  2. If you are currently running wineCellar on that device, quit out of it and re-run it to make sure all communication data is clear out of the pipeline.
  3. On iOS: press the “i” (Info) icon to access the Cloud Credentials panel.  On the desktop, choose “Show Account” from the File menu.
  4. Choose to “Detach” from the Cloud.
  5. On the desktop, when asked are you sure, say “yes”.
  6. When asked if all wine entries in the cloud should be deleted/cleared, say “yes”.
  7. When the detaching operation is complete, re-enter your username and password and select to attach to an existing account (iOS) or just to join the cloud (desktop).
  8. When the attaching is done, leave/close the window/pane and press the sync button.

Completing the above steps with one device will replace the old data in the syncing database with the contents of that device’s wine entries.  For all other devices you have running wineCellar, follow the above steps, but when asked if you want to clear the cloud, say no.  This will re-sync the other devices against the cloud where your first device’s data already exists.

I truly apologize for this situation.  I am very disappointed that this has happened and shaken my confidence in third party providers (including Cloud provider).  As such, I will be moving away from third party hosting and pulling everything on to a server I own and manage myself so I can be sure proper measures are being taken.  I’ll update this news channel as that work progresses.

Copyright © 1997-2015 Cadent Computing Inc. All rights reserved