18
Dec

Recent ChangeIP Outage

As you know, the last few days have been a significant challenge.  Now that we’ve recovered I wanted to take the time to share the details of what happened and what our plan is moving forward to ensure you receive the great service you know and deserve from us.

On Sunday, December 11 at 1PM PT, during a scheduled routine off-site backup procedure, a critical error was performed that caused an unprecedented complete outage of our backend systems.  Our inquiry and root cause determination efforts confirmed that a simple human error was responsible.  Specifically, while following the procedure on our primary backup server, inadvertently we ran instructions intended for that backup instance on our primary operational database.  The instruction dropped our primary database and cascaded to all redundant systems.

Why did take so long to recover?  Within one hour of the failure, the entire team convened to address the service outage and begin recovery efforts. Unfortunately, due to the primary failure affecting all replicated instances, we were required to conduct a transactional rebuild of the databases used to run the service.  This transaction record recovery was a massive data importation effort on the order of many billions of transactions.  On Monday, December 12th at 10:50PM PT, the main backup restore procedure was complete. However, this backup was missing the last 24 hours prior to the system failure. In order to avoid potential fatal systems failures and to data integrity, we decided to enable only DNS records at that time and proceed with recovering transactional logs. This process took much longer and was completed on Wednesday at 11AM PT and we were able to bring the systems completely back online with no data loss.

What did we learn?  We learned that when we started providing this service to the world, we made design and data layout decisions that made sense at the time but no longer do.  Over the years we’ve been fortunate to grow our service in ways we never expected.  That success resulted in data needs we hadn’t yet considered. We will be working diligently to increase our systems resilience and availability.

How will we ensure this doesn’t happen again?  We’ve developed a new database design that will address our need of faster recovery should we ever need to do a transactional recovery again.  Additionally, we’ve implemented a fault tolerant replication model that will allow us time to stop replications before complete outage can happen should something happen to the primary database instances.

We at ChangeIP are proud to be your DNS provider.  We couldn’t do any of this without your support.  Letting you down has felt absolutely horrible.  We’ve identified and are taking the steps to ensure this never happens again but want to express to you both our apology and gratitude for your business.  We will be issuing a free month of service to all of our paying customers. Please accept this credit as we work to regain your trust. To take advantage of the free month service please open a ticket through our helpdesk.

18
Aug

Awesome Upgrades to all Virtual Private Servers (SSD VPS)

ChangeIP, the rockstar, low-cost and high-touch web host, has today announced upgrades to its Virtual Private Server (VPS) hosting. All VPS packages now feature:
Read more

24
Jun

VPN and VPS Provider ChangeIP Starts Accepting Bitcoin and Alipay

ChangeIP.com now accepts for payment both the cryptographic currency Bitcoin and China’s leading third-party online payment solution, Alipay.
Read more

29
Apr

ChangeIP Announces Web & VPS Hosting, and the Fastest VPN

ChangeIP, the rockstar, low-cost and high-touch web host, today announced the launch of their new website and their new suite of products.
Read more

14
Apr

Internet Privacy Concerns? Use The Best VPN Service

Sitting behind a keyboard in the privacy of your home, it’s easy to feel that you’re invincible and invisible. You can do whatever you’d like and post whatever you’d like, wherever you’d like – and since you use a fake name or handle, no one will ever know it was you.
Read more

12
Apr

The VPS FAQ: Frequently Asked Questions about Virtual Private Servers

Those who have done a little reading about virtual private servers are usually intrigued by the low-cost option of having their own dedicated space on a web server to use as they please, without having to share resources with dozens or even hundreds of other clients.
Read more

11
Apr

The ABCs Of VPS: Virtual Private Servers And Choosing A Provider

You may have seen TV or online ads offering “COMPLETE BUSINESS WEB HOSTING FOR JUST $4.99 A MONTH!” or something similar, and you probably said to yourself, “That can’t be for real.” Well, it is – sort of. But a $4.99 introductory hosting package isn’t going to be suitable for most modern businesses. What you probably need is an SSD VPS: a virtual private server (known as a VPS) which will cost a little more than the ad promises, particularly if it has modern solid-state (SSD) storage capacity.
Read more

10
Apr

The VPN-Netflix Battle: How To Still Watch Your Favorite Movies And Shows

For many years, it was easy for Netflix customers anywhere in the world to watch all of the streaming service’s content. If they were traveling or living in a country where the content they wanted to watch was “blocked,” all they had to do was use a VPN (virtual private network) service which made it appear they were accessing Netflix from an allowed location. Customers in America could do the same thing if, for example, they were on a break at work but their company’s network wouldn’t let them access Netflix.
Read more

9
Apr

What Is A Virtual Private Network? How Do You Choose The Best VPN Service?

You probably know someone who won’t do any of their banking or shopping online for fear that their personal information will be stolen. You may also know someone who is very careful choosing the websites they visit or refuses to use public WiFi networks, because they’re convinced that their movements and activities can be easily compromised and used against them by hackers, criminals or the government. Those often-valid concerns may be yours as well.
Read more

8
Apr

Finding The Best Shared Hosting: Virtual Hosting Is Not A Commodity

So you’ve decided to put a website online, and you’ve checked out shared hosting packages offered by the big-name web hosts. You would be completely forgiven if your eyes glazed over pretty quickly – it happens to everyone. Unlimited this, unlimited that…one-click installation, free backup, concurrent connections…PHP, Perl, MySQL…not only is half of it unintelligible to most people, it looks like every host has exactly the same chart with exactly the same features.
Read more