Data is the lifeblood of businesses. With increasing reliance on the Snowflake Data Cloud to connect data to the people and applications that need it, it’s critical to have a resilient platform to enable continuous global operations. Organizations such as Fidelity and HD Supply who spoke at Summit 2022 shared how they used Snowflake to recover in the event of a disruption and to migrate data across clouds. They’re not alone. In the last 12 months, Snowflake customers have replicated over 300 PB of compressed data across regions and clouds to keep their businesses up and running.
Today, we are excited to further advance these capabilities with Account Replication and Failover, which is now available in public preview. Account Replication expands replication beyond databases to account metadata and integrations making business continuity truly turnkey. Paired with Client Redirect (which will be generally available soon), users are able to recover their account and client connections in seconds, at virtually any scale. For even higher resiliency, these features can even be used to perform failover across public clouds.
Turnkey Replication and Failover
During an outage or a disaster, recovery is not just limited to data. A data platform is the beating heart of enterprise analytics with a rich set of metadata and, akin to blood vessels transporting oxygen to the rest of the system, has integrations with several enterprise systems and client applications running against the data, such as business intelligence dashboards and data engineering pipelines.
Role-based access control, governance policies, compute resources, and network policies constitute but a few examples of the metadata organizations define in their data platform. And integrations with identity providers (such as Okta) for single sign-on, cloud storage (such as AWS S3) for loading or unloading data, and API proxies for invoking code running outside the platform constitute the blood vessels to the rest of the system. How does an organization build for continuity in the face of such complexity?
To maintain business continuity, organizations previously were faced with trade-offs. They could spend months developing custom code for replicating metadata and configuring integrations but that meant fewer resources were available for developing new applications and insights. To make matters worse, custom code could take hours to run at production scale, was brittle, and required constant maintenance. Alternatively, organizations could avoid investing in custom solutions but that meant users and client applications were left stranded during an outage or disaster.
With Account Replication, Snowflake enables replication of your account, including databases, account metadata, and integrations. Using a few commands, administrators can specify what to replicate and where, and Snowflake takes care of the rest. This eliminates custom code, shortens implementation to minutes, and accelerates adoption of Snowflake for mission-critical applications. Account Failover is equally turnkey. With a single command, a secondary can be promoted to serve as the new primary, enabling near instant recovery.
“Prior to replicating account metadata on Snowflake, we used to do all-or-nothing data center failovers. Now with these features, each application can be designed to failover and execute extended runs from out of region independently. This resilient and trusted architecture seamlessly processes tens of millions of financial transactions every day and is critical to keeping the global markets operating efficiently. We are a dependable leader that can be trusted in times of uncertainty and crisis—always prepared to manage fast-moving, unexpected crises that pose significant threats to the financial system.”
—Arjun Puri, Director of Cloud Engineering and Analytics, Depository Trust & Clearing Corporation (DTCC)
At Summit, we announced new replication support for data pipelines, including external stages, storage integrations, Snowpipe, and copy history, now in private preview. Pipelines can failover seamlessly without any code changes and can be replayed on the new primary without data being duplicated.
That’s not all. Business Continuity is not just limited to an outage or disaster. What happens when an organization’s cloud strategy changes? This could happen as a result of strategic realignment, price arbitrage, acquisitions, or mergers. Organizations must strive for maximum portability to ensure continuity in such scenarios and Account Replication delivers exactly that.
“With Snowflake and the ability to replicate both data and account objects across clouds, we were able to successfully migrate our 3.6 PB environment from AWS to Azure in a week without a single issue and zero downtime. With 37 applications and 1,100 end users relying on this data, the access to data was continuous.”
—Elliot Oshman, Senior Director of Technology, Office Depot
How It Works
To get started, you use a FAILOVER GROUP, a new Snowflake object to specify what you want to replicate, where, and when. You create a primary group in your Snowflake account and one or more linked secondary groups in accounts in other regions. Here’s an example:
That’s it. Snowflake will replicate these objects to the destination accounts at the specified schedule. This has several benefits:
1. Identity management and authentication is seamless across replicas
Users as well as security integrations for authentication are replicated. All forms of user authentication such as password, key pair, federated authentication with single sign-on (SSO), multi-factor authentication (MFA), and OAuth are supported.
2. Authorization (RBAC) is enforced consistently across replicas
Roles and access privileges are replicated. This means customers need to set up role-based access control only on their primary account and Snowflake will ensure the same roles and privileges are applied on secondary accounts.
3. Compute resources are ready to use on replicas
Virtual warehouses and resource monitors are replicated. Virtual warehouses are replicated in a suspended state to secondary accounts to optimize costs and can be resumed whenever you need. Resource monitors are replicated to help control costs and avoid unexpected credit usage on secondary accounts.
4. Point-in-time consistency across databases
Objects including databases can now be replicated as part of a Failover Group. This ensures point-in-time consistency across databases on the destination accounts. This is especially useful if you have views that join tables across databases, or multi-statement transactions that write to multiple tables across databases.
5. Centralized governance works well with replication
We have heard requests for a centralized approach to governance. Customers want to define tags, masking policies, and row access policies in one database and apply them to objects outside the database. Previously, database replication would fail because of such object references across databases. Now these object references can be replicated seamlessly with Failover Groups and ensure consistent governance across replicas.
6. Replication schedule is easier to manage
No need to create tasks to manage replication schedule for individual databases. Now you can specify a replication schedule in the Failover Group and Snowflake will replicate at the specified schedule for you.
7. Flexibility to replicate and failover by line of business
Often customers need to perform disaster recovery drills to assess readiness and satisfy regulators. But with multiple lines of business using Snowflake, it’s not always possible to get them ready for a drill at the same time. Therefore, customers want flexibility to failover by line of business. You can now define Failover Groups that align with your line of business and fail them over independently.
Get Started
Snowflake Account Replication and Failover is available on all Snowflake regions. To get started, you can review the documentation here and check out the “Replication & Failover Across Clouds” session from Summit 2022 here.
Forward Looking Statements
This post contains express and implied forwarding-looking statements, including statements regarding (i) Snowflake’s business strategy, (ii) Snowflake’s products, services, and technology offerings, including those that are under development or not generally available, (iii) market growth, trends, and competitive considerations, and (iv) the integration, interoperability, and availability of Snowflake’s products with and on third-party platforms. These forward-looking statements are subject to a number of risks, uncertainties and assumptions, including those described under the heading “Risk Factors” and elsewhere in the Quarterly Reports on Form 10-Q and the Annual Reports on Form 10-K that Snowflake files with the Securities and Exchange Commission. In light of these risks, uncertainties, and assumptions, actual results could differ materially and adversely from those anticipated or implied in the forward-looking statements. As a result, you should not rely on any forwarding-looking statements as predictions of future events.
© 2022 Snowflake Inc. All rights reserved. Snowflake, the Snowflake logo, and all other Snowflake product, feature and service names mentioned herein are registered trademarks or trademarks of Snowflake Inc. in the United States and other countries. All other brand names or logos mentioned or used herein are for identification purposes only and may be the trademarks of their respective holder(s). Snowflake may not be associated with, or be sponsored or endorsed by, any such holder(s).
Originally posted on August 30, 2022 @ 4:42 pm
