Migrate from Azure Cosmos DB to MongoDB Atlas Using Dsync

Migrating data between databases can be a complex and time-consuming process, especially for large-scale, mission-critical workloads. One increasingly common scenario is the online or live migration from Azure Cosmos DB to MongoDB Atlas, a high-performance, cloud-native NoSQL database with a wide range of advanced features. This use case presents unique challenges, including compatibility, downtime minimization, and ensuring data consistency throughout the process - see our previous post for more details.

While existing solutions like custom scripts, mongodump/mongorestore, and third-party tools are available, they often fall short in scalability, resiliency, or ease of use. Enter dsync—our open-source tool specifically designed to handle these complex migrations faster and with less risk.

Dsync: Built to Solve the Pain of Data Migration

Dsync is a powerful data migration and synchronization tool developed by Adiom, Inc. to address the needs of developers and DevOps teams dealing with large-scale migrations. We built dsync after seeing the limitations of other tools firsthand—either they required too much manual intervention, or they couldn’t scale effectively for production-level workloads. Dsync offers a simple, resilient, and highly performant solution that’s built with both ease of use and enterprise-grade reliability in mind.

Key Advantages of Dsync:

• Ease of Use: Deploy dsync with a single binary—no additional infrastructure required. Run it on VMs or Docker and manage migrations via CLI or web-based monitoring. All complexity of orchestrating a highly parallelized live migration is abstracted away from the user.

• Resiliency: Migrations can be interrupted and resumed without losing progress. Dsync ensures data consistency and integrity at every step.

• Speed: It significantly accelerates migration timelines, reducing hours-long processes to just minutes compared to traditional methods. What could've been a 2 hour wait becomes a 10-minute thing.

• Security: Dsync doesn’t store data or send it anywhere except the designated destination. It also supports network encryption to keep data secure.

• No Hidden Costs: As an open-source tool, dsync eliminates the need to onboard expensive SaaS solutions or purchase licenses in early stages of the process.

Prerequisites for Migration

Before you start migrating from Azure CosmosDB to MongoDB Atlas using dsync, make sure you have the following in place:

1. Provision the Source (Cosmos DB) and Destination (MongoDB Atlas) databases. For optimal performance and to avoid request timeouts, make sure to provision at least 4000 RU/s on your source Cosmos DB collections. On the Atlas side, we recommend M50 or better at least for the migration itself. Dsync supports load level configuration to control how fast it works.

2. Connection strings for both your Cosmos DB and MongoDB Atlas instances, including the necessary credentials and configuration. Don't worry, dsync doesn't transfer, store or log your credentials anywhere.

   1. Instructions for Cosmos DB

   2. Instructions for Atlas

3. Provision a VM for dsync. The most common VM configurations are 4 CPUs, 16 GB of RAM or 8 CPUs, 32 GB of RAM. Ensure that the network is properly configured - dsync should be able to connect to both the Cosmos DB instance and MongoDB Atlas from that host.

Steps for a Live or Online Migration Using Dsync

Once the prerequisites are set, follow these steps to execute your migration seamlessly. You can also find these steps and additional details in our documentation.

1. Download dsync

Head over to the dsync GitHub repository and download the latest release. Unpack the zip and choose the binary that matches your host.

2. Run the Migration

Use the CLI to start your migration by providing the necessary connection strings and any configuration settings (such as the databases and collections you want to migrate). Dsync will initiate the transfer of data from your Cosmos DB source to MongoDB Atlas destination.

./dsync -s <Cosmos> -d <Atlas> --progress --logfile dsync.log

By default, dsync will use the "adiom-internal" database in your MongoDB Atlas instance to store the metadata, but it's configurable.

3. Monitor the Process

You can monitor your migration in real-time via the web-based interface (accessible via localhost:8080/progress) or directly in the CLI. Dsync provides detailed logs and metrics on throughput, latency, and any potential errors, allowing you to track progress and troubleshoot as needed.

After the initial data copy, dsync will automatically switch to CDC-based incremental change replication.

When the reported replication lag ("Events to catch up") gets close to 0 and stays there, it means Cosmos DB and Atlas are synchronized.

You can initiate the cutover by stopping the writes on your source. After that, the lag should go down to 0 quickly. At this point, the data migration is complete.

4. Validate the Data

Dsync includes built-in validation mechanisms to ensure the data is transferred accurately. You can run post-migration checks to verify that all data matches between Cosmos DB and MongoDB Atlas, minimizing the risk of inconsistencies. All you need is to restart the original dsync process and add one new parameter at the end:

./dsync -s <Cosmos> -d <Atlas> --progress --logfile dsync.log --verify-quick-count

dsync will display the verification result in the CLI.

5. Cutover

Once the migration is complete and validated, you can cut over your applications to point to the new MongoDB Atlas database. The switchover can be done with minimal downtime, ensuring a smooth transition without disrupting your business operations.

Next Steps: Unlock the Power of MongoDB Atlas

With your data successfully migrated to MongoDB Atlas, you can now leverage its advanced features. MongoDB Atlas offers significant performance improvements with the new 8.0 release, including faster queries and enhanced scalability. You can also take advantage of native vector search for AI-driven applications and robust security features for compliance and data protection. Whether you're scaling globally or improving operational resilience, Atlas provides the tools to make the most of your data.

Conclusion

Migrating from Azure CosmosDB to MongoDB Atlas may seem daunting, but with dsync, the process becomes manageable, resilient, and fast. By following these steps, you can ensure your data is moved with minimal disruption and be ready to unlock the full potential of MongoDB Atlas’s advanced features. If you’re ready to modernize your database infrastructure, dsync is here to help you accelerate the transition and keep your data safe along the way.

You can see dsync in action in our live demo on Youtube.

At Adiom, we built dsync to help developers and DevOps perform live migrations and real-time replication easily. Dsync is a fast, reliable, easy-to-use and Open Source solution. Using dsync helps to accelerate and derisk projects, and allows teams to get their applications and services onto a new database in minutes instead of months.

Give dsync a try by downloading it from GitHub. It's distributed as a binary that you can just run anywhere, including your laptop, and it doesn't require specialized infrastructure or any complex setup.

Adiom offers help and commercial support on terms that suit your projects best. Get in touch with us here.