Best Practices for Migrating Data into Egnyte Using Migration App

Audience

Any user of the Migration App.

Context

Planning a migration is a daunting task. In this article, we'll talk you through it, based on the best practices we've identified for the process. 

Before You Begin

1. Be aware that Migration App can only migrate data from on-premises sources that can be accessed via a UNC path or mapped letter drive.
2. Run a speed test by going to https://<yourdomain>.egnyte.com/speed-test/. This checks for the available bandwidth from the on-premises server to the cloud. This doesn't guarantee throughput speeds, but it will at least give you a baseline. For more information on speed, see the Helpdesk article How Long Will it Take to Upload, Backup, or Download my Files?
3. Ensure that the desired Users and Groups are created in the Egnyte Cloud for your domain. For more information, refer to the Helpdesk article for Importing Users and Groups.
4. Log into your Egnyte domain with an Admin account.  You must also be sure you can access the entire data set on the local network share with admin level privileges.
5. Review the Migration App Product Guide and the Migration App FAQs to gain a basic understanding of the tool.
6. Communicate with your Users to help them understand the migration process. They can continue using the data on the Source for standard day-to-day file system activity. But it's critical that no large data sets be moved, renamed, or deleted during data migration. This type of activity can seriously endanger the success of your migration and the integrity of the final data set.  It is also imperative that no users be allowed access to Egnyte until the entire migration and cutover are complete.

Planning Your Migration Project

Data migration requires some initial planning and analysis. It’s important that you understand and plan your strategy around some of the fundamental limitations on Egnyte that could impact your migration project.

• Egnyte does not permit more than 50K immediate children in any folder. If possible, you should address this before initiating your migration, but Migration App will alert you and interrupt the migration if this limit is exceeded.
• Some characters in file and folder names are unsupported by Egnyte, but Migration App offers two ways to mitigate this. You can elect to “sanitize” such characters on the source before migrating, or on-the-fly during migration. See Unsupported Characters and File Types. 
• As a general rule, individual migration jobs should not exceed 4TB or 6-7 million objects, so you need to plan your project accordingly. Most domains are set at a default hard limit of 5TB and 8 million objects, so any migration job that exceeds this will be interrupted. In some exceptional cases where it is not possible to break jobs down into smaller subsets, these limits can be raised. But the overall complexity of the job is increased significantly, as is the probability of errors and problems.
• The Egnyte Cloud Platform currently cannot allow individual files of more than 100GB.
• If you have many thousands (or millions!) of small files to migrate, your job will take longer than if you have a smaller number of large files, even if the total amount of data is the same.
• If you have limited bandwidth, you may wish to plan your migrations over evenings and weekends to minimize negative impact to users on your production environment prior to cutover.
• Migration App does not migrate file types that are incompatible with Egnyte, such as TMP files, active database files, and package files. See Unsupported Characters and File Types.

Taking a Phased Approach

We recommend that customers take a phased approach when planning their migrations using the Migration App. 

Phase 1:

Create your Migration Jobs. When you create a migration job, Migration App automatically scans the Source and provides intelligence as to the size and nature of the job. This can advise you whether to proceed or instead restructure the migration project before beginning any actual migration.

This scan helps you to identify if there are unsupported characters that need to be sanitized, as well as any files that will not be migrated due to incompatibility with the Egnyte Cloud platform. After the scan is complete, you will have some basic information made available in the Migration Details page user interface. In addition, we recommend that you download the Migration Report and review the information provided in the Scan register of the file.

Phase 2:

Seed the data from the Source. This is the initial Data Migration to copy data to Egnyte and will take the most time. It will also give you visibility into whether you need to address any unsupported filenames or pathnames that are too long. Don’t worry about users making normal everyday changes to the source data; they can continue working with their content and you'll be able to true it up in a later phase. Be sure to communicate basic guidelines to your users, as described above in Before You Begin, point 6.

You will be offered several Data Migration Options after you select Migrate Data from the Migration Actions dropdown button. Be sure you understand how these options are applied by reading the following Helpdesk articles:

• Understanding the Sanitize Command in Migration App
• Migrating Empty Folders with Migration App

Important Notes about Case-Sensitivity:

1. We recommend you instruct users not to change the case of file or folder names on the source once the migration has begun. The Egnyte platform is not case-sensitive, so it sees LEGAL and Legal as the same file name. But the utility at the heart of the Migration App is case-sensitive, so it sees LEGAL and Legal as two different file names. The result is the potential to completely lose your data during Phase 3 (True-Ups.)
2. If you have enabled Canonical Support on your directories, we strongly recommend that you disable it. This is not compatible with the Egnyte CFS (Cloud File System.) 

Phase 3:

Run True-Ups. After some period of time has lapsed since you ran the Migrate Data command, it’s time to True-Up or Sync your data. To do this, return to your Migration Dashboard, navigate to the Migration Job in question and select True-Up Data from the migration actions menu. This generally will take much less time than the first Data Migration, as it will mirror the Source and only migrate new or changed files (sometimes called the "deltas"). It will also give you an estimate for how much data changes at the Source over that period of time, and therefore how long to plan for the final True-Up and Cutover.

Most migration jobs involve several True-Ups before the final Cutover.

Important Note about Syncing Deletes:
Migration App syncs deleted data by default. This means that if a file exists on Egnyte but no longer exists on the Source, running a True-Up will delete the file on Egnyte. 

During a True-Up, the following steps are performed:

• Scan the Source
• Scan the Destination
• Compare the Source and Destination to prepare to mirror the two
• Migrate the new data
• Delete the files that are no longer on the Source.

You can run True-Ups as many times as you like before you finally Cutover.

Phase 4:

Migrate Permissions (this is optional.) Keep in mind that permissions migration with Migration App doesn't recognize permissions that were previously migrated. So it's a fresh migration, whether or not you've migrated permissions previously, and it commonly takes quite a while. That's why we recommend doing it as the final step in your migration project before Cutover. You may want to review the Helpdesk article Permission Translation with Migration App before proceeding.

Phase 5:

Schedule Cutover. The final phase before you Go Live is the “Cutover." Most customers plan it over a weekend and require users to stop using the Source data while they run the final Sync. This means running the same Migration Job True-Up one final time to capture the last changes and migrate them to Egnyte. Once it has been completed, you can announce Go Live, when all users start using Egnyte and disable the Source.

Planning for the Cutover

Let's say you have 900GB of total data at your source. It's reasonable to assume only a small percentage will change over the course of a business day – for this example, we'll estimate it at 5% or 45GB. So if the initial Data Seeding (Phase 1) took 18 hours to migrate, you can assume that the True-Up would take about an hour. To test this, you go ahead and run a True-Up after a day, then again after a 3- or 5-day interval. This will start to give you a picture of how much time it will take to migrate the deltas when you plan your final Cutover. 

Before you schedule the Cutover, remember you need to communicate with all content users well in advance. In addition, for this migration to be successful, users must stop using the source data permanently and switch over to the Egnyte data only after you have completed the final True-Up and Permissions Migration.

Finally, be aware that not all Cutovers are successful the first time. Lots of things can happen, so be sure to have a Plan B in place if you can’t finalize the Cutover when planned. Our Technical Support Agents are diligent and will do everything in their power to keep you on track, but our Engineering team is not on call 24/7. If something comes up that requires Engineering intervention or troubleshooting, it may force you to reschedule your Cutover. Be aware of our Terms of Use for customers using Migration App in a Self-Service manner.

 

Questions? Feature Requests? Other Feedback?

If you have feature suggestions or requests, feel free to submit them here, and make sure you mention Migration App in the text.

For more complex requests that would benefit from providing screenshots or other attachments, you may submit them to support@egnyte.com. Be sure to include Migration App in the email title. Our Support team will push your comments to the Product team for consideration.