Audience
Advanced users of the Migration App.
Context
Several features are available only if Advanced Mode is enabled for your domain. These features require more technical knowledge and are recommended for use only by experienced users of Migration App.
Enabling Advanced Mode
Before you can access the Advanced Mode features in Migration App, you must have Advanced Mode enabled for your domain. To do this, submit a ticket to support@egnyte.com to request a Settings Update. This process can take up to two full business days to be implemented.
Features Available in Advanced Mode
- Controlling Threads
- Controlling Bandwidth Throughput
- Customizing the Agent Configuration
- Using Exclusion Rules
Controlling Threads
By default, the Agent running on the local Windows machine uses 4 transfers and 8 checkers (collectively referred to as the threads). Thread is a universal computing term, referring to one process in a multithreaded system that occurs simultaneously with other processes. "Checkers" and "transfers" in this context are the custom names of different threads implemented in the underlying technology. "Checkers" determine how parallel a given operation will run (e.g., when deleting files, how many deletes to do in parallel). "Transfers" control uploading (streaming) of files to the destination.
If you have Advanced Mode enabled for your domain, you can increase the number of threads and potentially improve your migration speed, assuming your Windows machine has enough compute capacity to accommodate the increase. This must be undertaken with caution, since allocating too many threads to Migration App will have a negative impact on the host machine’s compute capabilities.
Configuring Threads
Go to the Migration Details page. In the left column, click on the arrow next to Advanced Options to open the drop-down menu.
If a migration job is currently running a command, you may not modify any of the Advanced Options. You must wait until the command has completed before configuring advanced settings for the job.
Select Threads from the left menu.
Here, you can decrease or increase the number of transfers (from 1 to 10, inclusive) and checkers (from 1 to 20, inclusive) with a custom configuration. This can increase the speed of your migration job, assuming your Windows host machine has the compute power to accommodate the increase.
The recommended ratio for any configuration is 2x Checkers per Transfer.
You may return to this menu at any time to change your selection. Remember that before you can make any changes, the migration job cannot have a command running.
Your threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.
Controlling Bandwidth Throughput
For any migration job, you have the option to configure various advanced settings, including Empty Folders, Sanitization, Unattended Mode, Confluence. These are all available from the Migration Details Page. If you have Advanced Mode enabled for your domain, you will also have the ability to control Throughput from the same Advanced Options menu.
Configuring Throughput
Go to the Migration Details page. In the left column, click on the arrow next to Advanced Options to open the drop-down menu.
If a migration job is currently running a command, you may not modify any of the Advanced Options. You must wait until the command has completed before configuring advanced settings for the job.
Select Throughput from the left menu.
Select ON for Enable Bandwidth Controls and you will be presented with another drop-down menu.
You have two options for controlling bandwidth: 1Mb/second at all times or 1Mb/second during business hours. Make your selection and click Save.
You may return to this menu at any time to change your selection. Remember that before you can make any changes, the migration job cannot have a command running.
Your threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.
Customizing the Agent Configuration
Advanced Options may be customized for each individual migration job. But this can be tedious and time consuming if you have multiple jobs tied to a single Agent and want them all to have the same settings. Using a Custom Config file, you may control the settings at the Agent level, which will then be applied to every job created from that Agent.
Agent Settings
From the Migration Dashboard, click Settings to navigate to the Settings page.
From the Settings page, select the Agent you want to customize.
This opens the settings panel for that Agent.
Click on Download Default Configuration File.
Open the Configuration.json file in a text or code editor like Atom or Notepad++.
The default configuration file looks like this:
Modifying the Agent Configuration
Edit the configuration file using JSON syntax.
- Specify a configName within the empty brackets. In the example below, it has been specified as ABCConfig.
- The first section controls the Agent. You can change Transfers and Checkers, but this will change it for all jobs associated with this Agent, so you need to be mindful of how much actual compute is available on the host machine.
- Anything you specify under the “config” parameter will apply to all jobs associated with this agent.
In the example below, only the Agent configuration has been changed. It keeps the Agent at four Transfers, eight Checkers, but now Confluence and Symbolic Links are both enabled.
Once you have the configuration you want, upload the custom config file to Migration App by dragging and dropping it on the Settings page. Any future migration operations associated with this agent will reflect this custom configuration.
Using Exclusion Rules
Migration App offers the ability to use filters to granularly specify what will be migrated with a given Migration job. There are three types of file exclusions that may be specified.
Types Of Exclusions
Exclusion By File Extension
Migration Operators may specify file types to be excluded by entering the undesired extensions in a single text box. This is the simplest type of file exclusion and may be used by users with any experience level. Accordingly, Advanced Mode is not required for this level of exclusion.
Glob Pattern-Based Exclusion
Migration Operators may also exclude files based on glob patterns. The UI provides a separate text box for each pattern entry, and these entries are combined into a string array and validated before forwarding them to the agent. If a pattern cannot be validated, the UI will display an error. Rclone offers detailed documentation, but also see the Limitations and How Migration App Differs sections below.
Regex Pattern-Based Exclusion
Users can also exclude files based on regex patterns. Similar to the glob pattern-based exclusion, the UI provides a separate text box for each entry which are then combined into a string array, validated and forwarded to the agent. If a pattern cannot be validated, the UI will display an error. For further information about regex patterns, rclone offers detailed documentation for regular expressions.
Limitations
- If your exclusion rules exclude all items at the source, the scan will fail and the job will be aborted. If your exclusion rules exclude all items within only one or more folders, the job will run, but those folders will fail to be created on the destination during migration.
- Only PCRE/POSIX based regex/glob patterns (more specifically rclone recognized patterns) are supported.
- There is currently no scrolling in the UI. If there are too many patterns, they cannot all be displayed.
- For some seemingly valid patterns that are flagged as invalid, you may try putting them in curly braces ({ }) and then Save.
- We are validating a restrictive set of expressions which are supported by rclone. Furthermore, we are only validating syntax. We are not confirming it will ultimately match and successfully exclude one or more files at the source.
- For Glob and Regex expression-based exclusion, a certain level of competence is assumed about the Migration Operator.
- If you have a folder on the source that contains only unsupported files, none of these files will be migrated. As a result, this folder will not be created on the destination. Moreover, as this folder is not empty on the source, it will not be created as part of the empty folder migration step. So this folder will always be missing on the destination. There is no way presently to migrate it. The only solution is to put something inside that folder that can be migrated and run a True-Up. It will then appear on the destination containing only the one item that is supported.
How Migration App Differs From Native Rclone Functionality
For those familiar with rclone, you will see many parallels in how we have enabled Migration App versus native rclone functionality. The purpose of this section is to educate users on the difference between Migration App and what is defined in rclone documentation. The filtering functionality can be incredibly useful to ensure that only specific files or directories are transferred. The rclone documentation provides basic guidance on how to use filters, but there are some discrepancies between what is mentioned in the documentation and how rclone actually accepts patterns.
Nested Curly Braces
The rclone documentation includes some filter patterns that are nested within curly braces ({{}}). However, it's important to note that rclone does not accept nested curly braces. If you try to use a pattern with nested curly braces, you will receive an error message.
For example, the documentation includes the following pattern:
{{*.txt}}
However, to use this pattern with rclone, you need to remove the nested curly braces, so it becomes:
{*.txt}
Filtering Files With 3-Digit Extension
Rclone filtering documentation shows that the following pattern will filter files with extension with three digits:
*.\d\d\d
But in reality it is considered invalid. The same objective can be achieved by using the same pattern in curly braces like so:
{*.\d\d\d}
Incorrect Pattern Explanation
Another discrepancy in the rclone documentation is the explanation for some patterns. One example is the pattern *.jpe?g. The documentation states that this pattern will filter both jpg and jpeg files, but in reality, the pattern will filter files with the extension jpeXg, where X can be any character. To filter only jpg and jpeg files, you need to use 2 patterns *.jpg and *.jpeg. This pattern will match files with either the jpg or jpeg extension.
Questions? Feature Requests? Other Feedback?
If you have feature suggestions or requests, feel free to submit them here, and make sure you mention Migration App in the text.
For more complex requests that would benefit from providing screen shots or other attachments, you may submit them to support@egnyte.com. Be sure to include Migration App in the email title. Our Support team will push your comments to the Product team for consideration.