Advanced Mode in Migration App

Audience

Advanced users of the Migration App.

Context

Certain features are accessible exclusively when Advanced Mode is enabled for a domain. These features are intended for use by experienced Migration App users and require a higher level of technical expertise.

Enabling Advanced Mode

To gain access to Advanced Mode features in Migration App, Advanced Mode must be activated for the domain. This can be done by submitting a request to support@egnyte.com for a Settings Update. Note that this process may take up to two business days to complete.

Features Available in Advanced Mode

• Controlling Threads
• Controlling Bandwidth Throughput
• Customizing the Agent Configuration
• Using Exclusion Rules
• Email Recipients 

Controlling Threads

By default, the Agent running on the local Windows machine uses 4 transfers and 8 checkers (collectively referred to as the threads). Thread is a universal computing term, referring to one process in a multithreaded system that occurs simultaneously with other processes. Checkers and transfers in this context are the custom names of different threads implemented in the underlying technology. Checkers determine how parallel a given operation will run (e.g., when deleting files, how many deletes to do in parallel). Transfers control uploading (streaming) of files to the destination.

For domains with Advanced Mode enabled, it is possible to increase the number of threads to potentially improve migration speed, provided the Windows machine has sufficient compute capacity to accommodate the increase. However, this adjustment should be made cautiously, as allocating too many threads to the Migration App may negatively affect the host machine's overall performance.

Configuring Threads

1. Access the Migration Job Details page.
   
2. In the left column, click on the arrow next to Advanced Options to open the drop-down menu.
   
3. Select Threads from the left menu.
   Here, the user can decrease or increase the number of transfers (from 1 to 10, inclusive) and checkers (from 1 to 20, inclusive) with a custom configuration. This can increase the speed of the migration job, assuming the Windows host machine has the compute power to accommodate the increase.
   The recommended ratio for any configuration is 2x Checkers per Transfer.
   

Threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.

Controlling Bandwidth Throughput

For any migration job, various advanced settings can be configured, including Empty Folders, Sanitization, Unattended Mode, Confluence, all available from the Migration Details Page. With Advanced Mode enabled for a domain, control over Throughput is also available from the same Advanced Options menu.

Configuring Throughput

1. Navigate to Migration details page and expand Advanced options dropdown. Select Throughput from the left menu.
   
2. Select ON for Enable Bandwidth Controls and another drop-down menu will appear.
   
3. There are two options for controlling bandwidth: 1Mb/second at all times or 1Mb/second during business hours. Make the selection and click Save.

Threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.

Customizing the Agent Configuration

Advanced Options may be customized directly from the User Interface for each individual migration job. But this can be tedious and time consuming if there are multiple jobs tied to a single Agent and all of them are required have the same settings. Using a Custom Config file, the user can control settings at the Agent level, which will then be applied to every job created from that Agent. Users can also use the Custom Config to address individual job exceptions

Agent Settings

1. From the Migration Dashboard, click Settings to navigate to the Settings page.
   
2. From the Settings page, select the Agent that needs to be customized. This opens the settings panel for that Agent.
   
   
3. Click on Download Default Configuration File.
4. Open the Configuration.json file in a text or code editor like Atom or Notepad++. The default configuration file looks like this:
   

Modifying the Agent Configuration

Edit the configuration file using JSON syntax.

1. Specify a configName within the empty brackets. In the example below, it has been specified as LightConfig.
2. The first section controls the Agent. The user can change Transfers and Checkers, but this will change it for all jobs associated with this Agent, so they need to be mindful of how much actual compute is available on the host machine.
3. Anything specified under the config parameter will apply to all jobs associated with this agent.
   
   In the example below, only the Agent configuration has been changed. It keeps the Agent at four Transfers, eight Checkers, but now Confluence and Symbolic Links are both enabled.
   
4. Once the configuration changes have been made, upload the custom config file to Migration App by dragging and dropping it on the Settings page.
   Any future migration operations associated with this agent will reflect this custom configuration.
   

 Modifying Individual Migration Job Configurations

Users can also specify unique settings for individual jobs – or exceptions to the Agent settings – by calling them out under migrations.

1. Replace the “null” reference with brackets and follow the syntax provided below.
2. For “name”, enter the exact migration name
3. For “id”, copy the Migration Job ID from the Migration Details page.
4. Specify the unique exceptions for that job for Transfers, Checkers, Confluence and/or Symbolic Links.
   
   In the example below, Job A and Job B have unique configurations, but any other jobs associated with Agent Vitality will follow the configuration specified for the Agent.
   
   

   The migration job and scheduler options are summarized in the table below.

   
   

   Field	Valid Field Value		Impact On Upload
   	Migration Config	Scheduler Config
   transfers	Between 1 to 20	Not Applicable	Updates transfers Result – Improves Migration speed
   checkers	Between 1 to 20	Not Applicable	Updates checkers Result – Improves Migration speed
   min_diskspace_required	Minimum default is 10	Not Applicable	Not Applicable
   confluence	Boolean value - true or false	Not Applicable	Sets confluence mode for migration job Result - allows creating and performing copy/sync for conflicting destination job if confluence is set to true
   symlink	Boolean value - true or false	Not Applicable	Sets Symlink behaviour Result - allow symlink data upload from source if set to true
   sanitize	Not Applicable	Between 1 to 3.  1 - Never  2 - On the Fly  3 - On the Source	If value selected for sanitize is 2 then sanitize option will be set to “Always On Fly”and it cannot be changed further.  Result - If sanitize on fly/source is added then invalid file/folders will be sanitized and allowed to migrate.
   emptyFolders	Not Applicable	Boolean value - true or false	Sets the preference for empty folder migration. Result - if set to true then allows performing empty folder operations
   skipScanPeriod	Integer Value: 0 to 24	Not Applicable	Updates skip scan period Result - If Migration or true-up is executed with the specified period then scan will be skipped. Else the scan will run as part of Migration or true-up
   exclusionRules	Refer to using exclusion rules section	Not Applicable	Result - files and folders that match the exclusion rule will be skipped during upload
   startingFrom	Not Applicable	UTC time formatted as YYYY-MM-DDTHH:MM:SS	Set the start date and time for a Migration job
   frequency	Not Applicable	Integer Value: 1, 3, 5, 7, 14, 30	Sets Interval between jobs in number of days if job is periodic
   times	Not Applicable	Any Integer Value	Number of times job should run. 1 - for one time job
   cleanfutureTimestamp	Not Applicable	Boolean value - true or false	sets behaviour for future timestamps. Result - Changes future timestamp to present if set to true
   syncDeletes	Not Applicable	Boolean value - true or false	sets behaviour for syncing deletes    Result - If set to true will delete files in destination that have been deleted on the source
   allowInterruptionForSyncDeletes	Not Applicable	Boolean value - true or false	Sets the behaviour for interrupting true-up when they are a lot of deletes on source Result – will stop the true-up job if there are a lot of deletes on source allowing users to decide how to proceed.
   emailrecipients	Null (default) Email addresses of the recipients	Not Applicable	E-mail address updated for notifications. Result - Email updates are shared with the recipients added.

5. Once the configuration changes have been made, upload the custom config file to Migration App by dragging and dropping it on the Settings page.
   Any future migration operations associated with this agent and Jobs A and B will reflect this custom configuration.
   

Using Exclusion Rules

Migration App offers the ability to use filters to granularly specify what will be migrated with a given Migration job.

Types Of Exclusions

If Advanced Mode is not enabled, only the Exclusion by File Extension option will be visible in the Advanced Options menu.

Exclusion By File Extension

Migration Operators may specify file types to be excluded by entering the undesired extensions in a single text box. This is the simplest type of file exclusion and may be used by users with any experience level. Accordingly, Advanced Mode is not required for this level of exclusion.

Once Advanced Mode is enabled, two additional Exclusion Rules options will be available.

Glob Pattern-Based Exclusion

Migration Operators may also exclude files based on glob patterns. The UI provides a separate text box for each pattern entry, and these entries are combined into a string array and validated before forwarding them to the agent. If a pattern cannot be validated, the UI will display an error. Rclone offers detailed documentation, but also see the Limitations and How Migration App Differs sections below.

Regex Pattern-Based Exclusion

Users can also exclude files based on regex patterns. Similar to the glob pattern-based exclusion, the UI provides a separate text box for each entry which is then combined into a string array, validated, and forwarded to the agent. If a pattern cannot be validated, the UI will display an error. For further information about regex patterns, rclone offers detailed documentation for regular expressions.

Email Recipients 

Egnyte provides an option to include additional email recipients using Advanced options. The domain users receive email alerts for the updates related to the Migration statuses. Using advanced options, it is possible to add and save up to 10 recipients for them to receive the email alerts. 

Navigate to Advanced Options -> Email Recipients 

Enter any valid email address(es) and click on Save. The migration alerts will be received by the admins and the additional email recipients added. 

The users will be alerted if an invalid email address is entered. 

Limitations

• If exclusion rules exclude all items at the source, the scan will fail and the job will be aborted. If exclusion rules exclude all items within only one or more folders, the job will run, but those folders will fail to be created on the destination during migration.
• Only PCRE/POSIX based regex/glob patterns (more specifically rclone recognized patterns) are supported.
• There is currently no scrolling in the UI. If there are too many patterns, they cannot all be displayed.
• For some seemingly valid patterns that are flagged as invalid, try putting them in curly braces ({ }) and then Save.
• We are validating a restrictive set of expressions which are supported by rclone. Furthermore, we are only validating syntax. We are not confirming it will ultimately match and successfully exclude one or more files at the source.
• For Glob and Regex expression-based exclusion, a certain level of competence is assumed about the Migration Operator.
• If a folder on the source contains only unsupported files, none of these files will be migrated. As a result, this folder will not be created on the destination. Moreover, as this folder is not empty on the source, it will not be created as part of the empty folder migration step. So this folder will always be missing on the destination. There is no way presently to migrate it. The only solution is to put something inside that folder that can be migrated and run a True-Up. It will then appear on the destination containing only the one item that is supported.

How Migration App Differs From Native Rclone Functionality

Those familiar with rclone may notice several similarities in how Migration App has been enabled compared to the native rclone functionality. The purpose of this section is to educate users on the difference between Migration App and what is defined in rclone documentation. The filtering functionality can be incredibly useful to ensure that only specific files or directories are transferred. The rclone documentation provides basic guidance on how to use filters, but there are some discrepancies between what is mentioned in the documentation and how rclone actually accepts patterns.

Nested Curly Braces

The rclone documentation includes some filter patterns that are nested within curly braces ({{}}). However, it's important to note that rclone does not accept nested curly braces. If a user tries to use a pattern with nested curly braces, they will receive an error message.

For example, the documentation includes the following pattern:

{{*.txt}}

However, using this pattern with rclone requires removing the nested curly braces, so it becomes:

{*.txt}

Filtering Files With 3-Digit Extension

Rclone filtering documentation shows that the following pattern will filter files with extension with three digits:

*.\d\d\d

But in reality it is considered invalid. The same objective can be achieved by using the same pattern in curly braces like so:

{*.\d\d\d}

Incorrect Pattern Explanation

Another discrepancy in the rclone documentation is the explanation for some patterns. One example is the pattern *.jpe?g. The documentation states that this pattern will filter both jpg and jpeg files, but in reality, the pattern will filter files with the extension jpeXg, where X can be any character. To filter only jpg and jpeg files, use 2 patterns *.jpg and *.jpeg. This pattern will match files with either the jpg or jpeg extension.

Questions? Feature Requests? Other Feedback?

If there are any feature suggestions or requests, feel free to submit them here, and make sure to mention Migration App in the text.
For more complex requests that would benefit from providing screen shots or other attachments, submit them to support@egnyte.com. Be sure to include Migration App in the email title. Our Support team will push the comments to the Product team for consideration.