Welcome to
Help Desk

Product Updates
Training
Support
Ideas Contact Support

Advanced Mode in Migration App

Follow

Audience

Advanced users of the Migration App.

Context

Several features are available only if Advanced Mode is enabled for your domain. These features require more technical knowledge and are recommended for use only by experienced users of Migration App.

Enabling Advanced Mode

Before you can access the Advanced Mode features in Migration App, you must have Advanced Mode enabled for your domain. To do this, submit a ticket to support@egnyte.com to request a Settings Update. This process can take up to two full business days to be implemented.

Features Available in Advanced Mode

Controlling Threads

By default, the Agent running on the local Windows machine uses 4 transfers and 8 checkers (collectively referred to as the threads). Thread is a universal computing term, referring to one process in a multithreaded system that occurs simultaneously with other processes. "Checkers" and "transfers" in this context are the custom names of different threads implemented in the underlying technology. "Checkers" determine how parallel a given operation will run (e.g., when deleting files, how many deletes to do in parallel). "Transfers" control uploading (streaming) of files to the destination.

If you have Advanced Mode enabled for your domain, you can increase the number of threads and potentially improve your migration speed, assuming your Windows machine has enough compute capacity to accommodate the increase. This must be undertaken with caution, since allocating too many threads to Migration App will have a negative impact on the host machine’s compute capabilities.

Configuring Threads

Go to the Migration Details page. In the left column, click on the arrow next to Advanced Options to open the drop-down menu.

Migration App_Advanced Mode_1.png

If a migration job is currently running a command, you may not modify any of the Advanced Options. You must wait until the command has completed before configuring advanced settings for the job.

Migration App_Advanced Mode_7.png

Select Threads from the left menu.

Here, you can decrease or increase the number of transfers (from 1 to 10, inclusive) and checkers (from 1 to 20, inclusive) with a custom configuration. This can increase the speed of your migration job, assuming your Windows host machine has the compute power to accommodate the increase.

The recommended ratio for any configuration is 2x Checkers per Transfer.

Migration App_Advanced Mode_2.png

You may return to this menu at any time to change your selection. Remember that before you can make any changes, the migration job cannot have a command running.

Your threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.

Migration App_Advanced Mode_5.png

Controlling Bandwidth Throughput

For any migration job, you have the option to configure various advanced settings, including Empty Folders, Sanitization, Unattended Mode, Confluence. These are all available from the Migration Details Page. If you have Advanced Mode enabled for your domain, you will also have the ability to control Throughput from the same Advanced Options menu.

Configuring Throughput

Go to the Migration Details page. In the left column, click on the arrow next to Advanced Options to open the drop-down menu.

Migration App_Advanced Mode_1.png

If a migration job is currently running a command, you may not modify any of the Advanced Options. You must wait until the command has completed before configuring advanced settings for the job.

Migration App_Advanced Mode_7.png

Select Throughput from the left menu.

Migration App_Advanced Mode_6.png

Select ON for Enable Bandwidth Controls and you will be presented with another drop-down menu.

Migration App_Advanced Mode_9.png

You have two options for controlling bandwidth: 1Mb/second at all times or 1Mb/second during business hours. Make your selection and click Save.

You may return to this menu at any time to change your selection. Remember that before you can make any changes, the migration job cannot have a command running.

Your threshold for this job is displayed in the right column of the Migration Details page for each job, under Other Details.

Migration App_Advanced Mode_5.png

Customizing the Agent Configuration

Advanced Options may be customized directly from the User Interface for each individual migration job. But this can be tedious and time consuming if you have multiple jobs tied to a single Agent and want them all to have the same settings. Using a Custom Config file, you may control the settings at the Agent level, which will then be applied to every job created from that Agent. You can also use the Custom Config to address individual job exceptions

Agent Settings

From the Migration Dashboard, click Settings to navigate to the Settings page.

Migration App_Advanced Mode_12.png

From the Settings page, select the Agent you want to customize.

Migration App_Advanced Mode_10.png

This opens the settings panel for that Agent.

Migration App_Advanced Mode_11.png

Click on Download Default Configuration File.

Open the Configuration.json file in a text or code editor like Atom or Notepad++.

The default configuration file looks like this:

Migration App_Advanced Mode_13.png

Modifying the Agent Configuration

Edit the configuration file using JSON syntax.

  1. Specify a configName within the empty brackets. In the example below, it has been specified as LightConfig.
  2. The first section controls the Agent. You can change Transfers and Checkers, but this will change it for all jobs associated with this Agent, so you need to be mindful of how much actual compute is available on the host machine.
  3. Anything you specify under the “config” parameter will apply to all jobs associated with this agent.

In the example below, only the Agent configuration has been changed. It keeps the Agent at four Transfers, eight Checkers, but now Confluence and Symbolic Links are both enabled.

Migration App_Advanced Mode_14.png

Once you have the configuration you want, upload the custom config file to Migration App by dragging and dropping it on the Settings page. Any future migration operations associated with this agent will reflect this custom configuration.

Migration App_Advanced Mode_15.png

 Modifying Individual Migration Job Configurations

You can also specify unique settings for individual jobs – or exceptions to the Agent settings – by calling them out under “migrations”.

  1. Replace the “null” reference with brackets and follow the syntax provided below.
  2. For “name”, enter the exact migration name
  3. For “id”, copy the Migration Job ID from the Migration Details page.
  4. Specify the unique exceptions for that job for Transfers, Checkers, Confluence and/or Symbolic Links.

In the example below, Job A and Job B have unique configurations, but any other jobs associated with Agent Vitality will follow the configuration specified for the Agent.

A screen shot of a computer program

Description automatically generated

Once you have the configuration you want, upload the custom config file to Migration App by dragging and dropping it on the Settings page. Any future migration operations associated with this agent and Jobs A and B will reflect this custom configuration.

A screenshot of a computer

Description automatically generated

Creating New Migration Jobs Using the Custom Config File

If you want to initiate new migrations for a specific agent using a Custom Config file, see the example below.

A screenshot of a computer screen

Description automatically generated

JSON Format:

  • The field "id" under "migrations": brackets should be left empty for creating a new migration job. The App will assign a Job ID as part of initiation.
  • Fields "transfers", "checkers", "src-path" and "dst-path" are mandatory.
  • If the destination path is the same for multiple jobs within the same domain, set "confluence" to true.

Important Considerations:

  1. Creating multiple jobs per agent using the Custom Config file can slow down your migrations. The jobs will share CPU/memory because they will be running in parallel. We don’t recommend running more than three or four parallel migration jobs per agent.
  2. The feasibility of simultaneous job creation depends on the size of the migrations:
    1. For small to medium-sized migrations, creating up to 4 simultaneous jobs is acceptable.

    2. Creating more than 4 jobs simultaneously is strongly discouraged as it can lead to significant system strain. Upon successful upload of the Custom Config, it initiates multiple processes for creating migrations. Each successful migration creation triggers additional processes for executing scan commands. This can considerably overload memory and CPU on the host machine.

  3. Migration App currently can’t run jobs in sequence, although it is on our feature roadmap.
  4. Sync Deletes will not work with Confluence enabled.

Using Exclusion Rules

Migration App offers the ability to use filters to granularly specify what will be migrated with a given Migration job.

Types Of Exclusions

If Advanced Mode is not enabled, only the Exclusion by File Extension option will be visible in the Advanced Options menu.

Exclusion By File Extension

Migration Operators may specify file types to be excluded by entering the undesired extensions in a single text box. This is the simplest type of file exclusion and may be used by users with any experience level. Accordingly, Advanced Mode is not required for this level of exclusion.

blobid0.png

Once Advanced Mode is enabled, two additional Exclusion Rules options will be available.

Glob Pattern-Based Exclusion

Migration Operators may also exclude files based on glob patterns. The UI provides a separate text box for each pattern entry, and these entries are combined into a string array and validated before forwarding them to the agent. If a pattern cannot be validated, the UI will display an error. Rclone offers detailed documentation, but also see the Limitations and How Migration App Differs sections below.

Regex Pattern-Based Exclusion

Users can also exclude files based on regex patterns. Similar to the glob pattern-based exclusion, the UI provides a separate text box for each entry which is then combined into a string array, validated, and forwarded to the agent. If a pattern cannot be validated, the UI will display an error. For further information about regex patterns, rclone offers detailed documentation for regular expressions.

blobid1.png

Limitations

  • If your exclusion rules exclude all items at the source, the scan will fail and the job will be aborted. If your exclusion rules exclude all items within only one or more folders, the job will run, but those folders will fail to be created on the destination during migration.
  • Only PCRE/POSIX based regex/glob patterns (more specifically rclone recognized patterns) are supported.
  • There is currently no scrolling in the UI. If there are too many patterns, they cannot all be displayed.
  • For some seemingly valid patterns that are flagged as invalid, you may try putting them in curly braces ({ }) and then Save.
  • We are validating a restrictive set of expressions which are supported by rclone. Furthermore, we are only validating syntax. We are not confirming it will ultimately match and successfully exclude one or more files at the source.
  • For Glob and Regex expression-based exclusion, a certain level of competence is assumed about the Migration Operator.
  • If you have a folder on the source that contains only unsupported files, none of these files will be migrated. As a result, this folder will not be created on the destination. Moreover, as this folder is not empty on the source, it will not be created as part of the empty folder migration step. So this folder will always be missing on the destination. There is no way presently to migrate it. The only solution is to put something inside that folder that can be migrated and run a True-Up. It will then appear on the destination containing only the one item that is supported.

How Migration App Differs From Native Rclone Functionality

For those familiar with rclone, you will see many parallels in how we have enabled Migration App versus native rclone functionality. The purpose of this section is to educate users on the difference between Migration App and what is defined in rclone documentation. The filtering functionality can be incredibly useful to ensure that only specific files or directories are transferred. The rclone documentation provides basic guidance on how to use filters, but there are some discrepancies between what is mentioned in the documentation and how rclone actually accepts patterns.

Nested Curly Braces

The rclone documentation includes some filter patterns that are nested within curly braces ({{}}). However, it's important to note that rclone does not accept nested curly braces. If you try to use a pattern with nested curly braces, you will receive an error message.

For example, the documentation includes the following pattern:

{{*.txt}}

However, to use this pattern with rclone, you need to remove the nested curly braces, so it becomes:

{*.txt}

Filtering Files With 3-Digit Extension

Rclone filtering documentation shows that the following pattern will filter files with extension with three digits:

*.\d\d\d

But in reality it is considered invalid. The same objective can be achieved by using the same pattern in curly braces like so:

{*.\d\d\d}

Incorrect Pattern Explanation

Another discrepancy in the rclone documentation is the explanation for some patterns. One example is the pattern *.jpe?g. The documentation states that this pattern will filter both jpg and jpeg files, but in reality, the pattern will filter files with the extension jpeXg, where X can be any character. To filter only jpg and jpeg files, you need to use 2 patterns *.jpg and *.jpeg. This pattern will match files with either the jpg or jpeg extension.

Questions? Feature Requests? Other Feedback?

If you have feature suggestions or requests, feel free to submit them here, and make sure you mention Migration App in the text.
For more complex requests that would benefit from providing screen shots or other attachments, you may submit them to support@egnyte.com. Be sure to include Migration App in the email title. Our Support team will push your comments to the Product team for consideration.

Was this article helpful?
0 out of 0 found this helpful

For technical assistance, please contact us.