Using Filtering Patterns and Rules in Migration App

Audience

Any user of the Migration App.

Context

Migration App offers the ability to use filters to more granularly specify what will be migrated with a given Migration job. There are three types of file exclusions that may be specified.

Types of Exclusions

Exclusion by File Extension 

Migration Operators may specify file types to be excluded by entering the undesired extensions in a single text box. This is the simplest type of file exclusion and may be used by users with any experience level.

Glob Pattern-Based Exclusion

Migration Operators may also exclude files based on glob patterns. The UI provides a separate text box for each pattern entry, and these entries are combined into a string array and validated before forwarding them to the agent. If a pattern cannot be validated, the UI will display an error. Rclone offers detailed documentation, but also see the Limitations and How Migration App Differs sections below.

Regex Pattern-Based Exclusion

Users can also exclude files based on regex patterns. Similar to the glob pattern-based exclusion, the UI provides a separate text box for each entry which are then combined into a string array, validated and forwarded to the agent. If a pattern cannot be validated, the UI will display an error. For further information about regex patterns, rclone offers detailed documentation for regular expressions.

Limitations

1. Only PCRE/POSIX based regex/glob patterns (more specifically rclone recognized patterns) are supported.
2. There is currently no scrolling in the UI. If there are too many patterns, they cannot all be displayed.
3. For some seemingly valid patterns that are flagged as invalid, you may try putting them in curly braces ({ }) and then Save. 
4. We are validating a restrictive set of expressions which are supported by rclone. Furthermore we are only validating syntax. We are not confirming it will ultimately match and successfully exclude one or more files at the source.
5. For Glob and Regex expression based exclusion, a certain level of competence is assumed about the Migration Operator.

How Migration App Differs from Native Rclone Functionality

For those familiar with rclone, you will see many parallels in how we have enabled Migration App versus native rclone functionality. The purpose of this section is to educate users on the difference between Migration App and what is defined in rclone documentation. The filtering functionality can be incredibly useful to ensure that only specific files or directories are transferred. The rclone documentation provides basic guidance on how to use filters, but there are some discrepancies between what is mentioned in the documentation and how rclone actually accepts patterns. 

Nested Curly Braces

The rclone documentation includes some filter patterns that are nested within curly braces ({{}}). However, it's important to note that rclone does not accept nested curly braces. If you try to use a pattern with nested curly braces, you will receive an error message.

For example, the documentation includes the following pattern:

{{*.txt}}

However, to use this pattern with rclone, you need to remove the nested curly braces, so it becomes:

{*.txt}

Filtering Files with 3-Digit Extension

Rclone filtering documentation shows that the following pattern will filter files with extension with three digits:

*.\d\d\d

But in reality it is considered invalid. The same objective can be achieved by using the same pattern in curly braces like so:

{*.\d\d\d} 

Incorrect Pattern Explanation

Another discrepancy in the rclone documentation is the explanation for some patterns. One example is the pattern *.jpe?g. The documentation states that this pattern will filter both jpg and jpeg files, but in reality, the pattern will filter files with the extension jpeXg, where X can be any character. To filter only jpg and jpeg files, you need to use 2 patterns *.jpg and *.jpeg. This pattern will match files with either the jpg or jpeg extension.

Questions? Feature Requests? Other Feedback?

If you have feature requests or suggestions, feel free to submit them on our Product Board, and be sure to mention Migration App in the text.

For more complex requests that would benefit from providing screenshots or other attachments, you may submit them to support@egnyte.com. Be sure to include Migration App in the email title. Our Support team will push your comments to the Product team for consideration.