Regular Expressions

A regular expression is a special text string for describing a search pattern. You can think of regular expressions as wildcards on steroids. You are probably familiar with wildcard notations such as *.txt to find all text files in a file manager. The regular expression equivalent is .*\.txt$. As you can see it is more complicated to describe this simple case this way but regular expression are also far more powerful than simple globbing (the technical term for filesystem wildcards).

There are several variations of regular expressions used in the wild. The one Robo-FTP supports is Perl regular expressions. In Perl regular expressions, all characters match themselves except for the following special characters:

.[{}()\*+?|^$

Wildcards:

The single character '.' when used outside of a character set will match any single character except the newline character or null.

Anchors:

A '^' character shall match the start of a line.

A '$' character shall match the end of a line.

Marked Sub-Expressions:

A section beginning with ( and ending with ) acts as a marked sub-expression. Whatever matched the sub-expression is split out in a separate field by the matching algorithms. Marked sub-expressions can also be repeated, or referred to by a back-reference. In single line regex commands like IFREGEX and SETREPLACE the matched sub-expressions can be found in the %regex[n] variables where n is 1 to sub-expression count.

Non-Marking Group:

A marked sub-expression is useful to lexically group part of a regular expression, but has the side-effect of spitting out an extra field in the result. As an alternative you can lexically group part of a regular expression, without generating a marked sub-expression by using (?: and ) , for example (?:ab)+ will repeat ab without splitting out any separate sub-expressions.

Repeats:

Any atom (a single character, a marked sub-expression, or a character class) can be repeated with the *, +, ?, and {} operators.

The * operator will match the preceding atom zero or more times, for example the expression a*b will match:

"b" or "ab" or "aaaaaaaab"

The + operator will match the preceding atom one or more times, for example the expression a+b will match:

"ab" or "aaaaaaaab", but won't match: "b"

The ? operator will match the preceding atom zero or one times, for example the expression ca?b will match:

"cb" or "cab", but will not match "caab"

An atom can also be repeated with a bounded repeat:

a{n} Matches 'a' repeated exactly n times.

a{n,} Matches 'a' repeated n or more times.

a{n, m} Matches 'a' repeated between n and m times inclusive.

For example:

"^a{2,3}$" Will match either "aa" or "aaa" but neither of "a" or "aaaa"

Note that the "{" and "}" characters will be treated as ordinary literals when used in a context that is not a repeat: this matches Perl 5.x behavior. For example in the expressions "ab{1", "ab1}" and "a{b}c" the curly brackets are all treated as literals and no error will be raised.

It is an error to use a repeat operator, if the preceding construct can not be repeated, for example:

a(*) Will raise an error, as there is nothing for the * operator to be applied to.

Non-Greedy Repeats:

The normal repeat operators are "greedy", that is to say they will consume as much input as possible. There are non-greedy versions available that will consume as little input as possible while still producing a match.

*? Matches the previous atom zero or more times, while consuming as little input as possible.

+? Matches the previous atom one or more times, while consuming as little input as possible.

?? Matches the previous atom zero or one times, while consuming as little input as possible.

{n,}? Matches the previous atom n or more times, while consuming as little input as possible.

{n,m}? Matches the previous atom between n and m times, while consuming as little input as possible.

Possesive Repeats:

By default when a repeated pattern does not match then the engine will backtrack until a match is found. However, this behaviour can sometime be undesireble so there are also "possessive" repeats: these match as much as possible and do not then allow backtracking if the rest of the expression fails to match.

*+ Matches the previous atom zero or more times, while giving nothing back.

++ Matches the previous atom one or more times, while giving nothing back.

?+ Matches the previous atom zero or one times, while giving nothing back.

{n,}+ Matches the previous atom n or more times, while giving nothing back.

{n,m}+ Matches the previous atom between n and m times, while giving nothing back.

Back References:

An escape character followed by a digit n, where n is in the range 1-9, matches the same string that was matched by sub-expression n. For example the expression:

"^(a*).*\1$" Will match the string: "aaabbaaa" But not the string: "aaabba"

Back references can also be used in the replace string when using SETREPLACE with the /regex option.

You can also use the \g escape for the same function, for example:

Finally the \k escape can be used to refer to named subexpressions, for example \k<two> will match whatever matched the subexpression named "two".

Alternation:

The | operator will match either of its arguments, so for example: abc|def will match either "abc" or "def".

Parenthesis can be used to group alternations, for example: ab(d|ef) will match either of "abd" or "abef".

Empty alternatives are not allowed (these are almost always a mistake), but if you really want an empty alternative use (?:) as a placeholder, for example:

"|abc" is not a valid expression, but" (?:)|abc" is and is equivalent, also the expression: "(?:abc)??" has exactly the same effect.

Character Sets:

A character set is a bracket-expression starting with [] and ending with , it defines a set of characters, and matches any single character that is a member of that set.

A bracket expression may contain any combination of the following:

matches a \0 character.

Equivalence Classes:

An expression of the form [[=col=]], matches any character or collating element whose primary sort key is the same as that for collating element col, as with collating elements the name col may be a Symbolic Name. A primary sort key is one that ignores case, accentation, or locale-specific tailorings; so for example [[=a=]] matches any of the characters: a, À, Á, Â, Ã, Ä, Å, A, à, á, â, ã, ä and å. Unfortunately implementation of this is reliant on the platform's collation and localisation support; this feature can not be relied upon to work portably across all platforms, or even all locales on one platform.

Escaped Characters:

All the escape sequences that match a single character, or a single character class are permitted within a character class definition. For example [\[\]] would match either of [ or ] while [\W\d] would match any character that is either a "digit", or is not a "word" character.

Combinations:

All of the above can be combined in one character set declaration, for example: [[:digit:]a-c[.NUL.]].

Escapes:

Any special character preceded by an escape shall match itself.

The following escape sequences are all synonyms for single characters:

"Single Character" character classes:

Any escaped character x, if x is the name of a character class shall match any character that is a member of that class, and any escaped character X, if x is the name of a character class, shall match any character not in that class.

The following are supported by default:

Character Properties:

The character property names in the following table are all equivalent to the names used in character classes.

For example \pd matches any "digit" character, as does \p{digit}.

Word Boundaries:

The following escape sequences match the boundaries of words:

< Matches the start of a word.

> Matches the end of a word.

\b Matches a word boundary (the start or end of a word).

\B Matches only when not at a word boundary.

Buffer Boundaries:

The following match only at buffer boundaries: a "buffer" in this context is the whole of the input text that is being matched against (note that ^ and $ may match embedded newlines within the text).

\` Matches at the start of a buffer only.

\' Matches at the end of a buffer only.

\A Matches at the start of a buffer only (the same as \`).

\z Matches at the end of a buffer only (the same as \').

\Z Matches a zero-width assertion consisting of an optional sequence of newlines at the end of a buffer: equivalent to the regular expression (?=\v*\z). Note that this is subtly different from Perl which behaves as if matching (?=\n?\z).

Continuation Escape:

The sequence \G matches only at the end of the last match found, or at the start of the text being matched if no previous match was found. This escape useful if you're iterating over the matches contained within a text, and you want each subsequence match to start where the last one ended.

Quoting Escape:

The escape sequence \Q begins a "quoted sequence": all the subsequent characters are treated as literals, until either the end of the regular expression or \E is found. For example the expression: \Q*+\Ea+ would match either of:

\*+a

\*+aaa

Unicode Escapes:

\C Matches a single code point, this has exactly the same effect as a the "." operator. \X Matches a combining character sequence: that is any non-combining character followed by a sequence of zero or more combining characters.

Matching Line Endings:

The escape sequence \R matches any line ending character sequence, specifically it is identical to the expression (?>\x0D\x0A?|[\x0A-\x0C\x85\x{2028}\x{2029}]).

Keeping Back Some Text:

\K Resets the start location of $0 to the current text position: in other words everything to the left of \K is "kept back" and does not form part of the regular expression match. $` is updated accordingly.

For example foo\Kbar matched against the text "foobar" would return the match "bar" for $0 and "foo" for $`. This can be used to simulate variable width lookbehind assertions.

Any Other Escape:

Any other escape sequence matches the character that is escaped, for example \@ matches a literal '@'.

Perl Extended Patterns:

Perl-specific extensions to the regular expression syntax all start with (?.

Named Subexpressions:

You can create a named subexpression using:

(?<NAME>expression)

Which can be then be referred to by the name NAME. Alternatively you can delimit the name using 'NAME' as in:

(?'NAME'expression)

These named subexpressions can be referred to in a backreference using either \g{NAME} or \k<NAME>.

Comments:

(?# ... ) is treated as a comment, it's contents are ignored.

Modifiers:

(?imsx-imsx ... ) alters which of the perl modifiers are in effect within the pattern, changes take effect from the point that the block is first seen and extend to any enclosing ). Letters before a '-' turn that perl modifier on, letters afterward, turn it off.

(?imsx-imsx:pattern) applies the specified modifiers to pattern only.

Non-Markign Groups:

(?:pattern) lexically groups pattern, without generating an additional sub-expression.

Branch Reset:

(?|pattern) resets the subexpression count at the start of each "|" alternative within pattern.

The sub-expression count following this construct is that of whichever branch had the largest number of sub-expressions. This construct is useful when you want to capture one of a number of alternative matches in a single sub-expression index.

In the following example the index of each sub-expression is shown below the expression:

# before  ---------------branch-reset----------- after

/ ( a )  (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x

#  1             2         2  3       2     3      4

Lookahead:

(?=pattern) consumes zero characters, only if pattern matches.

(?!pattern) consumes zero characters, only if pattern does not match.

Lookahead is typically used to create the logical AND of two regular expressions, for example if a password must contain a lower case letter, an upper case letter, a punctuation symbol, and be at least 6 characters long, then the expression:

(?=.*[[:lower:]])(?=.*[[:upper:]])(?=.*[[:punct:]]).{6,}

could be used to validate the password.

Lookbehind:

(?<=pattern) consumes zero characters, only if pattern could be matched against the characters preceding the current position (pattern must be of fixed length).

(?<!pattern) consumes zero characters, only if pattern could not be matched against the characters preceding the current position (pattern must be of fixed length).

Independent sub-expressions:

(?>pattern) pattern is matched independently of the surrounding patterns, the expression will never backtrack into pattern. Independent sub-expressions are typically used to improve performance; only the best possible match for pattern will be considered, if this doesn't allow the expression as a whole to match then no match is found at all.

Recursive Expressions:

(?N) (?-N) (?+N) (?R) (?0) (?&NAME)

(?R) and (?0) recurse to the start of the entire pattern.

(?N) executes sub-expression N recursively, for example (?2) will recurse to sub-expression 2.

(?-N) and (?+N) are relative recursions, so for example (?-1) recurses to the last sub-expression to be declared, and (?+1) recurses to the next sub-expression to be declared.

(?&NAME) recurses to named sub-expression NAME.

Conditional Expressions:

These take the form of (?(condition)yes-pattern[|no-pattern]) which attempts to match yes-pattern if the condition is true, otherwise attempts to match no-pattern or NULL if no-pattern is absent.

condition may be either: a forward lookahead assert, the index of a marked sub-expression (the condition becomes true if the sub-expression has been matched), or an index of a recursion (the condition become true if we are executing directly inside the specified recursion).

Here is a summary of the possible predicates:

Backtracking Control Verbs:

Robo-FTP has partial support for Perl's backtracking control verbs, in particular (*MARK) is not supported. There may also be detail differences in behavior between this library and Perl, not least because Perl's behavior is rather under-documented and often somewhat random in how it behaves in practice. The verbs supported are:

Operator Precedence:

The order of precedence for of operators is as follows:

What Gets Matched:

If you view the regular expression as a directed (possibly cyclic) graph, then the best match found is the first match found by a depth-first-search performed on that graph, while matching the input text.

Alternatively:

The best match found is the leftmost match, with individual elements matched as follows;