ITEC dept promo banner

regular expressions

Unwieldy video from distance lecture, 2017-feb-21 (1h35m)
but you can watch just [0:24:40,1:33:00) (1h08m) about regular-expressions.
(You can skip over the first 25min (opening pause [0:00:00,0:00:30), some ERD review [0:00:30,0:04:30), git-review [0:04:30,0:22:00), and a few comments about projects-with-external-clients [0:22:00,0:24:40)), as well as the final few "shutting down" minutes at [1:33:00,1:35:00).)

regular expression intro

Representing regular expressions

Regular expressions are not the same type as strings ². Various languages have slightly different ways to represent regexps. video (14m23s)

For example:

Java: class Pattern represents regexps (e.g. new Pattern("ab*c+?d")).
As a convenience, many functions taking in regexps are also overloaded to take in a string, and it calls the Pattern constructor with that string for you.
The thing to be careful about is regexps that contain a backslash: To get the pattern a\sb, your Java would use have the string-literal "a\\sb", because to get a backslash into any string-literal requires second backslash.

This sometimes leads to backslash hell: If a regexp wants to match a backslash, the regexp will contain \\, and the java-string-to-represent-that-regexp would be written with \\\\.
Even where we don't have this situation, regexps that use a lot of backslashes end up looking even less readable as Java strings.
Javascript: the concept of regular-expression literals are built-in to the language! Instead of quote-marks, surround your literal regexps with a slash: var pat = /a\sb/.
This solution nice, because it emphasizes the difference between strings and regexps at the type level (which we like), without needing convenience functions which immediately obscure the difference and erode the helpfulness of the type-checker.

Note that internally, there is still a class Regexp, which these literals are instances of. In addition, Javascript also allows for modifying the meaning of a regexp by adding one of a few possible suffixes; e.g. /a\sb/i is a case-independent regexp.
python: There aren't regexp-literals built in, but even better: python has “raw-string literals, which can be used in many situations, including being terribly convenient for writing regular expressions: the raw-string-literal r"a\sb" is equivalent to writing "a\\sb". WIN!
PHP: Since PHP does have single-quotes for raw-string-literals, so you'd think it'd be cool like python. But not quite (of course): For PHP, the regular-expression libraries expect strings which begin and end with “/”³! So we'd use '/a\sb/' (or equivalently, "/a\\sb/").

They're the fine idea of raw-strings from python, but (for regexps) they then tack on a superfluous javascript-but-not requirement that detracts from its usefulness. phpsadness.com.
Racket has #px as a syntax for regular-expression literals, but (unlike javascript literals) those literals still require escaping a backslash. Thus #px"a\\sb" is shorthand for calling the constructor (pregexp "a\\sb"), and we can still suffer from backslash hell.
HTML (okay, it's more a data-representation format than a programming language, but… next week we'll talk about the pattern attribute for input tags):
There are no issues with backslashes in regexps-in-HTML, because backslashes aren't anything special in HTML. So something like <input type='text' pattern='a\sb'> is fine. The only character to wonder about getting inside regexps is a quote-mark itself. Since you can use either double- or single-quotes to delimit attribute values, it's usually straightforward: if you want " inside your attribute, use '…' to delimit your attribute, and vice versa. But what if you want both characters in your regexps? In that case, you can fall back to the known HTML way: use the entity " or '.

As a final annoyance to polyglots, languages differ on what they do with “illegal” character-escapes. For example, given "a\zb" as a string-literal in code:

Java throws an exception (illegal character-escape) (yay!)
javascript treats it as "azb" (as in “any character can be escaped; most characters escape to themselves”) (hmm)
PHP treats it as the four-character string "a\zb" (le sigh).

(And that leaves me further wondering: what do the regexp library implementations inside each of these languages choose to do with illegal escape-characters at the regexp level, after having converted strings into regexps. I would hope they throw an exception, rather than take either of the latter two approaches above.)

PHP regular expressions

PHP regular-expression matching: In php, regular expressions are strings delimited by a special character (usually /). Some examples; see playing with regexps in PHP for details.

      echo preg_match( '/abcd/', 'abcd' );
      echo preg_match( '/abcd/', 'azcd' );      // false

      echo preg_match( '/a..d/', 'azcd' );      // . matches any single character (besides newline, null)

      echo preg_match( '/ab*cd/', 'abbbbcd' );  // b*  matches 0-or-more-b's.
      echo preg_match( '/ab*cd/', 'abcd' );  
      echo preg_match( '/ab*cd/', 'acd'  ); 

      echo preg_match( '/ab+cd/', 'abbbbcd' );  // b+  matches 1-or-more-b's.
      echo preg_match( '/ab+cd/', 'abcd' );  
      echo preg_match( '/ab+cd/', 'acd'  );     // false

      echo preg_match( '/ab.*cd/', 'abcd' );
      echo preg_match( '/ab.*cd/', 'abXd' );               // .* and .+ are a common patterns.
      echo preg_match( '/ab.*cd/', 'abBlahBlahBlahcd' );
      echo preg_match( '/ab.+cd/', 'abcd' );               // false
      echo preg_match( '/ab.+cd/', 'abXcd' );

      
      echo preg_match( '/ab?cd/', 'acd' );  // b?  matches 0-or-1 b
      echo preg_match( '/ab?cd/', 'abcd' );  
      echo preg_match( '/ab?cd/', 'abbbbcd'  );     // false

      echo preg_match( '/(ab)*cd/', 'ababababcd' );  // parens do grouping
      echo preg_match( '/(ab)*cd/', 'abbbbcd' );  // false


      // WARNING: preg_match looks to see if the string *contains* a match!
      echo preg_match( '/row/', 'How now, brown cow?' );  // true
      echo preg_match( '/.ow/', 'How now, brown cow?' );  // true

      // Use "^" to specify the start-of-string, and "$" to specify end-of-string.
      echo preg_match( '/^.ow$/', 'How now, brown cow?' );  // false
      echo preg_match( '/^.ow$/', 'Zow' );
      echo preg_match( '/^.ow/', 'Zowee' );
      echo preg_match( '/w.e$/', 'Wowee Zowee' );

      
      echo preg_match( '/[WZ]ow/', 'Wowee Zowee' );  // square-brackets match any one character from the set
      echo preg_match( '/[WZ]ow/', 'Yow' );          // false
      echo preg_match( '/[W-Z]ow/', 'Yowee' );       // square-brackets can contain a *range*
      echo preg_match( '/ab[0-9]+de/', 'ab789de' );


      echo preg_match( '/[0-9]*/', '00047' );        // Beware: matching just a * expression (w/o ^,$)!

See the manual.
Note: the name “preg” comes from "Perl compatible"; earlier PHP used the POSIX regexp's but PHP decided to deprecate that.

Your task: What is a regular expression to match...

The name "Jon"?
The name "Jon" or "John"?
Any name starting with "Jon"?
Any name starting with: "Jon" or "John"?
Any name starting with "Jon"; or, the name "John"?
Any name containing "Jon"?
The name "Jon" or "John" or "Jonathon" or "Jonathan"?
(For the following: we'll take “names” to mean containing only the 26 unaccented roman letters, and capitalized as in western culture ⁴).
A hyphenated name.
A possibly-hyphenated name.
A doubly-hyphenated name.
An exactly-doubly-hyphenated name.
An at-most-doubly-hyphenated name.

or an underscore (?!)

A word starting with a lower-case letter.
A word not starting with a capital letter.
A word ending with a digit.
A word ending with a roman numeral.

False positive, and false negative:

		test result
		negative	positive
is test accurate?	false	false negative	false positive
is test accurate?	true	true negative	true positive

In our setting, the “test” is preg_match, and “accurate” means whether the value returned is what we want it to be.

Warning: Beware matching a top-level * expression: the empty-string matches it, and any string contains the empty-string! Thus preg_match_all( "/(xyz)*/", "uh-oh") === 6!!, since "uh-oh" has zero "xyz"'s at the start, followed by 'u', followed by zero more "xyz"'s, followed by 'h', followed by ….

Atomic regexps

^ : matches start-of-string
$ : matches end-of-string
. : matches any one character
Most characters match themselves (e.g. 'b'); you can escape special characters with backslash.
Character classes: We'll tend to prefer the unicode properties, since the other two don't account for characters like “é”. See www.regular-expressions.info/posixbrackets.html.
- \p{L} is better than [[:alpha:]] is better than \w.
- \p{Lu} is better than [[:upper:]] is better than [A-Z].
- \p{Ll} is better than [[:lower:]] is better than [a-z].
- For digits, [0-9] or [\d] (or [[:digit:]]) is arguably okay; use \p{N} if you want to handle any digit-ish character like or “𝟜” (double-struck script) or “٤” (arabic 4) or “⁴” (superscript). You can use \p{Nd} to exclude the last (but not the former two).
  bug?: preg_match_all( '/\p{N}/', "⁴٤𝟜4" ) is returning 1 for me (not 4), in php 5.4.3.
- \p{P} is better than [[:punct:]] is the same as [!.?:;"'].
- […] — build your own character set, e.g. [aeiou] matches any one lower-case vowel.
  Use ^ just inside the [ to negate the set. (Note that this overrides the beginning-of-string meaning of ^; it's clear from context).
- Checking for space (one of the most common situations) is the one that is hardest: We have \p{Z}, as well as [[:space:]] and \s. The first, unicode-property version checks for separating space characters (thin-space, etc), but doesn't catch tabs (or newlines). On the other hand, \s does catch tabs, newlines etc., but doesn't catch thin-space or nonbreaking-space or other space-looking characters.
  
  Also, we are probably least able to ignore this problem, because in web forms people might paste in text from web pages or Word documents, and those exper-typesetting programs do tend to use the various special-spaces.
  
  Solution: Either
  - use the pattern [\p{Z}\n\t\r\v\f] — long to type and read, and easy to get wrong; or
  - $str2 = preg_replace( '/\p{Z}+/', ' ', $str ); … if( preg_match(/\s+/',$str2)) …
Remember with the character-sets: Each of these match one single character. They are often used in conjunction with * and + below.

Compound regexps

There are also ways of building bigger regexps out of smaller ones:

(α) — same as α; used to dis-ambiguate precedence.
α* — 0 or more occurrences of α
α+ — 1 or more occurrences of α
α? — 0 or 1 occurrence of α
α{…} — you specify a range of #repetitions of α (not very common)
α|β — α OR β. This is the one low-priority regexp operator: ab|cd is interpreted as (ab)|(cd), not a(b|c)d. Compare that with any of the others: ab*c is interpreted as a(b*)c, not (ab)*c.

NOTE: In php, no trailing “g” allowed, as in javascript!
(There are other trailing modifiers however — e.g. i for case-insensitive, and more.)

Three helpful functions:

preg_match($pat,$target) (also, note: preg_match_all — not as common, but it motivates why preg_match's output is 0,1 and not a raw boolean).
Warning: preg_match_all("/x/","abc",$results) will always have $results be non-empty — it will be an array containing an empty array. (In particular, the condition of if ($results) … will always be true.) You can instead use preg_match_all("/x/","abc",$results,PREG_SET_ORDER) which will have zero elements for zero matches (and if ($results) … will behave as you want). Each element of $results will still be an another entire array though -- the full match plus any sub-patterns.
preg_replace($pat,$target,$str)
split($pat,$target) Returns an array (indexed numerically from 0)
trim — nothing to do with regular expressions, but a function you probably want to call on most user-entered text.

regexps vs. unicode

In unicode, we separate the notion of a byte from that of a character. (The term “code point” is sometimes used to emphasize unicode-character-that-might-be-more-than-a-char.)
In unicode, we have more generalized notions of letters: not just A-Z, but also accented characters like “é”, chinese characters, etc.. Also, whitespace is more complicated: There is a regular ol' space, thin whitespace  , and more.
Use “properties” — generalized character classes, when supported by your language:
- \p{Letter} or \p{L} — any letter, any language.
- \p{Separator} or \p{Z} — space-characters, including thin-space etc.. But it does not include newlines and tabs (!). Using \p{Z}|[\t\r\n\v\f] is better than \s though, since people might paste in whitespace on input forms. (Or, do a regexp-replace of \p{Z} to , and then use \s.)
- \p{Punctuation} or \p{P}
There are more refined versions of these properties, too (e.g. \p{Ll} or \p{Lowercase_Letter}).
Corner case: If you want to compare strings case-independently, use a library function that is unicode-aware. Note that the common trick of converting both strings to (say) uppercase doesn't always work in unicode, since (e.g.) “ß” has no (traditional) upper-case version. Moreover, for comparing German strings, “ß” should be considered equivalent to “ss”.
Corner case: Beware that characters like “é” can actually be encoded either as a single character (and thus match “.”), or as a regular ol' “e” combined with the accent “́” (and thus match, say, “..”)! See this page for advice on how to write regexps to try and cope with this.

² C, ML, and Haskell let you introduce new type-synonyms, but not Java -- you have to introduce a new class that wraps strings. ↩

¹ Btw, I'd love to have a language that lets me rename types ², so that I could have string (for raw-data), html-data (safe to concatentate to HTML), and sql-data (safe to splice into a SQL query as data). htmlspecialchars and mysql_real_escape_string can be thought of as constructors for these new types, but the type-system doesn't help protect me from giving a raw (unsanitized) string when it was expecting some already-sanitized data. ↩

³ Actually, they can begin/end with pretty much any punctuation character, or even matching parenthesis-like characters, so that's nice. But use / unless you have reason not to, since that's the standard/expected practice. ↩

⁴ Okay, that's actually vague: is “de Soto” or “da Vinci” all last name? Names are notoriously hard to characterize, especially across multiple cultures. My advice is to liberally accept what characters people say their name is; trimming and collapsing whitespace is about all I'd do. ↩

This page licensed CC-BY 4.0 Ian Barland
Page last generated 2018.Mar.12 (Mon) Please mail any suggestions
(incl. typos, broken links)
to ibarlandradford.edu