ITEC 325 2018fall ibarland

home—lects—hws
D2L—breeze (snow day)

regular expressions

intro

Review git usage; who ran into conflicts? Auto-resolved? Glance quickly at the git submits; obeserve: tabs can ruin indentation for others; do indent meaningfully! Poor indentation would be cause an interviewer to dismiss somebody immediately.

prof: Use colors effectively: Say, patterns in green, strings in black, and explaining-meaning-of-regexp-chars in blue?

• Say what you want about the William Shatner, the original Captain Kirk; that man can act!
  STII: The Wrath of KHAN (make sure volume is on)
  Q: Just how do you spell that?
• In 80s comic-strips, most animal and people were drawn cutesy-wootsy. Bloom County intentionally broke that mold, especially when introducing Bill the Cat: Although here his raspberry “thbbft” is fairly short, I imagine it was often longer: “thbbbbft” or “thbbbbbbbbbbbbbbft”.
• We'll say a pattern can match certain strings: the pattern kha*n can match "khan", "khaaaan", and "khaaaaaaaaaaaaaaan". It can also match "khn".
• The pattern “*” means “the preceding pattern can be repeated 0-or-more times”.
• Q: What is a pattern that will match things like “phbbbbbbbbft”, there there can be 3-or-more “b”s?
  A: phbbbb*ft
• The concept “1 or more of the preceding” is already derivable from “*”, but it's common enough that it does have its own special character: “+”.

• A . ("dot") means any one character.

  So if My favorite words include "bot", "bat", "blt", and even "b4t" and "b%t" and any letter between a b and a t, then b.t matches the things I like.

  
  Well, actually . matches any character except for a newline. So if a file contains a line ending in "b" and the next line starting in "t", it won't match. Fair enough.

• Hey Barland — Why regular expressions? I mean, when do you use these things?

  • In programs: verify that a string has a certain structure. (e.g. certain URLs might need to start with "http://" or "https://", then at least one "." in the , and then at least two later "/"s separated by letters.)

    In fact, we'll use regular expressions to validate data. E.g. a text-field for credit-card-numbers might need to be in a certain format.

  • scripts that process filenames or semi-regular data. (e.g. filter names that start with "hw" and end in "pdf" and contain some digit somewhere inbetweeen".)
  • A better search-replace, in my text editor.
    E.g. an apostrophe between two letters: When editing a web-page, I want to search-replace straight-apostrophes with curly-apostrophes. But I don’t want to replace html-attribute-values etc..
  • Grepping through text files: e.g. egrep " num.* =" manySources.java
  • shell commands? Well, shells have a different and primitive version they call “globbing”. (It actually pre-dates modern regexp libraries.) E.g. ls *.pdf will list all pdf files; ls hw*html will match filenames like hw03.html and hw02-soln.html. Things to keep in mind about shell globbing:
    • it only matches filenames;
    • it uses the one-character * instead of the two-character .* to mean "0 or more of anything".
    • It's performed by the shell, not be each program: ls *.pdf and echo *.pdf are nearly identical!
  Okay, let's get back to more regexps now.

• A “|” means "or".

  So Eric|Erik|Eriq can be useful, if I'm search for any of those variants in a file.

• You want to get rid of the duplicate "code" in the above regexp, don't you! Happily, we can: Eri(c|k|q).
• ( and ) do grouping, as the previous example shows.

• Say we want to have a regexp matching strings with hi occurring mid-word, in between any two other (lower-case) vowels. We can do this with what we've already seen: (a|e|i|o|u)hi(a|e|i|o|u)

  Conceptually the same, what about "hi" embedded between any to other lower-case letters? (a|b|c|…|z)hi(a|b|c|…|z) Oh, brother!

  It turns out, it's pretty common that you want to have an "|" of many single-letters. So, a special feature was added to regexps: character sets, between [ and ]:
  [aeiou]hi[aeiou].

  From here, it's a small jump to allow entire ranges, as a shorthand: [a-z]hi[a-z]. Here's trying to match "hi" between a possibly-uppercase letter in front, and a trailing letter-or-exclamation-point-or-prime-digit after: [A-Za-z]hi[235a-z!7] (which matches This. and chi5 and ahi!).

• It's not uncommon to see character classes like [A-Za-z] and [a-z0-9] and (for hex-digits) [A-Fa-f0-9].
• Don't use the character-set [A-z]. What characters come between "Z" and "a"?

  essentials: The two ascii codes every CS person should have on the tip of their tongue is for 'A' and ' ' (space). Knowing '0' is nice too; I don't find I need much beyond that. (Well, the hex ascii-code for space and / come up a lot in encoded URLs, so they're good to know.)

• Unicode: we'll talk more about better patterns for letters that account for different alphabets (which [A-Za-z] does not!)

Unwieldy video from distance lecture, 2017-feb-21 (1h35m)
but you can watch just [0:24:40,1:33:00) (1h08m) about regular-expressions.
(You can skip over the first 25min (opening pause [0:00:00,0:00:30), some ERD review [0:00:30,0:04:30), git-review [0:04:30,0:22:00), and a few comments about projects-with-external-clients [0:22:00,0:24:40)), as well as the final few "shutting down" minutes at [1:33:00,1:35:00).)

regular expression intro

Representing regular expressions

Regular expressions are not the same type as strings². Various languages have slightly different ways to represent regexps. video (14m23s)

For example:

• Java: class Pattern represents regexps (e.g. new Pattern("ab*c+?d")).
  As a convenience, many functions taking in regexps are also overloaded to take in a string, and it calls the Pattern constructor with that string for you.

  The thing to be careful about is regexps that contain a backslash: To get the pattern a\sb, your Java would use have the string-literal "a\\sb", because to get a backslash into any string-literal requires second backslash.

  This sometimes leads to backslash hell: If a regexp wants to match a backslash, the regexp will contain \\, and the java-string-to-represent-that-regexp would be written with \\\\.
  Even where we don’t have this situation, regexps that use a lot of backslashes end up looking even less readable as Java strings.

• Javascript: the concept of regular-expression literals are built-in to the language! Instead of quote-marks, surround your literal regexps with a slash: var pat = /a\sb/.

  This solution nice, because it emphasizes the difference between strings and regexps at the type level (which we like), without needing convenience functions which immediately obscure the difference and erode the helpfulness of the type-checker.

  Note that internally, there is still a class Regexp, which these literals are instances of. In addition, Javascript also allows for modifying the meaning of a regexp by adding one of a few possible suffixes; e.g. /a\sb/i is a case-independent regexp.

• python: There aren’t regexp-literals built in, but even better: python has “raw-string literals, which can be used in many situations, including being terribly convenient for writing regular expressions: the raw-string-literal r"a\sb" is equivalent to writing "a\\sb". WIN!

• PHP: Since PHP does have single-quotes for raw-string-literals, so you’d think it’d be cool like python. But not quite (of course): For PHP, the regular-expression libraries expect strings which begin and end with “/”³! So we’d use '/a\sb/' (or equivalently, "/a\\sb/").

  They’re the fine idea of raw-strings from python, but (for regexps) they then tack on a superfluous javascript-but-not requirement that detracts from its usefulness. phpsadness.com.

• Racket has #px as a syntax for regular-expression literals, but (unlike javascript literals) those literals still require escaping a backslash. Thus #px"a\\sb" is shorthand for calling the constructor (pregexp "a\\sb"), and we can still suffer from backslash hell.
• HTML (okay, it’s more a data-representation format than a programming language, but… next week we’ll talk about the pattern attribute for input tags):
  There are no issues with backslashes in regexps-in-HTML, because backslashes aren’t anything special in HTML. So something like <input type='text' pattern='a\sb'> is fine. The only character to wonder about getting inside regexps is a quote-mark itself. Since you can use either double- or single-quotes to delimit attribute values, it’s usually straightforward: if you want " inside your attribute, use '…' to delimit your attribute, and vice versa. But what if you want both characters in your regexps? In that case, you can fall back to the known HTML way: use the entity &quot; or &apos;.

• Java throws an exception (illegal character-escape) (yay!)
• javascript treats it as "azb" (as in “any character can be escaped; most characters escape to themselves”) (hmm)
• PHP treats it as the four-character string "a\zb" (le sigh).

PHP regular expressions

PHP regular-expression matching: In php, regular expressions are strings delimited by a special character (usually /). Some examples; see playing with regexps in PHP for details.

      echo preg_match( '/abcd/', 'abcd' );
      echo preg_match( '/abcd/', 'azcd' );      // false

      echo preg_match( '/a..d/', 'azcd' );      // . matches any single character (besides newline, null)

      echo preg_match( '/ab*cd/', 'abbbbcd' );  // b*  matches 0-or-more-b’s.
      echo preg_match( '/ab*cd/', 'abcd' );  
      echo preg_match( '/ab*cd/', 'acd'  ); 

      echo preg_match( '/ab+cd/', 'abbbbcd' );  // b+  matches 1-or-more-b’s.
      echo preg_match( '/ab+cd/', 'abcd' );  
      echo preg_match( '/ab+cd/', 'acd'  );     // false

      echo preg_match( '/ab.*cd/', 'abcd' );
      echo preg_match( '/ab.*cd/', 'abXd' );               // .* and .+ are a common patterns.
      echo preg_match( '/ab.*cd/', 'abBlahBlahBlahcd' );
      echo preg_match( '/ab.+cd/', 'abcd' );               // false
      echo preg_match( '/ab.+cd/', 'abXcd' );

      
      echo preg_match( '/ab?cd/', 'acd' );  // b?  matches 0-or-1 b
      echo preg_match( '/ab?cd/', 'abcd' );  
      echo preg_match( '/ab?cd/', 'abbbbcd'  );     // false

      echo preg_match( '/(ab)*cd/', 'ababababcd' );  // parens do grouping
      echo preg_match( '/(ab)*cd/', 'abbbbcd' );  // false


      // WARNING: preg_match looks to see if the string *contains* a match!
      echo preg_match( '/row/', 'How now, brown cow?' );  // true
      echo preg_match( '/.ow/', 'How now, brown cow?' );  // true

      // Use "^" to specify the start-of-string, and "$" to specify end-of-string.
      echo preg_match( '/^.ow$/', 'How now, brown cow?' );  // false
      echo preg_match( '/^.ow$/', 'Zow' );
      echo preg_match( '/^.ow/', 'Zowee' );
      echo preg_match( '/w.e$/', 'Wowee Zowee' );

      
      echo preg_match( '/[WZ]ow/', 'Wowee Zowee' );  // square-brackets match any one character from the set
      echo preg_match( '/[WZ]ow/', 'Yow' );          // false
      echo preg_match( '/[W-Z]ow/', 'Yowee' );       // square-brackets can contain a *range*
      echo preg_match( '/ab[0-9]+de/', 'ab789de' );


      echo preg_match( '/[0-9]*/', '00047' );        // Beware: matching just a * expression (w/o ^,$)!
    

Your task: What is a regular expression to match…

• An apostrophe between two letters
  (Why this? Editing a web-page, I want to search-replace straight-apostrophes with a curly-apostrophe. But I don’t want to replace html-attribute-values etc..
• The name "Jon"?
• The name "Jon" or "John"?
• Any name starting with "Jon"?
• Any name starting with: "Jon" or "John"?
• Any name containing "on" (any case)?
• The name "Jon" or "John" or "Jonathon" or "Jonathan"?
• (For the following: we’ll take “names” to mean containing only the 26 unaccented roman letters, and capitalized as in western culture⁴).
• A hyphenated name.
• A possibly-hyphenated name.
• A doubly-hyphenated name.
• An exactly-doubly-hyphenated name.
• An at-most-doubly-hyphenated name.
(For the following: we’ll take “words” to mean containing only unaccented roman letters and digits, or an underscore (?!).)
• A word starting with a lower-case letter.
• A word not starting with a capital letter.
• A word ending with a digit.
• A word ending with a roman numeral.

False positive, and false negative:

Warning: Beware matching a top-level * expression: the empty-string matches it, and any string contains the empty-string! Thus preg_match_all( "/(xyz)*/", "uh-oh") === 6!!, since "uh-oh" has zero "xyz"'s at the start, followed by 'u', followed by zero more "xyz"'s, followed by 'h', followed by ….

Atomic regexps

• ^ : matches start-of-string
• $ : matches end-of-string
• . : matches any one character
• Most characters match themselves (e.g. 'b'); you can escape special characters with backslash.
• Character classes: We’ll tend to prefer the unicode properties, since the other two don’t account for characters like “é”. See www.regular-expressions.info/posixbrackets.html.
  • \p{L} is better than [[:alpha:]] is better than \w.
  • \p{Lu} is better than [[:upper:]] is better than [A-Z].
  • \p{Ll} is better than [[:lower:]] is better than [a-z].
  • For digits, [0-9] or [\d] (or [[:digit:]]) is arguably okay; use \p{N} if you want to handle any digit-ish character like or “𝟜” (double-struck script) or “٤” (arabic 4) or “⁴” (superscript). You can use \p{Nd} to exclude the last (but not the former two).

    bug?: preg_match_all( '/\p{N}/', "⁴٤𝟜4" ) is returning 1 for me (not 4), in php 5.4.3.

  • \p{P} is better than [[:punct:]] is the same as [!.?:;"'].
  • […] — build your own character set, e.g. [aeiou] matches any one lower-case vowel.
    Use ^ just inside the [ to negate the set. (Note that this overrides the beginning-of-string meaning of ^; it’s clear from context).

  • Checking for space (one of the most common situations) is the one that is hardest: We have \p{Z}, as well as [[:space:]] and \s. The first, unicode-property version checks for separating space characters (thin-space, etc), but doesn’t catch tabs (or newlines). On the other hand, \s does catch tabs, newlines etc., but doesn’t catch thin-space or nonbreaking-space or other space-looking characters.

    Also, we are probably least able to ignore this problem, because in web forms people might paste in text from web pages or Word documents, and those exper-typesetting programs do tend to use the various special-spaces.

    Solution: Either

    • use the pattern [\p{Z}\n\t\r\v\f] — long to type and read, and easy to get wrong; or

    • $str2 = preg_replace( '/\p{Z}+/', ' ', $str ); … if( preg_match(/\s+/',$str2)) …

  Remember with the character-sets: Each of these match one single character. They are often used in conjunction with * and + below.

Compound regexps

• (α) — same as α; used to dis-ambiguate precedence.
• α* — 0 or more occurrences of α
• α+ — 1 or more occurrences of α
• α? — 0 or 1 occurrence of α
• α{…} — you specify a range of #repetitions of α (not very common)
• α|β — α OR β. This is the one low-priority regexp operator: ab|cd is interpreted as (ab)|(cd), not a(b|c)d. Compare that with any of the others: ab*c is interpreted as a(b*)c, not (ab)*c.

NOTE: In php, no trailing “g” allowed, as in javascript!
(There are other trailing modifiers however — e.g. i for case-insensitive, and more.)

• preg_match($pat,$target) (also, note: preg_match_all — not as common, but it motivates why preg_match's output is 0,1 and not a raw boolean).

  Warning: preg_match_all("/x/","abc",$results) will always have $results be non-empty — it will be an array containing an empty array. (In particular, the condition of if ($results) … will always be true.) You can instead use preg_match_all("/x/","abc",$results,PREG_SET_ORDER) which will have zero elements for zero matches (and if ($results) … will behave as you want). Each element of $results will still be an another entire array though -- the full match plus any sub-patterns.

• preg_replace($pat,$target,$str)
• split($pat,$target) Returns an array (indexed numerically from 0)
• trim — nothing to do with regular expressions, but a function you probably want to call on most user-entered text.

regexps vs. unicode

• In unicode, we separate the notion of a byte from that of a character. (The term “code point” is sometimes used to emphasize unicode-character-that-might-be-more-than-a-char.)
• In unicode, we have more generalized notions of letters: not just A-Z, but also accented characters like “é”, chinese characters, etc.. Also, whitespace is more complicated: There is a regular ol' space, thin whitespace &thinsp;, and more.
• Use “properties” — generalized character classes, when supported by your language:
  • \p{Letter} or \p{L} — any letter, any language.
  • \p{Separator} or \p{Z} — space-characters, including thin-space etc.. But it does not include newlines and tabs (!). Using \p{Z}|[\t\r\n\v\f] is better than \s though, since people might paste in whitespace on input forms. (Or, do a regexp-replace of \p{Z} to , and then use \s.)
  • \p{Punctuation} or \p{P}
  There are more refined versions of these properties, too (e.g. \p{Ll} or \p{Lowercase_Letter}).
• Corner case: If you want to compare strings case-independently, use a library function that is unicode-aware. Note that the common trick of converting both strings to (say) uppercase doesn’t always work in unicode, since (e.g.) “ß” has no (traditional) upper-case version. Moreover, for comparing German strings, “ß” should be considered equivalent to “ss”.
• Corner case: Beware that characters like “é” can actually be encoded either as a single character (and thus match “.”), or as a regular ol' “e” combined with the accent “́” (and thus match, say, “..”)! See this page for advice on how to write regexps to try and cope with this.

a warning

• It is provably impossible to use regular expressions for general parsing of HTML, or JSON, or even determining whether a sequence of open/close parentheses are balanced. Take Automata Theory (ITEC420) for proofs of this and much more!
  …Or, just believe the top answer to this question⁵.
• Roughly: Regular expressions have no "context", and they can't correlate what happens in one part of a string with what happens somewhere later, potentially very-far-away⁶.
• Use regular expressions for short patterns (usually contained within a single line). Although you can't parse general HTML, you might have a bunch of scraped web-data where a bunch of lines are in a simple, predictable (tagged) format. Regular expressions might shine, in that limited case.
• Don't use regular expressions for numeric checks. If you want to verify that a field is bewteen 73 and 125, don't try to come up with some regular expression that detects this. Just convert your strings to numbers, and use ≤

home—lects—hws
D2L—breeze (snow day)

This page licensed CC-BY 4.0 Ian Barland Page last generated

Please mail any suggestions (incl. typos, broken links) to ibarlandradford.edu