Readme

• License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-nc-sa/4.0/

• You are free to:
  • Share : copy and redistribute the material
  • Adapt : rebuild and transform the material
• Under the following conditions:
  • Attribution : You must give appropriate credit, provide a link to the license, and indicate if changes were made.
  • NonCommercial : You may not use this work for commercial purposes.
  • Share Alike : If you remix, transform, or build upon this work, you must distribute your contributions under the same license to this one.

Disclaimer and acknowledgements

• These slides have been prepared based on multiple sources: websites, blogs, courses. While it is hard to cite them all I wish to acknowledge those sources that have been particularly useful.

  • Tutorial-R-web-data, by Gaston Sanchez
  • Text Mining, Scraping and Sentiment Analysis with R Udemy.
  • Working with Web Data in R Datacamp Course

• This chapter makes a heavy use of strings so two additional sources are:

  • Handling Strings with R by Gaston Sanchez and
  • R for Data Science (Strings chapter) by Hadley Wickham and Garret Grolemund.
  • tidyverse vignette on Regular expressions

Introduction

• Web scraping is about collecting information from the web.
• Sometimes this will be contained in tables or other well structured containers,
• But most of the times we need to gather relevant information from heaps of unstructured textual data.
• This is usually done in three steps
  1. Gather the unstructured text.
  2. Determine which recurrent patterns in the data are we looking for.
  3. Apply the patterns to the unstructured text to obtain the desired information.
• We focus on steps 2 and 3 and introduce regular expressions as the tool for optimally doing this processing.
• Regular expressions provide a syntax for systematically accessing and operating on text patterns.

When would one need regular expressions?

• The example below shows a typical situation where regular expressions mat be useful: manipulating files by their names.
  • We are often faced with the problem of selecting/accessing a subset of files.
  • We can use string functions to extract certain (file)names, say all documents on a certain topic.
    • At a certain level standard string functions will be enough to describe the names we want to access: for example all files whose name includes the string dplyr.
    • For more sophisticated extractions this will not be enough. Here is where regular expressions enter the game allowing to describe almost any pattern one can imagine.
• Regular expressions are throroughly used by system administrators to manage computer files.
• A different but related situation is when one whishes to extract meaningful (or “given”) content from text.
  • Mining text to look for a certain type of expression
  • Mining twitter data

Functions for Pattern Matching

• R has various functions for regular expression based match and replaces.
• The grep(), grepl(), regexpr() and gregexpr() functions are used for searching for matches.

Detect Patterns

• grep(pattern, x, value = FALSE) returns an integer vector of the indices of the elements of x that yielded a match (or not, for invert = TRUE).

string <- c("Regular", "expression", "examples of R language")
grep("ex", string, value=F)

## [1] 2 3

• grep(pattern, x, value = TRUE) returns the specific strings that happen to have the match.

grep("ex", string, value=T)

## [1] "expression"             "examples of R language"

• grepl(pattern, x) returns a TRUE/FALSE vector indicating which elements of the character vector contain a match.

grepl("ex", string)

## [1] FALSE  TRUE  TRUE

Locate Patterns (1)

• regexpr(pattern, text) returns an integer vector of the same length as text giving the starting position of the first match or \(-1\) if there is none, with attribute “match.length”, an integer vector giving the length of the matched text (or -1 for no match).

string <- c("Regular", "expression", "examples of R language")
x <- regexpr("ex", string)
x

## [1] -1  1  1
## attr(,"match.length")
## [1] -1  2  2
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE

Locate Patterns (2)

• gregexpr(pattern, text) returns a list of the same length as text each element of which is of the same form as the return value for regexpr, except that the starting positions of every (disjoint) match are given.

string <- c("Regular", "expression", "examples of R language")
x <- gregexpr("x*ress", string)
x

## [[1]]
## [1] -1
## attr(,"match.length")
## [1] -1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[2]]
## [1] 4
## attr(,"match.length")
## [1] 4
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[3]]
## [1] -1
## attr(,"match.length")
## [1] -1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE

Replace Patterns

• sub(pattern, replacement, string) replace first match

string <- "He is now 25 years old, and weights 130lbs";
x <- sub("[[:digit:]]", "", string)
x

## [1] "He is now 5 years old, and weights 130lbs"

• gsub(pattern, replacement, string) replace all matches

x <- gsub("[[:digit:]]", "", string)
x

## [1] "He is now  years old, and weights lbs"

Extract Patterns (1)

• regmatches(string, regexpre()) extract first match

x <- c("Arkansas", "Alabama", "Calabash", "Washington")
pattern <- "[Aa][^a]*a" # sequences starting with A or a and continuing up to next a.
regmatches(x, regexpr(pattern, x))

## [1] "Arka" "Ala"  "ala"

regmatches(x, regexpr(pattern, x), invert = TRUE)

## [[1]]
## [1] ""     "nsas"
## 
## [[2]]
## [1] ""     "bama"
## 
## [[3]]
## [1] "C"    "bash"
## 
## [[4]]
## [1] "Washington"

Extract Patterns (2)

• regmatches(string, gregexpre()) extract all matches, outputs a list

x <- c("Arkansas", "Alabama", "Calabash", "Washington")
pattern <- "[Aa][^a]*a" # sequences starting with A or a and continuing up to next a.
regmatches(x, gregexpr(pattern, x)) 

## [[1]]
## [1] "Arka"
## 
## [[2]]
## [1] "Ala" "ama"
## 
## [[3]]
## [1] "ala"
## 
## [[4]]
## character(0)

regmatches(x, gregexpr(pattern, x), invert = TRUE) 

## [[1]]
## [1] ""     "nsas"
## 
## [[2]]
## [1] ""  "b" "" 
## 
## [[3]]
## [1] "C"    "bash"
## 
## [[4]]
## [1] "Washington"

So what are regular expressions?

• A regular expression is a special text string for describing a certain amount of text.
• This “certain amount of text” receives the formal name of pattern.
• A regular expression is a pattern that describes a set of strings.
• It is common to abbreviate the term “regular expression” as regex.
• Simply put, working with regular expressions is nothing more than pattern matching.

Forming regular expressions

• Regex patterns consist of a combination of alphanumeric characters as well as special characters.
  • e.g. [a-zA-Z0-9 .]*
• A regex pattern can be as simple as a single character.
• But it can also be formed by several characters with a more complex structure.
• Regular expressions are constructed from 3 types of components:
  • Literal characters are matched only by the character itself.
  • Character classes, matched by any single member of the specified class
  • Modifers that operate on literal characters, character classes, or combinations of the two.

String functions and patterns

• Notice that the enumeration above relies on two types of functions
  • Standard base R functions
  • Functions from the stringr package, developed for extending and simplifying R base functionalities.
• Information is available:
  • For base functions at: ENDMEMO/Regular Expressions in R
  • For stringr at: stringi regular expressions vignette
• All the functions require a pattern to describe the set of strings on which they operate.
• Regular expressions are not the functions but the rules used to build the patterns.

Common Regex tasks

• Detect match to a pattern:
  • grep(..., value = FALSE), grepl(),
  • stringr::str_detect()
• Locate pattern within a string, i.e. give the start position of matched patterns.
  • regexpr(), gregexpr(),
  • stringr::str_locate(), string::str_locate_all()
    
• Extract match to a pattern:
  • grep(..., value = TRUE),
  • stringr::str_extract(), stringr::str_extract_all()
    
• Replace a pattern:
  • sub(), gsub(),
  • stringr::str_replace(), stringr::str_replace_all()
    
• Split a string using a pattern:
  • strsplit(),
  • stringr::str_split()

Regular expression syntax

• Regular expressions typically specify characters or character classes to seek out, possibly with information about repeats and location within the string.
• This is accomplished with the help of metacharacters that have specific meaning:
  • $ * + . ? [ ] ^ { } | ( ) \.
• In this section, we will introduce the basic building blocks of extended regular expressions as implemented in R.

Basic exact character matching

• At the most basic level characters match characters, even in regular expressions.
• Thus, extracting a substring of a string will yield itself if present:

require(stringr)
x <- c("apple", "banana", "pear")
str_extract(x, "an")

## [1] NA   "an" NA

• You can perform a case-insensitive match using ignore_case = TRUE:

bananas <- c("banana", "Banana", "BANANA")
str_detect(bananas, "banana")

## [1]  TRUE FALSE FALSE

str_detect(bananas, regex("banana", ignore_case = TRUE))

## [1] TRUE TRUE TRUE

The str_whatever functions

• The stringr package offers both str_whatever() and str_whatever_all() in many instances.
  • The former addresses the first instance of a matching string,
  • the latter accesses all instances.
• The syntax of all these functions is such that:
  • the character vector in question is the first element,
  • the regular expression the second, and
  • all possible additional values come after that.

require(stringr)
example.obj <- "1. A small sentence. - 2. Another tiny sentence."
str_extract(example.obj, "e")

## [1] "e"

str_extract_all(example.obj, "e")

## [[1]]
## [1] "e" "e" "e" "e" "e" "e" "e"

str_extract vs grep

• Observe the different result:

x <- c("apple", "banana", "pear")
str_extract(x, "an")

## [1] NA   "an" NA

grep("an", x, value = TRUE)

## [1] "banana"

• To obtain the same result as with grep() one can use str_subset()

str_subset(x,"an")

## [1] "banana"

Refining the search for a character:
Specifying location

• Sometimes we do not simply care about finding a match anywhere in a string but are concerned about the specific location within a string.
• There are two simple additions we can make to our regular expression to specify locations.
  • The caret symbol (^) at the beginning of a regular expression marks the beginning of a string
  • The dollar symbol ($) at the end marks the end.

x <- c("apple", "banana", "pear")
str_detect(x, "^a")

## [1]  TRUE FALSE FALSE

Syntax (1) More about character matching: wildcards

• The next step up in complexity is to use the period character “.”, which matches any character except a newline:

x <- c("apple", "banana", "pear")
str_extract(x, ".a.")

## [1] NA    "ban" "ear"

• You can allow . to match everything, including \n (new line), by setting dotall = TRUE:

str_detect("\nX\n", ".X.")

## [1] FALSE

str_detect("\nX\n", regex(".X.", dotall = TRUE))

## [1] TRUE

• The wildcard “.” allow the search for any character

example.obj <- "1. A small sentence. - 2. Another tiny sentence."
str_extract(example.obj, "sm.ll")

## [1] "small"

example.obj.2 <- "The cat sat on the mat"
str_extract(example.obj.2, ".at")

## [1] "cat"

str_extract_all(example.obj.2, ".at")

## [[1]]
## [1] "cat" "sat" "mat"

• The power of regular expressions stems from the possibility to write flexible, generalized search queries.

Syntax (2): Escape sequences

• If “.” matches any character, how do you match a literal “.”?
  • You need to use an “escape” to tell the regular expression you want to match it exactly, not use its special behaviour.
  • Like strings, regexps use the backslash, “\”, to escape special behaviour.
  • So to match an “.”, you need the regexp “\.”. Unfortunately this creates a problem.
  • We use strings to represent regular expressions, and “\” is also used as an escape symbol in strings.
  • So to create the regular expression “\.” we need the string “\\.”.

str_extract(c("abc", "a.c", "bef"), "a\\.c")

## [1] NA    "a.c" NA

• If \ is used as an escape character in regular expressions, how do you match a literal \?
  • Well you need to escape it, creating the regular expression \\.
  • To create that regular expression, you need to use a string, which also needs to escape \.
  • That means to match a literal \ you need to write “\\\\” — you need four backslashes to match one!

x <- "a\\b" # There is only one \
writeLines(x)

## a\b

str_extract(x, "\\\\")

## [1] "\\"

• An alternative quoting mechanism is \Q...\E: all the characters in … are treated as exact matches. This is useful if you want to exactly match user input as part of a regular expression.

Syntax (3): More on Escape sequences

• There are other characters in R that require escaping, and this rule applies to all string functions in R, including regular expressions.
  • \': single quote. You don’t need to escape single quote inside a double-quoted string, so we can also use "'" in the previous example.
    
  • \": double quote. Similarly, double quotes can be used inside a single-quoted string, i.e. '"'.
    
  • \n: newline.
    
  • \r: carriage return.
  • \t: tab character.
    
• See here for a complete list of R escape sequences.
  
• Note: cat() and print() to handle escape sequences differently, if you want to print a string out with these sequences interpreted, use cat().

Example - Let’s say you specify your pattern with single quotes and you want to find countries with the single quote “’”. - You would have to “escape” the single quote in the pattern, by preceding it with “\”, so it’s clear it is not part of the string-specifying machinery:

library(XML)
library(RCurl)
url <- getURL("https://www.nationsonline.org/oneworld/countries_of_the_world.htm")
df <- readHTMLTable(url, header = T)
countries <- c(levels(df[[1]]$V2),levels(df[[2]]$V2),levels(df[[3]]$V2),
               levels(df[[4]]$V2),levels(df[[5]]$V2))
grep("\'", countries, value = TRUE)

## [1] "Côte D'ivoire (Ivory Coast)"                  
## [2] "Korea, Democratic People's Rep. (North Korea)"
## [3] "Lao, People's Democratic Republic"

Syntax (4): Quantifiers

• Sometimes we need to describe a sequence by the number of matches. This can be done using quanitifiers
• Quantifiers specify how many repetitions of the pattern.
  • *: matches at least 0 times (0 or more).
    
  • +: matches at least 1 times (1 or more).
    
  • ?: matches at most 1 times (0 or 1).
    
  • {n}: matches exactly n times.
    
  • {n,}: matches at least n times.
    
  • {n,m}: matches between n and m times.

Syntax (4): Quantifiers (examples)

(strings <- c("a", "ab", "acb", "accb", "acccb", "accccb"))
grep("ac*b", strings, value = TRUE) # "ab"     "acb"    "accb"   "acccb"  "accccb"
grep("ac+b", strings, value = TRUE) # "acb"    "accb"   "acccb"  "accccb"
grep("ac?b", strings, value = TRUE) # "ab"  "acb"
grep("ac{2}b", strings, value = TRUE) # "accb"
grep("ac{2,}b", strings, value = TRUE) # "accb"   "acccb"  "accccb"
grep("ac{2,3}b", strings, value = TRUE) # "accb"  "acccb"

Exercise

Find all countries with ee in their name using quantifiers.

## [1] "Cocos (Keeling) Islands"       "Greece"                       
## [3] "Greenland"                     "Holy See"                     
## [5] "Vatican City State (Holy See)"

Syntax (5): Position of pattern within the string:
anchors

• By default, regular expressions will match any part of a string.
• It’s often useful to anchor the regular expression so that it matches from the start or end of the string:
  • ^ matches the start of string.
  • $ matches the end of the string.
  • \b: matches the empty string at either edge of a word. Don’t confuse it with ^ $ which marks the edge of a string.
    
  • \B: matches the empty string provided it is not at an edge of a word.

(strings <- c("abcd", "cdab", "cabd", "c abd"))
grep("ab", strings, value = TRUE)
grep("^ab", strings, value = TRUE)
grep("ab$", strings, value = TRUE)
grep("\\bab", strings, value = TRUE)

str_replace_all("The quick brown fox", "\\b", "_")

## [1] "_The_ _quick_ _brown_ _fox_"

str_replace_all("The quick brown fox", "\\B", "_")

## [1] "T_h_e q_u_i_c_k b_r_o_w_n f_o_x"

Exercises

Find the countries that end up with land.

##  [1] "Christmas Island" "Finland"          "Greenland"        "Iceland"         
##  [5] "Ireland"          "New Zealand"      "Pitcairn Island"  "Poland"          
##  [9] "Reunion Island"   "Swaziland"        "Switzerland"      "Thailand"

Find the countries that have the word and in their name.

## [1] "Antigua and Barbuda"              "Bosnia and Herzegovina"          
## [3] "Saint Kitts and Nevis"            "Saint Vincent and the Grenadines"
## [5] "Sao Tome and Principe"            "Trinidad and Tobago"             
## [7] "Turks and Caicos Islands"         "Wallis and Futuna Islands"

Syntax (6): Operators

• Regular expressions are composed using operators.
  • .: matches any single character.
  • [...]: a character list, matches any one of the characters inside the square brackets.
    We can also use - inside the brackets to specify a range of characters.
    
  • [^...]: an inverted character list, similar to [...], but matches any characters except those inside the square brackets.
    
  • \: suppress the special meaning of metacharacters in regular expression, i.e. $ * + . ? [ ] ^ { } | ( ) \, similar to its usage in escape sequences. Since \ itself needs to be escaped in R, we need to escape these metacharacters with double backslash like \\$.
    
  • |: an “or” operator, matches patterns on either side of the |.
    
  • (...): grouping in regular expressions. This allows you to retrieve the bits that matched various parts of your regular expression so you can alter them or use them for building up a new string. Each group can than be refer using \\N, with N being the No. of (...) used. This is called backreference.

Syntax (6): Operators examples

(strings <- c("^ab", "ab", "abc", "abd", "abe", "ab 12", "acb"))
grep("ab.", strings, value = TRUE)
grep("ab[c-e]", strings, value = TRUE)
grep("ab[^c]", strings, value = TRUE)
grep("^ab", strings, value = TRUE)
grep("\\^ab", strings, value = TRUE)
grep("abc|abd", strings, value = TRUE)
gsub("(ab) 12", "\\1 34", strings)

url <- getURL("https://en.wikipedia.org/wiki/List_of_culinary_fruits")
df <- readHTMLTable(url, header = T)
fruits <- c(levels(df[[1]]$`Common name`), levels(df[[2]]$`Common name`),
            levels(df[[3]]$`Common name`), levels(df[[4]]$`Common name`),
            levels(df[[5]]$`Common name`), levels(df[[6]]$`Common name`),
            levels(df[[7]]$`Common name`), levels(df[[8]]$`Common name`))
pattern <- "(..)\\1"
str_subset(fruits, pattern)

## [1] "Bolivian mountain coconut" "King coconut"             
## [3] "Sea coconut"               "Salal"                    
## [5] "Cassabanana"               "Banana"

Exercise

Find countries with letter i or t, and ends with land, and replace land with LAND.

## [1] "Christmas IsLAND" "FinLAND"          "IceLAND"          "IreLAND"         
## [5] "Pitcairn IsLAND"  "Reunion IsLAND"   "SwaziLAND"        "SwitzerLAND"     
## [9] "ThaiLAND"

Syntax (7): Character classes

• Character classes allows to – surprise! – specify entire classes of characters, such as numbers, letters, etc.

• There are two flavors of character classes, one uses [: and :] around a predefined name inside square brackets and the other uses \ and a special character. They are sometimes interchangeable.

  • [:digit:] or \d: digits, 0 1 2 3 4 5 6 7 8 9, equivalent to [0-9].
    
  • \D: non-digits, equivalent to [^0-9].
    
  • [:lower:]: lower-case letters, equivalent to [a-z].
    
  • [:upper:]: upper-case letters, equivalent to [A-Z].
    
  • [:alpha:]: alphabetic characters, equivalent to [[:lower:][:upper:]] or [A-z].
  • [:alnum:]: alphanumeric characters, equivalent to [[:alpha:][:digit:]] or [A-z0-9].
    
  • [:blank:]: blank characters, i.e. space and tab.
    
  • [:space:]: space characters: tab, newline, vertical tab, form feed, carriage return, space.

More character classes

• Other character classes are described below:

  • \w: word characters, equivalent to [[:alnum:]_] or [A-z0-9_].
  • \W: not word, equivalent to [^A-z0-9_].
    
  • [:xdigit:]: hexadecimal digits (base 16), 0 1 2 3 4 5 6 7 8 9 A B C D E F a b c d e f, equivalent to [0-9A-Fa-f].
  • \s: space, .
    
  • \S: not space.
    
  • [:punct:]: punctuation characters, ! " # $ % & ’ ( ) * + , - . / : ; < = > ? @ [  ] ^ _ ` { | } ~.
  • [:graph:]: graphical (human readable) characters: equivalent to [[:alnum:][:punct:]].
  • [:print:]: printable characters, equivalent to [[:alnum:][:punct:]\\s].
  • [:cntrl:]: control characters, like \n or \r, [\x00-\x1F\x7F].

Note:
* [:...:] has to be used inside square brackets, e.g. [[:digit:]].
* \ itself is a special character that needs escape, e.g. \\d. Do not confuse these regular expressions with R escape sequences such as \t.

General modes for patterns

• There are different syntax standards for regular expressions, and R offers two:
  • POSIX extended regular expressions (default)
  • Perl-like regular expressions.
• You can easily switch between by specifying perl = FALSE/TRUE in base R functions, such as grep() and sub().
• For functions in the stringr package, wrap the pattern with perl().
• The syntax between these two standards are a bit different sometimes, see an example here.

Functions in the stringr package

Functions in the stringr package

Functions in stringr vs in functions in base R

Functions in stringr vs in functions in base R

Resources

• A Rstudio cheatsheet on string manipulation
• A basic cheatsheet on regular expressions
• Official document about Regular expression in R.
  
• Perl-like regular expression: regular expression in perl manual.
  
• qdapRegex package: a collection of handy regular expression tools, including handling abbreviations, dates, email addresses, hash tags, phone numbers, times, emoticons, and URL etc.
  
• There are some online tools to help learn, build and test regular expressions. On these websites, you can simply paste your test data and write regular expression, and matches will be highlighted.
  • regexpal
    
  • RegExr