---
title: "R Programming for Web Scraping"
author: "Alex Sanchez (asanchez@ub.edu) <br> Francesc Carmona (fcarmona@ub.edu) <br> GME Department. Universitat de Barcelona"
date: "March 2020"
output:
  slidy_presentation:
    fig_width: 7
    fig_height: 6
css: myStyles.css
footer: "R Programming"
keep_md: true
editor_options: 
  chunk_output_type: console
---

```{r setLicense, child = 'license.Rmd'}
```

```{r disclaimer, child = 'disclaimer2.Rmd'}
```

The dplyr package (1)
=================

- The data frame is a key data structure in statistics and in R. 
- The basic structure of a data frame is that there is one observation per row and each column represents a variable, a measure, feature, or characteristic of that observation.
- It’s important that we have good tools for dealing with data frames. In previous slides we have already discussed some tools like the `subset()` function and the use of `[` and `$` operators to extract subsets of data frames. 
- However, other operations, like filtering, re-ordering, and collapsing, can often be tedious operations in R whose syntax is not very intuitive. 
- The `dplyr` package is designed to mitigate a lot of these problems and to provide a highly optimized set of routines specifically for dealing with data frames.

The dplyr package (2)
=================

- The `dplyr` package was developed by **Hadley Wickham** of *RStudio* and is an optimized and distilled version of his `plyr` package. 
- The `dplyr` package does not provide any "new" functionality to R per se, in the sense that everything dplyr does could already be done with base R, but it greatly simplifies existing functionality in R.
- One important contribution of the `dplyr` package is that it provides a “grammar” (in particular, verbs) for data manipulation and for operating on data frames. 
- Another useful contribution is that the `dplyr` functions are very fast, as many key operations are coded in C++.

`dplyr` Grammar
=============

Some of the key "verbs" provided by the `dplyr` package are

- **select**: return a subset of the columns of a data frame, using a flexible notation

- **filter**: extract a subset of rows from a data frame based on logical conditions

- **arrange**: reorder rows of a data frame

- **rename**: rename variables in a data frame

- **mutate**: add new variables/columns or transform existing variables

- **summarise / summarize**: generate summary statistics of different variables in the data frame, possibly within strata

- **%>%**: the "pipe" operator is used to connect multiple verb actions together into a pipeline

The `dplyr` package as a number of its own data types that it takes advantage of.

`select()` (1)
==========

For the examples in this document we will be using a dataset containing air pollution and temperature data for the city of Chicago in the U.S.

```{r message=FALSE}
library(dplyr)
chicago <- readRDS("chicago.rds")
dim(chicago)
str(chicago)
```

The `select()` function can be used to select columns of a data frame that you want to focus on. 

Suppose we wanted to take the first 3 columns only. There are a few ways to do this. We could for example use numerical indices. But we can also use the names directly.
```{r}
names(chicago)[1:3]
subset <- select(chicago, city:dptp)
head(subset)
```

`select()` (2)
==========

Note that the `:` normally cannot be used with names or strings, but inside the `select()` function you can use it to specify a range of variable names.

You can also **omit variables** using the `select()` function by using the negative sign.
```{r eval=FALSE}
select(chicago, -(city:dptp))
```

The `select()` function also allows a special syntax that allows you to specify variable names based on patterns.
```{r}
subset <- select(chicago, ends_with("2"))
head(subset)
subset <- select(chicago, matches("^p.*2$"))
head(subset)
```

`filter()`
==========

- The `filter()` function is used to extract subsets of rows from a data frame. 
- This function is similar to the existing `subset()` function in R but is quite a bit faster.
```{r}
chic.f <- filter(chicago, pm25tmean2 > 30)
dim(chic.f)
chic.f <- filter(chicago, pm25tmean2 > 30 & tmpd > 80)
dim(select(chic.f, date, tmpd, pm25tmean2))
```

`arrange()`
===========

- The `arrange()` function is used to reorder rows of a data frame according to one of the variables/columns. 
- Reordering rows of a data frame (while preserving corresponding order of other columns) is normally a pain to do in R. The `arrange()` function simplifies the process quite a bit.

Here we can order the rows of the data frame by date, so that the first row is the earliest (oldest) observation and the last row is the latest (most recent) observation:
```{r}
chicago.ord <- arrange(chicago, date)
head(select(chicago.ord, date, pm25tmean2), 3) # the first few rows
tail(select(chicago.ord, date, pm25tmean2), 3) # the last few rows
chicago.desc <- arrange(chicago, desc(date))   # descending order
```

`rename()`
==========

Renaming a variable in a data frame in R is surprisingly hard to do! The `rename()` function is designed to make this process easier.
```{r}
names(chicago.desc)[1:5]
# names of the first five variables
chicago.desc <- rename(chicago.desc, dewpoint = dptp, pm25 = pm25tmean2)
names(chicago.desc)[1:5]
```

`mutate()`
==========

- The `mutate()` function exists to compute transformations of variables in a data frame. 
- Often, you want to create new variables that are derived from existing variables and `mutate()` provides a clean interface for doing that.

Here we create a pm25detrend variable that subtracts the mean from the pm25 variable.
```{r}
chicago.desc <- mutate(chicago.desc, pm25detrend = pm25 - mean(pm25, na.rm = TRUE))
head(chicago.desc[,6:9])
```

- There is also the related `transmute()` function, which does the same thing as `mutate()` but then *drops all non-transformed variables*.

Here we detrend the PM10 and ozone (O3) variables.
```{r eval=FALSE}
head(transmute(chicago.desc, 
               pm10detrend = pm10tmean2 - mean(pm10tmean2, na.rm = TRUE),
               o3detrend = o3tmean2 - mean(o3tmean2, na.rm = TRUE)))
```

`group_by()`
============

- The `group_by()` function is used to generate summary statistics from the data frame within strata defined by a variable.

- The general operation here is a combination of splitting a data frame into separate pieces defined by a variable or group of variables (`group_by()`), and then applying a summary function across those subsets (`summarize()`).

First, we can create a year variable using `as.POSIXlt()`.
```{r}
chicago.desc <- mutate(chicago.desc, year = as.POSIXlt(date)$year + 1900)
```
Now we can create a separate data frame that splits the original data frame by year.
```{r}
years <- group_by(chicago.desc, year)
```
Finally, we compute summary statistics for each year in the data frame with the `summarize()` function.
```{r eval=FALSE}
summarize(years, pm25 = mean(pm25, na.rm = TRUE), 
                 o3 = max(o3tmean2, na.rm = TRUE), 
                 no2 = median(no2tmean2, na.rm = TRUE))
```

`summarize()` returns a data frame with year as the first column, and then the annual averages of pm25, o3, and no2.

The pipeline operator `%>%`
===========================

- The pipeline operator `%>%` is very handy for stringing together multiple `dplyr` functions in a sequence of operations. 
- Notice above that every time we wanted to apply more than one function, the sequence gets buried in a sequence of nested function calls that is difficult to read, i.e.
```{r eval=FALSE}
third(second(first(x)))
```
- This nesting is not a natural way to think about a sequence of operations. 
- The `%>%` operator allows you to string operations in a left-to-right fashion, i.e.
```{r eval=FALSE}
first(x) %>% second %>% third
```

Take the example that we just did in the last section where we computed the mean of pm25, o3 and no2 by year. 

There we had to

* create a new variable year
* split the data frame by that new variable
* compute the mean of pm25, o3 and no2 in the sub-groups defined by year

That can be done with the following sequence in a single R expression.
```{r eval=FALSE}
mutate(chicago.desc, year = as.POSIXlt(date)$year + 1900) %>%    
       group_by(year) %>% 
       summarize(pm25 = mean(pm25, na.rm = TRUE),
                 o3 = mean(o3tmean2, na.rm = TRUE), 
                 no2 = mean(no2tmean2, na.rm = TRUE))
```

The keyboard shortcut for `%>%` is
<div align="center"> 
![](images/pipelineShortcut.png)
</div>

Tibbles
=======

- Almost all functions in the `tidyverse` package produce data in the **tibble** format, an alternative to the classic R data.frame.
```{r eval=FALSE}
library(tidyverse)
as_tibble(iris)
```
- You can create a new tibble from individual vectors with `tibble()`.
```{r}
tibble(
  x = 1:5, 
  y = 1, 
  z = x ^ 2 + y
)
```
- If you’re already familiar with `data.frame()`, note that `tibble()` does much less: it never changes the type of the inputs (e.g. it never converts strings to factors!), it never changes the names of variables, and it never creates row names.

Tibbles (2)
=======
- Another way to create a tibble is with `tribble()`, short for **t**ransposed tibble. 
- `tribble()` is customised for data entry in code: column headings are defined by formulas (i.e. they start with ~), and entries are separated by commas.
```{r}
tribble(
  ~x, ~y, ~z,
  #--|--|----
  "a", 2, 3.6,
  "b", 1, 8.5
)
```

Printing tibbles
================

- Tibbles have a refined print method that shows only the first 10 rows, and all the columns that fit on screen. This makes it much easier to work with large data. 
- In addition to its name, each column reports its type, a nice feature borrowed from str():
```{r}
nycflights13::flights
```

- But you can explicitly `print()` the data frame and control the number of rows (n) and the width of the display. `width = Inf` will display all columns:
```{r eval=FALSE}
nycflights13::flights %>% 
  print(n = 10, width = Inf)
```
- You can also control the default print behaviour by setting `options()`.
- A final option is to use RStudio’s built-in data viewer to get a scrollable view of the complete dataset.
```{r eval=FALSE}
nycflights13::flights %>% 
  View()
```

Subsetting tibbles
==================

- To use `$` and `[[` in a pipe, you’ll need to use the special placeholder `.`:
```{r eval=FALSE}
# Extract by name
df$x
df[["x"]]
# Extract by position
df[[1]]
df %>% .$x
df %>% .[["x"]]
```
- Compared to a data.frame, tibbles are more strict: they never do partial matching, and they will generate a warning if the column you are trying to access does not exist.
- Some older functions don’t work with tibbles. If you encounter one of these functions, use `as.data.frame()` to turn a tibble back to a data.frame.


Resources
==========
- [R Programming for Data Science](https://bookdown.org/rdpeng/rprogdatascience/managing-data-frames-with-the-dplyr-package.html)
- [RStudio Cheat Sheets](https://rstudio.com/resources/cheatsheets/)



