---
title: "Web APIs"
subtitle: "Lecture 13"
author: "Dr. Colin Rundel"
footer: "Sta 523 - Fall 2023"
format:
  revealjs:
    theme: slides.scss
    transition: fade
    slide-number: true
    self-contained: true
execute: 
  echo: true
---

```{r setup}
#| message: False
#| warning: False
#| include: False
options(
  width=80
)

knitr::opts_chunk$set(
  fig.align = "center", fig.retina = 2, dpi = 150,
  out.width = "100%",
  message = TRUE
)

library(tidyverse )
library(rvest)
```


## URLs

![](imgs/url-structure.png){fig-align="center" width="100%"}


::: {.aside}
From [HTTP: The Protocol Every Web Developer Must Know](http://code.tutsplus.com/tutorials/http-the-protocol-every-web-developer-must-know-part-1--net-31177)
:::


## Query Strings

Provides named parameter(s) and value(s) that modify the behavior of the resulting page. 

<br/>

Format generally follows:

::: {.center .large}
`?arg1=value1&arg2=value2&arg3=value3`
:::

. . .

<br/>

Some quick examples,

::: {.small}
* `http://lmgtfy.com/?q=hello%20world`

* `http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=1600+Amphitheatre+Parkway`

* `https://nomnom-prod-api.dennys.com/mapbox/geocoding/v5/mapbox.places/raleigh,%20nc.json?`
   `types=country,region,postcode,place&country=us,pr,vi,gu,mp,ca`
:::


## URL encoding

This is will often be handled automatically by your web browser or other tool, but it is useful to know a bit about what is happening

* Spaces will encoded as '+' or '%20'

* Certain characters are reserved and will be replaced with the percent-encoded version within a URL

::: {.small}
| !   | #   | $   | &   | '   | (   | )   |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| %21 | %23 | %24 | %26 | %27 | %28 | %29 |
| *   | +   | ,   | /   | :   | ;   | =   |
| %2A | %2B | %2C | %2F | %3A | %3B | %3D |
| ?   | @   | [   | ]   |
| %3F | %40 | %5B | %5D |
:::

* Characters that cannot be converted to the correct charset are replaced with HTML numeric character references (e.g. a &#931; would be encoded as &amp;#931; )

## Examples



::: {.small}
```{r}
URLencode("http://lmgtfy.com/?q=hello world")
URLdecode("http://lmgtfy.com/?q=hello%20world")
```
:::

. . .

::: {.small}
```{r}
URLencode("! # $ & ' ( ) * + , / : ; = ? @ [ ]")
URLdecode(URLencode("! # $ & ' ( ) *  + , / : ; = ? @ [ ]"))
```
:::

. . .

::: {.small}
```{r}
URLencode("Σ")
URLdecode("%CE%A3")
```
:::


# RESTful APIs


## REST

*RE*presentational *S*tate *T*ransfer 

* describes an architectural style for web services (not a standard)

* all communication via HTTP requests

* Key features: 
    - client–server architecture
    - addressible (specific URL endpoints)
    - stateless (no client information stored between requests)
    - layered / hierarchical
    - cacheability


## HTTP Methods / Verbs

* *GET* - fetch a resource

* *POST* - create a new resource

* *PUT* - full update of a resource

* *PATCH* - partial update of a resource

* *DELETE* - delete a resource.

Less common verbs: *HEAD*, *TRACE*, *OPTIONS*.

::: {.aside}
Based on [HTTP: The Protocol Every Web Developer Must Know](http://code.tutsplus.com/tutorials/http-the-protocol-every-web-developer-must-know-part-1--net-31177)
:::

## Structure of an HTTP Request

<br/>

![](imgs/HTTP_Request.png){fig-align="center" width="100%"}

::: {.aside}
[Source](https://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html)
:::

## Structure of an HTTP Response

<br/>

![](imgs/HTTP_Response.png){fig-align="center" width="100%"}

::: {.aside}
[Source](https://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html)
:::

## Status Codes

* 1xx: Informational Messages

* 2xx: Successful

* 3xx: Redirection

* 4xx: Client Error

* 5xx: Server Error


# Example 1: <br/> [An API of Ice And Fire](https://anapioficeandfire.com/)


## Documentation

While there is a lot of standardization, every API is different and you will need to review the documentation of each.

See documentation [here](https://anapioficeandfire.com/Documentation) for AAOIF.

. . .

<br/>

**Resources / endpoints**:

- Root - `https://www.anapioficeandfire.com/api`
- List books - `https://www.anapioficeandfire.com/api/books`
- Specific book - `https://www.anapioficeandfire.com/api/books/1`
- ...


## Pagination

> An API of Ice And Fire provides a lot of data about the world of Westeros. To prevent our servers from getting cranky, the API will automatically paginate the responses. You will learn how to create requests with pagination parameters and consume the response.
>
> <br/>
>
> **Things worth noting**: <br/>
> Information about the pagination is included in the Link header Page numbering is 1-based You can specify how many items you want to receive per page, the maximum is 50
>
> <br/>
>
> **Constructing a request with pagination** <br/>
> You specify which page you want to access with the `?page` parameter, if you don't provide the `?page` parameter the first page will be returned. You can also specify the size of the page with the `?pageSize` parameter, if you don't provide the `?pageSize` parameter the default size of 10 will be used.


# Demo 1 - Basic access & pagination

# httr2

## Background

`httr2` is a package designed around the construction and handling of HTTP requests and responses. It is a rewrite of the `httr` package and includes the following features:

* Pipeable API

* Explicit request object, with support for

    * rate limiting
    
    * retries
    
    * OAuth
    
    * Secrure secret storage

* Explicit response object, with support for

    * error codes / reporting
    
    * common body encoding (e.g. json, etc.)


## request objects

A new request object is constructed via `request()` which is then modifed via `req_*()` functions

Some useful `req_*()` functions:

* `req_method()` - set HTTP method

* `req_url_query()` - add query parameters to URL

* `req_url_*()` - add or modify URL

* `req_body_*()` - set body content (various formats and sources)

* `req_user_agent()` - set user-agent

* `req_dry_run()` - shows the exact request that will be made


## response objects

A request is made via `req_perform()` which then returns a response object (the most recent response can also be retrieved via `last_response()`). Content of the response are accessed via the `resp_*()` functions

Some useful `resp_*()` functions:

* `resp_status()` - extract HTTP status code (`resp_status_desc()` for a text description)

* `resp_content_type()` - extract content type and encoding

* `resp_body_*()` - extract body from a specific format (json, html, xml, etc.)

* `resp_headers()` - extract response headers


## Example 2 - rottentomatoes.com

::: {.small}
```{r}
#| error: true
read_html("https://www.rottentomatoes.com")
```
:::

. . .

::: {.small}
```{r}
library(httr2)
req = request("https://www.rottentomatoes.com")
```

```{r}
#| message: true
req
```

```{r}
req |> req_user_agent()
```
:::


## Response

```{r}
#| echo: false
options(HTTPUserAgent = 'RStudio Desktop (2023.9.0.463); R (4.3.1 aarch64-apple-darwin23.0.0 aarch64 darwin23.0.0)')
```

:::: {.columns .small}
::: {.column width='50%'}
```{r}
(req_good = req |> 
  req_user_agent())
```
:::

::: {.column width='50%' .fragment}
```{r}
(req_bad = req |> 
  req_user_agent(options()$HTTPUserAgent))
```
:::
::::

. . .

:::: {.columns .small}
::: {.column width='50%'}
```{r}
(res_good = req_good |> req_perform())
```
:::

::: {.column width='50%' .fragment}
```{r}
#| error: true
req_bad |> req_perform()
```
:::
::::


## Response body

:::: {.columns .small}
::: {.column width='50%'}
```{r}
res_good |> resp_body_html()
```


:::

::: {.column width='50%'}
```{r}
res_good |> resp_body_string()
```
:::
::::



## Debugging base http via libcurl

::: {.small}
```{r}
#| eval: false
options(internet.info = 0)
readLines(url("https://www.rottentomatoes.com"))
```

```
* WARNING: failed to open cookie file ""
*   Trying 23.223.142.65:443...
* Connected to www.rottentomatoes.com (23.223.142.65) port 443 (#0)
* ALPN: offers h2,http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=Pennsylvania; O=Comcast Corporation; CN=*.rottentomatoes.com
*  start date: Mar 31 00:00:00 2023 GMT
*  expire date: Mar 30 23:59:59 2024 GMT
*  subjectAltName: host "www.rottentomatoes.com" matched cert's "*.rottentomatoes.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Organization Validation Secure Server CA
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: www.rottentomatoes.com]
* h2 [:path: /]
* h2 [user-agent: RStudio Desktop (2023.9.0.463); R (4.3.1 aarch64-apple-darwin23.0.0 aarch64 darwin23.0.0)]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x1573a0e00)
> GET / HTTP/2
Host: www.rottentomatoes.com
User-Agent: RStudio Desktop (2023.9.0.463); R (4.3.1 aarch64-apple-darwin23.0.0 aarch64 darwin23.0.0)
Accept: */*

< HTTP/2 403 
< mime-version: 1.0
< content-type: text/html
< content-length: 272
< cache-control: max-age=0
< expires: Fri, 13 Oct 2023 00:48:33 GMT
< date: Fri, 13 Oct 2023 00:48:33 GMT
* Added cookie akamai_generated_location="{"zip":"27601-27617+27619-27629+27634-27636+27640+27650+27656+27658+27661+27668+27675-27676+27690+27695+27697-27699","city":"RALEIGH","state":"NC","county":"WAKE","areacode":"919","lat":"35.7750","long":"-78.6336","countrycode":"US"}" for domain rottentomatoes.com, path /, expire 1699750113
< set-cookie: akamai_generated_location={"zip":"27601-27617+27619-27629+27634-27636+27640+27650+27656+27658+27661+27668+27675-27676+27690+27695+27697-27699","city":"RALEIGH","state":"NC","county":"WAKE","areacode":"919","lat":"35.7750","long":"-78.6336","countrycode":"US"}; expires=Sun, 12-Nov-2023 00:48:33 GMT; path=/; domain=.rottentomatoes.com; secure
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000 ; includeSubDomains
* Added cookie akacd_RTReplatform="2177452799~rv=88~id=c32ab070ed3d3c6433fcd5f301c4ab77" for domain www.rottentomatoes.com, path /, expire 2177452799
< set-cookie: akacd_RTReplatform=2177452799~rv=88~id=c32ab070ed3d3c6433fcd5f301c4ab77; path=/; Expires=Mon, 31 Dec 2038 23:59:59 GMT; Secure; SameSite=None
* The requested URL returned error: 403
* Connection #0 to host www.rottentomatoes.com left intact
Error in readLines(url("https://www.rottentomatoes.com")) : 
  cannot open the connection to 'https://www.rottentomatoes.com'
In addition: Warning message:
In readLines(url("https://www.rottentomatoes.com")) :
  cannot open URL 'https://www.rottentomatoes.com/': HTTP status was '403 Forbidden'
```
:::



# Demo 2 - httr2 + headers

## 

::: {.small}
```{r}
aaoif = function(
  resource = c("root", "books", "characters", "houses"), ..., 
  base_url = "https://www.anapioficeandfire.com/api/", verbose = TRUE
) {
  resource = match.arg(resource)
  
  if (resource == "root")
    resource = ""
  
  resp = request(base_url) |>
    req_url_path_append(resource) |>
    req_url_query(...) |>
    req_perform()
  
  full = list()
  page = 1
  
  repeat {
    if (verbose) message("Grabbing page ", page)
    full = c(full, resp_body_json(resp))
    
    links = get_links(resp)
    if (is.null(links[["next"]]))
      break
    
    resp = request(links[["next"]]) |>
      req_perform()
    
    page = page+1
  }
  
  full |>
    tibble(data = _) |>
    unnest_wider(data)
}
```
:::


## Exercise 1

Using the AAOIF API answer the following questions:

1. How many characters are included in this API?

2. What percentage of the characters are dead?

3. How many houses have an ancestral weapon?

```{r}
#| echo: false
countdown::countdown(5)
```

# Demo 3 - [GitHub API](https://docs.github.com/en/rest/overview)

