---
title: "Underrated Tidyverse Functions"
author: "Ted Laderas"
date: "12/1/2020"
output: 
  html_document:
    toc: true
    toc_float: true
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(tidyverse)
tweets <- read.csv("tweets_long.csv")
DT::datatable(tweets)
```

# The Assignment

I'm teaching an R Programming course next term. Jessica Minnier and I are developing the [Ready for R Materials](https://ready4r.netlify.app/labbook) into a longer and more involved course.

I think one of the most important things is to teach people how to self-learn. As learning to program is a lifelong learning activity, it's critically important to give them these meta-learning skills.  So that's the motivation behind the *Tidyverse function of the Week* assignment. 

I asked on Twitter:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Hi Everyone. I&#39;m teaching an <a href="https://twitter.com/hashtag/rstats?src=hash&amp;ref_src=twsrc%5Etfw">#rstats</a> course next quarter. <br><br>One assignment is to have each student write about a <a href="https://twitter.com/hashtag/tidyverse?src=hash&amp;ref_src=twsrc%5Etfw">#tidyverse</a> function. What it&#39;s for and an example.<br><br>What are some less known <a href="https://twitter.com/hashtag/tidyverse?src=hash&amp;ref_src=twsrc%5Etfw">#tidyverse</a> functions that do a job you find useful?</p>&mdash; Ted Laderas, PhD 🏳️‍🌈 (@tladeras) <a href="https://twitter.com/tladeras/status/1333480918158254082?ref_src=twsrc%5Etfw">November 30, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

## Some of my favorite suggestions

Here are some of the highlights from the thread.

I loved all of these. Danielle Quinn wins the MVP award for naming so many useful functions:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">dplyr::uncount()<br>tidyr::complete()<br>tidyr::fill() / replace_na()<br>stringr::str_detect() / str_which()<br>lubridate::ymd_hms() and related functions<br>ggplot2::labs() - so simple, yet under appreciated!</p>&mdash; Danielle Quinn (she/her) (@daniellequinn88) <a href="https://twitter.com/daniellequinn88/status/1333825406739345410?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

`fill()` was highly suggested:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">tidyr::fill() - extremely useful when creating a usable dataset out of a spreadsheet originally built for data entry, in which redundant informations are only reported once at the beginning of the group they refer to, rather than in every row as needed for the analysis.</p>&mdash; Luca Foppoli (@foppoli_luca) <a href="https://twitter.com/foppoli_luca/status/1333780245896335360?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

Many people suggested the window functions, including `lead()` and `lag()` and the cumulative functions:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Check out the dplyr window functions, cummin, cummax, cumany and cumall. They don&#39;t seen useful at first but they can solve really tricky aggregation problems. <a href="https://t.co/aDpXqSB2Vx">https://t.co/aDpXqSB2Vx</a></p>&mdash; Robert Kubinec (@rmkubinec) <a href="https://twitter.com/rmkubinec/status/1333763578428526593?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

Alison Hill suggested `problems()`, which helps you diagnose why your data isn't loading:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Ooh problems is a good function for importing rx <a href="https://t.co/P4ZR57PgOG">https://t.co/P4ZR57PgOG</a></p>&mdash; Alison Presmanes Hill (@apreshill) <a href="https://twitter.com/apreshill/status/1333602216318537728?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

I think that `deframe()` and `enframe()` are really exciting, since I do this operation all the time:

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">tibble::deframe(), tibble::deframe()<br>coercing a two-column df to named vector, which I prefer immensely to names(df) &lt;- vec_of_names</p>&mdash; E. David Aja (@PeeltothePithy) <a href="https://twitter.com/PeeltothePithy/status/1333563043649900544?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

`unite()`, `separate()` and `separate_rows()` also had their own contingent: 

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I find myself using tidyr::unite() a lot to clean messy data - particularly useful for making unique and informative ID&#39;s for each row. coalesce() and fill() are also little known gems! :)</p>&mdash; Guy Sutton🐝🌾🇿🇦🇿🇼 (@Guy_F_Sutton) <a href="https://twitter.com/Guy_F_Sutton/status/1333668370668003329?ref_src=twsrc%5Etfw">December 1, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


## Wow! Let's Grab All the Tweets and Replies

I was bowled over by all of the replies. This was an unexpectedly really fun thread, and lots of recommendations from others. 

I thought I would try and summarize everyone's suggestions and compile a list of recommended functions. I used [this script](https://github.com/gmellini/twitter-scraper) with some modifications to pull all the replies to my tweet. In particular, I had to request for `extended` tweet mode, and I extracted a few more fields from the returned JSON.

This wrote the tweet information into a CSV file.

Then I started parsing the data. I wrote a couple of functions, `remove_users_from_text()`, which removes the users from a tweet (by looking for words that begin with `@`) and `get_funcs()`, which uses a relatively simple regular expression to try to return the function (it looks for paired parentheses `()` or an underscore "-" to extract the functions). It actually works pretty well, and grabs most of the functions.

Then I use `separate_rows()` to split the multiple functions into their separate rows. This makes it easier to tally all the functions.

```{r}
remove_users_from_text <- function(col){
  str_replace_all(col, "\\@\\w*", "")
  
}

get_funcs <- function(col){
  out <- str_extract_all(col, "\\w*\\(\\)|\\w*_\\w*")
  paste(out[[1]], collapse=", ")  
}

parsed_tweets <- tweets %>%
  rowwise() %>%
  mutate(text = remove_users_from_text(text)) %>%
  mutate(funcs = get_funcs(text)) %>%
  ungroup() %>%
  separate_rows(funcs, sep=", ") %>%
  select(date, user, funcs, text, reply, parent_thread) %>%
  distinct()

write_csv(parsed_tweets, file = "cleaned_tweets_incomplete.csv")

knitr::kable(parsed_tweets[1:10,-c(5:6)])
```

At this point, I realized that I just needed to hand annotate the rest of the tweets, rather than wasting my time trying to parse the rest of the cases. So I pulled everything into Excel and just annotated the ones which I couldn't pull from.

## Functions by frequency

Here are the function suggestions by frequency. Unsurprisingly, `case_when()` (which I cover in the main course), has the most number of suggestions, because it's so useful. `tidyr::pivot_wider()` and `tidyr::pivot_longer()` are also covered in the course.


There are some others which were new to me, and a bit of a surprise, such as `coalesce()`, `fill()`. 


```{r}
cleaned_tweets <- read_csv("cleaned_tweets.csv") %>% select(-parent_thread) %>%
  mutate(user = paste0("[",user,"](",reply,")")) %>%
  select(-reply)

functions_by_freq <- cleaned_tweets %>%
  janitor::tabyl(funcs) %>%
  filter(!is.na(funcs)) %>%
  arrange(desc(n)) 

write_csv(functions_by_freq, "functions_by_frequency.csv")

functions_by_freq %>%
  knitr::kable()
```


## Cleaned Tweets and Threads

Here's all of the tweets from this thread (naysayers included). They are in somewhat order (longer threads are grouped).

Here's a [link to the cleaned CSV file](cleaned_tweets.csv)

```{r message=FALSE}


knitr::kable(cleaned_tweets)
```

## Source Code and Data

Feel free to use and modify. 

- [RMarkdown file](index.Rmd) used to generate this post
- [Python Twitter Scraper (by Giovanni Mellini)](https://github.com/gmellini/twitter-scraper) - I used this because there wasn't a ready made recipe in `rtweet` to extract replies - you have to use recursion to extract all of the thread replies that belong to a tweet, and this was easily modifiable. 
- [Cleaned Tweets File (CSV)](cleaned_tweets.csv)


## Thank You

This post is my thank you for everyone who contributed to this thread. Thank you!

