bongo.ai hex stickerbongo.aiHack Your Data

Day 4 · 11:15 · 90 min

Social and web intelligence

Scraping, listening, and not fooling yourself

Day 4 · 11:15

The open web is a dataset: brand mentions, competitor prices, app-store reviews. Scraping it is easy. Not fooling yourself with it is the craft.

You will leave able to

  • Know what you are allowed to collect, and what you are not.
  • Tidy a mentions extract into tone × brand.
  • Build a listening pipeline that a communications director can run weekly.

On the desk

The ancestor of this session used Twitter’s API. The API has changed; the question has not: what is the public saying, in what tone, and is it concentrated enough to act on? You will work from an anonymised extract of 400 posts across X, LinkedIn and Nairaland, mentioning four teaching brands.

A tidy listening extract

Download brand-mentions.csv· 400 rows, four brands, four tones
R
library(tidyverse)
library(tidytext)
 
mentions <- read_csv("brand-mentions.csv")
 
mentions %>%
  count(brand, tone) %>%
  ggplot(aes(brand, n, fill = tone)) +
  geom_col() +
  labs(
    title = "Tone mix, 14-day window",
    subtitle = "Anonymised public mentions — not a census of customers",
    y = "Posts"
  )
Tone mix of 400 anonymised brand mentions. Northridge Bank's complaint bar is the one a communications director should take to EXCO — not the rumour count.

Northridge Bank’s complaint bar is the one that should travel to EXCO. Rumour volume is a communications issue; complaint volume is an operations issue. Mixing them in a ‘sentiment score’ is how a bank spends the week chasing a rumour while the app still crashes at payday.

From posts to words

R
mentions %>%
  filter(tone == "complaint") %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words, by = "word") %>%
  count(brand, word, sort = TRUE) %>%
  group_by(brand) %>%
  slice_head(n = 6)

A weekly pipeline, not a hero scrape

  1. Export (API or vendor) into data/raw/YYYY-MM-DD.csv. Never overwrite.
  2. A Quarto brief reads only that folder, with a date parameter.
  3. Tone is either human-coded on a sample, or a model you have checked against a gold sample of 200 posts. Off-the-shelf sentiment lexicons were not built on Nigerian English.
  4. The output is a page on the briefing site, not a screenshot in a WhatsApp group.

Exercise 4.2

Rumour versus complaint

Using brand-mentions.csv, compute the complaint share and the rumour share for each brand. Which brand is a CX problem? Which is a comms problem? Write one sentence each for the CMO and the COO.

All of Day 4