Do I need to
bring a coat?

What's the weather like?

Location: Berlin
Date: 2019-09-24
Temperature: 15°

Is it cold in Berlin today?

Yes, you'll need a coat

Should I take a coat?

✓

Big Data !=
Big Information

Why should you care about taming your data?

Understand what information is contained in your data

Discover what's really valuable about your data

Learn how to structure and organise it effectively

Structure of the talk

1. Theory: Information entropy
2. Practice: Tame your data for analysis

Information Entropy

So how do we measure choice or uncertainty?

How can knowing the distribution of responses reduce the amount of data we need to communicate?

Sending one toss at a time

$$\begin{align} \text{Average bits per toss} = & P(H) \times 1 \text{ bit} + \\ & P(T) \times 1 \text{ bit} \\ = & 1 \text{ bit} \end{align}$$

Information Entropy (motivation)

Tells us the theoretical limit for compression

The irreducible information content of a source/ variable

A way of thinking about how valuable data is

Information Content

Bits of information required to communicate an outcome$$\begin{aligned} I(\text{outcome}) &= log_2 \frac{1}{P(\text{outcome})} \\ &= - log_2 P(\text{outcome}) \end{aligned}$$

Information Entropy

Average bits of information per outcome$$H(\text{source}) = \sum_{\text{outcome} \in \text{source}} {-P(\text{outcome}) \log_2 {P(\text{outcome})}}$$

Information Entropy (interpretation)

• more information requires more data
• more data doesn't mean more information
• rare things are more informative
• data is part of a conversation
• add value by resolving uncertainty

More information
requires
more data

Retain precision

Keep more decimal places than you need to show

Don't convert continuous variables (numbers) to discrete ones (categories)

You can't unsimplify

More data doesn't
mean more information

Normalisation

A way to structure your data to reduce redundancy

1NF: one value per cell

2NF: cells depend on the whole key

3NF: cells only depend on the key

What data scientists want...

Tidy Data

Orthongonality

Informative variables correlate with the objective
but not with each other

Rarity is
informative

Visualise your probability distributions

Data is part of
a conversation

Start by defining the question

Defining the Question...

What decision needs to be taken?

If in doubt, follow the money!

Incoporate Context

e.g. Unitless measures

Provide Metadata

Describe and explain your data

Aid discovery and interpretation

Track provenance

Handles to manipulate your data

Don't put data in the keys

{
  "2017":  8000,
  "2018": 10000,
  "2019": 15000
}

Data frames are easier to manipulate

[
  { "date": "2017", "pageviews":  8000 },
  { "date": "2018", "pageviews": 10000 },
  { "date": "2019", "pageviews": 15000 }
]

Add value by
resolving
uncertainty

Information Entropy

• more information requires more data - retain precision, link-up data
• more data doesn't mean more information - normalised tidy-data, mece, orthogonality
• rare things are more informative - visualise distributions
• data is part of a conversation - start by defining the question/ decision, incorporate context, use metadata
• add value by resolving uncertainty