Reading Data into R

Importing data feels like the trivial step before the real work, and it is where a surprising share of analytical errors are born. The reason is that R almost never refuses to import a file. It makes a decision about what your data mean, tells you nothing, and carries on. A column of numbers arrives as text, a missing value code of -99 is treated as a real observation, and everything downstream is quietly wrong.

This chapter covers the formats you will actually meet and, more importantly, the check that catches the failures.

CSV, the ordinary case

A comma-separated (CSV) file is plain text with one row per line and commas between fields. It is the format most political science data arrive in, and R reads it out of the box.

fl <- read.csv("data/FearonLaitin2003.csv")

There is also a tidyverse version, read_csv(), with an underscore rather than a period.

library(tidyverse)
fl <- read_csv("data/FearonLaitin2003.csv")

Three differences matter in practice. read_csv() is faster on large files, it prints a summary of the column types it guessed, which is useful information rather than clutter, and it returns a tibble rather than a data frame. A tibble is a data frame with better printing behavior, and everything you learned in the previous chapter applies to it unchanged.

Either is acceptable in this course. The handouts use read.csv() because it needs no package, so the first line of a chapter always runs.

Straight from a web address

Both functions accept a URL wherever they accept a filename, which means data can be loaded with nothing downloaded.

fl <- read.csv("https://raw.githubusercontent.com/babakrezaee/AppliedAI-PoliSci/main/data/FearonLaitin2003.csv")

This is how every dataset in these handouts is loaded. It has two advantages worth naming: the code runs identically on any machine, and there is no file path to get wrong.

It has one disadvantage. If the address ever changes, the code stops working, and if the file behind the address is edited, your results change without your knowing. For the final project, download the file once and put it on your cloud-based platform, then you have control over your data source than relying on someone else’s server.

Note the address on Github must be the raw one. Clicking a CSV on GitHub shows a formatted preview page, and its address returns HTML rather than data. The Raw button gives the address you want, and it always begins raw.githubusercontent.com. You can also upload your files on Dropbox, Googledrive, and Onedrive; just make sure that you have the permission to put the files on these public cloud-based platforms.

Check the import, every time

This is the habit the chapter exists for. Three lines, immediately after loading anything.

dim(fl)
str(fl)
summary(fl)

You are looking for three things.

Did every column get the type you expected? A numeric variable listed as chr in the str() output has a non-numeric value hiding in it somewhere.

Are the row and column counts right? If the file should have 6,610 rows and you have 6,609, the header row was probably read as data, or the last line lacks a line break.

Do the ranges make sense? summary() reports the minimum and maximum of each numeric column. A polity2 score of -88 or a population of -99 is a missing value code, not an observation, and it will wreck any model that takes it seriously.

Missing value codes

Older datasets frequently encode missing values as numbers, with -99, -999, and -88 being the common choices. R has no way of knowing this and will treat them as real.

Tell it explicitly at import.

dat <- read.csv("data/survey.csv", na.strings = c("NA", "", "-99", "-999"))

Or fix it afterward, which is clearer when only some columns are affected.

dat$income[dat$income == -99] <- NA

Either way, do it before anything else. A mean computed over a column containing -99 is not slightly wrong, it is meaningless, and nothing in the output will tell you.

When the commas are not commas

Files exported from Excel on a European system are often separated by semicolons rather than commas, and they use commas as decimal points. Opening one with read.csv() produces a single column of text.

dat <- read.csv2("data/dutch_export.csv")   # semicolons, comma decimals
dat <- read_csv2("data/dutch_export.csv")   # tidyverse equivalent

Worth knowing here rather than elsewhere, since Dutch-locale Excel does this by default and several of you will hit it with data you export yourselves.

If the separator is something else entirely, such as a tab, say so:

dat <- read.delim("data/file.txt")                    # tab separated
dat <- read.csv("data/file.txt", sep = "|")           # anything else

The fastest way to diagnose any of this is to open the file in a plain text editor and look at the first two lines. Two minutes of looking beats an hour of guessing.

Accented characters

Country names, party names, and any text data will contain characters outside plain English. If they arrive mangled, as é where é belongs, the file’s encoding was misread.

dat <- read.csv("data/parties.csv", encoding = "UTF-8")

UTF-8 is correct for the large majority of files you will meet. This will matter in Week 4, when text becomes data.

Excel files

Excel files are not text and cannot be read by read.csv(). Use readxl.

# install.packages("readxl")
library(readxl)

dat <- read_excel("data/data.xlsx")
dat <- read_excel("data/data.xlsx", sheet = "Coding")   # a named sheet
dat <- read_excel("data/data.xlsx", skip = 3)           # skip title rows

skip earns its place. Spreadsheets kept by humans very often carry a title, a blank line, and a note before the actual header row, and without skip all of that is read as data.

A note on Excel more generally. It is a reasonable place to receive data and a poor place to edit it, because changes made by hand leave no record. If a value must change, change it in R, where the change is visible in your code and reversible.

Stata and SPSS

Political science data arrive as .dta and .sav more often than anyone would like. The haven package reads both.

# install.packages("haven")
library(haven)

dat <- read_dta("data/anes.dta")   # Stata
dat <- read_sav("data/ess.sav")    # SPSS

Older tutorials recommend foreign and Hmisc for this. Both still work, and haven handles recent file versions and preserves variable labels better, so prefer it.

Labelled columns from these formats sometimes behave unexpectedly in models. If one gives you trouble, strip the labels:

dat$party <- haven::zap_labels(dat$party)

Saving

To hand data to someone who may not use R, write a CSV.

write.csv(dat, "data/processed/cleaned.csv", row.names = FALSE)

row.names = FALSE matters. Without it, R adds a column of row numbers, and a file that is written and read a few times accumulates several of them.

To save an R object for your own later use, including model objects and anything else a CSV cannot hold, use saveRDS().

saveRDS(model, "output/model.rds")
model <- readRDS("output/model.rds")

This is worth knowing for the final project, or algorithms that take a longer time to run, such as Random Forest. A model that takes several minutes to fit can be fitted once, saved, and read back, so that re-rendering the document does not refit it.

Exercise

About twenty minutes. Work in the ai-practice project from the previous chapter.

1. Load the Fearon and Laitin data from the raw URL and confirm its dimensions.

2. Run str() on it. List two columns that arrived as text. For each, say whether that is correct or whether it should have been numeric.

3. Run summary(fl$polity2). Look at the minimum. Polity scores are defined on a range of minus ten to plus ten. Is anything outside it, and if so, what do you think those values mean?

4. Make a deliberately awkward file. In the console:

   messy <- data.frame(
     country = c("França", "Türkiye", "España"),
     score   = c(1.5, -99, 3.2)
   )
   write.csv(messy, "data/messy.csv", row.names = FALSE)

   Read it back with read.csv("data/messy.csv"), then compute mean(score). Now read it again treating -99 as missing, and compute the mean a second time. Report both numbers and say which one you would put in a paper.

5. Check whether the accented characters survived the round trip. If they did not, read the file again with encoding = "UTF-8".

Step 4 is the point. Two defensible-looking numbers come out of the same file, and only one of them means anything. Nothing in R’s output distinguishes them for you.