Brendan Cooley

Read and Clean ICEWS Coded Event Data

2018-04-11T00:00:00+00:00

Last updated 11 April 2018. Lightly edited for reproducibility 30 November 2025.

Datasets that researchers use to measure conflict in international relations are generally coarse. The Correlates of War (COW) Project’s</a> Militarized Interstate Dispute</a> data records threats, displays, and uses of military force between 1816 and 2010 and COW’s interstate war</a> data records conflicts which resulted in at least 1,000 battle deaths. Both MIDs and wars are exceedingly unusual. Yet the “stuff” of international relations happens every day. Governments are bargaining and communicating all the time – sometimes cooperatively and sometimes conflictually. These interactions almost certaintly contain information about their proclivity to experience armed conflict. New data might help us measure and understand this “stuff” better.

In recent years, several very-large-n (>1,000,000 observation) dyadic event datasets have become available for public use. An “event” takes the form of “[actor x] undertook [action z] toward [actor y] on [date w].” Natural language processors scrape newswires and map events into preexisting event and actor ontologies. The Integrated Crisis Early Warning System (ICEWS)</a> is one such dataset. You can find a nice discussion of the project’s history by Phil Shrodt here</a>. Andreas Beger</a> and David Masad</a> have nice writeups on what the data look like. It’s still pretty rare to see these data used in political science, however. See Gallop (2016), Minhas, Hoff, and Ward (2016), and Roberts and Tellez (2017) for notable exceptions.1</a>

This may be because it’s still a little tricky to get these data into a format suitable for empirical analyses. Having struggled myself to clean ICEWS, I figured it’d be worth sharing my experience (working in R).

Note: This post is an illustrative tutorial showing the workflow for processing ICEWS data. The code examples below are for demonstration purposes. For the complete, executable implementation, see the icews-clean</a> repository on GitHub.

I show three steps in the process here:

Grabbing the data from dataverse</li>
Converting it ‘reduced’ form with conflict cooperation scores and COW codes, employing Phil Shrodt’s software</a></li>
Converting the ‘reduced’ data into date-dyad counts</li> </ol>
First, get the environment setup:
packages <- c('dataverse', 'dplyr', 'zoo', 'lubridate', 'tidyr', 'bibtex', 'knitcitations') lapply(packages, require, character.only = TRUE) write.bib(packages) bib <- read.bib('Rpackages.bib') </code></pre> Then, grab the data straight from Harvard’s dataverse
Sys.setenv("DATAVERSE_SERVER" = "dataverse.harvard.edu") doi <- "doi:10.7910/DVN/28075" dv <- get_dataset(doi) </code></pre> We can take a look at the files included in the dataverse repository
dvFiles <- dv$files$label dvFiles </code></pre> We want to get the files prefixed with ‘events.’ We’ll unzip them and put them in a folder called /rawICEWS.
dir.create('rawICEWS', showWarnings = FALSE) # search through all files in dataverse repo for (i in dvFiles) { # for those that start with events if (substr(i, 1, 6) == 'events') { dest <- paste0('rawICEWS/', i) writeBin(get_file(i, doi), dest) # unzip those that are compressed if (substr(i, nchar(i) - 2, nchar(i)) == 'zip') { unzip(dest, exdir='rawICEWS/') file.remove(dest) # trash zipfile } } } # store list of files in .txt fNames <- paste0('rawICEWS/', list.files('rawICEWS')) lapply(fNames, write, 'fNames.txt', append=TRUE) </code></pre> This will take a minute or two. Go get some coffee. The nice thing about this code is that it should dynamically grab new data as the ICEWS project uploads it to dataverse.
Either way, now we have all of our raw data ready to go sitting in /rawICEWS. The raw ICEWS data is clunky on several dimensions. Phil Shrodt provides software to get it into a format that looks recognizable to empirical international relations researchers. If you’re interested in the machinery, check it out here</a>.
For our purposes, we just need the script text_to_CAMEO.py</code> and the ontology files agentnames.txt</code> and countrynames.txt</code>. It’ll take the list of filenames and convert them into “reduced” form. Just navigate to the current working directory and run
python text_to_CAMEO.py -c -t fNames.txt </code></pre> After this you should have a bunch of .txt files sitting in your working directory. Move them to a /reducedICEWS folder and you can load these into memory and get to work.# helper to replace empty cells with NAs empty_as_na <- function(x) { ifelse(as.character(x)!="", x, NA) } reducedFiles <- list.files("reducedICEWS") events.Y <- list() # list holding data frames for each year # for each of the reduced files for (i in 1:length(reducedFiles)) { # append to list events.Y[[i]] <- read.delim(paste('reducedICEWS/', reducedFiles[i], sep=""), header = F) # convert column names colnames(events.Y[[i]]) <- c("date", "sourceName", "sourceCOW", "sourceSec", "tarName", "tarCOW", "tarSec", "CAMEO", "Goldstein", "quad") # replace empty cells with NAs events.Y[[i]] %>% mutate_if(is.factor, as.character) %>% mutate_all(funs(empty_as_na)) %>% as_tibble() -> events.Y[[i]] } # bind everything together events <- bind_rows(events.Y) # add year and month fields events$month <- as.yearmon(events$date) events$year <- year(events$date) events <- events %>% select(date, year, month, everything()) </code></pre> This will take a little while too (this is BIG data, people!). But once it’s done, we can take a look at what we’ve got: head(events) </code></pre> Now we have COW-coded countries (sourceCOW, tarCOW), the date actors within these countries interacted, and the sector identity of the actors (the data includes government-government interactions but also interactions between subnational actors). The CAMEO/Goldstein/quad fields describe the nature of the interaction under different event categorization systems. See the Shrodt documentation for more detail on the event and sector classifications. The quad score (ranging from 1-4) classifies events as either conflictual or cooperative and as verbal or material in nature. A score of 1 corresponds to verbal cooperation. So the first event can be read as follows: “On Jan. 1, 1995, rebels in Russia verbally cooperated with the government in Russia.” The CAMEO scores provide a much richer event classification. The first question we almost always want to answer – how many times did each pair of countries experience each type of interaction with each other? The following function returns the directed dyadic matrix of counts, aggregated by ‘year’ or ‘month’. event.counts <- function(events, agg.date=c('month', 'year'), code=c('quad', 'CAMEO')) { counts <- events %>% group_by_(agg.date, 'sourceCOW', 'tarCOW', code) %>% summarise(n = n()) %>% ungroup() # this seems trivial but screws up a lot of stuff if you don't do it output <- spread_(counts, code, 'n') output[is.na(output)] <- 0 return(output) } counts <- event.counts(events, 'year', 'quad') head(counts) </code></pre> COW code 0 is international organizations (IOs), so we can get a sense of how many times IOs are interacting with themselves and other countries in the data from the top of this table. And now that we have the count data we can start doing fun stuff. Over here</a> I’ve built an app that allows users to scale subsets of the data, following Lowe</a>.2</a> The procedures implemented there allow us to recover what can be thought of as a conflict-cooperation “score” for each dyad-year, based on the underlying count data. R Packages</h3> Leeper TJ (2017). dataverse: R Client for Dataverse 4. R package version 0.2.0. Wickham H, Francois R, Henry L and Müller K (2017). dplyr: A Grammar of Data Manipulation. R package version 0.7.4, <URL: https://CRAN.R-project.org/package=dplyr</a>>. Zeileis A and Grothendieck G (2005). “zoo: S3 Infrastructure for Regular and Irregular Time Series.” Journal of Statistical Software, 14(6), pp. 1-27. doi: 10.18637/jss.v014.i06 (URL: http://doi.org/10.18637/jss.v014.i06</a>). Grolemund G and Wickham H (2011). “Dates and Times Made Easy with lubridate.” Journal of Statistical Software, 40(3), pp. 1-25. <URL: http://www.jstatsoft.org/v40/i03/</a>>. Wickham H and Henry L (2018). tidyr: Easily Tidy Data with ‘spread()’ and ‘gather()’ Functions. R package version 0.8.0, <URL: https://CRAN.R-project.org/package=tidyr</a>>. Francois R (2017). bibtex: Bibtex Parser. R package version 0.4.2, <URL: https://CRAN.R-project.org/package=bibtex</a>>. Boettiger C (2017). knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.8, <URL: https://CRAN.R-project.org/package=knitcitations</a>>. References</h3> Gallop, Max B. 2016. “Endogenous networks and international cooperation.” Journal of Peace Research 53 (3):310–24. Minhas, Shahryar, Peter D Hoff, and Michael D Ward. 2016. “A new approach to analyzing coevolving longitudinal networks in international relations.” Journal of Peace Research 53 (3):491–505. Roberts, Jordan, and Juan Tellez. 2017. “Freedom House ’s Scarlet Letter: Assessment Power through Transnational Pressure.” ^{1 Please feel free point out others I’m missing! </div> ^{2 This is still in-progress, I’ve been meaning to go back and update it.] </div>}}

Calculating Historical Intercapital Distances 2018-01-10T00:00:00+00:00 Last updated 10 January 2018. Lightly edited for reproducibility 14 September 2025. Military power degrades with distance. Fighting a war on another continent is, for most militaries, more difficult than fighting at home. Studies of conflict frequently employ capital-to-capital distance (or some transformation of this metric) as one proxy for this loss of strength from power projection (see, for example, (Gartzke and Braithwaite (2011)). Combined with data on territorial contiguity</a>, these metrics can provide us with a picture of the geographic constraints facing countries contemplating war with one another. Historical intercapital distance data proved difficult to find however. Most researchers appear to rely on EuGene</a> to generate this data. I wasn’t able to track down any documentation on exactly how EuGene does this, however.1</a> Gleditsch and Ward (2001) generated a minimum interstate distance dataset, but their data only covers the post-1875 period. For researchers using Correlates of War</a> data, distance data would ideally cover 1816 to the present. It turns out that it’s not too difficult to build intercapital distance data from scratch, however. All that is required is data on the names of capital cities for each state system member for every year between 1816 and the present. Paul Hensel’s ICOW Historical State Names dataset</a> provides this information. We can then use Google’s Geocoding API</a> through the ggmap</code> R package to get the coordinates of each historical capital, which can then be used to generate intercapital distance matrices. Here, I show how to conduct this exercise in R. A clean version of Hensel’s historical capital dataset is available here</a>. I’ve included all of the data and software necessary to generate this data on Github</a>. Feel free to send along questions, comments, or suggestions for improvement to bcooley@princeton.edu</a>. Capital City Coordinates</h4> Start by loading up the packages we’ll need for analysis: # Install missing packages automatically required_packages <- c('readr', 'dplyr', 'tidyr', 'geosphere', 'ggplot2', 'maps', 'knitr') missing_packages <- required_packages[!required_packages %in% installed.packages()[,"Package"]] if(length(missing_packages) > 0) { install.packages(missing_packages, repos = "https://cran.rstudio.com/") } # Load all required libraries library(readr) library(dplyr) library(tidyr) library(geosphere) library(ggplot2) library(maps) library(knitr) </code></pre> I started by cleaning up Hensel’s data a bit in excel. Each capital city is listed as its own observation, along with the country, its COW code, and the first and last year the city served as a capital. Because the Hensel data is coded annually, I take the country’s capital at the start of any given year to be its capital for that entire year. For countries that no longer exist (e.g. Mecklenburg-Schwerin) I provide a contemporary alternative country name (aName) to help Google locate the city’s coordinates. The assumption underlying this procedure is that the cities that served as capitals historically have not moved from their historical location (if I’m missing any instances where this occured please let me know). We can take a look at the data below: # Load data directly from GitHub repository capitals_url <- "https://raw.githubusercontent.com/brendancooley/intercapital-distances/master/capitals.csv" dists <- read_csv(capitals_url, show_col_types = FALSE) # Hensel's data run from 1800-2016, set bounds dists$startDate <- ifelse(is.na(dists$startDate), 1800, dists$startDate) dists$endDate <- ifelse(is.na(dists$endDate), 2016, dists$endDate) head(dists) </code></pre> ## # A tibble: 6 × 6 ## ccode Name aName startDate endDate Capital ## <dbl> <chr> <chr> <dbl> <dbl> <chr> ## 1 2 United States of America <NA> 1800 2016 Washington D.C. ## 2 20 Canada <NA> 1800 1841 Ottawa ## 3 20 Canada <NA> 1841 1843 Kingston ## 4 20 Canada <NA> 1843 1849 Montreal ## 5 20 Canada <NA> 1849 1859 Toronto ## 6 20 Canada <NA> 1859 1865 Quebec City </code></pre> For this tutorial, I’ll use the pre-geocoded dataset that already contains latitude and longitude coordinates for all capital cities. This saves time and avoids the need for Google API keys. In practice, you would use ggmap</code>’s geocode</code> function to get coordinates, but that requires API setup and rate limiting. # Load the pre-geocoded dataset with coordinates geocoded_url <- "https://raw.githubusercontent.com/brendancooley/intercapital-distances/master/dists.csv" dists <- read_csv(geocoded_url, show_col_types = FALSE) # Clean up any missing values dists$startDate <- ifelse(is.na(dists$startDate), 1800, dists$startDate) dists$endDate <- ifelse(is.na(dists$endDate), 2016, dists$endDate) # Check that all cities have coordinates cat("Number of cities with missing coordinates:", sum(is.na(dists$lat)), "\n") </code></pre> ## Number of cities with missing coordinates: 0 </code></pre> cat("Total number of capital city records:", nrow(dists), "\n") </code></pre> ## Total number of capital city records: 253 </code></pre> We can verify the geocoding worked correctly by visualizing all capital cities on a world map: Historical Capital Cities Map</figcaption> </figure> The geocoded capitals are available as the dists</code> dataset we just loaded. You can find the original data here</a>. Intercapital Distances</h4> Remember that the point of all this was to generate intercapital distance data for all countries in the COW data. To do this, we want to convert our geocoded capital city data into a dyadic time series that gives the distance between any two countries’ capitals for a given year. If there are $N</code>$ countries in the system in a given year $t</code>$, we want to be able to generate an $N \times N</code>$ matrix for that year where each entry is the distance between countries $i</code>$ and $j</code>$. I start by loading up COW’s state system membership data, so we know which countries were members of the system for each year. The field “styear” denotes the year the country entered the system, and the field “endyear” gives the year it exited. I then append this data to the distance data. The output is shown below. # get state system membership (COW) sysMemUrl <- "https://correlatesofwar.org/wp-content/uploads/states2016.csv" sysMem <- read_csv(sysMemUrl, show_col_types = FALSE) %>% select(ccode, styear, endyear) dists <- left_join(dists, sysMem, by="ccode") head(dists) </code></pre> ## # A tibble: 6 × 10 ## ccode Name aName startDate endDate Capital lat lng styear endyear ## <dbl> <chr> <chr> <dbl> <dbl> <chr> <dbl> <dbl> <dbl> <dbl> ## 1 2 United State… <NA> 1800 2016 Washin… 38.9 -77.0 1816 2016 ## 2 20 Canada <NA> 1800 1841 Ottawa 45.4 -75.7 1920 2016 ## 3 20 Canada <NA> 1841 1843 Kingst… 44.2 -76.5 1920 2016 ## 4 20 Canada <NA> 1843 1849 Montre… 45.5 -73.6 1920 2016 ## 5 20 Canada <NA> 1849 1859 Toronto 43.7 -79.4 1920 2016 ## 6 20 Canada <NA> 1859 1865 Quebec… 46.8 -71.2 1920 2016 </code></pre> From this dataframe, I can build the $N \times N</code>$ intercapital distance matrix for any year. I wrap this in a function coord2DistM</code>, which takes the desired year and the dists dataframe as arguments. It filters the distance data to include only states that were active in that year. It then feeds the latitude and longitude coordinates to the distm</code> function from the geosphere</code> package, which retuns the desired distance matrix. I convert all distances to kilometers. The function distM2dydist</code> takes this distance matrix and a pair of COW country codes and returns the dyadic distance. Below, I show how to use these functions to build the intercapital distance matrix for the year 1816 and get the distance between Britain and Saxony. # Note: assigns capital to city that was capital at beginning of year coord2DistM <- function(dists, year) { # filter by system membership, then relevant capital distsY <- dists %>% filter(styear <= year, endyear >= year) %>% filter(startDate < year & endDate >= year) # check that one capital returned per country if (length(unique(distsY$Capital)) != length(unique(distsY$Name))) { print('error: nCountries != nCapitals, check underlying coordinate data') } else { # get distance matrix for selected year latlng <- distsY %>% select('lng', 'lat') %>% as.matrix() distsYmatrix <- distm(latlng, latlng, fun=distVincentySphere) distsYmatrix <- distsYmatrix / 1000 # convert to km rownames(distsYmatrix) <- colnames(distsYmatrix) <- distsY$ccode return(distsYmatrix) } } # get distance between i and j distM2dydist <- function(distM, ccode1, ccode2) { return(distM[ccode1, ccode2]) } # application year <- 1816 Britain <- "200" Saxony <- "269" distM1816 <- coord2DistM(dists, year) distM2dydist(distM1816, Britain, Saxony) </code></pre> ## [1] 965.3628 </code></pre> These functions can be used in tandem to grab intercapital distances for arbitrary year, dyad pairings. Merge with COW War Data</h4> Conflict researchers often want this data to analyze wars. Now I show how to merge intercapital distance data with COW’s inter-state-war data. I clean up the COW war data a bit, which you can see below post-cleaning. In the most basic leve, the data give information about the belligerents in every war and how long each war lasted. # append to COW wars data warUrl <- "https://correlatesofwar.org/wp-content/uploads/Inter-StateWarData_v4.0.csv" cowWars <- read_csv(warUrl, show_col_types = FALSE) # for simplicity, ignore armistices cowWars$StartYear <- cowWars$StartYear1 cowWars$EndYear <- ifelse(cowWars$EndYear2 == -8, cowWars$EndYear1, cowWars$EndYear2) cowWars <- cowWars %>% select(WarName, ccode, StateName, Side, StartYear, EndYear) head(cowWars) </code></pre> ## # A tibble: 6 × 6 ## WarName ccode StateName Side StartYear EndYear ## <chr> <dbl> <chr> <dbl> <dbl> <dbl> ## 1 Franco-Spanish War 230 Spain 2 1823 1823 ## 2 Franco-Spanish War 220 France 1 1823 1823 ## 3 First Russo-Turkish 640 Ottoman Empire 2 1828 1829 ## 4 First Russo-Turkish 365 Russia 1 1828 1829 ## 5 Mexican-American 70 Mexico 2 1846 1847 ## 6 Mexican-American 2 United States of America 1 1846 1847 </code></pre> We want to know the intercapital distance between each pair of belligerents between 1816 and the present. We could use the coord2DistM</code> and distM2dydist</code> but this would require calculating the intercapital distance matrix for every year in which there was a war. A simpler solution is to build a dataframe of capital-years, along with their coordinates, merge this with the war data, and calculate the distance between belligerents, given their capitals’ coordinates. I used the procedure in this stackoverflow post</a> to create the capital-year dataframe. Once merged, I use geosphere</code>’s distVicentySphere</code> function to calculate the distances. The resulting data is shown below the code. # convert capital data to yearly observations # https://stackoverflow.com/questions/28553762/expand-year-range-in-r distsYear <- dists distsYear$year <- mapply(seq, distsYear$startDate, distsYear$endDate, SIMPLIFY=FALSE) distsYear <- distsYear %>% unnest(year) %>% select(ccode, year, lat, lng) # get capital in start year cowWars$year <- cowWars$StartYear # append coords for each side cowWars1 <- cowWars %>% filter(Side == 1) %>% left_join(distsYear, by=c("ccode", "year")) %>% rename(State1 = StateName, ccode1 = ccode, lat1 = lat, lng1 = lng) %>% select(-Side) cowWars2 <- cowWars %>% filter(Side == 2) %>% left_join(distsYear, by=c("ccode", "year")) %>% rename(State2 = StateName, ccode2 = ccode, lat2 = lat, lng2 = lng) %>% select(WarName, State2, ccode2, lat2, lng2) cowWarsDyadic <- left_join(cowWars1, cowWars2, by="WarName") %>% select(WarName, ccode1, State1, ccode2, State2, year, lng1, lat1, lng2, lat2) # calculate distance latlng1 <- cowWarsDyadic %>% select(lng1, lat1) latlng2 <- cowWarsDyadic %>% select(lng2, lat2) cowWarsDyadic$distance <- distVincentySphere(latlng1, latlng2) / 1000 # convert to km cowWarsDyadic %>% select(WarName, ccode1, State1, ccode2, State2, distance) </code></pre> ## # A tibble: 813 × 6 ## WarName ccode1 State1 ccode2 State2 distance ## <chr> <dbl> <chr> <dbl> <chr> <dbl> ## 1 Franco-Spanish War 220 France 230 Spain 1054. ## 2 Franco-Spanish War 220 France 230 Spain 1054. ## 3 First Russo-Turkish 365 Russia 640 Ottom… 2233. ## 4 Mexican-American 2 United States of Amer… 70 Mexico 3035. ## 5 Austro-Sardinian 300 Austria 337 Tusca… 632. ## 6 Austro-Sardinian 300 Austria 325 Italy 765. ## 7 Austro-Sardinian 300 Austria 332 Modena 575. ## 8 First Schleswig-Holstein 255 Prussia 390 Denma… 356. ## 9 First Schleswig-Holstein 255 Prussia 390 Denma… 356. ## 10 First Schleswig-Holstein 255 Prussia 390 Denma… 356. ## # ℹ 803 more rows </code></pre> The resulting data can be found here</a>. R Packages</h4> Wickham H, Hester J and Francois R (2017). readr: Read Rectangular Text Data. R package version 1.1.1, <URL: https://CRAN.R-project.org/package=readr</a>>. Wickham H, Francois R, Henry L and Müller K (2017). dplyr: A Grammar of Data Manipulation. R package version 0.7.4, <URL: https://CRAN.R-project.org/package=dplyr</a>>. Wickham H and Henry L (2017). tidyr: Easily Tidy Data with ‘spread()’ and ‘gather()’ Functions. R package version 0.7.2, <URL: https://CRAN.R-project.org/package=tidyr</a>>. Kahle D and Wickham H (2013). “ggmap: Spatial Visualization with ggplot2.” The R Journal, 5(1), pp. 144-161. <URL: http://journal.r-project.org/archive/2013-1/kahle-wickham.pdf</a>>. Hijmans RJ (2016). geosphere: Spherical Trigonometry. R package version 1.5-5, <URL: https://CRAN.R-project.org/package=geosphere</a>>. Cheng J, Karambelkar B and Xie Y (2017). leaflet: Create Interactive Web Maps with the JavaScript ‘Leaflet’ Library. R package version 1.1.0, <URL: https://CRAN.R-project.org/package=leaflet</a>>. Xie Y (2017). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.18, <URL: https://yihui.name/knitr/</a>>. Xie Y (2015). Dynamic Documents with R and knitr, 2nd edition. Chapman and Hall/CRC, Boca Raton, Florida. ISBN 978-1498716963, <URL: https://yihui.name/knitr/</a>>. Xie Y (2014). “knitr: A Comprehensive Tool for Reproducible Research in R.” In Stodden V, Leisch F and Peng RD (eds.), Implementing Reproducible Computational Research. Chapman and Hall/CRC. ISBN 978-1466561595, <URL: http://www.crcpress.com/product/isbn/9781466561595</a>>. Francois R (2017). bibtex: Bibtex Parser. R package version 0.4.2, <URL: https://CRAN.R-project.org/package=bibtex</a>>. Boettiger C (2017). knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.8, <URL: https://CRAN.R-project.org/package=knitcitations</a>>. References</h4> Gartzke, Erik, and Alex Braithwaite. 2011. “Power, Parity and Proximity.” Gleditsch, Kristian S, and Michael D Ward. 2001. “Measuring space: A minimum-distance database and applications to international studies.” Journal of Peace Research 38 (6). ^{1 Tips on where to find this documentation are more than welcome. </div>}