Still Playing Catch-Up

As I was flipping through the February 2014 issue of the American Historical Review I was encouraged to see that American historical profession’s flagship journal seems to be doing a pretty decent job of publishing the impressive work of female historians. Three out of its four main articles were written by women and four out of the five books in its “Featured Reviews” section were also by women. That’s encouraging. But what about the rest of the February issue? Figuring out how many women are in the 176 contributors for this single issue is a lot harder. And what about not just this issue, but all five issues it publishes annually? And what about not just this year, but every year since its inception in 1895?

Looking at gender representation in the American Historical Review is exactly the kind of historical project that lends itself well towards digital analysis. Collecting individual author information from 120 years of publication history would take an enormous amount of tedious labor. Fortunately the information is already online. I wrote a Python script to scrape the table-of-contents from every AHR issue and then, with the help of Bridget Baird, began to process all of this text to try and extract the books that were reviewed in the AHR, their authors, and the names of the person reviewing them. The data was something of a nightmare, but we were eventually able to get everything we wanted: around 60,000 books, authors, and reviewers. The challenge turned to: was there a way to automatically identify the gender of all of these different people? Especially for a dataset that spanned more than a hundred years we needed a way to take into account potential changes in naming conventions. A historian named Leslie who was born before 1950 was likely to be a man, but if that same Leslie was born after 1950 the person was likely to be a woman. Bridget’s solution was for us to write a program that relies on a database of names from the Social Security Administration dating back to 1880 to account for these changes. This approach is not without problems. It only includes American names while subtly reinforcing an insidious gender binary framework. Nevertheless, it does contribute a useful new digital humanities methodology and one that we are planning to explore with Lincoln Mullen in more depth.

This might come as a real shock, but the American Historical Review didn’t feature very many women for much of its publication history. Over the first eighty years of the AHR‘s existence there were rarely more than a handful of books written by female authors in any given issue – as a percentage of all authors, women made up less than 10% of reviewed books through the 1970s. But things began to change in the late 1970s, when female authors began a steady ascent in the AHR‘s reviews. By the end of the 1980s women’s books had nearly doubled in the journal. By the twenty-first century there were three times as many women as there had been in the 1970s.

gender_percent_byyear

Gender of book authors (as a percent of all authors) in the American Historical Review between 1895 and 2013. The number of authors categorized as “Unknown” in the early years stems from the widespread use of initials (ex. K. T. Drew). Most of these authors were likely men, but we’ve erred on the safe side in categorizing them as Unknown. In the later years, many of the “Unknowns” stem from non-U.S. names.

But other numbers paint a less rosy picture. Lincoln Mullen’s recent work on history dissertations showed a similarly steady upwards trajectory in the number of female-authored history dissertations since 1950. Although it has plateaued in recent years, women have very nearly closed the gap in terms of newly completed history dissertations. But the glass ceiling remains stubbornly low in terms of what happens from that point onwards. In book reviews published in the AHR male authors continue to outnumber female authors by a factor of nearly 2 to 1. Whereas there is now a gap of around 3-5% separating the proportion of male and female dissertation authors, that gap jumps to 25-35% in terms of the proportion of male and female book authors being reviewed in the American Historical Review.

mf_diss_book_bluegreen

Gender of dissertation authors and of book authors in the American Historical Review. Note: The above chart only looks at authors whose gender was successfully identified by the program. It is also something of an apples-to-oranges comparison given that Lincoln and I were using slightly different methods, but it gives a rough sense for the gap between dissertations and the AHR.

On the reviewer side of the equation, things aren’t much better. There are still more than twice as many male reviewers as female reviewers in the AHR. But gender inflects this relationship in less direct ways. In particular, we can look at the gender dynamics of who reviews who. About three times as many men write reviews of male-authored books as do women. In the case of female-authored books, there are slightly more male reviewers than female reviewers but the ratio is much closer to 50/50. In short, women are much more likely to write reviews of other women. And while men still write reviews of the majority of female-authored books, they tend to gravitate towards male authors – who are, of course, already over-represented in the AHR.

male_authors_withreviewers

Gender of reviewers for male-authored books. Note: The above chart only looks at authors and reviewers whose gender was successfully identified by the program.

female_authors_withreviewers

Gender of reviewers for female-authored books. Note: The above chart only looks at authors and reviewers whose gender was successfully identified by the program.

Bridget and I were also able to extract the subjects used by the AHR to categorize their reviews. Although these conventions changed quite a bit over time, I took a stab at aggregating them into some broad categories for the past forty years. Essentially, I wanted to find out the gender representation within different historical fields. As you can see in the chart below, the proportion of men and women is not the same for all fields. Caribbean/Latin American history has had something approaching equal representation for the past decade-and-a-half. In both African history and Ancient/Medieval history female historians made some quite dramatic gains during the late-nineties and aughts. The guiltiest parties, however, are also the two subject categories that publish the most book reviews: Modern/Early Modern Europe and the United States/Canada. Both of them have made steady progress but still hover at around two-thirds male.

categories_gender_bytime

The different subjects are sorted left-to-right by the number of reviews in the AHR. Again, please note that the above chart only looks at authors whose gender was successfully identified by the program.

Women are now producing history dissertations at nearly the same rate as men, but the flagship journal of the American historical profession has yet to catch up. There are, of course, a lot of factors at play. This gap might reflect a substantial time-lag as a younger, more evenly-balanced generation gradually moves its way through the ranks even as an older, male-skewed generation continues to publish monographs. It might reflect biases in the wider publishing industry, or the fact that female historians continue to bear a disproportionate amount of the time-burden of caring for families. That the AHR continues to publish far more reviews of male authors than female authors is depressing, but unfortunately not surprising given the systemic inequalities that continue to exist across the profession.

The County Problem in the West

Happy GIS Day! Below is a version of a lightning talk I’m giving today at Stanford’s GIS Day.

Historians of the American West have a county problem. It’s primarily one of geographic size: counties in the West are really, really big. A “List of the Largest Counties in the United States” might as well be titled “Counties in the Western United States (and a few others)” – you have to go all the way to #30 before you find one that falls east of the 100th meridian. The problem this poses to historians is that a lot of historical data was captured at a county level, including the U.S. Census.

521px-Map_of_California_highlighting_San_Bernardino_County.svg

San Bernardino County

San Bernardino County is famous for this – the nation’s largest county by geographic area, it includes the densely populated urban sprawl of the greater Los Angeles metropolis along with vast swathes of the uninhabited Mojave Desert. Assigning a single count of anything to San Bernardino county to is to teeter on geographic absurdity. But, for nineteenth-century population counts in the national census, that’s all we’ve got.

TheWest_1871_Population-01-01

Here’s a basic map of population figures from the 1870 census. You can see some general patterns: central California is by far the most heavily populated area, with some moderate settlement around Los Angeles, Portland, Salt Lake City, and Santa Fe. But for anything more detailed, it’s not terribly useful. What if there was a way to get a more fine-grained look at settlement patterns in these gigantic western counties? This is where my work on the postal system comes in. There was a post office in (almost) every nineteenth-century American town. And because the department kept records for all of these offices – the name of the office, its county and state, and the date it was established or discontinued – a post office becomes a useful proxy to study patterns over time and space. I assembled this data for a single year (1871) and then wrote a program to geocode each office, or to identify its location by looking it up in a large database of known place-names. I then supplemented it with the the salaries of postmasters at each office for 1871. From there, I could finally put it all onto a map:

TheWest_1871_PostOffices

The result is a much more detailed regional geography than that of the U.S. Census. Look at Wyoming in both maps. In 1870, the territory was divided into five giant rectangular counties, all of them containing less than 5,000 people. But its distribution of post offices paints a different picture: rather than vertical units, it consisted largely of a single horizontal stripe along its southern border.

Wyoming_census-02   Wyoming_postoffices-02

Similarly, our view of Utah changes from a population core of Salt Lake City to a line of settlement running down the center of the territory, with a cluster in the southwestern corner completely obscured in the census map.

Utah_census-01   Utah_postoffices-01

Post offices can also reveal transportation patterns: witness the clear skeletal arc of a stage-line that ran from the Oregon/Washington border southeast to Boise, Idaho.

Dalles_Boise

Connections that didn’t mirror the geographic unit of a state or county tended to get lost in the census. One instance of this was the major cross-border corridor running from central Colorado into New Mexico. A map of post offices illustrate its size and shape; the 1870 census map can only gesture vaguely at both.

ColoradoNewMexico_census-02   ColoradoNewMexico_postoffices-02

The following question, of course, should be asked of my (and any) map: what’s missing? Well, for one, a few dozen post offices. This speaks to the challenges of geocoding more than 1,300 historical post offices, many of which might have only been in existence for a single year or two. I used a database of more than 2 million U.S. place-names and wrote a program that tried to account for messy data (spelling variations, altered state or county boundaries, etc.). The program found locations for about 90% of post offices, while the remaining offices I had to locate by hand. Not surprisingly, they were missing from the database for a reason: these post offices were extremely obscure. Finding them entailed searching through county histories, genealogy message boards, and ghost town websites – a process that is simply not scalable beyond a single year. By 1880, the number of post offices in the West had doubled. By 1890, and it doubled again. I could conceivably spend years trying to locate all of these offices. So, what are the implications of incomplete data? Is automated, 90% accuracy “good enough”?

What else is missing? Differentiation. The salary of a postmaster partially addresses this problem, as the department used a formula to determine compensation based partially on the amount of business an office conducted. But it was not perfectly proportional. If it was, the map would be one giant circle covering everything: San Francisco conducted more business than any other office by several orders of magnitude. As it is, the map downplays urban centers while highlighting tiny rural offices. A post office operates in a kind of binary schema: no office, no people (well, at least very few). If there was an office, there were people there. We just don’t know how many. The map isn’t perfect, but it does start to tackle the county problem in the West.

*Note: You can download a CSV file containing post offices, postmaster salaries, and latitude/longitude coordinates here.*

Ada Lovelace Day 2013

October 15th marks Ada Lovelace Day, an annual celebration of the achievements of women in science, technology, engineering and maths. As I read through posts commemorating the day, it got me reflecting on my own experience. It’s not just that I admire Ada Lovelace and the women that followed after her. It’s that I quite literally wouldn’t be here without them.

My mom, Bridget Baird, went to an all-women’s college in the late 1960s where she considered majoring in philosophy before switching to mathematics. After getting her PhD, she took a job in the early 1980s at Connecticut College in the math department. She got interested in computer programming, and eventually moved into a joint appointment in the computer science department. Over a three-decade career, her curiosity led her (and her thousands of students along with her) to the intersection of computer science with disciplines as far afield as archaeology, music, dance, and art. Along the way she faced the kinds of systemic discrimination that plagued the entire cohort of women entering male-dominated fields in the 1970s and 1980s. In other ways, she was lucky to have grown up during a time of transition when women began carving out new possibilities to enter those fields. She has spent her entire career mentoring female students and colleagues while vocally pushing her institution and discipline to take a more active role in tackling gender equity.

Although I missed the boat entirely on my mom’s math gene, she did manage to impress on me her fascination with applying computers to solve problems. Five years ago I wrote personal statements for history graduate programs structured around my interest in using technology to study the past. My mom since helped me learn how to program and we eventually ended up collaborating on a couple of projects. I’m one of the few graduate students I know who can call their mother to ask her about Thanksgiving plans and Python modules. I am, in ways I can’t even begin to articulate, a direct beneficiary of the legacy left by women like Ada Lovelace.

Which is why I oscillate between hope and discouragement when I look at around my own disciplinary homes of history and the digital humanities. On the one hand, women have made significant inroads in both fields. There are roughly equal numbers of male and female graduate students in my department. Many of the thought leaders and rising stars of the digital humanities are women, with opportunities and support growing all the time. The kinds of daily overt sexism faced by my mom and other women in her generation have, for the most part, gone the way of transistor radios. But that’s the problem: what remains is an insidious, covert sexism that is much, much harder to uproot.

And it’s everywhere. The proportion of female faculty in history departments is far lower than other fields, with the proportion of new female PhDs hovering stubbornly around 40%. Male historians continue to enjoy more time to spend on the kind of research that will get them tenure (as opposed to female historians spending more time on teaching and instruction), while men and women express completely different perceptions of gender equity at their institutions. The digital humanities have unfortunately inherited many of the gender problems endemic to computer science. These problems rear their ugly head everywhere, from the assumptions of a privileged male coding culture to the language of “hard” STEM fields vs. “soft” humanities. When I look around the room at digital humanities meetings and conferences I see the faces of a whole lot of people who look a whole lot like me. At a digital humanities conference on women’s history, though, I found that those same faces all but disappeared. I think about my mom every time I watch a female student grow increasingly silent during a discussion section or read the names of this year’s Nobel Prize winners. We can do better.

Who Picked Up The Check?

Adventures in Data Exploration

In November 2012 the United States Postal Service reported a staggering deficit of $15.9 billion. For the historian, this begs the question: was it always this bad? Others have penned far more nuanced answers to this question, but my starting point is a lot less sophisticated: a table of yearly expenses and income.

SurplusDeficitByYear

US Postal Department Surplus (Gray) or Deficit (Red) by Year

So, was the postal department always in such terrible fiscal shape? No, not at first. But from the 1840s onward, putting aside the 1990s and early 2000s, deficits were the norm. The next question: What was the geography of deficits? Which states paid more than others? Essentially, who picked up the check?

Every year the Postmaster General issued a report containing a table of receipts and revenues broken down by state. Let’s take a look at 1871:

AnnualReportTableReceiptsExpenditruesByState

1871 Annual Report of the Postmaster General – Receipts and Expenditures

Because it’s only one table, I manually transcribed the columns into a spreadsheet. At this point, I could turn to ArcGIS to start analyzing the data, maybe merging the table with a shapefile of state boundaries provided by NHGIS. But ArcGIS is a relatively high-powered tool better geared for sophisticated geospatial analysis. What I’m doing doesn’t require all that much horsepower. And, in fact, quantitative spatial relationships (ex. measurements of distance or area) aren’t all that important for answering the questions I’ve posed. There are a number of different software packages for exploring data, but Tableau provides a quick-and-dirty, drag-and-drop interface. In keeping with the nature of data exploration, I’ve purposefully left the following visualizations rough around the edges. Below is a bar graph, for instance, showing the surplus or deficit of each state, grouped into rough geographic regions:

SurplusDeficitBar_Crop

Postal Surplus or Deficit by State – 1871

Or, in map form:

SurplusDeficitMap_Crop

Postal Surplus (Black) or Deficit (Red) by State – 1871

Between the map and the bar graph, it’s immediately apparent that:
a) Most states ran a deficit in 1871
b) The Northeast was the only region that emerged with a surplus

So who picked up the check? States with large urban, literate populations: New York, Pennsylvania, Massachusetts, Illinois. Who skipped out on the bill? The South and the West. But these are absolute figures. Maybe Texas and California simply spent more money than Arizona and Idaho because they had more people. So let’s normalize our data by analyzing it on a per-capita basis, using census data from 1870.

SurplusDeficitBar_PerCapita_Crop

Postal Surplus or Deficit per Person by State – 1871

The South and the West may have both skipped out on the bill, but it was the West that ordered prime rib and lobster before it left the table. Relative to the number of its inhabitants, western states bled the system dry. A new question emerges: how? What was causing this extreme imbalance of receipts and expenditures in the West? Were westerners simply not paying into the system?

ReceiptsExpendituresByRegion

Postal Receipts and Expenditures per Person by Region – 1871

Actually, no. The story was a bit more complicated. On a per-capita basis, westerners were paying slightly more money into the system than any other region. The problem was that providing service to each of those westerners cost substantially more than in any other region: $38 per person, or roughly 4-5 times the cost of service in the east. For all of its lore of rugged individualism and a mistrust of big government, the West received the most bloated government “hand-out” of any region in the country. This point has been driven home by a generation of “New Western” historians who demonstrated the region’s dependence on the federal government, ranging from massive railroad subsidies to the U.S. Army’s forcible removal of Indians and the opening of their lands to western settlers. Add the postal service to that long list of federal largesse in the West.

But what made mail service in the West so expensive? The original 1871 table further breaks down expenses by category (postmaster salaries, equipment, buildings, etc.). Some more mucking around in the data reveals a particular kind of expense that dominated the western mail system: transportation.

TransportationMap_PerCapita_Crop

Transportation Expenses per Person by State (State surplus in black, deficit in red) – 1871

High transport costs were partially a function of population density. Many western states like Idaho or Montana consisted of small, isolated communities connected by long mail routes. But there’s more to the story. Beginning in the 1870s, a series of scandals wracked the postal department over its “star” routes (designated as any non-steamboat, non-railroad mail route). A handful of “star” route carriers routinely inflated their contracts and defrauded the government of millions of dollars. These scandals culminated in the criminal trial of high-level postal officials, contractors, and a former United States Senator. In 1881, the New York Times printed a list of the ninety-three routes under investigation for fraud. Every single one of these routes lay west of the Mississippi.

1881_StarRouteFrauds_Crop

Annual Cost of “Star” Routes Under Investigation for Fraud – 1881 (Locations of Route Start/End Termini)

The rest of the country wasn’t just subsidizing the West. It was subsidizing a regional communications system steeped in fraud and corruption. The original question – “Who picked up the check?” – leads to a final cliffhanger: why did all of these frauds occur in the West?

Learning by Doing: Labs and Pedagogy in the Digital Humanities

The digital humanities adore labs. Labs both symbolize and enable many of the field’s overarching themes: interdisciplinary teamwork, making/building, and the computing process itself. Labs give digital humanists a science-y legitimation that, whether we admit it or not, we find appealing. Labs aren’t necessary for doing digital humanities research, but in terms of infrastructure, collaboration, and institutional backing they certainly help. Along with “collaboration” and “open” (and possibly “nice“), “lab” is one of the field’s power words. With a period of accelerated growth over the past five years, world-wide digital humanities labs and centers now run into the hundreds. We overwhelmingly focus on labs in this kind of context: labs as physical research spaces. I’d like to move away from this familiar ground to discuss the role of lab assignments within a digital humanities curriculum. While reflecting on my own recent experience of designing and using labs in the classroom, I realized it spoke to many of the current issues facing the digital humanities.

Let me start with some background. This past autumn I taught my first college course, “The Digital Historian’s Toolkit: Studying the West in an Age of Big Data.” It was one of Stanford History Department’s Sources & Methods seminars, which are classes aimed at history majors to get them working intensively with primary sources. When I was designing my course a year ago, I decided to blend a digital humanities curriculum with more traditional historical pedagogy. Under the broad umbrella of the nineteenth-century American West, I used a specific historical theme each week (mining, communications, tourism, etc.) to tie together both traditional analysis and digital methodology. As part of this, over five different class periods students met in the Center for Spatial and Textual Analysis to complete a weekly lab assignment.

In designing the course, I wrestled with a problem that faces every digital humanist: the balancing of “traditional” (for lack of a better term) and “digital.” How much of my curriculum should follow a seminar model based on reading and discussion? How much should it follow a lab model based on technical tools and techniques? As is often the case, pragmatism partially informed my decision. Because my class was part of a required series of courses offered by the department, I couldn’t simply design a full-blown digital humanities methods course. It had to have a strong historical component in order to get approved. This juggling act is not uncommon for digital humanists. But more philosophically, I believed that digital tools were best learned in the context of historical inquiry. An overarching theme (in my case, the late nineteenth-century West) helped answer the question of why a student was learning a particular piece of software. Without it, digital pedagogy can stray into the bugaboo waved about by skeptics: teaching technology for technology’s sake.

I designed my labs with three goals in mind. First, I wanted my students to come away with at least an introduction to technical skills they wouldn’t otherwise get in a typical history course. Given my background, I focused largely on GIS, textual analysis, and visual design. I didn’t expect my students to become geospatial technicians in ten weeks, but I did want them to try out these kinds of methods and understand how they could be applied to historical problems. This first goal speaks to the alarmist rhetoric of a “crisis in the humanities,” of falling enrollments and shrinking budgets and growing irrelevance. In this caricature, the digital humanities often get remade as a life-boat for a sinking ship. This view is obviously overblown. But it is important to remember that the vast majority of our students are not going to end up as professors of history, literature, or philosophy. While there is a strong case to be made for the value of the humanities, I also think we need to do a better job of grafting other kinds of skills onto the field’s reading/writing/thinking foundation.

Second, I wanted students to learn technical skills as part of a larger intellectual framework. I pursued this in part by assigning specific techniques to answer larger questions. For instance, how does Mark Twain’s western novel Roughing It compare to other iconic nineteenth-century works of literature? Instead of assigning thousands of pages of text, I had my students use topic modeling to compare Roughing It to other books such as Uncle Tom’s Cabin and Little Women. But labs were also an effective way to concretize some of the contemporary issues swirling around technology. In one of the labs, students applied different kinds OCR software to a sampling of pages from an Overland Trail diary they had read earlier in the week. This gave them a chance to peer behind the curtain of large-scale digitization projects. When you experience first-hand just how many words and characters the OCR process can miss, it makes you think more critically about resources like Google Books or LexisNexis. Teaching in the digital humanities should, in part, force students to think critically about the issues surrounding the tools we use: copyrightaccessmarginalization.

Finally, I wanted students to learn by doing. There’s a certain passive mode of learning endemic to so many humanities courses: go to lectures, write a few papers, study for an exam, make comments in discussion. Student passivity can be inherent to both the pedagogical form itself and how it’s practiced, as anyone who has sat in a lecture hall or watched a student coast through discussion can tell you. Don’t get me wrong: bad labs can be just as passive as lectures. But done right, they emphasize active learning based on immediate feedback. As much as I’ve soured on the term “hacking” and all the privileged baggage it can carry, it is a useful term to describe the type of learning I want my students to engage in. Try something out. If it doesn’t work, try something else. Under this rubric, mistakes are a necessary part of the process. Feedback is more immediate in a way that enables exploration, tinkering, tangents, and restarts. It’s a lot harder to do this with traditional assignments; trying out something new in a paper is riskier than trying out something new in a lab.

This last goal proved the hardest to meet and constitutes one of the major hurdles facing digital humanities pedagogy. We want to teach digital methods not for their own sake, but to fit them within a broader framework, such as how they help us understand the past. But to get to that point, students need to make a fairly substantial investment of time and energy into learning the basics of a particular tool or technique. I tried to scaffold my lab assignments so that they became less and less prescriptive and more and more open-ended with each passing week. The idea was that students needed heavy doses of step-by-step instruction when they were still unfamiliar with the technology. My first lab, for instance, spelled out instructions in excruciating detail. Unfortunately, this led to exactly the kind of passive learning I wanted to avoid. I liken it to the “tutorial glaze” – focusing so much on getting through individual tasks that you lose track of how they all fit together or how you would apply them beyond the dataset at hand. The ability to teach early-stage technical skills involves a litany of pedagogical challenges that humanities instructors are simply not used to tackling.

By contrast, my final lab gave students a dataset (a map of Denver and enumeration district data from the 1880 census) and asked them to formulate and then answer a historical question through GIS. By nearly any metric – enthusiasm, results, feedback – this proved to be the most effective lab. It forced students to engage in the messy process of digital history: exploring the data enough to formulate a question, returning to the data to answer that question, realizing the data can’t even begin to answer that question, formulating a different question, figuring out how to answer it, and deciding how to visualize an argument. I was even more satisfied with their reflections on the process. Some described the frustrations that came with discovering the limits or gaps in census data. Others remarked on how their own mapmaking decisions, such as changing classification breaks or using different symbology, could completely alter the presentation of their argument. It’s one thing for students to read an essay by J.B. Harley on the subjectivity of maps (which they did). It’s another for students to experience the subjective process of map-making for themselves. Learning by doing: this is what was labs are all about.

To try and help others who want to integrate labs into their curriculum, I’ve made the labs and datasets available to download on the course website. Even as I posted them, though, I was reminded of one last problem facing the digital humanities: the problem of ephemerality. I spent hours and hours designing labs that will likely be unusable in a matter of years. Some of them require expensive software licenses, others rely on tools that could fall completely out of development. That’s one of the downside of labs. Ten years from now, I’ll still be able to re-use my lesson plan for discussing Roughing It. The lab on topic-modeling Twain and other novelists? Doubtful. But ephemerality is one of the necessary costs of teaching digital humanities. Because labs, and the broader pedagogical ethos of the digital humanities they embody, are ultimately worth it.

A Dissertation’s Infancy: The Geography of the Post

A history PhD can be thought of as a collection of overlapping areas: coursework, teaching, qualifying exams, and the dissertation itself. The first three are fairly structured. You have syllabi, reading lists, papers, classes, deadlines. The fourth? Not so much. Once you’re advanced to candidacy there’s a sense of finally being cut loose. Go forth, conquer the archive, and return triumphantly to pen a groundbreaking dissertation. It’s exhilarating, empowering, and also terrifying as hell. I’ve been swimming through the initial research stage of the dissertation for the past several months and thought it would be a good time to articulate what, exactly, I’m trying to find. Note: if you are less interested in American history and more interested in maps and visualizations, I would skip to the end.

The Elevator Speech

I’m studying communications networks in the late nineteenth-century American West by mapping the geography of the U.S. postal system.*

The Elevator-Stuck-Between-Floors Speech

From the end of the Civil War until the end of the nineteenth century the US. Post steadily expanded into a vast communications network that spanned the continent. By the turn of the century the department was one of the largest organizational units in the world. More than 200,000 postmasters, clerks, and carriers were involved in shuttling billions of pounds of material between 75,000 offices at the cost of more than $100 million dollars a year. As a spatial network the post followed a particular geography. And nowhere was this more apparent than in the West, where the region’s miners, ranchers, settlers, and farmers led their lives on the network’s periphery. My dissertation aims to uncover the geography of the post on its western periphery: where it spread, how it operated, and its role in shaping the space and place of the region.

My project rests on the interplay between center and periphery. The postal network hinged on the relationship between its bureaucratic center in Washington, DC and the thousands of communities that constituted the nodes of that network. In the case of the West, this relationship was a contentious one. Departmental bureaucrats found themselves buffeted with demands to reign in ballooning deficits. Yet they were also required by law to provide service to every corner of the country, no matter how expensive. And few regions were costlier than the West, where a sparsely settled population scattered across a huge area was constantly rearranged by the boom-and-bust cycles of the late nineteenth century. From the top-down perspective of the network’s center, providing service in the West was a major headache. From the bottom-up perspective of westerners the post was one of the bedrocks of society. For most, it was the only affordable and accessible form of long-distance communication. In a region marked by transience and instability, local post offices were the main conduits for contact with the wider world. Western communities loudly petitioned their Congressmen and the department for more offices, better post roads, and speedier service. In doing so, they redefined the shape and contours of both the network and the wider geography of the region.

The post offers an important entry point into some of the major forces shaping American society in the late nineteenth century. First, it helped define the role of the federal government. On a day-to-day basis, for many Americans the post was the federal government. Articulating the geographic size and scale of the postal system will offer a corrective to persistent caricatures of the nineteenth-century federal government as weak and decentralized. More specifically, a generation of “New Western” historians have articulated the omnipresent role of the state in the West. Analyzing the relationship between center and periphery through the post’s geography provides a means of mapping the reach of federal power in the region. With the postal system as a proxy for state presence, I can begin to answer questions such as: where and how quickly did the state penetrate the West? How closely did it follow on the heels of settler migration, railroad development, or mining industries? Finally, the post was deeply enmeshed in a system of political patronage, with postmasterships disbursed as spoils of office. What was the relationship between a communications network and the geography of regional and national politics?

Second, the post rested on an often contentious marriage between the public and private spheres. Western agrarian groups upheld the post as a model public monopoly. Nevertheless, private hands guided the system’s day-to-day operations on its periphery. Payments to mail-carrying railroad companies became the department’s single largest expenditure, and it doled out millions of dollars each year to private contractors to carry the mail in rural areas. This private/public marriage came with costs – in the early 1880s, for instance, the department was rocked by corruption scandals when it discovered that rural private contractors had paid kickbacks to department officials in exchange for lavish carrying contracts. How did this uneasy alliance of public and private alter the geography of the network? And how did the department’s need to extend service in the rural West reframe wider debates over monopoly, competition, and the nation’s political economy?

Getting Off The History Elevator

That’s the idea, at least. Rather than delve into even greater detail on historiography or sources, I’ll skip to a topic probably more relevant for readers who aren’t U.S. historians: methodology. Digital tools will be the primary way in which I explore the themes outlined above. Most obviously, I’m going to map the postal network. This entails creating a spatial database of post offices, routes, and timetables. Unsurprisingly, that process will be incredibly labor intensive: scanning and georeferencing postal route maps, or transcribing handwritten microfilmed records into a database of thousands of geocoded offices. But once I’ve constructed the database, there are any number of ways to interrogate it.

To demonstrate, I’ll start with lower-hanging fruit. The Postmaster General issues an annual report providing (among other information) data on how many offices were established and discontinued in each state. These numbers are fairly straightforward to put into a table and throw onto a map. Doing so provides a top-down view of the system from the perspective of a bureaucrat in Washington, D.C. For instance, by looking at the number of post offices discontinued each year it’s possible to see the wrenching reverberations of the Civil War as the postal system struggled to reintegrate southern states into its network in 1867:

Post Offices Discontinued By State, 1867
(Source: Annual Report of the Postmaster General, 1867)

The West, meanwhile, was arguably the system’s most unstable region. As measured by the percentage of its total offices that were either established or discontinued each year, states such as New Mexico, Colorado, and Montana were continually building and dismantling new nodes in the network.

Post Offices Established or Discontinued as a Percentage of Total Post Offices in State, 1882
(Source: Annual Report of the Postmaster General, 1882)

Of course, the broad brush strokes of national, year-by-year data only provide a generalized snapshot of the system. I plan on drilling down to far more detail  by charting where and when specific post offices were established and discontinued. This will provide a much more fine-grained (both spatially and temporally) view of how the system evolved. Geographer Derek Watkins has employed exactly this approach:

Screenshot from Derek Watkins, “Posted: U.S. Growth Visualized Through Post Offices” (25 September 2011)

Derek’s map demonstrates the power of data visualization: it is compelling, interactive, and conveys an enormous amount of information far more effectively than text alone. Unfortunately, it also relies on an incomplete dataset. Derek scraped the USPS Postmaster Finder, which the USPS built as a tool for genealogists to look up postmaster ancestors. The USPS historian adds to it on an ad-hoc basis depending on specific requests by genealogists. In a conversation with me, she estimated that it encompasses only 10-15% of post offices, and there is no record of what has been completed and what remains to be done. Derek has, however, created a robust data visualization infrastructure. In a wonderful demonstration of generosity, he has sent me the code behind the visualization. Rather than spending hours duplicating Derek’s design work, I’ll be able to plug my own, more complete, post office data into a beautiful existing interface.

Derek’s generosity brings me back to my ongoing personal commitment to scholarly sharing. I plan on making the dissertation process as open as possible from start to finish. Specifically, the data and information I collect has broad potential for applications beyond my own project. As the backbone of the nation’s communications infrastructure, the postal system provides rich geographic context for any number of other historical inquiries. Cameron Ormsby, a researcher in Stanford’s Spatial History Lab, has already used post office data I collected as a proxy for measuring community development in order to analyze the impact of land speculation and railroad construction in Fresno and Tulare counties.

To kick things off, I’ve posted the state-level data I referenced above on my website as a series of CSV files. I also used Tableau Public to generate a quick-and-dirty way for people to interact with and explore the data in map form. This is an initial step in sharing data and I hope to refine the process as I go. Similarly, I plan on occasionally blogging about the project as it develops. Rather than narrowly focusing on the history of the U.S. Post, my goal (at least for now) is to use my topic as a launchpad to write about broader themes: research and writing advice, discussions of digital methodology, or data and visualization releases.

*By far the most common response I’ve received so far: “Like the Pony Express?” Interestingly, the Pony Express was a temporary experiment that only existed for about eighteen months in 1860-1861. In terms of mail carried, cost, and time in existence, it was a tiny blip within the postal department’s operations. Yet it has come to occupy a lofty position in America’s historical memory and encapsulates a remarkable number of the contradictions and mythologies of the West.

Coding a Middle Ground: ImageGrid

Openness is the sacred cow of the digital humanities. Making data publicly available, writing open-source code, or publishing in open-access journals are not just ideals, but often the very glue that binds the field together. It’s one of the aspects of digital humanities that I find most appealing. Despite this, I have only slowly begun to put this ideal into practice. Earlier this year, for instance, I posted over one hundred book summaries I had compiled while studying for my qualifying exams. Now I’m venturing into the world of open-source by releasing a program I used in a recent research project.

The program tries to tackle one of the fundamental problem facing many digital humanists who analyze text: the gap between manual “close reading” and computational “distant reading.” In my case, I was trying to study the geography within a large corpus of nineteenth-century Texas newspapers. First I wrote Python scripts to extract place-names from the papers and calculate their frequencies. Although I had some success with this approach, I still ran into the all-too-familiar limit of historical sources: their messiness. Namely, nineteenth-century newspapers are extremely challenging to translate into machine-readable text. When performing Optical Character Recognition (OCR), the smorgasbord nature of newspapers poses real problems. Inconsistent column widths, a potpourri of advertisements, vast disparities in text size and layout, stories running from one page to another – the challenges go on and on and on. Consequently, extracting the word “Havana” from OCR’d text is not terribly difficult, but writing a program that identifies whether it occurs in a news story versus an advertisement is much harder. Given the quality of the OCR’d text in my particular corpus, deriving this kind of context proved next-to-impossible.

The messy nature of digitized sources illustrates a broader criticism I’ve heard of computational distant reading: that it is too empirical, too precise, and too neat. Messiness, after all, is the coin of the realm in the humanities - we revel in things like context, subtlety, perspective, and interpretation. Computers are good at generating numbers, but not so good at generating all that other stuff. My computer program could tell me precisely how many times “Chicago” was printed in every issue of every newspaper in my corpus. What it couldn’t tell me was the context in which it occurred. Was it more likely to appear in commercial news? Political stories? Classified ads? Although I could read a sample of newspapers and manually track these geographic patterns, even this task proved daunting: the average issue contained close to one thousand place-names and stretched more than 67,000 words (or, longer than Mrs. Dalloway, Fahrenheit 451, and All Quiet on the Western Front). I needed a middle ground. I decided to move backwards, from the machine-readable text of the papers to the images of the newspapers themselves. What if I could broadly categorize each column of text according both to its geography (local, regional, national, etc.) and its type of content (news, editorial, advertisement, etc.)? I settled on the idea of overlaying a grid onto the page image. A human reader could visually skim across the page and select cells in the grid to block off each chunk of content, whether it was a news column or a political cartoon or a classified ad. Once the grid was divided up into blocks, the reader could easily calculate the proportions of each kind of content.

My collaborator, Bridget Baird, used the open-source programming language Processing to develop a visual interface to do just that. We wrote a program called ImageGrid that overlaid a grid onto an image, with each cell in the grid containing attributes. This “middle-reading” approach allowed me a new access point into the meaning and context of the paper’s geography without laboriously reading every word of every page. A news story on the debate in Congress over the Spanish-American War could be categorized primarily as “News” and secondarily as both “National” and “International” geography. By repeating this process across a random sample of issues, I began to find spatial patterns.

Grid with primary categories as colors and secondary categories as letters

For instance, I discovered that a Texas paper from the 1840s dedicated proportionally more of its advertising “page space” to local geography (such as city grocers, merchants, or tailors) than did a later paper from the 1890s. This confirmed what we might expect, as a growing national consumer market by the end of the century gave rise to more and more advertisements originating from outside of Texas. More surprising, however, was the pattern of international news. The earlier paper contained three times as much foreign news (relative “page space” categorized as news content and international geography) as did the later paper in the 1890s. This was entirely unexpected. The 1840s should have been a period of relative geographic parochialism compared to the ascendant imperialism of the 1890s that marked the United States’s noisy emergence as a global power. Yet the later paper dedicated proportionally less of its news to the international sphere than the earlier paper. This pattern would have been otherwise hidden if I had used either a close-reading or distant-reading approach. Instead, a blended “middle-reading” through ImageGrid brought it into view.

We realized that this “middle-reading” approach could be readily adapted not just to my project, but to other kinds of humanities research. A cultural historian studying American consumption might use the program to analyze dozens of mail-order catalogs and quickly categorize the various kinds of goods – housekeeping, farming, entertainment, etc. – marketed by companies such as Sears-Roebuck. A classicist could analyze hundreds of Roman mosaics to quantify the average percentage of each mosaic dedicated to religious or military figures and the different colors used to portray each one.

Inspired by the example set by scholars such as Bethany NowviskieJeremy Boggs, Julie Meloni, Shane Landrum, Tim Sherratt, and many, many others, we released ImageGrid as an open-source program. A more detailed description of the program is on my website, along with a web-based applet that provides an interactive introduction to the ImageGrid interface. The program itself can be downloaded either on my website or on its GitHub repository, where it can be modified, improved, and adapted to other projects.