historying

Yesterday I handed in my final papers to officially conclude my first academic quarter at Stanford. With that in mind, here are my initial thoughts on being a graduate student:

1. I read a lot.

I’ve never read this much in my life, and it took me most of the quarter to learn how to sit down and read books for hours and hours every day.

Given that this was my first term and that I was only enrolled in two reading-intensive courses, I’m a tiny bit terrified of what the future (cough, studying for orals) holds. Reading for so many hours has sadly all but demolished any inclination towards pleasure reading. On the flip side, I’ve gained a new appreciation for Hulu.

2. I’ve met amazing people.

It took me a little while to get used to the fact that I was surrounded by ridiculously intelligent people, including a classmate with his own Wikipedia page and an advisor who won a Macarthur Genius Award and who’s also way cooler than I am. All thirteen of my fellow cohort members are incredibly bright, inquisitive, and quite inspiring to think of as future colleagues. Everyone I’ve met outside of the history department has been just as impressive. Meanwhile, I have a blog that I struggle to update more than once a month.

3. I still don’t know what I want to study.

This is going to be a recurring problem. The downside to having a focus on digital methodology is that I have yet to figure out what I actually want to use that methodology to study. I don’t like the idea of confining myself to a narrow thematic topic, but it’s a step I’ll have to take eventually (one I plan on putting off for as long as possible). In the meantime, I’ll just have to keep answering the questions of “American history? Anything more specific?” with “Um, no. Not yet.”

4. I made the right decision.

Stanford has been a phenomenal fit so far. The best example I can give is that I had the opportunity to take a non-history course titled Literary Studies and the Digital Library taught by Matt Jockers. It was a seminar class in the English department, and included undergraduates, graduate students, and a fairly broad sample of various disciplines. The course centered around designing a digital text mining project to examine over one thousand American and British works from the nineteenth century, culminating in a three-paper proposal to the Digital Humanities 2010 conference in London (fingers crossed). Matt has since extended the course into the next quarter to let us continue to work on the project.

I could not have been happier with the course, and it embodies many of what I consider to be the core tenets of the digital humanities: hands-on technical problem-solving, interdisciplinarity, and intensive collaboration. I have almost no background in English – during class discussion I at one point googled “buildings romane,” which a helpful classmate had to correct to “Bildungsroman.” Despite this, or maybe because of this, I learned not only about cool things like epistolary novels and the rise of serialization, but also the differences in how literary critics and historians approach questions and problems. This interdisciplinary experience will serve me well in the future.

At the end of the day, I feel incredibly lucky to be given the opportunity to be doing something that I love. As one of my classmates said to me over lunch, “Isn’t it absurd that we’re somehow getting paid to be here?” Not yet jaded from TA’ing, orals, dissertation writing, or the job market, I could not agree more.

One of the most basic applications of text mining is simply counting words. I began by stripping out punctuation (in order to avoid differentiating mend and mend. as two separate words), put every word into lowercase, and then ignored a list of stop words (the, and, for, etc.). By writing a program to count occurrences of the 500 most common words, I could get a general (and more quantitative) sense for what general topics Martha Ballard wrote about in her diary. Unsurprisingly, her vocabulary usage followed a standard path of exponential decay: like most people, she utilized a relatively small number of words with extreme frequency. For example, the most common word (mr) occurred 10,050 times, while her 500th most common word (relief) occurred 67 times:

Because each word has information attached to it – specifically what date it was written – we can look at long-term patterns for a particular word’s usage. However, looking at only raw word frequencies can be problematic. For example, if Ballard wrote the word yarn twice as often in 1801 as 1791, it could mean that she was doing a lot more knitting in her old age. But it could also mean that she was writing a lot more words in her diary overall. In order to address this issue, for any word I was examining I made sure to normalize its frequency – first by dividing it by the total word count for that year, then by dividing it by the average usage of the word over the entire diary. This allowed me to visualize how a word’s relative frequency changed from year to year.

In order to visualize the information, I settled on trying out sparklines: “small, intense, simple datawords” advocated by infographics guru Edward Tufte and meant to give a quick, somewhat qualitative snapshot of information. To test my method, I used a theme that Laurel Ulrich describes in A Midwife’s Tale: land surveying. In particular, during the late 1790s Martha’s husband Ephraim became heavily involved in surveying property. In the raw word count list, both survey and surveying appear in the top 500 words, so I combined the two and looked at how Martha’s use of them in her diary changed over the years (1785-1812):

survey(ing)

Looking at the sparkline, we get a visual sense for when surveying played a larger role in Martha’s diary – around the middle third, or roughly 1795-1805, which corresponds relatively well to Ulrich’s description of Ephraim’s surveying adventures. As a basis for comparison, the word clear appeared with numbing regularity (almost always in reference to the weather):

clear

Using word frequencies and sparklines, I could investigate and visualize other themes in the diary as well.

Religion

Out of the 500 most frequent words in the diary, only three of them relate directly to religion: meeting (#28), worship (#143), and god (#220).

meeting

worship

god

Meeting, which was used largely in a religious context (going to a church meeting), but also in a socio-political context (attending town meetings), had a relatively consistent rate of use, although it trended slightly upwards over time. Worship (which Martha largely used in the sense of “went to publick worship”), meanwhile, was more erratic and trended slightly downwards. Finally, and perhaps most interestingly, was Martha’s use of the word god. Almost non-existent in the first third of her diary, it then occurred much more frequently, but also more erratically over the final two-thirds of the diary. Not only was it a relatively infrequent word overall (flax, horse, and apples occur more often), but its usage pattern suggests that Martha Ballard did not directly invoke a higher power on a personal level with any kind of regularity (at least in her diary). Instead, she was much more comfortable referring to the more socially and community-based activity of attending a religious service. While a qualitative close reading of the text would give a richer impression of Martha’s spirituality, a quantitative approach demonstrates how little “real estate” she dedicates to religious themes in her diary.

Death

death

dead

funeral

expired

interd

Most of the words related to death show an erratic pattern. There are peaks and valleys across the years without much correlation between the different words, and the only word that appears with any kind of consistency is interd (interred). In this case, word frequency and sparklines are relatively weak as an analytical tool. They don’t speak to any kind of coherent pattern, and at most they vaguely point towards additional questions for study – what causes the various extreme peaks in usage? Is there a common context with which Martha uses each of the words? Why was interd so much flatter than the others?

Family

In this final section, I’ll offer up a small taste of how analyzing word frequency can reveal interpersonal relationships. I used the particular example of Dolly (Martha’s youngest daughter):

dolly

The sparkline does a phenomenal job of driving home a drastic change in how Martha refers to her daughter. In a matter of a year or two in the mid 1790s, she goes from writing about Dolly frequently to almost never mentioning her. Why? Some quick detective work (or reading page 145 in A Midwife’s Tale) shows that the plummet coincides almost perfectly with Dolly’s marriage to a man named Barnabas Lambart in 1795. But why on earth would Martha go from mentioning Dolly all the time in her diary to going entire years without writing her name? Did Martha disapprove of her daughter’s marriage? Was it a shotgun wedding?

The answer, while not so scandalous, is an interesting one nonetheless that text analysis and visualization helps to elucidate. In short, Martha still writes about her daughter after 1795, but instead of referring to her as Dolly, she begins to refer to her as Dagt Lambd (Daughter Lambert). This is a fascinating shift, and one whose full significance might get lost by a traditional reading. A human poring over these detailed entries might get a vague impression that Martha has started calling her daughter something different, but the sparkline above drives home just how abrupt and dramatic that transformation really was. Martha, by and large, stopped calling her youngest daughter by her first name and instead adopted the new husband’s proper name. Such a vivid symbolic shift opens up a window into an array of broader issues, including marriage patterns, familial relationships, and gender dynamics.

Conclusions

Counting word frequency is a somewhat blunt instrument that, if used carefully, can certainly yield meaningful results. In particular, utilizing sparklines to visualize individual word frequencies offers up two advantages for historical inquiry:

1. Coherently display general trends
2. Reveal outliers and anomalies

First, sparklines are a great way to get a quick impression of how a word’s use changes over time. For example, we can see above that the frequency of the word expired steadily increases throughout the diary. While this can often simply reiterate suspected trends, it can ground these hunches in refreshingly hard data. By the end of the diary, a reader might have a general sense for how certain themes appear, but a text analysis can visualize meaningful patterns and augment a close reading of the text.

Second, sparklines can vividly reveal outliers. In the course of reading hundreds of thousands of words over the course of nearly 10,000 entries, it’s quite easy to lose sight of the forest for the trees (to use a tired metaphor). Visualizing word frequencies allows historians to gain a broader perspective on a piece of the text, and they also act as signposts pointing the viewer towards a specific area for further investigation (such the red-flag-raising rupture in how frequently Dolly appears). Relatively basic word frequency by itself (such as what I’ve done here) does not necessarily explain anomalies, but it can do an impressive job of highlighting important ones.

Given Martha Ballard’s profession as a midwife, it is no surprise that she carefully recorded the 814 births she attended between 1785 and 1812. These events were given precedence over more mundane occurrences by noting them in a separate column from the main entry. Doing so allowed her to keep track not only of the births, but also record payments and restitution for her work. These hundreds of births constituted one of the bedrocks of Ballard’s experience as a skilled and prolific midwife, and this is reflected in her diary.

As births were such a consistent and methodically recorded theme in Ballard’s life, I decided to begin my programming with a basic examination of the deliveries she attended. This examination would take the form of counting the number of deliveries throughout the course of the diary and grouping them by various time-related characteristics, namely: year, month, and day of the week.

Process and Results

The first basic step for performing a more detailed text analysis of Martha Ballard’s diary was to begin cleaning up the data. One step was to take all the words and (temporarily) turn every uppercase letter into a lowercase letter. This kept Python from seeing “Birth” and “birth” as two separate words. For the purposes of this particular program, it was more important to distill words into a basic unit rather than maintain the complexity of capitalized characters.

Once the data was scrubbed, we could turn to writing a program that would count the number of deliveries recorded in the diary. The program we wrote does the following:

1. Checks to see if Ballard wrote anything in the “birth” column (the first column of the entries that she also used to keep track of deliveries)
2. If she did write anything in that column, check to see if it contains any of the words: “birth”, “brt”, or “born”.
3. I then printed the remainder of the entries that contained text in the “birth” column but did not contain one of the above words. From this short list I manually added an additional seven entries into the program, in which she appeared to have attended a delivery but did not record it using the above words.

Using these parameters, the program could iterate through the text and recognize the occurrence of a delivery. Now we could begin to organize these births.

First, we returned the birth counts for each year of the diary, which were then inserted into a table and charted in Excel:

At the risk of turning my analysis into a John Henry-esque woman vs. machine, I compared my figures to the chart that Laurel Ulrich created in A Midwife’s Tale that tallied the births Ballard attended (on page 232 of the soft-cover edition). The two charts follow the same broad pattern:

Note: I reverse-built her chart by creating a table from the printed chart, then making my own bar graph. Somewhere in the translation I seem to have misplaced one of the deliveries (Ulrich lists 814 total, whereas I keep counting 813 on her graph). Sorry!

However, a closer look reveals small discrepancies in the numbers for each individual year. I calculated each year’s discrepancy as follows, using Ulrich’s numbers as the “true” figures (she is the acting President of the AHA, after all) from which my own figures deviated, and found that the average deviation for a given year was 4.86%. Apologies for the poor formatting, I had trouble inserting tables into WordPress:

Keeping the knowledge in the back of my mind that my birth analysis differed slightly from Ulrich’s, I went on to compare my figures with other factors, including the frequency of deliveries by month over the course of the diary.

If we extend the results of this chart and assume a standard nine-month pregnancy, we can also determine roughly which months that Ballard’s neighbors were most likely to be having sex. Unsurprisingly, the warmer period between May and August appears to be a particularly fertile time:

Finally, I looked at how often births occurred on different days of the week. There wasn’t a strong pattern, beyond the fact that Sunday and Thursday seemed to be abnormally common days for deliveries. I’m not sure why that was the case, but would love to hear speculation from any readers.

Analysis

The discrepancies between the program’s tally of deliveries and Ulrich’s delivery count speak to broader issues in “digital” text mining versus “manual” text mining:

Data Quality

Ulrich’s analysis is a result of countless hours spent eye-to-page with the original text. And as every history teacher drills into their students when conducting research, looking directly at the primary documents minimizes the degrees of interpretation that can alter the original documents.  In comparison, my analysis is the result of the original text going through several levels of transformation, like a game of telephone:

Original text -> Typed transcription -> HTML tables -> Python list -> Text file -> Excel table/chart

Each level increases the chance of a mistake.  For instance, a quick manual examination using the online version of the diary for 1785 finds an instance of a delivery (marked by ‘Birth’) showing up in the online HTML, but which does not appear in the “raw” HTML files our program is processing and analyzing.

On the other hand, a machine doesn’t get tired and miscount a word tally or accidently skip an entry.

Context

Ulrich brings to bear on the her textual analysis years of historical training and experience along with a deeply intimate understanding of Ballard’s diary. This allows her to take into account one of the most important aspects of reading a document: context. Meanwhile, our program’s ability to understand context is limited quite specifically to the criteria we use to build it. If Ballard attended a delivery but did not mark it in the standard “birth” column like the others, she might mention it more subtly in the main body of the entry. Whereas Ulrich could recognize this and count it as a delivery, our program cannot (at least with the current criteria).

Where the “traditional” skills of a historian come into play with data mining is in the arena of defining these criteria. Using her understanding of the text on a traditional level, Ulrich could create far, far superior criteria than I could for counting the number of deliveries Martha Ballard attends. The trick comes in translating a historian’s instinctual eye into a carefully spelled-out list of criteria for the program.

Revision

One area that is advantageous for digital text mining is that of revising the program. Hypothetically, if I realized at a later point that Ballard was also tallying births using another method (maybe a different abbreviated word), it’s fairly simple to add this to the program’s criteria, hit the “Run” button, and immediately see the updated figures for the number of deliveries. In contrast, it would be much, much more difficult to do so manually, especially if the realization came at, say, entry number 7,819. The prospect of re-skimming thousands of entries to update your totals would be fairly daunting.

“mr Ballard left home bound for Oxford. I had been Sick with the Collic. mrs Savage went home. mrs foster Came at Evening. it snowd a little.”

This is the first entry in the diary of Martha Ballard. Martha Ballard was a rural Maine midwife who kept an extensive diary between 1785 and 1812 and whose life was immortalized in 1990 by the historian Laurel Thatcher Ulrich‘s award-winning A Midwife’s Tale. Over the course of three decades, Ballard kept a meticulous, near-daily accounting of her life spanning over 10,000 entries.

When reading A Midwife’s Tale, I was struck by how readily the text would seem to lend itself to digital analysis. In an interview, Ulrich noted, “The very thing that had attracted me to the diary in the first place was also the thing that made it difficult to work with. I mean there’s just so much.” To ground herself, she began by simply counting things: “And I would go day by day for every other year of the diary, and I would tick off what was in each entry: baking or brewing, spinning or washing, or trading, sewing, mending, deliveries, general medical accounts, going to church, visitors, people coming for meals, etc.” Because of the sprawling scope, she took this quantitative approach only for the even-numbered years in the diary. The fact that she was working in the late eighties without a computer makes her work even more impressive.

After poking around online I came across DoHistory.org, a website developed and maintained by the Film Study Center at Harvard University and hosted by (who else, really) George Mason’s CHNM. The website presents the diary to the public in two formats: the viewer can either browse through photographed pages of the diary or read the transcript of the pages (transcribed through a monumental effort by Robert R. McCausland and Cynthia MacAlman McCausland):

When I realized the entire diary was online, it got me thinking about possibilities for text mining. As an aspiring digital humanist with little “hard” skills beyond basic GIS, I had been meaning to learn how to program for quite some time. In Martha Ballard’s diary, I had an intriguing source of data with which to learn how to do so. Now I just had to learn how to program. With the patient help of several programming-savvy family members, I gradually learned the basics of Python and how to apply it to Martha Ballard’s diary. What follows are the first steps we took to process the diary’s raw data into an accessible digital format.

Process

At first, I briefly considered learning how to scrape the text of the diary off the website. After some investigation, I decided that was a little beyond my abilities, so I copped out to the much easier route of sending an email to Kelly Schrum at CHNM, who kindly forwarded my request to Ammon Shepherd, who emailed me a zip file containing 1,431 html documents, one for each page of the diary. The html files of the transcribed diary are a basic, 3-column table that look this. My first step was to find a way to strip out the html tags and organize the text into a systematic database of individual entries. Fortunately, Ballard’s meticulousness and consistency lent itself well to such an approach.

The diary’s format translates quite nicely into creating a list of lists – the “main” diary being a list of all the entries, and each entry being a list in and of itself. The first program we wrote was to open each html file and begin extracting the different sections of text (which were conveniently marked by html tags). Iterating through each entry allowed us to separate the different columns in her diary into different items in the list. Here is the breakdown of our “list of lists”:

1. Diary
   1. Entry
      1. Date
         1. Month
         2. Day
         3. Year
      2. Day of the Week
      3. Main Text of Entry
      4. Day Summaries (Column 3 of actual diary entry)
      5. Birth(s) (Recorded in Column 1 of actual diary entry)

In creating the list, we had to separate out the raw data from the html tags that formatted it. Fortunately, the folks who built the html files originally used an extremely systematic formatting process that actually made the job of distilling one from the other quite straightforward. A Python module called Pickle allowed us to export the list of entries as a manageable single file that we could then easily import into future programs to manipulate.

For example, the third entry in the diary would translate a bit into something like this:

1. Diary
   1. Entry (3)
      
      1. Date
         1. 1 (January)
            
         2. 3
            
         3. 1785
   2. 3 (Tuesday – Ballard numbered the weekdays, beginning with Sunday as 1)
   3. “Tuesday. mrs. Foster went home. I had threats of thee Collic; by takein peper found releif.”
      
   4. Empty
   5. Empty

The list allows us to access pieces of information by “calling” their position. It helped me to think of the entire diary list as a warehouse containing almost 10,000 boxes (entries) inside it, with each box containing five compartments, with the first of those compartments divided into three sub-compartments. If you were to open any of the boxes (entries) and look inside the first compartment, then inside sub-compartment number two, you would always find a number that represented the month of that particular entry. If you were to look inside the third compartment of the entry/box, you would always find the main text for that day’s entry.

The advantages of setting up the data in a list structure is the ability to access these specific pieces of information easily and to compare them across entries. In many ways, processing the text to make it readable and programmable is one of the biggest challenges to text mining. Deciding on the most logical way to organize and break down over 1,400 files will lay the groundwork for the fun part: writing programs to actually analyze the diary of Martha Ballard.

***Special-edition sneak preview of future posts in this series***

A simple counting program reveals that the main text of Martha Ballard’s diary alone contains 377,315 words, spanning I-couldn’t-make-this-number-up 9,999 entries. That is a lot of data to play with.

Andrew Jackson = bad.

Whigs = good.

That’s my five-word summary of Daniel Walker Howe’s Pulitzer-Prize winning, 900-page, career-defining work What Hath God Wrought: The Transformation of America, 1815-1848. Of course, the tome is monumental in every sense of the word, in its subject matter, scope, weight, and approach. Howe is fearless in shouldering the daunting task to chart the the United States’ tumultuous adolescence. In an academic climate of sometimes numbing specialization, What Hath God Wrought is boldly and refreshingly big.

Howe approaches this task by looking at the nation’s growth through the eyes of the two ideological competitors fighting for its future. In one corner sat the Democrat ideals of Andrew Jackson, which lay out a path of white male individualism, territorial and racial conquest, and limited federal government. In the other corner sat the Whig’s ideology of national improvement through an active government and strong internal commercial expansion. Howe maintains this general dichotomy as he wanders down thirty-odd years of national history. The thematic path he treads most commonly is that of the communications and transportation revolution, although he winds enthusiastically down every manner of side trail and parallel road.

Daniel Walker Howe detests Andrew Jackson. His characterization of our seventh president throws all notions of bland academic sterility to the winds, instead engaging in fierce and almost-personal skirmishes against Jackson. First and foremost, Howe argues that Jackson’s legacy is marked by the insidious morally debilitating advancement of white male supremacy.  “…historians have variously pointed to free enterprise, manhood suffrage, the labor movement, and resistance to the market economy. But in its origins, Jacksonian Democracy…was not primarily about any of these, though it came to intersect with all of them in due course. In the first place it was about the extension of white supremacy across the North American continent.”

Howe is absolutely relentless in his campaign against Jackson’s apologists, and in many ways he makes a convincing argument. Indian Removal remains one of the most shameful aspects of our nation’s history, and Andrew Jackson was one of its most effective champions. In the eloquent and caustic words of Howe, “During the Removal process the president personally intervened frequently, always on behalf of haste, sometimes on behalf of economy, but never on behalf of humanity, honesty, or careful planning.” In doing so, Jackson became the figurehead for a growing ideology of American imperialism, as our national borders expanded both literally and figuratively. Howe argues convincingly that the Jacksonian flavor of imperialism included not only territorial expansion, but a dissolution of the rule of law. Concepts such as treaties, property rights, or basic civil liberties were swept aside when it came to the forcible and violent relocation of Indian tribes from their land.

Jackson also gets taken to task for his assault on that pesky thorn in the side of ideologues everywhere: freedom of speech. Specifically, Jackson attempted to stifle the increasingly loud voices of abolitionists, whom he deemed “monsters.” He took action by using federal control of the postal system to hinder abolitionist material from being transported through the mail. Jackson also went further, instructing his postmaster general to publish the names of abolitionists guilty of “exciting the negroes to insurrection and to massacre.” These measures were met with mixed results, but regardless, Howe fulminates that restricting abolitionist material through the post quite possibly constituted “the largest peacetime violation of civil liberty in U.S. history.” Umm, Mr. Howe? Didn’t you just describe the violent removal of tens of thousands of people from their rightfully-owned property? Might that not constitute a bigger “peacetime violation of civil liberty”?

Big, bold statements such as these make What Hath God Wrought fun and refreshing, but Howe’s argumentation can sometimes stray into the realm of hyperbole. I appreciate the fact that he makes no bones about having an overt historiographical agenda, but it can lead to rather blatant side-taking. For instance, in Howe’s view, Jackson’s administration had a hand in causing the Panic of 1837: “Democrats blamed the banks. Whigs blamed Jackson.” Which side do you think Howe falls on? Thankfully, he doesn’t play coy: “There is more truth in the Whig argument.” Meanwhile, Howe spends ample time cheerleading the expansion of federally-funded internal improvements under John Quincy Adams (a Whig), but when that same federal funding for internal improvement soars even higher under Jackson, Howe dismisses the administration’s efforts as hypocritical, ad-hoc, and ambiguous.

In comparison to Jackson, Howe takes a decidedly rose-tinted view of the Whigs. Given his scholarly background as the author of The Political Culture of the American Whigs, this isn’t necessarily surprising. In Howe’s opinion, “the Whigs, though not the dominant party of their own time, were the party of America’s future.” He (rightfully) champions their role in connecting a disparate young collection of states and communities into an integrated whole. Howe even opens the “what-if” door to offer a brief glimpse into what might have been different had John Quincy Adams won the 1828 election instead of Andrew Jackson. Although he doesn’t come out and say it, Howe delicately insinuates that maybe, just maybe, an Adams administration could have pre-emptively prevented the Civil War: “[The Whigs'] strong central government would have held long-term potential for helping the peaceful resolution of the slavery problem…” I mean, come on…really? For that small segment of the American population interested in historiography (like myself), nuggets such as these contribute to the juicy platter of provocative scholarly interpretation that Howe serves up to his readers.

Howe’s romp through the historical period is done with a stylistic flair and easy grace that continously impressed me. In the hands of many writers, a three-decade survey of America would devolve into a yawn-inducing litany of dates, names, and events. Instead, Howe nimbly leaps from micro to macro across a dizzying geography, effortlessly mixing anecdotes, analysis, and arguments. His writing is evocative and unpretentious, allowing him to open a paragraph bluntly: “Then the whole thing blew up in the administration’s face,” and to end a chapter with a playful cliffhanger: “Why an abolitionist believed Texas annexation presented a moral crisis requires explanation.” The skill with which he crafts language allows him to show off his stunning grasp of the subject matter – a fantastic combination that contributes to his humble (tongue-in-cheek?) claim that “This book tells a story” – and, I might add, one told by a supremely skilled storyteller.

Almost two years ago, I made the decision to go to graduate school. At the time I was basking in what could only be called a history nerd’s dream summer break, spending my workdays as an intern at the New-York Historical Society‘s public programs department doing background research on professors and authors we could invite to give talks. While learning about traditional history occupied my days, learning about digital history began to occupy my evenings (at least when I wasn’t occupied with being a 21 year-old enjoying NYC). I had used GIS extensively the previous summer through a research project and had caught passing glimpses of the broader digital history universe, but I hadn’t fully explored it as a possibility for future study or (gasp) a future career. By the end of the summer, I had realized with a crystal-eyed clarity that digital history is what I wanted to “do” – in the airplane conversation sense of, “So, what do you do?”

By the time I began the actual application process, things had become even clearer, but certainly not easier. I loved history – I loved reading it, researching it, writing it, speaking it, teaching it. The idea that I could potentially spend my life doing these things made me embarrassingly giddy. At the same time, I was endlessly fascinated by the potential that lay in digital scholarship as an exciting frontier with seemingly limitless possibilities. When I sat down to my computer to start looking at schools, I began to feel the intense tug-of-war between these two impulses that would become a constant throughout the next nine months.

On the one side of the spectrum lay the traditional academic program, ivy-wrapped in prestige and brimming with names that fly off the jackets of some of my favorite books. On the other side lay the digital history program, sleekly packaged in technology and humming with voices that build the blogs and websites I trawl. On the one side, my college professors and the academic job market counseling me to apply to the very best schools I could. On the other side, my own geeky impulses were urging me to take a chance and apply somewhere new and different and exciting. I did my best to split the difference, and in the end, I was lucky enough to be accepted to a school with a fantastic combination of these two sides.

My experience led me to the conclusion that just as digital methodology is shifting the scholarly landscape of historical study, it is also altering the competitive landscape of the field. Schools are rapidly carving out digital niches for themselves, and this will prove increasingly attractive to successive waves of graduate applicants and job candidates. Most of these individuals will be reliant on accessing databases and articles online, many will be familiar with new forms of media and technology, and some will be interested in areas of visual design, data mining, or spatial analysis. However, if any of them were to ask their advisor or mentor for suggestions on programs that are strong in digital history, they’d likely hear a one-name (if any) reply: George Mason. Most advisors wouldn’t be able to point towards Nebraska-Lincoln or University of Virginia as digital history strongholds, the same way they would be able to point towards Duke and North Carolina for their strength in African-American history.

This will change. At first glance, the general structure will remain the same. “Top-tier” schools aren’t likely to start hemorrhaging applicants to less-established programs immediately. Innovative schools like UNL will continue to fight the persistent prestige-and-name-recognition battle. Nonetheless, subtler transformations will occur. Even five years ago, it would have been inconceivable that a school like George Mason (whose doctoral program has only been in existence for eight years) would be able to compete on an even footing for history applicants with a school like Stanford. Now, their success in establishing themselves as the dominant industry leader gives them an unparalleled advantage for anyone interested in digital history.

Even for schools on traditionally similar footing, an established track record of integrating new media can easily tip the scales in their favor. Much like special collections and on-campus archives, showing off a sparkling digital infrastructure will emerge as a “sexy” way to pull in both applicants and candidates. Wealthier schools may begin to invest in humanities computing centers and new kinds of software, even if for the simple goal of keeping pace with their competitors. The day is not far off when the mainstream academic history-verse buzzes with the news that an ivy-league school has “poached” a leading history professor in media and technology. Grant proposals for more applicable digital initiatives will cut out bigger slices from an expanding NEH pie.

The schools that will truly separate themselves from the pack, however, will be the ones that demonstrate their support for digital scholarship on an ideological level.  Those programs that establish a sustained committment to encourage, guide, and reward the members of their department (both faculty and students) for digital methodological inquiry will be the ones that will emerge in the best position to attract and train historians eager to tackle the technological opportunities inherent in today’s world.

The rocketing ascent of mobile technology was one of the fundamental shifts of 2008, and many market analysts predict it will only continue throughout 2009. Its rise seems to be following a two-tracked progression: individuals in developing countries are latching onto increasingly affordable mobile phones as a way to log in to a wider network, while wealthier consumers fascinated by the ability to take their online experience on-the-go are snatching up smartphones at a shocking rate (to the point where the smartphone industry appears to be recession resistant). This environment creates an intriguing medium for historians to refine and improve their craft, and the time is ripe for innovation.

Some historians have been leading the charge in utilizing this technology. Bill Turkel has been a pioneer in applying new methods in place-based computing to the field of history. Meanwhile, the majority of similar efforts fall under the sphere of public history. Some museums have long been experimenting with “electronic curators,” or hand-held audio devices that emit information about an aspect of the exhibit depending on where its carrier is standing. Cultural heritage sites, particularly battlefields and/or national parks, have quickly recognized the potential for GPS-enabled devices that guide visitors through a site. Finally, some history educators are experimenting with ways to engage their students using portable technology, including fieldwork and visitations.

Dave Lester, of George Mason University’s CHNM, presented “Mobile Historical Landscapes: Exposing and Crowdsourcing Historical Landmarks” in early April at the American Association for History and Computing conference. Dave’s is currently working on a project called HistoryPlot to encourage user participation in exploring and contributing to a knowledge bank of historical places. The idea is that roving bands of history enthusiasts could visit sites, pull out their iPhone, learn about some of its history, and possibly add both information and multimedia to the site by snapping pictures and/or uploading content – creating a kind of Yelp for the historically-minded. Dave’s project draws upon two specific advantages: 1) the participatory culture of crowdsourcing, and 2) the increasing ubiquitousness of mobile technology

Dan Cohen recently explored the advantage of crowdsourcing when he posted a historical puzzle on his blog at the start of a presentation, which asked people to identify the following picture using minimal clues:

He simultaneously sent out the puzzle via Twitter by asking his 1,600 followers to try to solve it in the next hour. The speed with which Dan got answers was impressive, with an initial correct answer coming in 9 minutes. Although he admits he should have made the puzzle a bit more difficult, the process was successful in highlighting the immense advantages of crowdsourcing historical problems using a fluid and mobile platform such as Twitter.

The growth of a mobile culture in which users are constantly connected magnifies the power of crowdsourcing. Dan’s experiment rested on the assumption that a certain number of his followers would be online and checking their tweets, and enough of them would then be able to use the internet to access his blog, read the clue, and search for the answer online. Two or three years ago, the chances of receiving an answer in 9 minutes would be much, much slimmer. A mobile culture removes barriers to accessing information, and simultaneously increases users’ expectations for accessing that information, many of whom no longer tolerate being shackled by outlets, ethernet cords, or wireless signals.

Consequently, mobile technology is redefining our social conception of space and place, and this has corresponding ramifications for historians. It revisits the fundamental relationship between a physical location and what happened in the past within that space, a relationship with which spatial and geographic historians continuously grapple. This shift is opening up a two-way street for historical researchers. On the one hand, a mobile culture allows efforts such as Dave Lester’s to shed light on previously inaccessible areas. Suddenly, a historian researching a far-away site might be able to “travel” there by looking at uploaded pictures and documents, trading emails or tweets with other researchers who have visited the place, or watching the video of a history enthusiast on vacation at the site.

On the other hand, those shifting expectations that accompany a mobile culture can also turn themselves on historical researchers. A mobile society might question the reliability of a solitary historian writing abstractly about a place they have never actually been to. A constantly connected audience will start to expect the kind of intimate access and exploration that can only be gained from hands-on visitation. A readership conditioned to read reviews on Amazon or tourists’ travel blogs will increasingly dismiss the authority of a specialist who has never visited a location they describe, even if they are describing its past. Audiences will continue to tolerate a historian’s inability to time-travel; they will not continue to tolerate an inability to place-travel.

Fortunately, mobile technology can also create a mobile historian. Imagine a historian writing about shifting gender roles on the Oklahoma Chickasaw reservation during the Dust Bowl. Armed with a laptop, digital camera, and smartphone, the historian can travel to Oklahoma and go to the reservation itself. Once there, traditional archival research is greatly enhanced by technology. Instead of lugging around 3×5 index cards, Zotero can speed up and digitize the note-taking process. The digital camera can capture documents for later perusal, allowing them to find more sources in a shorter amount of time. Is the researcher suddenly curious about gender demographics for a particular town near the reservation, or wants to understand the background to a religious ceremony referenced in a court record? They can use their smartphone to look up census data or send out queries to colleagues likely receive a rapid answer to their question.

Leaving the archives, the historian can dip into oral history by interviewing locals and recording their memories on the smartphone or digital recorder. The smartphone’s GPS capabilities allow him or her to not only locate the homes of the interviewees, but to flag and mark locations to look for spatial patterns at a later date – what if all the traditional “male” venues on a reservation were located on a specific street, while “female” venues were spread over a greater area? The GPS ability of a smartphone can capture these on-the-ground patterns. Finally, the mobile historian can quickly send out updates on their progress, receiving feedback and suggestions from a remote crowd of like-minded researchers, students, assistants, or colleagues.

Mobile technology (like all technology) is not a magic pill that will suddenly transform the historical profession. There are certainly drawbacks. First and foremost exists a strong economic barrier to entry. Already struggling for travel stipends and fellowship money, many historians won’t be able to afford a brand-new iPhone or high-quality digital camera. Those who aren’t already comfortable with mobile technology will often feel overwhelmed or at an unfair disadvantage. On a more abstract level, technology and its inherent distractions can sometimes construct blinders to one of the most important advantages to visiting a place in person: the ability to feel the sense of place, to listen to the wind and hear the accents and taste the food, a decidedly fuzzy process that adds crucial depth and richness to the historian’s understanding of their subject.

As technology itself becomes more refined and more sophisticated, the possibilities for innovation and exploration will continue to expand. As with any new methodology, the traditional skills and strengths of a historian will not fade into obsolescence. Instead, they’ll be ever more critical to the process of responsibly incorporating new techniques and approaches into the broader historical fold. If this process is even moderately successful, the future of the mobile historian appears bright.

At 3AM on April 25, 2009, I will join several hundred other participants and attempt to walk 100 kilometers (62 miles) in one day along the C&O Canal towpath, from Georgetown, D.C. to Harper’s Ferry, West Virgina. To mentally prepare for this foolhardy attempt, I thought I’d briefly walk through the canal’s history.

Courtesy of the NPS

The canal’s history as a functional transport artery is something of a prolonged tragedy. The story begins, like most tragedies, with a seemingly idyllic birthright: the canal was first championed by none other than George Washington, who also, it should be remembered, championed such things as non-partisanship, presidential term limits, and the United States of America. Apparently in his younger years, George was something of a canal enthusiast, going so far as to introduce a 1774 bill in the Virginia legislature to construct canals around obstacles in the Potomac River. Even in his presidency, George continued to lobby hard for a canal extending from the nation’s capitol and viewed it as a critical piece of his broader vision for consolidating national cohesion and opening up commerce and communications with the west:

‘”Extend the inland navigation of the eastern waters; communicate them as near as possible with those which run westward; open these to the Ohio; open also such as extend from the Ohio towards Lake Erie; and we shall not only draw the produce of the western settlers, but the peltry and fur trade of the lakes also, to our ports: thus adding an immense increase to our exports, and binding those people to us by a chain which never can be broken.’”

From Memoirs of De Witt Clinton, transcribed by Bill Carr

Armed with the ringing endorsement of one America’s most hagiograph-ied figures, the future C&O Canal looked destined to join the ranks of Route 66, the Union Pacific Railroad, and, yes, the Erie Canal, in the annals of transportation lore. By the 1820′s, a perfect storm brewed. The case for the C&O Canal was buoyed by the lucrative success of its northern cousin. Proposed in 1808 and beginning construction in 1817, the Erie Canal took little time to start turning a profit as toll revenues poured in from already-completed sections. The Canal had a profound effect on the region and country, establishing New York City as the nation’s premiere commercial center and acting as a poster child for the infrastructure boom of the 1820′s.

The Erie Canal

Digging a little deeper (pun intended) however, revealed fundamental weaknesses underlying the C&O Canal’s prospects to follow in the Erie’s footsteps. Construction began in 1828, as John Quincy Adams dug the first spadeful of earth. Fittingly for the canal, his attempt at ceremonial ground-breaking met with an unforeseen obstacle:

“It happened that at first stroke of the spade it met immediately under the surface with a large stump of tree after repeating the stroke three or four times without making any impression threw off my coat and resuming the spade, raised a shovel full of the earth at which a general shout broke forth from the surrounding multitude and I completed my address which occupied about fifteen minutes.”

John Quincy Adams

JQA Diary, number 36, 1 January 1825-30 September 1830, page 21

Beginning with this nearly-bungled case of presidential midwifery, it was all downhill from there. Five years after its construction, the canal had only progressed 60 miles, and was not yet fully operational. In hindsight, beginning construction on a canal in 1828 was poor timing. The meteoric pace of internal improvements during this period may have spurred the impetus for building the canal, but also created an infrastructure bubble that left it open to significant competition and unrealistic expectations.

In particular, the canal found itself embroiled with the Baltimore & Ohio Railroad over land-use rights along the Potomac. Although the two sides settled their dispute, the legal wrangling was symptomatic of the canal’s slow-going construction and never-ending obstacles. The B&O Railroad, which began construction on the same day as the C&O Canal, progressed far smoother, and provided a significant advantage to its clients in speed of transport. The C&O finally reached Cumberland, Maryland (linking it to the National Road) and promptly ceased construction. The B&O Railroad completed the all-important link to the Ohio River in West Virginia in 1852, usurping the canal’s purpose and rendering the slow-moving waterway somewhat moot. Which is why the B&O Railroad, instead of the C&O Canal, garnered eternal fame as a one of Monopoly’s prized properties:

The canal remained nominally operational until 1924, when a catastrophic flood wreaked havoc on the transport system, and  in 1938 the canal was sold to the federal government. By the 1950′s, in the wake of another national transportation boom, Congress entertained the idea of paving over the canal to build a scenic road for automobiles. Fortunately for canal lovers, Supreme Court Justice William O. Douglas assumed the mantle of George Washington to become the canal’s next great champion in Washington.

In an impassioned letter to the Washington Post in 1954, the nature-loving Douglas channeled his inner John Muir when he wrote, “It is a refuge, a place of retreat, a long stretch of quiet and peace at the Capitol’s back door – a wilderness area where we can commune with God and with nature, a place not yet marred by the roar of wheels and the sound of horns.” He then threw down the gauntlet as only a tree-hugging Supreme Court Justice of the 1950′s could do: by challenging proponents of paving the canal to walk its entire 184.5 miles with Douglas as their guide.

Perhaps to Douglas’s surprise, his challenge was not only accepted, but snowballed into a media spectacle. Fifty-seven people joined him on March 20, 1954 to begin hiking the trail. Although only nine of them finished the eight-day hike (many were discouraged by a snowstorm on the second day of the trip), Douglas’s expedition garnered national attention, as flocks of reporters interviewed the justice and his hikers, and onlookers showered the walkers with cheers and food. By the conclusion of the hiking circus, public support had swung towards preserving the C&O Canal.

The C&O Canal Expedition

The publicity stunt ended up saving the canal, and in 1971 the federal government designated the C&O Canal a National Historic Park. Today the canal remains a popular get-away for D.C.-area residents, who can walk, bike, jog, kayak, or in Douglas’s words (if they are in a particularly spiritual mood), “commune with God and with nature.” When plodding along the canal on Saturday, I will try to remember a poem stanza composed by Douglas’s hikers while on trail:

The knees are slowly playing out
The arches start to drop;
If we had John Brown’s body here,
We’d like to make a swap.

William "Keep Your Hands Off My Canal" Douglas

The first session of the afternoon at the AAHC conference was by Patrick Murray-John as he discussed “Interlinking Who, What, Where, When, Why, and How We Teach into a Giant EduGraph.” He gave an entertaining presentation that described the benefits and challenges of a semantic web within education. One story stood out in particular as an example for the need for better efforts in the field. Two courses might use Frankenstein as an assigned text: a class in the chemistry department that examined the portrayal of scientists in literature and an introductory English literature course. Unfortunately, the only people who know that both classes are using the same text is the school book store. If the course syllabi were located within a strong semantic web, there could be tremendous potential for interdisciplinary discovery and interaction within a university.

The second session of the afternoon was “Mapping Our Archives,” by Tim Sherratt of the National Archives of Australia. Sherratt gave an incredible presentation, as he demonstrated his work with the records of Australian servicemembers during the First World War in Mapping Our Anzacas. First up, he used Google Earth to show how he mapped where each of the 300,000 plus soldiers were from. It was remarkably powerful, as he used the platform dynamically in order to visualize the origins and density of this segment of the population. In particular, I was struck at how effectively it took advantage of the interface of Google Earth, and its re-population of point data when you zoom in, so even as you get more and more detailed, it seems like the dots just keep multiplying. The map was one point of entry for users of the database, with another critical point of entry being the opportunity for users to upload multimedia to a “scrapbook” of individual servicemembers profiles. Although some curators expressed (standard) anxiety about losing credibility and content control, they were overwhelmed at the number and quality of contributions from “laypeople,” such as this one of Donald Addison:

Tim also shared his crafting and tweaking of the database for the servicemembers’ records. Each record was scanned, and many of them included photographic sets of the individuals along with vitals and descriptions of their service. He then used CoolIris (which I discussed in one of my first blog posts) to create a virtual gallery of these archival documents, as you could move along a 3-D wall with rather haunting profile photographs and corresponding information (almost like an early 20th-century version of Facebook). He also showed a similar application of CoolIris for other archival records, including those of immigrants and laborers. I’ve always been somewhat skeptical of CoolIris as a visualization technique, always thinking it did little more than create a kind of cool way to zoom around and look at pictures. But Tim substantially changed my mind by demonstrating its power as an interface – without a doubt, being able to see an array of documents displayed in front of you, seeing the faces of Chinese immigrant children looking out at you, was a singularly powerful use of archival material that significantly enhanced the viewer’s interaction with them.

Tim’s presentation got rave reviews from its audience members, and I think I speak for all of them when I say that his use of dynamic technical interfaces (which included not only Google Earth and CoolIris, but also Tumblr and Greasemonkey scripts) was one of the most elegant, powerful, and accessible examples of bringing archival material to life that I’ve seen yet. Tim offered a fantastic glimpse into what is possible when you combine the power of digital applications with the subtlety and thoughtfulness of historical exploration.

The final session of the afternoon was a presentation from Susan Garfinkel, Judy Graves, and Jurretta Heckscher from the Library of Congress. Although I was a little fried from seven hours of absorbing and processing a host of new thoughts and ideas, I always love supporting our friends at the LOC, who have put a lot of time, thought, and effort into advancing and experimenting with digital history. Their presentations brought up several interesting points. First, they stressed that a series of surveys have shown that faculty members are less than enthusiastic about actually utilizing digitization tools and resources that their libraries provide. Additionally, the LOC has decided to expand their earlier initiative to post photographic content on Flickr Commons by posting audio and video content to iTunes and YouTube. The presentation reinforced my optimism that our national library has the courage to pursue such a variety of digital initiatives.

I drove out of George Mason’s campus feeling exhausted and inspired. The conference did a superb job of bringing together and enhancing a community of digital humanists (made even stronger by the constant Twittering activity during the day). This community isn’t necessarily representative of either the general public or even the wider historical community – in particular, I was struck at the rather overwhelming whiteness of the conference participants. I think more efforts could be made to encourage a wider diversity of perspectives and backgrounds within the field of digital history (a subject for another blog post). Nevertheless, the range, depth, and passion with which so many people are pursuing their projects gives me a strong sense of hope. I know that over the next several years I will undoubtedly struggle to balance the demands of training in a traditional graduate history program with my own passion for digital methodology and exporation. But events such as the AAHC conference bolster my faith in the incredible support system of fellow digital humanists that I can lean on as I walk down that path.

Today I attended the American Association for Historical Computing‘s 2009 annual conference, hosted by the Center for History and New Media at George Mason University. For someone interested in the field of digital history, it was a phenomenal opportunity to meet fellow enthusiasts and explore a variety of topics within the field.

The first session, a presentation by Amanda French of NYU on “Basic Digital History Skills for Historians,” came from her experience in teaching courses in digital history, many geared towards archival and library studies. Of particular interest was a survey she administered to 25 students that measured their comfort and ability in a wide variety of digital skills – everything from using social media to knowledge of metadata systems. She spoke about the fact that there was a gap between the skills being taught to public historians and archivists, and those being taught to traditional historians. Namely, those in the former group usually gain a stronger digital literacy. One of the major action points she drew from the survey was the need to teach students in the following fields: website creation, metadata, and multimedia.

Besides being the first conference presentation that I’ve live-tweeted, it brought up some interesting questions. The biggest one (that recurred throughout the day) was the question of teaching students what I’ll term hard vs. soft skills in gaining digital literacy. Should teachers expect their college students to have a basic skill set (uploading videos onto YouTube, using RSS feed readers, etc.) already? Should you spend the majority of your time teaching the skills and habits that they can then adapt to specific platforms? Is it possible to impart broader concepts of digital history without a concrete base in technical proficiency? My first instinct was to come down on the side of a liberal-artsy instruction of soft skills and underlying “big-picture” principles. But the more I thought about the issue, the more I realized that for many people, the best way of learning these soft skills is by putting on your work gloves and diving into starting a blog, using Zotero, or generating a KML file.

The second session was Dave Lester‘s “Mobile Historical Landscapes: Exposing and Crowdsourcing Historical Landmarks.” Dave explained his ongoing project (History Plot) to create a means for people to contribute to a geolocated database. He compared it to Yelp, in that he dreams of a centralized platform through which people can look up historical landmarks and their metadata (primarily their location). In order to start seeding History Plot, Dave turned to 80,000 historic sites listed in the National Registry of Historic Places. Other resources could include Wikipedia, Flickr, and partnerships with local historical societies.

Dave’s enthusiasm was downright infectious, as he spoke about being able to walk down a city street, use your iPhone to locate a nearby historical building, look up information about it, then take a photograph of it and immediately upload it to the database. Possibly the most exciting aspect, for me, was his idea of leveraging community-based history volunteers (he calls them “street teams”) to crowdsource the project. I think this has tremendous potential. History remains one of the foremost fields for armchair enthusiasts, as legions of geneologists and Civil War re-enactors would provide an incredible resource for this kind of geo-based crowd sourcing. It’s easy to imagine groups of history buffs meeting up on the weekends to explore cities and sites, snapping pictures and contributing research tidbits. I’d love for this to get off the ground, and would jump at the chance to found a local chapter.

Dan Cohen brought up a good point at the end of Dave’s talk: that the issue is finding an incentive structure so that people will actually participate in the project. In particular, there’s a gap between the (usually) younger tech-savvy crowd that lacks a strong interest in local history, and the (usually) older, less tech-savvy crowd that could potentially be the strongest source for knowledge seeding. I think it’s a manageable problem, but one that increases the need for highly accessible mobile technology and platforms that makes the barriers to entry as low as possible, even if it has to come at the cost of losing some technical robustness.

The last session of the morning was “Teaching, History, and Digital Tools: A Roundtable Discussion,” by Jeremy Boggs, Jeff McClurken, and Josh Sternfeld. All of them brought different perspectives to the topic, although each of them came from the similar experience of having taught a digital history course. It was a similar presentation to the one given by Jeremy and Jeff at the AHA convention, and it reinforced a lot of the lessons they had previously given (chief among these is Jeff’s great refrain about trying to make students “uncomfortable but not paralyzed”). One point that it made me think about was the issue of how to value historical scholarship. I’ve been thinking a lot more about not only how the historical academy values research in digital history, but how it values teaching in digital history as well. Does listing “Creating History in New Media” on your C.V. as a course you taught carry more weight than listing an American history survey? Would a tenure review board be impressed with your tech-savvy literacy, or put off because they don’t understand it?