historying

“mr Ballard left home bound for Oxford. I had been Sick with the Collic. mrs Savage went home. mrs foster Came at Evening. it snowd a little.”

This is the first entry in the diary of Martha Ballard. Martha Ballard was a rural Maine midwife who kept an extensive diary between 1785 and 1812 and whose life was immortalized in 1990 by the historian Laurel Thatcher Ulrich‘s award-winning A Midwife’s Tale. Over the course of three decades, Ballard kept a meticulous, near-daily accounting of her life spanning over 10,000 entries.

When reading A Midwife’s Tale, I was struck by how readily the text would seem to lend itself to digital analysis. In an interview, Ulrich noted, “The very thing that had attracted me to the diary in the first place was also the thing that made it difficult to work with. I mean there’s just so much.” To ground herself, she began by simply counting things: “And I would go day by day for every other year of the diary, and I would tick off what was in each entry: baking or brewing, spinning or washing, or trading, sewing, mending, deliveries, general medical accounts, going to church, visitors, people coming for meals, etc.” Because of the sprawling scope, she took this quantitative approach only for the even-numbered years in the diary. The fact that she was working in the late eighties without a computer makes her work even more impressive.

After poking around online I came across DoHistory.org, a website developed and maintained by the Film Study Center at Harvard University and hosted by (who else, really) George Mason’s CHNM. The website presents the diary to the public in two formats: the viewer can either browse through photographed pages of the diary or read the transcript of the pages (transcribed through a monumental effort by Robert R. McCausland and Cynthia MacAlman McCausland):

When I realized the entire diary was online, it got me thinking about possibilities for text mining. As an aspiring digital humanist with little “hard” skills beyond basic GIS, I had been meaning to learn how to program for quite some time. In Martha Ballard’s diary, I had an intriguing source of data with which to learn how to do so. Now I just had to learn how to program. With the patient help of several programming-savvy family members, I gradually learned the basics of Python and how to apply it to Martha Ballard’s diary. What follows are the first steps we took to process the diary’s raw data into an accessible digital format.

Process

At first, I briefly considered learning how to scrape the text of the diary off the website. After some investigation, I decided that was a little beyond my abilities, so I copped out to the much easier route of sending an email to Kelly Schrum at CHNM, who kindly forwarded my request to Ammon Shepherd, who emailed me a zip file containing 1,431 html documents, one for each page of the diary. The html files of the transcribed diary are a basic, 3-column table that look this. My first step was to find a way to strip out the html tags and organize the text into a systematic database of individual entries. Fortunately, Ballard’s meticulousness and consistency lent itself well to such an approach.

The diary’s format translates quite nicely into creating a list of lists – the “main” diary being a list of all the entries, and each entry being a list in and of itself. The first program we wrote was to open each html file and begin extracting the different sections of text (which were conveniently marked by html tags). Iterating through each entry allowed us to separate the different columns in her diary into different items in the list. Here is the breakdown of our “list of lists”:

1. Diary
   1. Entry
      1. Date
         1. Month
         2. Day
         3. Year
      2. Day of the Week
      3. Main Text of Entry
      4. Day Summaries (Column 3 of actual diary entry)
      5. Birth(s) (Recorded in Column 1 of actual diary entry)

In creating the list, we had to separate out the raw data from the html tags that formatted it. Fortunately, the folks who built the html files originally used an extremely systematic formatting process that actually made the job of distilling one from the other quite straightforward. A Python module called Pickle allowed us to export the list of entries as a manageable single file that we could then easily import into future programs to manipulate.

For example, the third entry in the diary would translate a bit into something like this:

1. Diary
   1. Entry (3)
      
      1. Date
         1. 1 (January)
            
         2. 3
            
         3. 1785
   2. 3 (Tuesday – Ballard numbered the weekdays, beginning with Sunday as 1)
   3. “Tuesday. mrs. Foster went home. I had threats of thee Collic; by takein peper found releif.”
      
   4. Empty
   5. Empty

The list allows us to access pieces of information by “calling” their position. It helped me to think of the entire diary list as a warehouse containing almost 10,000 boxes (entries) inside it, with each box containing five compartments, with the first of those compartments divided into three sub-compartments. If you were to open any of the boxes (entries) and look inside the first compartment, then inside sub-compartment number two, you would always find a number that represented the month of that particular entry. If you were to look inside the third compartment of the entry/box, you would always find the main text for that day’s entry.

The advantages of setting up the data in a list structure is the ability to access these specific pieces of information easily and to compare them across entries. In many ways, processing the text to make it readable and programmable is one of the biggest challenges to text mining. Deciding on the most logical way to organize and break down over 1,400 files will lay the groundwork for the fun part: writing programs to actually analyze the diary of Martha Ballard.

***Special-edition sneak preview of future posts in this series***

A simple counting program reveals that the main text of Martha Ballard’s diary alone contains 377,315 words, spanning I-couldn’t-make-this-number-up 9,999 entries. That is a lot of data to play with.

One of the sharper distinctions between the digital humanities and traditional scholars is an acceptance and emphasis on collaboration. Lisa Spiro has written several convincing posts that detail how scholars in the digital humanities are far more likely to work together and co-author essays, along with some examples of collaborative projects. At the NEH’s Office for Digital Humanities, the first requirement for applying to a grant for a fellowship at a Digital Humanities Center is to: “support innovative collaboration on outstanding digital research projects.” Meanwhile, many disciplines within the humanities cling to the notion of the individual scholar. Cathy Davidson of HASTAC tells the story of job-seeking and being told that collaborative work didn’t “count” as legitimate scholarship: “I felt like Hester Prynne wearing her Scarlet A . . . for Adulterous Authorship.” The academy remains enamored with putting a single face and a single name to research; the vast majority of the annual prizes given by the AHA are presented to individual historians for individual work.

The reasons for this distinction are easy to understand. Most digital humanities initiatives are inherently multidisciplinary. There are those among us lucky or hard-working enough to possess both “soft” humanistic talent and “hard” technical skills, but for the majority of us it is much more efficient and effective to split the workload of multiple, and often very different, approaches between more than one person. Why spend six months trying to master the intricacies of MySQL when you can team up with a colleague who already knows how to implement it? Teaming up with other people across disciplines is a form of self-preservation that saves everyone time and energy.

Another reason for the distinction often stems from the basic nature of the projects – many digital humanists have focused on building tools, online collections, and interactive media. Whereas as most academic monographs are aimed at an audience of fellow academics, these projects are inherently designed with a broader public in mind. With that overarching goal, collaboration during the production phase becomes an almost instinctive (and necessary) pursuit. Similarly, scholarly specialization leads to (often) intense intellectual turf wars. If you are struggling to make your academic mark on a very specific focus within a very specific sub-field, other people working on that same field can often seem more like a threat than a resource. These jealously guarded barriers are less prevalent within the digital humanities community, given its emphasis on greater transparency and a broader scope of study.

This is not to say that traditional humanists are allergic to collaboration. Established (read: tenured) professors are often much more willing to edit volumes, co-author essays, and work together on research projects. When you are a successful author and Harvard historian like Jill Lepore, you can afford to take a chance and co-write a work of historical fiction. An associate professor at a small state school struggling to get tenure? Not so much. Younger scholars are still plagued by the never-ending issue of digital scholarship not “counting” as a valid accomplishment.

Most graduate (particularly Ph.D) programs in the humanities simply do not train their students to play well (or at all) with others. Writing a dissertation is still viewed as an infamously lonesome pursuit. Doing so establishes your credentials as an individual scholar capable of producing original work. Unfortunately, this not only reinforces the conception that anything other than individual research is somehow less valued, but it also does a terrible job of preparing students to do any kind of future collaborative work. Learning how to take notes in an archive or write manuscript chapters are critical skills, but so is learning how to delegate tasks to research partners or co-author a grant proposal.

There is no reason why the traditional humanities cannot begin to embrace scholarly collaboration. Even for those with no interest in digital initiatives, increased collaboration creates a ripple effect. There are the obvious benefits: different perspectives add richness and depth to studies, a division of labor and specialization can lead to greater efficiency, and more collaborators often facilitates future connections across otherwise-insular academic networks. Almost every scholar has the story of a single conversation, comment, or idea from a colleague, friend, or family member sparking a revelation or major advancement in their work. Official collaboration only magnifies this effect, and the academy as a whole would benefit.

Collaboration is not a cure-all, and it presents its own set of quite-formidable challenges. As every high-schooler working on a group project or cubicle-dweller sitting in a meeting can tell you, working with other people can often be a frustrating experience. How do you divide up responsibilities, reconcile different opinions, share both criticism and credit? A professor of literature sitting across the table from a computer scientist will probably have a lot of trouble communicating effectively with each other. All of these issues have the potential to be even sharper inside the humanities, where most scholars have been given little to no official instruction or practical experience in how to work together. Nevertheless, the potential for concerted collaboration to spur on academic discovery within the humanities is simply too high to ignore.

It’s now been over a year since I started history-ing and over a month since my last post, so I thought I’d ease back into writing by reflecting on a year in the blogosphere.

1. Intellectual stimulation

One of the most jarring changes going from a college lifestyle to the workforce was the lack of academic stimulation on a daily basis. and problem-solving transitioned from a classroom to the office. Having a blog gave me an impetus to really think about issues. It forced me to write (semi) regularly, to think about issues, to engage in at least a limited conversation on intellectual topics I cared about. Instead of being a passive consumer of ideas, posts, articles, essays, and books, I became an active one.

The knowledge that my writing would be open and available for anyone to read and judge made me think even harder to develop my own ideas and opinions. If you write a shitty paper in a college seminar, the professor gives you a shitty grade and you file it away. If you write a shitty post, it’s out there for anyone to read. Employers, colleagues, professors, admissions people – all of them now have a growing body of my writing to read, disagree with, and critique if they’re so inclined. For an unestablished scholar like myself, this provides some major motivation to really think and work at what I write.

2. Joining a community

Blogging also let me jump into a vibrant online community of digital historians and humanists. Instead of being something of a sideline observer, I laced up and joined the fray. Doing so not only exposed me to a wide range of new ideas and possibilities, but also introduced me to a number of fascinating and inspiring people – many of whom I met in person at the AAHC and THATCamp conferences. Especially for a younger scholar like myself, having a blog gave me confidence in my credentials and allowed me to participate in a wider dialogue.

Moving forward, the connections I’ve made through blogging (and on a noisier level, Twitter) will serve me for a long time to come. I’ve been lucky in that before I’ve even stepped foot inside a graduate classroom, I’ve have had the opportunity to interact with so many people who I (hope) will be my future colleagues and collaborators. In the insular world of traditional academics, this is a relative rarity.

3. Feedback

I’m a firm believer that there’s no point in writing into a void. While much of my blogging was “for myself,” in that I wrote about what interested me, the most rewarding part by far is the response I’ve received. There is certainly an egotistical and superficial element to checking  site-visit stats. But there is some validity to the point that my writing has already reached a larger audience in a year than all of my undergraduate writing put together. By a long shot. For example, my most popular post, by almost a 2:1 factor, is a rudimentary text analysis of Venture Smith’s narrative. As of today, it had been viewed over a thousand times. This metric might be a tiny drop in the blogosphere bucket, but it will certainly eclipse any audience I’ll have for my traditional academic research, at least in the near future.

One of the more rewarding episodes occurred recently, when a local Connecticut writer contacted me through my blog because she was interested in  Venture Smith. She had stumbled across my posts talking about my undergraduate research on Venture Smith, and had been inspired to do some truly remarkable research on her own. We met yesterday, and I was thrilled to find that not only had she uncovered a fascinating new development, but that it directly related to work I had done. I was humbled to hear that my blog had been an impetus for her to get involved in the Venture Smith community. It served as a great reminder of how blogs can increase transparency and lower barriers between academics and the wider public.

I’m not sure what the future will hold for history-ing. There are bad as well as good aspects of maintaining a blog, and it remains to be seen whether it will survive the time-drain of graduate school. Regardless, blogging at history-ing has been, and I hope will continue to be, an enriching experience.