By the late 1940s and early 1950s, working-class whites felt themselves under siege from what they saw as a black invasion of their neighborhoods and public spaces such as parks and swimming pools. Working-class whites at first turned to organized violence and intimidation, but soon realized the importance of winning the battle for public image. In Kruse’s words, “In time, they would learn to put aside the brown shirts of the [white supremacist] Columbians and the white sheets of the Klan and instead present themselves as simple homeowners and concerned citizens.” (44) On an ideological level, they moved from trying to protect the integrity of their communities (a cohesion that Kruse convincingly undermines), and instead began to emphasize their individual rights and liberties to live amongst whomever they chose. In many neighborhoods, their struggle was not enough, as the first wave of black homeowners caused a stampede of white individuals rushing to sell their homes before property values decreased.

Meanwhile, a similar battle over the desegregation of public schools led middle-class whites into the fray during the 1950s. Segregationist leaders quickly picked up on a central theme that ran through their movement (and one that runs through White Flight as well): “freedom of association.” For a middle-class white father, barring blacks from attending the same school as his daughter was purportedly less about denying black people rights as it was preserving his own right to determine who his daughter could and should interact with. Even as this line of reasoning proved ineffectual at halting desegregation, white families fled from public schools into private ones, creating a second-wave of de facto segregation in Atlanta’s school system.

The third stage of white flight came in the early 1960s. As working and middle-class whites faced the integration of their neighborhoods, parks, and schools, many upper-class whites observed the conflict form a distance, safely ensconced in their wealthy neighborhoods, country clubs, and private schools. But with the passage of the Civil Rights Act, suddenly their businesses came under direct assault. Elite businessman, hitherto allied in a moderate coalition with white politicians and black leaders, bitterly struggled against organized sit-in protests and later government injunctions that aimed to desegregate their restaurants and department stores. It was during their struggle that the earlier shifts towards individual rights and privatization crystallized into an organized and increasingly powerful conservative ideology.

The strength of Kruse’s argument lies in tracing this conservative political crystallization, sometimes at the expense of a more rigorous analysis of white flight as a spatial phenomenon. While maps are scattered throughout White Flight, most of them serve as modest visual signposts, when they have the potential to more deeply enrich the project. Nevertheless, Kruse persuasively argues that this tandem of political and spatial movements had profound historical implications. As white Americans increasingly coalesced into white suburban (and later exurban) enclaves, they eventually became the backbone of the Republican party. This “politics of suburban secession,” maintained the traditional tenets of white flight: retreating from any and all interaction with the black community (now synonymous the city itself) and championing minimal government, headlong privatization, and the primacy of the individual.

Kruse is an adept narrator, weaving together a host of characters and events into a compelling storyline of the racial landscape of Atlanta during the mid-20^th century. He paints a convincing portrait of a coalescing conservative movement based on withdrawal and charts the distinctive class divisions within this movement. The reader is sometimes left wishing for the kind of broader analysis that mainly occupies the final chapter and epilogue of his book. Atlanta’s patterns of white flight were simultaneously taking place in spaces across the country, yet Kruse offers only passing glimpses of how the city fit within a national framework. Despite this, White Flight remains a compelling case study on the origins of the modern conservative movement within the social and political backlash against the civil rights movement.

Of course, my proselytizing is a bit disingenuous. The dirty little secret is that not all data is created equal. And especially within the humanist’s turf, most digitized sources are rarely “machine-ready”. The more projects I work on, the more and more convinced I become that there is one real constant to them: I always spend far more time than I expect preparing, cleaning, and improving my data. Why? Because I can.

A crucial advantage to digital information is that it’s dynamic and malleable. You can clean up a book’s XML tags, or tweak the coordinates of a georectified map, or expand the shorthand abbreviations in a digitized letter. Which is all well and good, but comes with a pricetag. In a way that is fundamentally different from the analog world, perfection is theoretically attainable. And that’s where an addictive element creeps into the picture. When you can see mistakes and know you can fix them, the temptation to both find and fix every single one is overwhelming.

In many respects, cleaning your data is absolutely crucial to good scholarship. The historian reading an 18th-century newspaper might know that “Gorge Washington” refers to the first president of the United States, but unless the spelling error gets fixed, that name probably won’t get identified correctly by a computer. Of course, it’s relatively easy to change “Gorge” to “George”, but what happens when you are working with 30,000 newspaper pages? Manually going through and fixing spelling mistakes (or, more likely, OCR mistakes) defeats the purpose and neuters the advantage of large-scale text mining. While there are ways to automate this kind of data cleaning, most methods are going to be surprisingly time-intensive. And once you start down the path of data cleaning, it can turn into whack-a-mole, with five “Thoms Jefferson”s poking their heads up out of the hole for every one “Gorge Washington” you fix.

Chasing the “perfect data” dragon becomes an addictive cycle, one fueled by equal parts optimism and fear. Having a set of flawlessly-encoded Gothic novels could very well lead to the next big breakthrough in genre classification. On the other hand, what if all those missed “Gorge Washingtons” are the final puzzle pieces that will illuminate early popular conceptions of presidential power? The problem is compounded by the fact that, in many cases, the specific errors can be fixed. But in breathlessly attempting to meet the “data deluge” problem, the number and kind of specific errors get multiplied by several orders of magnitude over increasingly larger and larger bodies of information and material – which severely complicates the ability to both locate and rectify all of them.

At some point, the digital material has to simply be “good enough”. But breaking out of the “perfect data” dragon-chasing is easier said than done. “How accurate does my dataset have to be to in order to be statistically relevant?” “How do I even know how clean my data actually is?” “How many hours of my time is it worth to bump up the data accuracy from 96% to 98%?” These are the kinds of questions that DH researchers suddenly struggle with – questions that a background in the humanities ill-prepares them to answer. Just like so many aspects of doing this kind of work, there is a lot to learn from other disciplines.

Certain kinds of data quality issues get mitigated by the “safety in numbers” approach. Pinpointing the exact cross-streets of a rail depot is pretty important if you’re creating a map of a small city. But if you’re looking at all the rail depots in, say, the Midwest, the “good enough” degree of locational error gets substantially bigger. Over the course of thirty million words, the number of “George Washingtons” are going to far outweigh and balance out the number of “Gorge Washingtons”. With large-scale digital projects, it’s easier to see that chasing the “perfect data” dragon is both impossible and unnecessary. On the other hand, certain kinds of data quality problems get magnified with a larger scale. Small discrepancies get flattened out with bigger datasets. But foundational or commonly-repeated errors get exaggerated with a larger dataset, particularly if some errors have been fixed and others not. For instance, if you fixed every “Gorge Washington” but didn’t catch the more frequently misspelled “Thoms Jefferson”, comparing the textual appearances of the two presidents over those thirty million words is going to be heavily skewed in George’s direction.

As non-humanities scholars have been demonstrating for years, these problems aren’t new and they aren’t unmanageable. But as digital humanists sort through larger and larger sets of data, it will become increasingly important to know when to ignore the dragon and when to give chase.

By any measure, Valley of the Shadow has been a phenomenal success. Over the course of a decade and a half, it has provided the catalyst for a host of books, essays, CD-ROM’s, teaching aids, and articles – not to mention more than a few careers. At times it seems that everyone and their mother in the digital history world has some kind of connection to Valley of the Shadow. The impact the project has had, both within and outside of the academy, is a bit overwhelming. In this light, I decided to revisit Valley of the Shadow with a more critical lens and examine how it has held up over the years.

At the bottom of the Valley’s portal, it reads “Copyright 1993-2007.” There aren’t many academic sites that can claim that kind of longevity, but this also carries a price. In short, the website already feels a bit dated. The structure of the website is linear, vertical, and tree-like. The parent portal opens up into a choice between three separated sections: The Eve of War (Fall 1859 – Spring 1861), The War Years (Spring 1861 – Spring 1865), and The Aftermath (Spring 1865 – Spring 1870). Each of these are divided into different repositories of source material, from church records to tax and census data to battle maps. Clicking on a repository leads to different links (for instance, two links leading to the two counties’ letters). A few more clicks can lead to, say, a letter from Benjamin Franklin Cochran to his mother in which he leads off with the delicious detail of lived experience that historians love: “I am now writing on a bucket turned wrong side up.”

In this sense, the database is geared towards a vertical experience, in which users “drill down” (largely through hyperlinks) to reach a fine-grained level of detail: Portal -> Time Period -> Source Material Type -> County -> Letter. What this approach lacks is the kind of flexible, horizontal experience that has become a hallmark of today’s online user experience. If one wanted to jump from Cochran’s letter to see, for instance, battle maps of the skirmishes he was referencing or if local newspapers described any of the events he wrote about, the process is disjointed, requiring the user to “drill up” to the appropriate level and then “drill down” again to find battle maps or newspapers. This emphasis on verticality is largely due to the partitioned nature of the website, divided as it is into so many boxed categories. This makes finding a specific source a bit easier, but restricts the exploratory ability of a user to cross boundaries between the sites’ different eras, geography, and source types.

If different sections of the website are partitioned from one another, what kind of options exist for opening the database itself beyond the websites own walls? In October of 2009, NiCHE held a conference on Application Programming Interfaces (APIs) for the Digital Humanities, with the problem it was tackling outlined as follows:

To date, however, most of these resources have been developed with human-friendly web interfaces. This makes it easy for individual researchers to access material from one site at a time, while hindering the kind of machine-to-machine exchange that is required for record linkage across repositories, text and data mining initiatives, geospatial analysis, advanced visualization, or social computing.

This description highlights the major weakness of Valley of the Shadow: its (relative) lack of interactiveness and interoperability. A human researcher can access specific information from the website, but it remains a major challenge to employ more advanced digital research techniques on that information. Every database is inherently incomplete. But one way to mitigate this problem is to open up the contents of a database beyond the confines of the database itself. The following scenario might fall under the “pipe-dream” category, but it illustrates the potential for an online database: a researcher writes a programming script to pull out every letter in Valley of the Shadow written by John Taggart, search both the Valley’s database and national census records in order to identify the letters’s recipients, capture each household’s location and income level, and use that data to plot Taggart’s social world on a geo-referenced historical map or in a noded social network visualization. Again, this might be a pipe-dream, but it does highlight the possibilities for opening up Valley of the Shadow’s phenomenally rich historical content into a more interactive and interoperable database.

At the end of the day, Valley of the Shadow deserves every ounce of acclaim it has received. Beyond making a staggering array of primary sources available and accessible to researchers, educators, and students, it helped pave the way for the current generation of digital humanists. Valley of the Shadow embodies many of the tenets of this kind of scholarship: multi-modal, innovative, and most importantly, collaborative. Its longevity and success speaks to the potential of digital history projects, and should continue to serve as a resource and model moving forward.

1. I read a lot.

I’ve never read this much in my life, and it took me most of the quarter to learn how to sit down and read books for hours and hours every day.

Given that this was my first term and that I was only enrolled in two reading-intensive courses, I’m a tiny bit terrified of what the future (cough, studying for orals) holds. Reading for so many hours has sadly all but demolished any inclination towards pleasure reading. On the flip side, I’ve gained a new appreciation for Hulu.

2. I’ve met amazing people.

It took me a little while to get used to the fact that I was surrounded by ridiculously intelligent people, including a classmate with his own Wikipedia page and an advisor who won a Macarthur Genius Award and who’s also way cooler than I am. All thirteen of my fellow cohort members are incredibly bright, inquisitive, and quite inspiring to think of as future colleagues. Everyone I’ve met outside of the history department has been just as impressive. Meanwhile, I have a blog that I struggle to update more than once a month.

3. I still don’t know what I want to study.

This is going to be a recurring problem. The downside to having a focus on digital methodology is that I have yet to figure out what I actually want to use that methodology to study. I don’t like the idea of confining myself to a narrow thematic topic, but it’s a step I’ll have to take eventually (one I plan on putting off for as long as possible). In the meantime, I’ll just have to keep answering the questions of “American history? Anything more specific?” with “Um, no. Not yet.”

4. I made the right decision.

Stanford has been a phenomenal fit so far. The best example I can give is that I had the opportunity to take a non-history course titled Literary Studies and the Digital Library taught by Matt Jockers. It was a seminar class in the English department, and included undergraduates, graduate students, and a fairly broad sample of various disciplines. The course centered around designing a digital text mining project to examine over one thousand American and British works from the nineteenth century, culminating in a three-paper proposal to the Digital Humanities 2010 conference in London (fingers crossed). Matt has since extended the course into the next quarter to let us continue to work on the project.

I could not have been happier with the course, and it embodies many of what I consider to be the core tenets of the digital humanities: hands-on technical problem-solving, interdisciplinarity, and intensive collaboration. I have almost no background in English – during class discussion I at one point googled “buildings romane,” which a helpful classmate had to correct to “Bildungsroman.” Despite this, or maybe because of this, I learned not only about cool things like epistolary novels and the rise of serialization, but also the differences in how literary critics and historians approach questions and problems. This interdisciplinary experience will serve me well in the future.

At the end of the day, I feel incredibly lucky to be given the opportunity to be doing something that I love. As one of my classmates said to me over lunch, “Isn’t it absurd that we’re somehow getting paid to be here?” Not yet jaded from TA’ing, orals, dissertation writing, or the job market, I could not agree more.

Beyond being visually arresting and creative, the campaign offers up a vision of America that (by mainstream Madison Avenue standards) is fresh and edgy. The basic set-up of the sixty-second commercials is flashing imagery of denim-clad youngsters moving frenetically. Sounds like a fairly typical clothing ad. Except that it includes footage of post-Katrina New Orleans and is set to a Walt Whitman poem – in one of the spots, (supposedly) the reading comes from a wax cylinder recording of Whitman himself. Put in comparison to a concurrently-running ad campaign by Wrangler that involves Brett Favre tossing a football to a George Thorogood soundtrack, and you really get a sense for just how different this campaign is:

The imagery isn’t super sophisticated – a neon AMERICA sign half-submerged in flood water opens and closes the “America” spot. Some people might feel that throwing in a kissing interracial couple (or a kissing gay couple in “OPioneers!”) is tokenizing. But Levi’s has managed to construct a divergent conception of what exactly is America, no small feat for a corporate ad campaign. The new commercials are oddly triumphant, but with a disquieting edge to them. Children are running through fields, but in this new world they’re doing so under a looming electrical grid. There is laughter and muscle-flexing and vibrancy, but it’s against a backdrop of chain-link fences or broken down buildings. Blue jeans have constituted an enduring symbol of rural, down-to-earth, industrial America, an image that Levi’s has helped to cultivate in its lengthy, 130+ year-old history.  The fact that the same company would now stake itself to such a contrasting campaign speaks volumes. Is Levi’s banking on a collective shift in the American psyche? That we are open to moving beyond a cornfields-and-cowboys idea of American denim? What exactly is the alternative vision they’re hoping the American consumer will identify with? I have no idea, and that’s part of what makes this campaign intriguing.

Because each word has information attached to it – specifically what date it was written – we can look at long-term patterns for a particular word’s usage. However, looking at only raw word frequencies can be problematic. For example, if Ballard wrote the word yarn twice as often in 1801 as 1791, it could mean that she was doing a lot more knitting in her old age. But it could also mean that she was writing a lot more words in her diary overall. In order to address this issue, for any word I was examining I made sure to normalize its frequency – first by dividing it by the total word count for that year, then by dividing it by the average usage of the word over the entire diary. This allowed me to visualize how a word’s relative frequency changed from year to year.

In order to visualize the information, I settled on trying out sparklines: “small, intense, simple datawords” advocated by infographics guru Edward Tufte and meant to give a quick, somewhat qualitative snapshot of information. To test my method, I used a theme that Laurel Ulrich describes in A Midwife’s Tale: land surveying. In particular, during the late 1790s Martha’s husband Ephraim became heavily involved in surveying property. In the raw word count list, both survey and surveying appear in the top 500 words, so I combined the two and looked at how Martha’s use of them in her diary changed over the years (1785-1812):

survey(ing)

Looking at the sparkline, we get a visual sense for when surveying played a larger role in Martha’s diary – around the middle third, or roughly 1795-1805, which corresponds relatively well to Ulrich’s description of Ephraim’s surveying adventures. As a basis for comparison, the word clear appeared with numbing regularity (almost always in reference to the weather):

clear

Using word frequencies and sparklines, I could investigate and visualize other themes in the diary as well.

Religion

Out of the 500 most frequent words in the diary, only three of them relate directly to religion: meeting (#28), worship (#143), and god (#220).

meeting

worship

god

Meeting, which was used largely in a religious context (going to a church meeting), but also in a socio-political context (attending town meetings), had a relatively consistent rate of use, although it trended slightly upwards over time. Worship (which Martha largely used in the sense of “went to publick worship”), meanwhile, was more erratic and trended slightly downwards. Finally, and perhaps most interestingly, was Martha’s use of the word god. Almost non-existent in the first third of her diary, it then occurred much more frequently, but also more erratically over the final two-thirds of the diary. Not only was it a relatively infrequent word overall (flax, horse, and apples occur more often), but its usage pattern suggests that Martha Ballard did not directly invoke a higher power on a personal level with any kind of regularity (at least in her diary). Instead, she was much more comfortable referring to the more socially and community-based activity of attending a religious service. While a qualitative close reading of the text would give a richer impression of Martha’s spirituality, a quantitative approach demonstrates how little “real estate” she dedicates to religious themes in her diary.

Death

death

dead

funeral

expired

interd

Most of the words related to death show an erratic pattern. There are peaks and valleys across the years without much correlation between the different words, and the only word that appears with any kind of consistency is interd (interred). In this case, word frequency and sparklines are relatively weak as an analytical tool. They don’t speak to any kind of coherent pattern, and at most they vaguely point towards additional questions for study – what causes the various extreme peaks in usage? Is there a common context with which Martha uses each of the words? Why was interd so much flatter than the others?

Family

In this final section, I’ll offer up a small taste of how analyzing word frequency can reveal interpersonal relationships. I used the particular example of Dolly (Martha’s youngest daughter):

dolly

The sparkline does a phenomenal job of driving home a drastic change in how Martha refers to her daughter. In a matter of a year or two in the mid 1790s, she goes from writing about Dolly frequently to almost never mentioning her. Why? Some quick detective work (or reading page 145 in A Midwife’s Tale) shows that the plummet coincides almost perfectly with Dolly’s marriage to a man named Barnabas Lambart in 1795. But why on earth would Martha go from mentioning Dolly all the time in her diary to going entire years without writing her name? Did Martha disapprove of her daughter’s marriage? Was it a shotgun wedding?

The answer, while not so scandalous, is an interesting one nonetheless that text analysis and visualization helps to elucidate. In short, Martha still writes about her daughter after 1795, but instead of referring to her as Dolly, she begins to refer to her as Dagt Lambd (Daughter Lambert). This is a fascinating shift, and one whose full significance might get lost by a traditional reading. A human poring over these detailed entries might get a vague impression that Martha has started calling her daughter something different, but the sparkline above drives home just how abrupt and dramatic that transformation really was. Martha, by and large, stopped calling her youngest daughter by her first name and instead adopted the new husband’s proper name. Such a vivid symbolic shift opens up a window into an array of broader issues, including marriage patterns, familial relationships, and gender dynamics.

Conclusions

Counting word frequency is a somewhat blunt instrument that, if used carefully, can certainly yield meaningful results. In particular, utilizing sparklines to visualize individual word frequencies offers up two advantages for historical inquiry:

1. Coherently display general trends
2. Reveal outliers and anomalies

First, sparklines are a great way to get a quick impression of how a word’s use changes over time. For example, we can see above that the frequency of the word expired steadily increases throughout the diary. While this can often simply reiterate suspected trends, it can ground these hunches in refreshingly hard data. By the end of the diary, a reader might have a general sense for how certain themes appear, but a text analysis can visualize meaningful patterns and augment a close reading of the text.

Second, sparklines can vividly reveal outliers. In the course of reading hundreds of thousands of words over the course of nearly 10,000 entries, it’s quite easy to lose sight of the forest for the trees (to use a tired metaphor). Visualizing word frequencies allows historians to gain a broader perspective on a piece of the text, and they also act as signposts pointing the viewer towards a specific area for further investigation (such the red-flag-raising rupture in how frequently Dolly appears). Relatively basic word frequency by itself (such as what I’ve done here) does not necessarily explain anomalies, but it can do an impressive job of highlighting important ones.

As births were such a consistent and methodically recorded theme in Ballard’s life, I decided to begin my programming with a basic examination of the deliveries she attended. This examination would take the form of counting the number of deliveries throughout the course of the diary and grouping them by various time-related characteristics, namely: year, month, and day of the week.

Process and Results

The first basic step for performing a more detailed text analysis of Martha Ballard’s diary was to begin cleaning up the data. One step was to take all the words and (temporarily) turn every uppercase letter into a lowercase letter. This kept Python from seeing “Birth” and “birth” as two separate words. For the purposes of this particular program, it was more important to distill words into a basic unit rather than maintain the complexity of capitalized characters.

Once the data was scrubbed, we could turn to writing a program that would count the number of deliveries recorded in the diary. The program we wrote does the following:

1. Checks to see if Ballard wrote anything in the “birth” column (the first column of the entries that she also used to keep track of deliveries)
2. If she did write anything in that column, check to see if it contains any of the words: “birth”, “brt”, or “born”.
3. I then printed the remainder of the entries that contained text in the “birth” column but did not contain one of the above words. From this short list I manually added an additional seven entries into the program, in which she appeared to have attended a delivery but did not record it using the above words.

Using these parameters, the program could iterate through the text and recognize the occurrence of a delivery. Now we could begin to organize these births.

First, we returned the birth counts for each year of the diary, which were then inserted into a table and charted in Excel:

At the risk of turning my analysis into a John Henry-esque woman vs. machine, I compared my figures to the chart that Laurel Ulrich created in A Midwife’s Tale that tallied the births Ballard attended (on page 232 of the soft-cover edition). The two charts follow the same broad pattern:

Note: I reverse-built her chart by creating a table from the printed chart, then making my own bar graph. Somewhere in the translation I seem to have misplaced one of the deliveries (Ulrich lists 814 total, whereas I keep counting 813 on her graph). Sorry!

However, a closer look reveals small discrepancies in the numbers for each individual year. I calculated each year’s discrepancy as follows, using Ulrich’s numbers as the “true” figures (she is the acting President of the AHA, after all) from which my own figures deviated, and found that the average deviation for a given year was 4.86%. Apologies for the poor formatting, I had trouble inserting tables into WordPress:

Keeping the knowledge in the back of my mind that my birth analysis differed slightly from Ulrich’s, I went on to compare my figures with other factors, including the frequency of deliveries by month over the course of the diary.

If we extend the results of this chart and assume a standard nine-month pregnancy, we can also determine roughly which months that Ballard’s neighbors were most likely to be having sex. Unsurprisingly, the warmer period between May and August appears to be a particularly fertile time:

Finally, I looked at how often births occurred on different days of the week. There wasn’t a strong pattern, beyond the fact that Sunday and Thursday seemed to be abnormally common days for deliveries. I’m not sure why that was the case, but would love to hear speculation from any readers.

Analysis

The discrepancies between the program’s tally of deliveries and Ulrich’s delivery count speak to broader issues in “digital” text mining versus “manual” text mining:

Data Quality

Ulrich’s analysis is a result of countless hours spent eye-to-page with the original text. And as every history teacher drills into their students when conducting research, looking directly at the primary documents minimizes the degrees of interpretation that can alter the original documents.  In comparison, my analysis is the result of the original text going through several levels of transformation, like a game of telephone:

Original text -> Typed transcription -> HTML tables -> Python list -> Text file -> Excel table/chart

Each level increases the chance of a mistake.  For instance, a quick manual examination using the online version of the diary for 1785 finds an instance of a delivery (marked by ‘Birth’) showing up in the online HTML, but which does not appear in the “raw” HTML files our program is processing and analyzing.

On the other hand, a machine doesn’t get tired and miscount a word tally or accidently skip an entry.

Context

Ulrich brings to bear on the her textual analysis years of historical training and experience along with a deeply intimate understanding of Ballard’s diary. This allows her to take into account one of the most important aspects of reading a document: context. Meanwhile, our program’s ability to understand context is limited quite specifically to the criteria we use to build it. If Ballard attended a delivery but did not mark it in the standard “birth” column like the others, she might mention it more subtly in the main body of the entry. Whereas Ulrich could recognize this and count it as a delivery, our program cannot (at least with the current criteria).

Where the “traditional” skills of a historian come into play with data mining is in the arena of defining these criteria. Using her understanding of the text on a traditional level, Ulrich could create far, far superior criteria than I could for counting the number of deliveries Martha Ballard attends. The trick comes in translating a historian’s instinctual eye into a carefully spelled-out list of criteria for the program.

Revision

One area that is advantageous for digital text mining is that of revising the program. Hypothetically, if I realized at a later point that Ballard was also tallying births using another method (maybe a different abbreviated word), it’s fairly simple to add this to the program’s criteria, hit the “Run” button, and immediately see the updated figures for the number of deliveries. In contrast, it would be much, much more difficult to do so manually, especially if the realization came at, say, entry number 7,819. The prospect of re-skimming thousands of entries to update your totals would be fairly daunting.

This is the first entry in the diary of Martha Ballard. Martha Ballard was a rural Maine midwife who kept an extensive diary between 1785 and 1812 and whose life was immortalized in 1990 by the historian Laurel Thatcher Ulrich’s award-winning A Midwife’s Tale. Over the course of three decades, Ballard kept a meticulous, near-daily accounting of her life spanning over 10,000 entries.

When reading A Midwife’s Tale, I was struck by how readily the text would seem to lend itself to digital analysis. In an interview, Ulrich noted, “The very thing that had attracted me to the diary in the first place was also the thing that made it difficult to work with. I mean there’s just so much.” To ground herself, she began by simply counting things: “And I would go day by day for every other year of the diary, and I would tick off what was in each entry: baking or brewing, spinning or washing, or trading, sewing, mending, deliveries, general medical accounts, going to church, visitors, people coming for meals, etc.” Because of the sprawling scope, she took this quantitative approach only for the even-numbered years in the diary. The fact that she was working in the late eighties without a computer makes her work even more impressive.

After poking around online I came across DoHistory.org, a website developed and maintained by the Film Study Center at Harvard University and hosted by (who else, really) George Mason’s CHNM. The website presents the diary to the public in two formats: the viewer can either browse through photographed pages of the diary or read the transcript of the pages (transcribed through a monumental effort by Robert R. McCausland and Cynthia MacAlman McCausland):

When I realized the entire diary was online, it got me thinking about possibilities for text mining. As an aspiring digital humanist with little “hard” skills beyond basic GIS, I had been meaning to learn how to program for quite some time. In Martha Ballard’s diary, I had an intriguing source of data with which to learn how to do so. Now I just had to learn how to program. With the patient help of several programming-savvy family members, I gradually learned the basics of Python and how to apply it to Martha Ballard’s diary. What follows are the first steps we took to process the diary’s raw data into an accessible digital format.

Process

At first, I briefly considered learning how to scrape the text of the diary off the website. After some investigation, I decided that was a little beyond my abilities, so I copped out to the much easier route of sending an email to Kelly Schrum at CHNM, who kindly forwarded my request to Ammon Shepherd, who emailed me a zip file containing 1,431 html documents, one for each page of the diary. The html files of the transcribed diary are a basic, 3-column table that look this. My first step was to find a way to strip out the html tags and organize the text into a systematic database of individual entries. Fortunately, Ballard’s meticulousness and consistency lent itself well to such an approach.

The diary’s format translates quite nicely into creating a list of lists – the “main” diary being a list of all the entries, and each entry being a list in and of itself. The first program we wrote was to open each html file and begin extracting the different sections of text (which were conveniently marked by html tags). Iterating through each entry allowed us to separate the different columns in her diary into different items in the list. Here is the breakdown of our “list of lists”:

1. Diary
   1. Entry
      1. Date
         1. Month
         2. Day
         3. Year
      2. Day of the Week
      3. Main Text of Entry
      4. Day Summaries (Column 3 of actual diary entry)
      5. Birth(s) (Recorded in Column 1 of actual diary entry)

In creating the list, we had to separate out the raw data from the html tags that formatted it. Fortunately, the folks who built the html files originally used an extremely systematic formatting process that actually made the job of distilling one from the other quite straightforward. A Python module called Pickle allowed us to export the list of entries as a manageable single file that we could then easily import into future programs to manipulate.

For example, the third entry in the diary would translate a bit into something like this:

1. Diary
   1. Entry (3)
      
      1. Date
         1. 1 (January)
            
         2. 3
            
         3. 1785
   2. 3 (Tuesday – Ballard numbered the weekdays, beginning with Sunday as 1)
   3. “Tuesday. mrs. Foster went home. I had threats of thee Collic; by takein peper found releif.”
      
   4. Empty
   5. Empty

The list allows us to access pieces of information by “calling” their position. It helped me to think of the entire diary list as a warehouse containing almost 10,000 boxes (entries) inside it, with each box containing five compartments, with the first of those compartments divided into three sub-compartments. If you were to open any of the boxes (entries) and look inside the first compartment, then inside sub-compartment number two, you would always find a number that represented the month of that particular entry. If you were to look inside the third compartment of the entry/box, you would always find the main text for that day’s entry.

The advantages of setting up the data in a list structure is the ability to access these specific pieces of information easily and to compare them across entries. In many ways, processing the text to make it readable and programmable is one of the biggest challenges to text mining. Deciding on the most logical way to organize and break down over 1,400 files will lay the groundwork for the fun part: writing programs to actually analyze the diary of Martha Ballard.

***Special-edition sneak preview of future posts in this series***

A simple counting program reveals that the main text of Martha Ballard’s diary alone contains 377,315 words, spanning I-couldn’t-make-this-number-up 9,999 entries. That is a lot of data to play with.

The reasons for this distinction are easy to understand. Most digital humanities initiatives are inherently multidisciplinary. There are those among us lucky or hard-working enough to possess both “soft” humanistic talent and “hard” technical skills, but for the majority of us it is much more efficient and effective to split the workload of multiple, and often very different, approaches between more than one person. Why spend six months trying to master the intricacies of MySQL when you can team up with a colleague who already knows how to implement it? Teaming up with other people across disciplines is a form of self-preservation that saves everyone time and energy.

Another reason for the distinction often stems from the basic nature of the projects – many digital humanists have focused on building tools, online collections, and interactive media. Whereas as most academic monographs are aimed at an audience of fellow academics, these projects are inherently designed with a broader public in mind. With that overarching goal, collaboration during the production phase becomes an almost instinctive (and necessary) pursuit. Similarly, scholarly specialization leads to (often) intense intellectual turf wars. If you are struggling to make your academic mark on a very specific focus within a very specific sub-field, other people working on that same field can often seem more like a threat than a resource. These jealously guarded barriers are less prevalent within the digital humanities community, given its emphasis on greater transparency and a broader scope of study.

This is not to say that traditional humanists are allergic to collaboration. Established (read: tenured) professors are often much more willing to edit volumes, co-author essays, and work together on research projects. When you are a successful author and Harvard historian like Jill Lepore, you can afford to take a chance and co-write a work of historical fiction. An associate professor at a small state school struggling to get tenure? Not so much. Younger scholars are still plagued by the never-ending issue of digital scholarship not “counting” as a valid accomplishment.

Most graduate (particularly Ph.D) programs in the humanities simply do not train their students to play well (or at all) with others. Writing a dissertation is still viewed as an infamously lonesome pursuit. Doing so establishes your credentials as an individual scholar capable of producing original work. Unfortunately, this not only reinforces the conception that anything other than individual research is somehow less valued, but it also does a terrible job of preparing students to do any kind of future collaborative work. Learning how to take notes in an archive or write manuscript chapters are critical skills, but so is learning how to delegate tasks to research partners or co-author a grant proposal.

There is no reason why the traditional humanities cannot begin to embrace scholarly collaboration. Even for those with no interest in digital initiatives, increased collaboration creates a ripple effect. There are the obvious benefits: different perspectives add richness and depth to studies, a division of labor and specialization can lead to greater efficiency, and more collaborators often facilitates future connections across otherwise-insular academic networks. Almost every scholar has the story of a single conversation, comment, or idea from a colleague, friend, or family member sparking a revelation or major advancement in their work. Official collaboration only magnifies this effect, and the academy as a whole would benefit.

Collaboration is not a cure-all, and it presents its own set of quite-formidable challenges. As every high-schooler working on a group project or cubicle-dweller sitting in a meeting can tell you, working with other people can often be a frustrating experience. How do you divide up responsibilities, reconcile different opinions, share both criticism and credit? A professor of literature sitting across the table from a computer scientist will probably have a lot of trouble communicating effectively with each other. All of these issues have the potential to be even sharper inside the humanities, where most scholars have been given little to no official instruction or practical experience in how to work together. Nevertheless, the potential for concerted collaboration to spur on academic discovery within the humanities is simply too high to ignore.

1. Intellectual stimulation

One of the most jarring changes going from a college lifestyle to the workforce was the lack of academic stimulation on a daily basis. and problem-solving transitioned from a classroom to the office. Having a blog gave me an impetus to really think about issues. It forced me to write (semi) regularly, to think about issues, to engage in at least a limited conversation on intellectual topics I cared about. Instead of being a passive consumer of ideas, posts, articles, essays, and books, I became an active one.

The knowledge that my writing would be open and available for anyone to read and judge made me think even harder to develop my own ideas and opinions. If you write a shitty paper in a college seminar, the professor gives you a shitty grade and you file it away. If you write a shitty post, it’s out there for anyone to read. Employers, colleagues, professors, admissions people – all of them now have a growing body of my writing to read, disagree with, and critique if they’re so inclined. For an unestablished scholar like myself, this provides some major motivation to really think and work at what I write.

2. Joining a community

Blogging also let me jump into a vibrant online community of digital historians and humanists. Instead of being something of a sideline observer, I laced up and joined the fray. Doing so not only exposed me to a wide range of new ideas and possibilities, but also introduced me to a number of fascinating and inspiring people – many of whom I met in person at the AAHC and THATCamp conferences. Especially for a younger scholar like myself, having a blog gave me confidence in my credentials and allowed me to participate in a wider dialogue.

Moving forward, the connections I’ve made through blogging (and on a noisier level, Twitter) will serve me for a long time to come. I’ve been lucky in that before I’ve even stepped foot inside a graduate classroom, I’ve have had the opportunity to interact with so many people who I (hope) will be my future colleagues and collaborators. In the insular world of traditional academics, this is a relative rarity.

3. Feedback

I’m a firm believer that there’s no point in writing into a void. While much of my blogging was “for myself,” in that I wrote about what interested me, the most rewarding part by far is the response I’ve received. There is certainly an egotistical and superficial element to checking  site-visit stats. But there is some validity to the point that my writing has already reached a larger audience in a year than all of my undergraduate writing put together. By a long shot. For example, my most popular post, by almost a 2:1 factor, is a rudimentary text analysis of Venture Smith’s narrative. As of today, it had been viewed over a thousand times. This metric might be a tiny drop in the blogosphere bucket, but it will certainly eclipse any audience I’ll have for my traditional academic research, at least in the near future.

One of the more rewarding episodes occurred recently, when a local Connecticut writer contacted me through my blog because she was interested in  Venture Smith. She had stumbled across my posts talking about my undergraduate research on Venture Smith, and had been inspired to do some truly remarkable research on her own. We met yesterday, and I was thrilled to find that not only had she uncovered a fascinating new development, but that it directly related to work I had done. I was humbled to hear that my blog had been an impetus for her to get involved in the Venture Smith community. It served as a great reminder of how blogs can increase transparency and lower barriers between academics and the wider public.

I’m not sure what the future will hold for history-ing. There are bad as well as good aspects of maintaining a blog, and it remains to be seen whether it will survive the time-drain of graduate school. Regardless, blogging at history-ing has been, and I hope will continue to be, an enriching experience.