Prelude to a graffiti analysis: data, methodology, sampling

Over the next six weeks, I’ll be posting a set of results from my recent graffiti analysis, done with some degree of seriousness this time. The last time I put together a (tongue-in-cheek) analysis of the data, it was a smash hit that got the attention of Slashdot, the Wall Street Journal tech blog, and elsewhere. Fuck stats, Regenstein Library, March 2010What amused and disturbed me was how seriously people took this pseudo-scientific analysis, and how no one seemed inclined to point out the many obvious flaws that I myself described on the blog. Pie charts and graphs seem to have the ability to short-circuit people’s critical thinking skills*.

As usual, the results of a serious analysis are less clear-cut, and less flashy. That said, I think the results might be legitimately insightful this time. With the goal of being as transparent as possible about how I came to my conclusions, here’s how I went about the task:

The data

I chose the five schools where I had the biggest transcribed graffiti corpora. This included UChicago (1455 pieces of graffiti), Brown (930), Berkeley (142), University of Colorado – Boulder (262), and Arizona State University (507). I have an enormous corpus from McGill, but I haven’t transcribed any of it and that part takes to long for me to have it ready in time. It is also Canadian, an Anglophone school in an otherwise Francophone environment, and culturally set apart from my other schools. I have a large untranscribed corpus from University of Michigan – Ann Arbor, but when I was taking pictures there, I was only looking for interesting graffiti, which would throw off all my metrics.

My original transcriptions were based on the unit of the photograph, and often combined multiple pieces of graffiti in different hands in a single spreadsheet cell. For this analysis, I broke the pieces apart, linking connected pieces with an identifying number and a unique letter that indicated its known (in the case of the UChicago graffiti) or hypothesized place in the conversation (e.g. AS-20B is the presumed second piece of graffiti in a conversation numbered 20 at Arizona State; the numbers are based on the conversation’s position in the original transcription, and don’t actually mean anything.)

I only counted the graffiti in English– I didn’t want to privilege languages I could read/easily get translations for.

Almost all of the graffiti is from public study areas in the main library of the university. The exception is the graffiti from the Regenstein men’s bathrooms at UChicago. After a long debate with my husband (him pro, me anti) I decided to include them, even though I didn’t check out the men’s bathrooms at the other schools. For the sake of full disclosure, including them did raise UofC’s overall interestingness by more than .1 (which, as you’ll see with the results, is a non-trivial amount.)

The methodology

There were multiple things I looked at, including sources quoted and referenced, sexual use of words, and things loved and hated, but the main focus of the analysis was the topics discussed in graffiti and overall interestingness. Each piece of graffiti was classified and ranked for interestingness within one or more of the following 22 categories:

  • Advice
  • Classes
  • [Intellectual] Commentary
  • Despair
  • Drugs
  • Greek [fraternity]
  • Insults
  • Love
  • Meta [about graffiti, the surface it's written on, etc]
  • Misc
  • Orthography [spelling and/or grammar corrections]
  • Politics
  • Presence [variations on "X was here"]
  • Quotes [quoting things directly from other sources]
  • Reference [making reference to another source]
  • Religion
  • Reply
  • School
  • Self
  • Sex
  • Social [social issues]
  • Time

“Interestingness” sounds subjective, and while there’s plenty of room to nitpick on individual pieces, I’ve found that people tend to largely agree with the assessment I’ve made. (Perhaps it would’ve been better to make up an abbreviation, like GIR — Graffiti Interestingness Ranking– because nobody argues with a number associated with an abbreviation. But I’m trying to be transparent here.)

In general, I used the following guidelines for rankings:

  • 1: there are one or more words written, but there’s not much more you can say for it. It may a single, disconnected word without any context. It may be an obvious reply (“me too”), it may be someone’s initials, it may be a simple declaration of love (“I love X”).
  • 2: there’s a little more substance there– a complete thought, a non-obvious reply, use of non-obvious phrasing (“Physics wants me dead” rather than “I hate physics”).
  • 3: the piece has some real substance or a spark to it– wordplay, a complete thought that really says something or elicits a response from the reader.

There were ten pieces in the UChicago corpus that received a 4 in their categories– a mark of distinction, something truly clever or memorable, a step above the 3′s.

The ranking worked slightly differently for a couple of categories:

  • Quotes and References: the ranking was based on the source referenced or quoted, where songs or bands received a 1, TV/movies/pop literature received a 2, and literature/plays received a 3
  • Greek: frat letters alone (the most common manifestation) got a 1, saying something about the frat got a 2

Working with the idea that rare genres are interesting, I looked at how frequently each genre occurs in each corpus. To do this, some of the pieces of graffiti were double-counted (e.g. a single piece of graffiti could be both “Reply” and “Greek”). For calculating the final interestingness score for each corpus, I eliminated duplicate entries for pieces of graffiti. I chose their final classification based on whatever would give them the highest interestingness score, or if the scores were equal, using the less common classification.

I decided to give a .5 bonus to scores in the genres that occur with <2% frequency across all five corpora, as a way to mark the interestingness of rare genres. Intellectual Commentary (1.5%), Drugs (1.32%), Orthography (.53%), Politics (1.53%), School (1.94%), Self (1.58%), Social (1.85%), and Time (1.2%) occur less in less than <2% of graffiti each, but I threw out Drugs and Politics after reviewing the individual pieces of graffiti and concluding they were not actually interesting. It's great to base decisions off of numbers when you can, but even though I can't in this case, I feel no regret in not providing an interestingness bonus to things like "Smoke that KUSH" or "Bush knocked down the towers".

You could argue that my choice of categories influences the scores, and you wouldn't be wrong. That said, I think all the categories are valid on the basis of the UChicago data, and all but one occur in at least 3 of the 5 corpora. The exception is Time, which I only have data for from Brown and UChicago. Still, there are 9 examples from Brown (more than Intellectual Commentary, or Orthography, or Greek), so maybe it's just a concern for the higher-ranked schools.

The addition of the bonus .5 point did raise scores overall, but didn't result in any changes to the schools' rankings relative to each other.

Sampling

One of the questions people should ask about is the effect of sampling. The data from UChicago was collected over more than three years, whereas the data from the other schools was collected on a single visit, at different times of year: February for Berkeley, June for Brown and Arizona State, and July for University of Colorado. What about the effect of wall cleaning? How can I assert that the graffiti that happened to be there on that particular day is representative?

The answer: I can’t be certain, but I do have an interesting bit of data from a time-based study of UChicago graffiti. Convinced that the graffiti here is getting less interesting over time, I calculated the interestingness score for every quarter that I’ve been working on this project, and it didn’t vary by more than maybe .15– details coming in the post on UChicago.

That said, when the results from Berkeley (check back the week of 12/17) came in, they seemed to be skewed by the contents of a single, extended conversation, leading me to think that 142 isn’t a big enough corpus for the results to be entirely valid. I didn’t have that problem with the University of Colorado data, so maybe 250 would be a better cut-off in the future.

Next up: Arizona State

I’m publishing the data from least interesting to most interesting, to end the year on a good note. For a preview of the upcoming horror that is the graffiti at the Arizona State University library, check out the blog post from earlier this year.

12/4/10: Addendum on quotes and genres

As I’ve been doing the write-ups for each corpus, I realized I didn’t describe my method for identifying quotes, or the genres of the music referenced or quoted. I Googled every piece of graffiti that went beyond a simple, predictable statement– there were definitely some things that I thought were just creative that ended up being song lyrics. I marked every quote or reference with a classification (“music”, “TV”, “literature”, etc.) and then went back to identify the name of the source work and the author/artist (if relevant). For music reference/quote genres, I used the information provided by Wikipedia.

* To the extent that there was a negative response, it was almost always along the lines of “Why would anyone waste their time doing an analysis of graffiti?”, to which I’d be inclined to answer that it’s a fascinating look into the lives of college students, and I’ve found that inquiry into the small, everyday things that often get overlooked is a more fulfilling use of time than watching TV, playing computer games, or posting trollish comments on-line.

Add new comment