Partly because habits are just hard to break; partly because I actually really love “your mom” jokes.

Not because I think making fun of people’s moms is funny; I pretty much stay away from anything resembling a plausible attempt to comment on anyone’s actual mom.

I love your mom jokes because I love playing with language, and the process of yourmomization (for lack of a better word) is a seriously flexible one that lends itself to a wide variety of induced nonce euphemisms from the literal to the absurd. I like how it’s possible to take even the most innocuous sentence and yourmomize it, how the cultural association of “your mom” jokes with implications of e.g. sexual impropriety means even a totally absurd substitution (A: “I collated those files”, B: “I collated your MOM” or B: “Your MOM collated those DICKS”) reads as an acceptable (if often deeply stupid) semantic transformation.

I’ve even put serious thought in the past into, as a project in amateur computational linguistics and natural language processing, building a YourmomBot that would be purpose-built to parse natural language input, identify potential substitutions, and generate a comeback response from those candidates. (It could even be a learning machine: using either an explicit rating mechanism or an NLP heuristic that tries to gauge the positive/negative valence of responses to its comeback, it could build up a model of what substitutions work and use that to weight future candidate selections. As the basis for a joke entry in a future Loebner Prize competition it’d probably at least get a few laughs.)

I love dumb jokes for their own sake, but fundamentally the humor I find in absurdist yourmomization is not so much in the lowbrow implications of any given joke as in the sort of ready-made, Mad Lib universality of the pattern of jokes when made in series; it’s in the way yourmomization, when employed not as a personal attack of opportunity but rather as an always-on regime, is revealed (to paraphrase Stanley Kubrick) not to be hostile so much as indifferent.

A very specific riff on Chomsky, a modified theory of deep linguistic structures: every sentence was actually a your mom jokes all along.

—

But what I’ve realized is that what may be for me a personal exercise in long-form absurdism may as well be, for those around me, an exercise in littering every single conversation with really banal, repetitive only-barely-jokes. Which isn’t really fair to everybody who isn’t me, and isn’t really how I want to come off.

And I have tolerant friends and a deeply tolerant wife; no one is going to tell me I have to cut it out, nobody thinks I’m actually trying to diss their mom. They might groan a bit, which is the least they’re entitled to do, but that’s about all. They’re kind people.

But having a couple of friends visit for a few days at the end of last year and realizing that I was yourmomizing everything, even literally reflexively yourmomizing random snippets of half-overheard conversation from the other room, made me think about whether maybe it’s time to reel it in some, to recalibrate the meter. And probably the most effective way to start that process is to just kick cold turkey for a while.

And so I’ve stopped making “your mom” jokes for the moment. For a few days now.

All I can moderate so far is what comes out of my mouth (or mostly, considering most of my daily conversation is over the internet, what comes out of my fingers), not what comes into my head, and so I’m not really at a psycholinguistic level making any fewer of them, but now instead of having reflexive yourmomization thoughts and then producing them out into the world, I’m actively quashing the production part of process. Sometimes I mutter them quietly to myself instead of typing them out, but mostly I’ve been getting out ahead of that even.

It feels a bit like stifling a sneeze, one of those stifles where your sinuses get blasted by the blowback and feel unhappy. It’s not a process I’m enjoying. But it’s educational. It’s interesting. And I’m probably annoying fewer people.

And if I bottle up enough of this antiyourmomization frustration it might push me over the edge into actually implementing that chatbot.

Mapstalgia is a growing collection of drawn-from-memory maps of video games. Essentially, it’s a library of memories of fictional worlds: people doing their best to render what they recall of shared pop-culture video game spaces.

It’s open for submissions, so if you have even a dim recollection of a game you’ve played, whether a week ago or twenty years ago, go scratch out a map on a napkin if nothing else and drop it into Mapstalgia’s submission form.

The site is still brand new but it’s gotten some nice mentions on twitter (cheers to @liquidindian, @auntiepixelante, and, as I am writing this, Xbox heavy @majornelson) and I woke up this morning to a nice writeup from Rock Paper Shotgun, so I’m pretty excited about the response.

I’d love to see this just keep growing. Our collective time spent with games is such a huge well of spatial and experiential memory that this feels like one of those things that anyone can, and everyone should, have a crack at.

I’ve always enjoyed photography; I grew up around cameras, learned dark room workflow in high school, and have posted thousands of photos over the years to my flickr account. But I have trouble being consistent about shooting; I’ll get excited for a few days, or document an event or a trip, but I won’t keep shooting daily.

And so this is an attempt to give myself a little bit of structure, a reason to go out every day and look around and try to find a half-dozen good shots, to think about composition and about getting more than just a good-enough snapshot out of my camera.

It’s also a way for me to be more conscious of where I live — my street, my neighborhood, my town. I love Portland but it’s easy to get used to a place and stop thinking about all the little weird details and bits of character that define it. Familiarity leads to taking for granted; I’m hoping I’ll do a little less of that now. I’m enjoying walking around an noticing new details for the first time, but I’m also excited to go back to places I’ve been, to photograph the things that I already think of as being Portland.

The Square Foot is already shaping up to be a pretty satisfying personal project; I hope others, Portlanders and not, get a kick out of it as well.

So, what do I do, I just, I mean, do you wanna ask me questions? I never been to a shrink, all I ever seen is movies and stuff, so–

Okay, uh, yeah. I been having weird dreams.

Yeah, for, jesus, I dunno, years now. Years and years. Seems like I have the same dream again and again, like it’s on repeat, except–

Yeah, yeah, recurring dream, I guess that’s what you call it yeah. But I mean, does it have to be the same every time or it’s something different that you call it? Because there’s all these, y’know, these changes from one time to the next, it’s not like its the exact same thing every time. Sometimes it’s all in black and white, sometimes I can fly, sometimes everything feels like it’s just, I dunno, it’s flat, like the world’s got no depth. Izzat still count?

Okay, it’s, the basic thing is every time I realize my girlfriend, she’s been kidnapped.

Sometimes I see it happen, sometimes it just, it’s like I knew somehow it had happened. Or I’ll get a letter — hey, can people read stuff in their dreams? I heard that wasn’t something that you can do, like your brain when you’re asleep doesn’t know how–

Yeah, unusual? I guess I’m lucky then or something. I dunno. I’m not much of a reader in general, I gotta be honest. Feels like maybe its wasted on me.

Right, yeah, I’ll get letters from her. And it’s like, you got kidnapped but you’re writing me a nice note, what is this guy who kidnapped you? You’re buddies, he’s like, sure, I’ll just stick that in the mail for you?

No, I don’t know who the guy is. Nobody I know. He’s not really a guy, I mean, he’s actually this lizard thing, like a big lizard man. I end up fighting with him sometimes, though most of the dreams I’m just running around, uh, I dunno. It’s crazy bullshit, it’s just dreams, you know?

Well, it’s like…look. I’m not a violent person. Okay? I’m not some asshole who likes hurting people, I’m a talk you out of a fight if you get drunk and get in my face, we’ll end up having a drink together ninety-nine times out of a hundred. Just the kind of guy I am. I’m not a fuckin’ thug.

But these dreams, I dunno, I stomp the shit out of a lot of stuff, and it don’t feel right. I mean, it’s not people, I’m talking like, I’ll see a giant turtle walking by, big as a St. Bernard, and I just get so fuckin’ scared and I don’t know what to do and I just crush the shit out of the thing, breaks its shell under my feet. Over and over again, one after another. Beat one to death with the carcass of the last one I killed.

It ain’t right, it’s just, I’m not a fuckin’ thug, I’m a good guy, I–

No, I, uh, look, sorry. I don’t mean to get all worked up about it, you know, I don’t mean to be so defensive. It’s just, like I said I’m not used to this therapy deal, you know?

My family, we don’t talk about stuff. You got a problem, you figure it out for yourself. I know probably everybody’s a little fucked up but you just deal with, you don’t put it on other people. I only came here because this shit’s affecting my work, I’m losing clients because I can’t sleep for shit and I’m fucking up my schedule. I mean, I can’t really afford this but I can’t afford fucking up my business either, so whaddaya do.

Plumbing, yeah, I run a little plumbing company. Me and my brother, just the two of us, basically a freelance thing, yeah. It pays okay but it gets pretty slow sometimes.

Yeah, I got a good relationship with him I guess. He’s my kid brother, right? Lou, he’s a good guy. We’ve always been close. Like twins almost, my folks always said, ‘cept he’s too tall and I’m too fat so it’s not exactly like people are gonna get us confused.

Yeah, he does show up in the dreams sometimes. Like sometimes I’ll be me and then sometimes I’ll be him, but I don’t think it’s that I’m him really so much as I’m just watching. I mean, you put it that way, really a lot of the time it feels like I’m just watching even when I’m me, like it’s something happening to someone else. Like I’m watching it all on TV or something.

And you know the worst part, the dreams are always, you know, they’re terrible, I’m killing all these things and my girlfriend’s missing and nothing makes any fucking sense and it’s like I’m racing against some clock so I have to keep moving, I gotta keep moving. And then I’ll fuck up, I’ll fall down some goddam hole in the ground or I’ll get shot or some fucking thing will get its jaws on me, and I’ll die. Just, fuck, I’m dead now, and I think that’s it, that’s the end, I’m dead, but then it’ll just start all over again and I worry that it’ll never end and I’ll never wake up and I’m just terrified the whole time.

But everything’s got a smile on its face, you know, it’s like some sort of sugar high, nothing makes sense but everything’s glowing cheerful as shit and even when I look up, the fucking sun has this look on its face like it’s gonna die laughing. Except then it gets angry and starts chasing me too.

Every night lately it’s like this. I don’t want to go to sleep, you know? I just drink as much coffee as I can stand and watch shitty TV until its five in the morning and I end up passed out on the couch anyway and I have bad dreams anyway.

My girlfriend. Yeah, we’ve been together a long time. Forever, yeah, we met back when I was trying to go to school for a couple years. She’s a peach, I love her like crazy, I don’t know what she sees in me a lot of the time. Fuckin’ beautiful, always dolled up, I dunno how she ended up with some dropout pipe jockey. Says I make her laugh. I’m not gonna complain, what am I, crazy? You get lucky sometimes, you just gotta hold onto that long as you can.

Do I worry? What, like she’s messing around on me? She’s not some slut, whaddaya–

No, sorry. Sorry.

Nah, no, it’s not okay, I shouldn’t snap like that. You’re just askin’, I know. I guess you mean is it something I worry about mentally, like does it make me anxious? Like she’ll leave me? I dunno. I dunno, maybe, I mean, I ask myself how I got so lucky, you know, it feels like I could just wake up one day and she’d be gone, leave me a note, “Dear Mario, I just realized I’m dating a fucking plumber, gonna go find a good looking rich asshole with a Lambo, have a nice life”. I mean, she wouldn’t do that, we’re happy, she’s not some gold-digger, but, yeah, I dunno. I guess I worry. Who wouldn’t, gorgeous girl like her. But it’s just bullshit, it’s just in my head, you know. I’m just fucked up a little, everybody is.

Drugs? Nah. Nah, not really, no. I drink, I like to have a drink or two, sure, at night, watch some TV and unwind, maybe go out with Lou and have a couple down at The Toadstool. I mean, back in the day, a few times maybe, I, uh.

I mean, hey, uh, this is, this is confidential, right? You aren’t gonna, you can’t like rat me out if I say something about a little dope or whatever back whenever, young and stupid and all that shit, right? I can’t afford to get in trouble, you know, I–

Yeah, okay, good, the thing is, I smoked a little weed in college, not a big thing, not really my deal, but me and Lou were roomies with this guy one year, Cooper, he was always smoking up, he’d use whatever he could find for rolling paper, he had this thing where he liked to use a leaf and he’d get high, shit like that. Or he had this raccoon-shaped bong he’d used to get stoned. And that was pretty much every day with that guy. And we partook sometimes, you know, it was weird to always say no. Just made me feel kinda foggy though.

But one time Cooper’s got a couple friends over, one of them this bullet-nosed shitheel named Bill who was always like chasing you around the room trying to have a conversation with you no matter where you went. Really pushy asshole, never liked him. But so he pulls out a baggie and Cooper’s all excited suddenly and he’s all “yeah, let’s do this”. And we’d been drinking pretty much all day, I was far gone already or I woulda said no, but it was like, fuck it, you know?

So I guess it was mushrooms. Magic mushrooms, that what they call it? Psychedelics, whatever, hippie shit, not something I’d normally go in for, but we ate a bunch of ‘em. And I don’t really remember the rest of the night too well other than I spent an hour half-awake with my head in the toilet at one point. Cooper had this turtle he kept in a glass fish tank in the bathroom, I remember just staring at that thing between heaves, like it was just me and it and fuckall else in the whole universe. I dunno, it was pretty fucked, I never touched that stuff again.

I pretty much dropped out after that year. I’d just met my girlfriend, and college wasn’t going so good for me anyway and it was costing money my folks didn’t have. Lou was always the smart one, if he had any confidence he’d be doing his own thing and not working for me. But yeah, I knew a guy who was looking for a second hand in his plumbing deal, needed someone to handle the overflow, I was always pretty good with a wrench growing up anyway. So I was maybe, you know, it was a pretty stressful deal all around for me I guess, all that shit going on, maybe that’s why it started.

But I been having these dreams on and off ever since.

“If in the first act you have hung a pistol on the wall, then in the following one it should go off. Otherwise don’t put it there.”

– Anton Pavlovich Chekhov, as related by Ilia Gurlyand of a conversation from 1899

“Rules are made to be broken.”

– Ladies Home Journal, 1899

~ The setup ~

ACT I – SCENE 1
(The living room of a modest home. There is a fire burning in the fireplace; hanging above the mantle is pistol.)

ALICE
Oh, Uncle Bob, it is so good to see you again after all these years.

BOB
(embracing ALICE)
I have missed this old house. How is your mother?

~

~ The variations ~

1. The Vanilla

ACT III – SCENE IV

ALICE
You’ve betrayed me, and Mother! Everything this family stands for!

BOB
(grabs pistol from mantle, fires it at ALICE, killing her)
Oh god, what have I done?

~

2. The Ladies Home Journal

ACT III – SCENE IV

ALICE
You’ve betrayed us all, Uncle Bob!

BOB
(walks to fireplace, looks at pistol on the wall)
Yes, well, nobody’s perfect. Hey, where’d you find this neat gun?

ALICE
Mother bought it on eBay.

BOB
It’s really cool looking.

ALICE
Yeah, I like it a lot.

~

3. The Continuity Error

ACT III – SCENE IV

(BOB enters, wearing a green tanktop.)

ALICE
You’ve betrayed me, and Mother, and everything this family stands for!

BOB
(grabs pistol from mantle, points it at ALICE)
Alice, don’t make me do something I’ll regret!

ALICE
No, please!

BOB
(is not holding a gun, is now wearing an orange sweatshirt)

~

4. The Euphemism

ACT III – SCENE IV

ALICE
Yes, hold the gun just like that, oh, oh…

BOB
Alice, this is wrong, this is — oh, it’s going to go off!

~

5. The Infestation

ACT II – SCENE I

ALICE
Why are there two guns on the mantle now? Uncle Bob?

BOB
Don’t look at me, I don’t even like guns.

…

ACT III – SCENE I

BOB
(glances at mantle, does a double take at trio of guns on mantle)

ALICE
I suppose you think this is funny.

…

ACT III – SCENE IV

ALICE
Oh help, oh someone help!

BOB
(running into the room)
Alice, what is it, what’s –

(BOB follows ALICE’s gaze, and stops in horror. The mantle is covered in hundreds of guns, teeming, an undulating blanket of pistols.)

ALICE
So many! So many!

BOB
(screams, tears eyes from sockets)

~

6. The Soliloquy

ACT III – SCENE IV

ALICE
You’ve betrayed us all, Uncle Bob!
(runs toward the pistol on the mantle)

BOB
Oh, but you all betrayed me long ago!
(also runs toward the pistol)

(action freezes, PISTOL steps down from mantle and addresses the audience)

PISTOL
To Alice, or to Bob? Whose grasping hands
Will first encircle ’round my polished grip
And turn my muzzle ‘pon embattled kin?
The choice is fate’s, it is out of my hands,
Not least because a gun does not have hands.
Nor can guns rhyme, and they say “hands” too much.

~

7. The David Lynch

ACT III – SCENE IV

ALICE
(laying on the floor in a pool of blood, speaking oddly)
The leaves are blowing freely again.

BOB
(awkwardly grapples with gun from mantle, fires a bullet into ALICE’s wound)

ALICE
(stands up, blood flowing into the closing wound)
Wow, Bob, Wow.

~

8. The Anticlimax

ACT III – SCENE IV

ALICE
Betrayer, vile betrayer!

BOB
(grabs gun, point it at ALICE, pulls the trigger; the gun only clicks)
Huh.

ALICE
Man oh man. Anton is gonna straight shit himself.

~

9. The Spoiler

ACT III – SCENE IV

ALICE
Oh, how could you betray us like this?

BOB
Enough!
(moves to mantle, grabs pistol, points it at ALICE)
You could never understand my pain! You could — oh holy goddam.

ALICE
What’s, what’s the –

BOB
It’s the gun. Ugh.
(winces, holds the gun out at arm’s length)
Wow. Ugh, ugh, I can’t stand it. Come here, smell this.

ALICE
(approach, sniffs gun, nearly retches)
Oh, god, yeah, wow. Wow, that’s, uh. That gun is no good.

BOB
It’s gone off, alright.

~

10. The Bruckheimer

ACT III – SCENE IV

BOB
It’s been an amusing little game, Alice, but I’m afraid you’ve lost.

ALICE
You’ll never get away with this, Uncle Bob!

BOB
(laughing maniacally)
But I already have! The Socrates Diamond is mine, and soon the world will tremble before my power. But first, it’s time for you to die.
(retrieves pistol from mantle, points it at ALICE)
Any last words?

ALICE
Just five.

BOB
(raises eyebrow, cocks gun)

ALICE
The gun’s a bomb, Bob.

(ALICE leaps through a nearby window as BOB notices a countdown timer on the pistol. The timer counts 0:03, 0:02, 0:01 as BOB screams. The house explodes, and explodes, and explodes again. ALICE is thrown clear of the blast, her hair and makeup only superficially mussed; she retrieves the diamond and smiles as the fire of the explosions glitters on its facets.)

~

11. The Jumping the You Know What

ACT II – SCENE I

BOB
(tentatively brushes his fingers over the gun’s barrel. The gun fires!)

ALICE
Holy shit, Uncle, be more careful!

…

ACT III – SCENE II

(BOB stands several feet from the mantle, looking at the gun. The gun fires!)

BOB
Oh for god’s sake.

~

12. The Twist

ACT III – SCENE V

(The POLICE INSPECTOR and his DEPUTY stand over the bodies of ALICE and BOB, in front of the fireplace.)

DEPUTY
Oh god, I’ve never seen anything like this.

INSPECTOR
Murder-suicide. Look at the scratch marks on John Doe, blood under Jane Doe’s fingernails. She tried to fight him off.

DEPUTY
And then, he — Jesus Christ, Inspector, who does this sort of thing?

INSPECTOR
You mean, who murders a woman by lodging an unloaded pistol in her air passage until she suffocates, and then asphyxiates himself in the same fashion? Before today I would’ve said nobody, that’s who.

DEPUTY
It’s, it’s just so unbelievable!

INSPECTOR
I know, son. I think we can all agree that it’s…
(puts on sunglasses)
…hard to swallow.

I’ve updated the sidebar content and Music and Project page stuff as well; two years out of date is pretty out of date for me, considering how ephemeral a lot of my projects are. So some obsolete stuff has been nixed and a lot of new stuff form the last couple years is now properly featured.

This was originally a slideshow presentation, but my slides weren’t anything special so I’ve left most of those out of this re-run of the talk’s presentation in favor of running the text itself. If this reads a bit odd in spots, it’s partly a reflection of that transition.

On Metafilter

I’m Josh Millard, aka “cortex“, and I’ve worked for the last four and a half years or so for Metafilter.com, where I’m a moderator and community manager and data nerd. I’m going to talk today about a data project called the Metafilter Infodump, but since all of us on this panel are discussing work related to Metafilter I thought I’d start things off by giving a little background on the site itself.

Metafilter is a community website started by Matt Haughey in 1999, as a place to share interesting links from the web and discuss them with other Metafilter users, or “mefites”. It’s a generalist site with no given topical, ideological, or demographic focus or mission statement. Joining the site has, since 2004, cost a one-time $5 fee. The userbase is majority North American but significantly international as well. English is the common language on the site.

The front page of the Metafilter functions as a sort of low volume link aggregator, with about thirty posts a day made by individual users of the site. Posts range in scope and tone from funny cat videos to encyclopedic examinations of medieval sacred architecture, from election scandals to indie comic art. Each post features a comment thread, which was a definitive feature of the site back in 1999 and remains the fundamental venue for members to interact—while anyone can read the site, only members can comment in threads, and for most regulars it’s the discussion in threads on the front page and other parts of the site that makes Metafilter an interesting place to hang out.

Aside from the front page and it’s link-sharing ethos, we’ve built other subsets as well; there’s Music, where users can post their own recorded work; Jobs, where users can list job openings or note their own availability for work; IRL, a meetup organization area; and Projects, a site for users to post about creative projects they’ve undertaken, since self-linking elsewhere on the site is discouraged.

But most importantly there’s Ask Metafilter and Metatalk, the two other big subsites; including Metafilter proper, these three subsets account for the bulk of the posts and discussion that takes place in the community. The subsets are often referred to by color nicknames, based on their background colors: Metafilter is “the blue”, Ask Metafilter is “the green”, and Metatalk is “the grey”.

Metafilter I’ve talked about already. In addition to “the blue”, it’s also often called “the front page” or “mefi”, all of which are useful in discussing that link-sharing part of the site specifically as distinct from the overall entity that is Metafilter-as-a-community-and-collection-of-subsites.

Ask Metafilter, “the green”, is a question-and-answer site launched in late 2003 after it became clear that there was a lot of desire within the community to have a place where mefites could specifically interact with one another as a relatively trusted group. Users can ask at most one question a week but can answer as many as they like; on the order of a hundred questions and two thousand answers go up daily.

Metatalk came along much earlier, in early 2000, about the time that Metafilter was making the transition from being Matt’s de facto personal link blog to being a small community with several hundred registered accounts and an active core group of dozens of daily participants. People were talking about the site on the front page — making feature requests or suggestions, reporting bugs, discussing site policy or user behavior — and there was a clear need to segregate all that metacommentary from the actual “here’s a neat link” business of the front page.

Today, Metafilter has about 70,000 registered user accounts, about ten thousand active users depending on how you define your terms (more on that below), fields about three thousand comments a day from logged in users, and sees something like 25 million pageviews on a typical month from users and web readers in general.

The Metafilter Infodump

So that’s Metafilter in a nutshell. Let’s talk about the Infodump.

The Infodump is well-structured, weekly-updated collection of data files that distill the bulk of user activity on Metafilter down to a skeleton of activity records. It contains information like the datestamp, posting user, and unique IDs of every post and comment made to each subsite, as well as category information where applicable and whether a given post was closed to new comments or deleted by a moderator. It also contains raw data about the use of the favorites system, a multi-use and sometimes contentious bookmark-slash-thumbs-up features that allows users to mark individual posts and comments as a “favorite”.

The Infodump contains mostly numeric data; it elides the actual text of comments and posts, partly to keep the size of the data files a lot more manageable and their structure simpler, and partly to get away from simply reproducing the entire content of the database in slightly altered form. It does however contain some text-related content, including information about every folksonomic tag associated with a post on the site, the title text of posts, the character count of every comment, and the actual userids and usernames of registered users.

(A parallel project I’ve been working this year, the Metafilter Corpus, is more explicitly focused on the text content of the site; currently it consists of a variety of 1-gram frequency tables for comment text across the site over the first eleven years of comments, which is about 457 million words of source text. I plan to expand that into higher order n-gram files in the near future.)

The files that make up the Infodump are generated by some fairly simple perl scripts I wrote that pull data directly from our database every Sunday night and sort and clean that data a bit before writing it out to tab-delimited, zipped-up text files. It’s a straightforward, fire-and-forget procedure that our code wizard Paul Bausch has running as a service behind the scenes, which is good for everyone because I’m pretty distractible and back when I originally had to run it manually it would get pretty out of date until someone poked me over email.

Before I came to work for Metafilter, I’d been fascinated for years by questions about how the Metafilter functioned, about what the actual quantitative answers might be to questions otherwise only answerable with speculation when they came up in discussions on Metatalk. I had pestered Matt Haughey on a few occasions for behind-the-scenes data, and had done some scraping of the site to collect data manually, but both felt like impositions and I’d hoped there was a better way to get at that stuff.

Other folks had been in the same boat as well; academic mefites and even folks outside the community had done research projects about the site, some trivial and some as serious as graduate-level theses, and much of that work was complicated by the need to either beg for or scrape data as step one.

The Infodump was an attempt to answer that need, to simplify data collection and make it easier for people, whether serious researchers or curious community members, to dig into this stuff via an accessible and reliable outlet straight from the site itself.

My co-panelists here will all have some interesting stuff to talk about along the lines of that serious research side of things, but I’ll give you some quick examples here of the more informal intracommunity navelgazing stuff our users have done with this data over the years.

Datawankery In Action

When should you ask a question?

I mentioned that Ask Metafilter has a limit on questions a user can ask: one question per week per user at a maximum. This helps keep the volume of questions manageable, but it also makes a question feel a little precious sometimes — you don’t want to waste it, you want to get the most out of it. And so one of the recurring questions in Metatalk is “when is the best time to ask a question?” Surely some days are better than others, or some times are better than others? There was a lot of speculation and folk wisdom about this for years, but the Infodump made it easy for some data-minded users to actually answer the question quantitatively.

And it turned out that the answer, to pretty much everyone’s surprise, was that it really doesn’t matter. If you’re looking to maximize answers given to your question, you can post literally any time of the day, any day of the week, and expect an average of 14 answers or so. This despite the Metafilter community’s majority demographic being in North American timezones and weekends being significantly quieter than weekdays.

One partial explanation: Metafilter has a heartbeat. Folks who visit the site more or less daily are likely to look at the things that interest them whenever they get around to visiting, and to catch up on what they’ve missed since the day before, whether it’s something posted noon yesterday or three o’clock this morning.

Beyond that, fewer people visiting during weekends or quiet hours might imply fewer answers, but it also implies fewer questions; while both kinds of activity drop measurably on Saturday and Sunday, a lower number of questions asked means that it takes fewer total answers to get a fair number or responses to all the questions that have been posted.

How many active users does the site have?

How many active users does Metafilter have? If I wanted to brag the numbers up I’d just point out that we assign userids sequentially and we’re inches away from userid 140,000 right now, but in actual practice that number is pretty much bullshit. For one thing, not everyone who starts signing up completes the process, and we reserve those partially-registered accounts in case the inchoate user decides to come back and finish signing up later. About half our signups are false starts, so we’ve got maybe 70,000 active users.

Except a lot of those folks don’t do much on the site. A large chunk signed up but never did anything; many others have only commented a few times and then disappeared; others still were once very active but have since closed their accounts or walked away or even in some cases been banned.

So how many for real, ACTIVE active users do we have? One of our members, FishBike, put together a nice graph that uses the benchmark that a user is “active” if and only if they’ve had some sort of site activity (posting, commenting, favoring) in the last month; by that metric, there were on the order of 7000 active Metafilter members at the time that he did these calculations, with steady slow growth in the active user population over time even as there was a smaller level of steady slow growth in attrition.

It’s not a perfect metric — folks taking a months-long break from the site or who only comment sporadically but long-term are treated as having left, hence the big red spike at the right end of the graph, and a month is an arbitrary choice in any case — but thanks to the Infodump it at least gave us an actual figure based on data and made it possible for a user to answer the question himself instead of relying on a mod running the numbers. And it’s not a terrible estimate, though it fails to account for lurkers; looking at internal site logs, we can tell that in fact it’s about twelve thousand distinct logged-in users who at least visit the site each month, so Plutor’s metric mostly just misses the large chunk who are actively around but don’t leave any traces in the Infodump by leaving comments or so on.

Are there cliques on Metafilter?

I said our favorites system is contentious; people disagree about what they’re for and how they should be used, and beyond that some folks worry that for example they encourage cliquish behavior. A clique as a qualitative thing is hard to test for, but numerically the Infodump makes it possible to at least look at who favorites who to look for interesting patterns.

And so we’ve had a few different looks at some of these ideas. One user analyzed mutual favoriting behavior to see what pairs of users favorited one another’s content more often than average. It turns out that, for whatever reason, there are definite pairings and groupings of people who favorite one another more often than the baseline average for a user who engages in favoriting, often correlating to notable real-life relationships or strong shared topical interests.

We also have a contact feature on the site, where users can explicitly set up one-directional relationships with other users’ accounts, with modifier tags like “spouse” or “muse” or “met” as in met in real life. That data’s in the Infodump as well. And so another user compared favoriting behavior to the user contacts data and established that in fact folks are far more likely to favorite content by people who they have listed as a contact on the site, by a factor of thirteen or so.

And lots more

Mefites have even built tools to make the Infodump data more accessible to folks not as practically equipped to get into hardcore DIY datawankery (which is the site jargon for this kind of info wrangling stuff). One user, Bill Zeller who went by “null terminated” on the site, had maintained for a while a web-facing SQL query interface for the Infodump data; more recently, Combustible Edison Lighthouse built the Infodumpster, a friendly fill-in-the-blanks engine for a variety of common query templates that lets folks play with the basics without needing to know anything about data manipulation.

There’s a wealth of further examples but I can’t go through them all; if you’re interested in seeing more, there’s a page on the Metafilter wiki that rounds up a lot of the user-driven research over the years and points to a number or discussion threads on the site; user FishBike maintains his own charts and graph page as well.

How To Build An Infodump

As far as I have seen, there’s not a lot of stuff out there like the Infodump, which is a shame because it’s not necessarily a difficult thing to produce. I’m barely literate with SQL and ask our coder Paul for help with basic stuff like JOINs a lot of the time; if I can manage to hack out some perl and some basic SELECT statements to accomplish this, pretty much anybody who runs a database-driven website can, and if a site wants to accomplish this but doesn’t feel confident about creating a data dump procedure internally it’s a pretty simple consulting gig to hire out.

The big challenge is more in deciding what data to make available and thinking about how that impacts your users. Private stuff needs to stay private — we don’t put the content of our users’ profile pages or their email or payment details into the Infodump, we don’t include internal stuff like data about the flagging system folks use to report problematic comments or posts, and we do provide a munge feature so that users who would rather not have their ID directly correlated with their site activity in the Infodump files can have that dissociated automatically. Ultimately, everything in the Metafilter Infodump is already publicly viewable on the site, and not everything publicly viewable or derivable is in the Infodump, but it’s still important to think about what your userbase might expect to be out there and in what form. In the case of Metafilter, we’ve talked about this stuff at length with the Metafilter community in Metatalk and provide a clear channel for feedback from users about any ideas or concerns about the data, and that communication is key to not having a lot of folks feel like they’ve had the rug pulled out from under them.

Communities Are Great Research Venues

It really is striking to me that even as cohesive social communities have become a very significant part of the landscape of the web, things like the Infodump seem to be uncommon at best. There’s just a lot to recommend it.

A tool like this is a benefit to the community itself in terms of transparency in administration and giving folks back what they’ve invested — users are the lifeblood of a community site, this is their activity and their data in the first place and they should have access to it. At Metafilter we offer users direct access to a flat dump of their own commenting activity and internal mail as well, for the same reason.

An Infodump is also good for a community’s sense of self-awareness and involvement in its own state of being, of its context and history — a data dump like this gives them the ability to get their questions answered, to explore the various introspective “who”s and “how”s and “what if”s that naturally arise in any self-organizing group of smart people, and it gives them the autonomy to pursue those answers without waiting on the ability or availability of the site’s administrators to do analytical work for them.

As a practical logistical benefit, it can help a site avoid extra unplanned load from the methodical (and sometimes poorly-throttled) full-page scraping that has traditionally been the sole avenue for data collection for curious users and researchers, and to host the data independently from the main site to reduce the impact of large downloads.

And the data itself is cleaner and more reliable — scraping is often a miserable experience for the person collecting the data, as what’s useful or pragmatic for live webpage layout in a cross-browser, cross-device context is often a hard-to-clean-up mess for the person only looking for skeletal data. A conference like this is a testament to the fact that people have been, and will more and more as time goes by be, looking seriously at what people are doing with their time on the internet as individuals and groups, and if a community can help make that process more practical by offering up tools like the Metafilter Infodump or the Metafilter Corpus I think that’s an important thing to do.

It’s my hope that more sites will embrace this idea; there’s lots of smart, introspective communities on the web worth taking a close look at and I think if the folks running those sites and the researchers who see the value in those close looks can talk to each other about these ideas, this sort of thing can become less a rarity and more just an obvious and sensible way to deal with this wealth of data.

But I play a lot of games and don’t mention it here much at all, so why am I bringing up this one? Because of a plate of beans, is why.

It goes like this:

1. I found out about Glitch because I work for and hang out at Metafilter, and a lot of folks on Metafilter are playing it. And so while my little yellow dude has been mining beryl and cavorting with pigs in exchange for steaks and harvesting bubbles off of bubble trees, I’ve been chatting with my fellow Mefites. And not only are the are a lot of us playing the game, there’s even a couple folks who work for Tiny Speck, the company that makes Glitch.

2. There’s a long-running joke on Metafilter about how we’re the kind of people who could overthink a plate of beans. It started as a joke someone made during a thread years ago in which people were earnestly deconstructing the performative elements of Alanis Morissette’s cover of Black Eyed Peas’ My Humps, and it caught on, to the point where “beanplating” and “to beanplate” are a commonly understood derived verb forms used to describe maybe-needlessly-in-depth analysis of one thing or another. There’s even a song.

3. Glitch has bean trees, on which grow beans. One can harvest beans, and eat them or use them as constituents in recipes. What Glitch hasn’t had is beans arranged tastefully on a plate.

4. Except, well, that now exists, thanks to one of those Mefite Glitchers who happens to work on the game.

Here it is:

Click on it and you get, as with most items in Glitch, a context menu that gives you some basic options and a special option unique to this item: overthinking.

So what happens when you decide to overthink your plate of beans? Well, overthinking isn’t something you want to just jump into without a little bit of consideration, so the game helpfully asks if you’re sure you want do this:

But being sure about overthinking isn’t something to settle on trivially, so after a moment it asks if you’re really sure:

Any seasoned overthinker knows that settling on being sure about something takes serious contemplation, and so, well:

And, well:

Given that energy slowly goes down over time in Glitch, I had to resort to frequent bouts of meditation with my Focusing Orb just to keep myself from starving to death while waiting for the eleventh confirmation screen:

I have no reason to believe this would end. But I haven’t got the patience to find out; in fact, I don’t have the lifespan to find out, because the pause between each confirmation step grows literally exponentially: the wait between confirmation prompt n and confirmation prompt n+1 appears to be 2ⁿ seconds, which means that while it only took one second for the “really sure” prompt to appear, it took four seconds for the “really, really, really sure?” to appear.

And because I don’t want to keep tying out big strings of “really”s, let’s just note that the number of times “really” appears in a confirmation prompt can also be expressed mathematically as n – 1, and call this value really_n.

So, as above, the wait on really₁ is 2^{(n – 1)} = 2⁰ = 1 second; the wait on really₃ is 2² = 4 seconds.

The wait on that last one pictured above, really₁₀, was 2¹⁰ = 1024 seconds, or about seventeen goddam minutes. Or about one and three quarter hours of in-game time.

If you wanted to get as far as really₃₁, you’d have to play Glitch for about 50 years straight, every day, without sleeping. And that’s a lot of bubbles and pig steaks to eat.

You’ll never see the really₃₂ milestone yourself; you’ll need to establish a trust for that.

I can’t think of a better in-game manifestation of this little bit of Metafilter in-joke humor, and it’s just a goddamned magical bit of internet wonderfulness that I have this silly item in my inventory on this game I didn’t even know about until I blundered into it a few days ago and lost my mind to it.

Thanks for the plate of beans, ericost. You’re a mensch.

It uses Markov chains, one of my favorite bits of applied mathematics in the whole dang world, and it is fairly silly and you can go play with it here:

~ The Big Markovski ~

If you pay enough attention to the odd stuff I get up to, you might recognize this as a port of a very similar previous project, Previously, On The X-Files, which was itself a significant elaboration on an even previouserly project called Garkov that mashed up Markov chains and Jim Davis’ Garfield comics.

One interesting point of comparison between Markovski and the X-Files project is that there’s a great deal less raw material for this new one to work with—Spooky Mulder had something like 15,000 lines of dialogue over 9 seasons, whereas The Dude has maybe 450 total in the Lebowski script. So you see less mid-sentence swerves with this one than with the POTX project; a smaller corpus of input means less resynthesis and more verbatim regurgitation for any given generated statement.

Clearly, the brothers Coen need to address this by writing a long-running TV series in the Duderverse. Until then, we’ll just have to make do with this as is.