metrics
Original: metrics on Saturday Morning Breakfast Cereal
Transcript
Title: HOW SOCIAL SCIENCE METRICS WORK:
Step 1:
SOME VICTORIAN GUY COMES UP WITH AN AD HOC TEST.
Victorian man (top hat, monocle, mustache, holding a paper): "IT'S NOT A PERFECT MEASURE BUT IT'LL WORK UNTIL WE KNOW MORE."
Step 2:
THE METHOD, BEING THE ONLY ONE AVAILABLE, IS HANDED TO YOUNG RESEARCHERS.
A seated researcher to a standing one: "TRY USING VISCOUNT FITZJIBBLE'S PROTOCOL. IT'S THE BEST WE HAVE SO FAR."
Step 3:
THE METHOD PROLIFERATES AS ITS INITIAL AD HOC NATURE IS FORGOTTEN.
Speaker: "GOOD DATA BUT YOU NEED TO RUN VISCOUNT FITZJIBBLE'S PROTOCOL TO BE CONSISTENT WITH THE LITERATURE."
Step 4:
THE METHOD BECOMES ENTRENCHED.
Speaker: "YOU DIDN'T RUN A VFP? CHRIST JESUS CAN YOU EVEN SPELL SCIENCE?"
Step 5:
AFTER A SERIES OF FAILURES, NEW RESEARCHERS CALL THE METHOD INTO QUESTION.
Speaker: "...AND IN FACT FITZJIBBLE NEVER INTENDED IT TO BE A PERMANENT METHOD. WHO WOULDA THOUGHT?"
Step 6:
IT IS TOO LATE. THE METHOD IS MORE POWERFUL THAN THE SCIENTISTS. ALL HAIL THE METHOD.
First person: "NONE OF US BELIEVE IN IT, BUT NONE OF US WILL FUND PROJECTS THAT DON'T USE IT."
Second person: "BUT... BUT WHY?"
First person (whispering): "SHHH! VFP IS LISTENING! VFP HEARS ALL!"
Votey:
(THE REAL FUN STARTS WHEN IT LEAKS INTO POLICY DISCUSSIONS)
Step 1:
SOME VICTORIAN GUY COMES UP WITH AN AD HOC TEST.
Victorian man (top hat, monocle, mustache, holding a paper): "IT'S NOT A PERFECT MEASURE BUT IT'LL WORK UNTIL WE KNOW MORE."
Step 2:
THE METHOD, BEING THE ONLY ONE AVAILABLE, IS HANDED TO YOUNG RESEARCHERS.
A seated researcher to a standing one: "TRY USING VISCOUNT FITZJIBBLE'S PROTOCOL. IT'S THE BEST WE HAVE SO FAR."
Step 3:
THE METHOD PROLIFERATES AS ITS INITIAL AD HOC NATURE IS FORGOTTEN.
Speaker: "GOOD DATA BUT YOU NEED TO RUN VISCOUNT FITZJIBBLE'S PROTOCOL TO BE CONSISTENT WITH THE LITERATURE."
Step 4:
THE METHOD BECOMES ENTRENCHED.
Speaker: "YOU DIDN'T RUN A VFP? CHRIST JESUS CAN YOU EVEN SPELL SCIENCE?"
Step 5:
AFTER A SERIES OF FAILURES, NEW RESEARCHERS CALL THE METHOD INTO QUESTION.
Speaker: "...AND IN FACT FITZJIBBLE NEVER INTENDED IT TO BE A PERMANENT METHOD. WHO WOULDA THOUGHT?"
Step 6:
IT IS TOO LATE. THE METHOD IS MORE POWERFUL THAN THE SCIENTISTS. ALL HAIL THE METHOD.
First person: "NONE OF US BELIEVE IN IT, BUT NONE OF US WILL FUND PROJECTS THAT DON'T USE IT."
Second person: "BUT... BUT WHY?"
First person (whispering): "SHHH! VFP IS LISTENING! VFP HEARS ALL!"
Votey:
(THE REAL FUN STARTS WHEN IT LEAKS INTO POLICY DISCUSSIONS)
Alt text
A six-panel SMBC comic titled "How social science metrics work:" tracing the life cycle of a flawed research method. Step 1: a Victorian man in a top hat and monocle invents an ad hoc test, saying it's not perfect but will work until they know more. Step 2: because it's the only method available, it's handed to young researchers as "Viscount Fitzjibble's Protocol," the best available so far. Step 3: the method spreads and its ad hoc origins are forgotten; researchers now treat it as required for consistency with the literature. Step 4: it becomes entrenched, with someone furiously demanding "You didn't run a VFP? Christ Jesus can you even SPELL science?" Step 5: after failures, new researchers question it, noting Fitzjibble never meant it to be permanent. Step 6: it's too late, the method is more powerful than the scientists. One person admits nobody believes in it but nobody will fund projects without it; when asked why, they whisper in fear, "Shhh! VFP is listening! VFP hears all!" The joke: a stopgap measurement tool becomes an unkillable, almost cult-like institution that researchers fear. Votey (bonus panel): caption reading "(The real fun starts when it leaks into policy discussions)."
Transcribed by Claude Opus 4.8.