How To Be A Proper Fucking Scientist – A Short Quiz

Answers here, complaints about the answers can go here

Recently, this little quiz from Time Magazine has been doing the rounds. Reactions to it have mostly gone along the lines of – and I’ll paraphrase here – “Holy fucking shit how the fuck can people not answer these piss easy questions how fucking stupid are people?!?!?”

Yeah, yeah, whatever. You know stuff. Cool story, bro. But science education isn’t (read: shouldn’t be) about declarative knowledge. Memorising facts isn’t really science. For my students, I don’t really care if they can memorise the reactivity trends down the alkali metals and down the halogens – because you can simply deduce them by understanding the trends in electronegativity of the elements. And even the trends in electronegativity can be simply deduced by knowing atomic structure.

It’s safe to say that I’ve done pretty much minimal memorisation in my career so far, because you don’t need to “know” stuff (in the sense of rote-learning facts) to be a scientist. Well, I once thought about memorising the periodic table as an exercising in pegging and memory palaces, but – eugh – effort, and besides, I own this beautiful thing and have a caffeine addiction.

Instead, I propose the following exercise to test your real, actual factual, scientific know-how. Let’s see how pedantry over the number of planets helps you with this.

Question 1

A national newspaper prints a story on how an expert believes the chaya leaf will be a cure for cancer. What is the most important thing to look for in the story:

  • a) The expert’s qualifications
  • b) Which institution the expert comes from
  • c) The experiments they have done
  • d) Whether the reporter writing the story has a positive opinion on the subject
  • e) Whether your friends have tried the cure yet

Question 2

You are presented with the following graph:

What does this graph tell you?

  • a) Nothing in particular
  • b) That organic food causes autism
  • c) That autistic people eat more organic food
  • d) That there is a hidden third relationship at work
  • e) That the person making the graph is a shill for a pharmaceutical company

Question 3

A company is offering a programme to make you rich and successful in business. They say they take thousands of people each year on this course. To prove it works, they invite four people to speak about the benefits of the programme and how it transformed their lives. Of the following choices, which further evidence best indicates that the programme works as advertised?

  • a) At least three more speakers to make it statistically significant
  • b) Another testimony from an independent source
  • c) Nothing, the number of testimonies is sufficient
  • d) Interviews and testimonies from the people for whom the program didn’t work
  • e) Background information on the people offering testimony

Question 4

A shop has an automated alarm by the doors to test if people are walking out with stolen goods. The alarm sounds. Given that the alarm is guaranteed to catch 95% of all thieves, what is the probability that the person caught by the alarm is a thief?

  • a) 0%
  • b) Less than 95%
  • c) More than 95%
  • d) 100%
  • e) More information needed

Question 5

The following is a logical syllogism. The two premises lead to a conclusion:

  • Premise 1) Some doctors are tall
  • Premise 2) Some men are doctors
  • Conclusion) Some men are tall

The above syllogism is valid: true or false?

Question 6

A medical trial for a new drug to treat cold symptoms is undertaken. 20 people are placed in the control group and given a placebo pill. 40 people are put in the intervention group and given the drug, but 15 drop out over the course of the study. Does this:

  • a) Strengthen the study because the control and intervention groups are now nearly the same size
  • b) Weaken the study because we lose information on the drop outs
  • c) Weaken the study because the statistical significance is affected
  • d) Strengthen the study because people who the drug didn’t work on are weeded out
  • e) Have no effect on the study’s outcomes

Question 7

You are presented with the following graph on the news. It represents unemployment figures for two small groups of people in contrast to the previous year. The news report uses it to demonstrate a massive difference in the relative unemployment rate between the two groups.

graph

Why is the graph misleading?

  • a) It lacks a title on the Y axis saying that it’s a percentage
  • b) The data should be continuous, like a line chart
  • c) The graph doesn’t start from zero
  • d) “Group A” and “Group B” should be labelled better
  • e) The colour choice produces a sub-conscious bias in the viewer

Question 8

A new pre-screening test for breast cancer is announced in the news. The experts who made it claim a 99% success rate in detecting the condition early, an improvement on a previous test which had only an 65% success rate.  In the old test, 1 in 100 people would be falsely diagnosed, in the new test 1 in 50 would be falsely diagnosed. Which test is better?

  • a) The old test
  • b) The new test
  • c) They’re about the same
  • d) More data needed
  • e) It’s complicated

Question 9

All toupee’s look fake. How do you evaluate this statement?

  • a) Examine fake-looking hair
  • b) Examine real-looking hair
  • c) Examine hair whether it looks “fake” or not
  • d) You cannot evaluate this statement
  • e) The statement is true by definition

Question 10

A newspaper publishes a piece on the safety of a new vaccine given to combat the flu in people over 60. Their story centres around two people who died shortly after receiving the vaccine, and states that it should be recalled. What additional evidence is required to demonstrate that the vaccine is dangerous:

  • a) The rate of flu infections in people over 60
  • b) The rate of vaccine uptake in people over 60
  • c) The rate of flu mortality in people over 60
  • d) All of the above
  • e) Nothing, as the story is sufficient evidence to indicate a problem

Answers shortly. No cheating please.

Advertisements

16 thoughts on “How To Be A Proper Fucking Scientist – A Short Quiz

  1. I got most of them right. Just some notes:

    About 2, I admit I was thinking “d) That there is a hidden third relationship at work” because when I saw the graph I thought: “population increased and people with autism eat.” At the end I answered right, though.

    About 8, I did the math and answered (c). I medical trials are quite outside my line of work, so I did not thought about how the second test is usually more invasive or the psychological effects. Nice one.

    About 9, I misinterpreted “All toupee’s look fake” as “All and only the toupee’s look fake” so I answered (c). Totally my fault, but I imagined that if I actually find a person with real hair that looks fake I am sorry for him and all the idea of what is toupee or not is wrong.

    Here some more similar questions I like:
    Monday you meet a guy who tells you “The shares of X are going up today.” You check and it happens. Tuesday the same guy tells “The shares of Y are going down today.” You check and it happens. The same scene happens again until Saturday. They guy is always right.
    Sunday he present himself stating he works for a financial company, he shown how good they are predicting the market and so he asks for money assuring you that all the financial gains will be shared.
    Of course, it is a scam. The guy cannot predict the market, how he did it?

    You ask to print cards for a board game you are developing. First rule: the cards have an integer in one side and one letter on the other side. Second rule: if there is a vowel in one side, there must be an even integer on the other side.
    They show you four cards as prototype, the cards are on the table like this:
    |A| |K| |2| |7|
    Which one do you need to flip to see if the cards respect the second rule?

    Reply
    • Imagine you are a plane engineer during WWII (historical accuracy is unnecessary, just imagine an older war) and the other engineer team managed to make a slightly more powerful engine for the planes. So since it’s possible to make the plane heavier they ask your team to make the plane armor stronger, however you don’t have enough extra power to cover the whole plane so you have to focus somewhere.
      To decide where add the armor you check the planes who serviced in the war until now and you notice the bullets holes are not uniformly distributed, some zones are heavily damaged and some other zones are almost mint.

      Where do you think it is better to add the extra armor? Where there are lots of bullets hole or where there are almost none?

      Reply
    • Presumably the first one is that he find 256 people, tells 128 that the stocks will go up and 128 that the stocks will go down. The next day he finds the ones who he correctly “predicted” the market to, and tells 64 of them that the stocks go up and the others that the stocks go down. And so on, until you have 1 person he’s correctly predicted the market to.

      The second one is the Wason card problem and based on confirmation bias. You need to perform the test that disproves the hypothesis. You need to flip the ‘A’ as the rule is disproved if there’s an odd number. And you need to flip the ‘7’ because if there’s a vowel the rule is disproved. Flipping the other two have nothing to do with the hypothesis as ‘K’ isn’t a vowel, whereas it says nothing about consonants having even numbers. There’s a similar problem featuring underage drinkers that the majority of people get first time, but because it’s not abstract they can see the logic at work intuitively. The abstract card problem, though, is harder to spot.

      Your third one below is about silent evidence. If planes can survive with bullet holes in some regions, they obviously don’t need the extra armour there – so the model answer would be to put it in other places. But the reality of that situation would likely be much more complex and need a lot more information. It might be that more planes survive with hits there because that’s where they’re most commonly hit.

      Reply
      • Spot on, but why 256 people? 64 is enough, isn’t it? 2 to the power of 6 for giving the correct “prediction” to one person in six days.
        About the last quiz, of course reality is more complicated, but the idea of the “older war” was exactly to push that shootings were not precise so the holes were expected to be uniformly distributed, but they are not because you see only the surviving planes.
        I hope you liked 🙂 and thanks for the keeping this blog.

        Reply
  2. 1) I want a and c. C could be considered part of his qualifications, but if he’s got good qualifications, I’m more interested with fewer experiments, whereas his qualifications don’t matter as much if he has done a rigorous experiment.
    2) None of the above: It tells me that two things have gone up with a similar curve over the past several years. It does not tell me there is causation, but in the absence of other evidence, it suggests that looking for causation between the two things (whether direct or both as a byproduct of some third factor) may be worthwhile.
    3) B, of the options given. Realistically I want a study of people who were enrolled in the program, giving information on rates of success and failure, but none of the options involve that. Talking to the people for whom the program failed may be helpful also, but what I want to do is talk to lots of people who went through the program to get a sense of how successful it is, and if there is something that makes the program more or less likely to succeed for a given person.
    4) E, we don’t know the rate of false positives.
    5) False.
    6) B
    7) C
    8) B. The new test gives a false positive rate of 1/50, but running it a second time drops the probability of two false positives to 1/2500, and further testing even further. On the other hand, catching cancer early is extremely important, and improves outcomes, so 99% is vastly superior to 65%. Ultimately you could argue E, but I argue that decreasing the odds of a false negative from 35% to 1% is a huge improvement that vastly outweighs the downside of increasing the odds of a false positive from 1% to 2%.
    9) Examine real-looking hair. Examining toupees may cause biases in examination, better to examine hair that looks real, as finding any toupees in that group disproves the statement.
    10) D, and more (we need to know other negative reactions, as well as the odds the deaths of those two persons were due to the vaccine, that their relationship is more than temporal.)

    Reply
    • Yeah, pretty much spot on.

      There are a couple of added complications I’d add, though:

      I think your answer to 1 does highlight the more subtle problem with the argument from authority – even if an authority does present evidence, or experiments, or good reasons, people might be more willing to accept poor or non-rigorous claims just because of the individual and their previous qualifications. For example, take Roger Penrose’s claims about the universe being cyclic because of patterns in the CMB (http://physicsworld.com/cws/article/news/2010/nov/19/penrose-claims-to-have-glimpsed-universe-before-big-bang). People got all excited about it, it was smothered across the popular press, and the impression given was “hey, this is totally legit!”. But the experiments were flawed – almost pathetically so – and the conclusion wasn’t supported even in the slightest (they ran a statistical analysis to find concentric rings, but when someone replicated it they also managed to find triangles, squares, squiggles – so it was basically just noise). It’s not so much that it got there because of the name only, but because many people were willing to overlook how poor it really was, and the fact it was only a pre-print and not a reviewed piece of work, because of the name. It’s a decent rule of thumb to say people with the qualifications produce better work, but even just treating it as a rule of thumb runs the risk of this particular bias.

      With 8, I think you’re assuming that a false positive is always completely random. I don’t think this is ever truly the case unless you’re looking at noise (as Penrose and co. did above). If, for example, a blood test is used to detect the presence of a particular chemical then that test isn’t going to be just triggered randomly. It might be fooled by the presence of a similar chemical or fooled by the presence of that chemical for a different ultimate cause. That suggests that if you run a false positive test once more, the odds of it coming up as a false positive a second time are vastly increased. So a more realistic figure is that running two false positives in a row will be closer to the original 1-in-50 odds than 1-in-50×50. That means you should move to a second, different test for your second attempt at a diagnosis – usually something closer to a “gold standard”, where there are zero false positives/negatives by definition, but is unfortunately more time-consuming or expensive to run. In the case of cancers, this often means (after, say, CT/MRI scans) a biopsy – which is an invasive, often painful, time-consuming process that leaves a patient waiting days or even weeks for solid results and in a state of being unsure (this is weighing in psychological trauma of the test itself along with the simple biological outcomes). So when you double your false positives – and with a low background incidence this can be your largest demographic – you double the potential trauma experienced (unnecessarily, as they’re false positives) for people. This all needs to be weighed into the ethical considerations along with just the raw figures. This is why, despite campaigns, it’s probably a bad idea with a net negative effect to reduce the age of regular breast cancer screenings – you’ll catch the very occasional cancer earlier, but the background incidence in lower age demographics is so low that the main effect would be to put literally thousands more women through more invasive and traumatic procedures just to get an all-clear. That’s not to say this is a trump card, it’s just that it needs factored into the ethical decision making. Some people might choose to assign it a very low weighting in their decision making process and assume better-safe-than-sorry, others might consider it to be more important because medical interventions need to look at more than just their immediate effects.

      Reply
  3. Pingback: Grounded Parents | Weekend Reads: Academic Steroids, Critical Thinking Quiz, Homework Help, etc

  4. Ooops. Never mind … I just saw the other link to the answers. I’ll be back with my answers before I read it. Please delete this, and my post immediately preceeding this one.

    Reply
  5. Pingback: How To Be A Proper Fucking Scientist – Answers Sheet | Spherical Bullshit

  6. OK, time to make a potential fool of myself … but since I was one of the people going “OMG how can they not know that” it seems only fair …

    1. I’d be looking for the expert’s qualifications and institution, honestly, it’s not as strongly relevant as what experiments they’ve done but a BSc I did two decades ago (followed by zero time studying or working in science) isn’t enough to let me actually review experiments properly. Actually what I would probably do if I was really interested was come back and google “Dr Expert Name chaya leaf cancer skeptic” or close variations on that theme to see if science based medicine or Orac or Rational Wiki or somebody I trusted had evaluated it.

    2. Nothing in particular. Or nothing relevant, anyway – it tells me that both of those things increased between 1998 and 2007 with similar shaped curves if you get the Y axes right.

    3. I’d want to see information for people who the program didn’t work for – knowing whether or not the actually followed the instructions could be useful information. I wouldn’t trust the program developers to source them though!

    4. More information – “95% of thieves” tells us nothing about false positive rates. I use a power wheelchair when I go out, and I set off shop alarms frequently, so I am well versed in false positives of this kind.

    5. I think it’s false, but it took a fair bit of thinking! I tried subbing in variables like this to figure it out:
    1. Some red objects are fire trucks
    2. Some apples are red objects
    3. Therefore some apples are fire trucks

    6. Losing people is always losing information, so it must weaken the study. You don’t get only random people dropping out of drug studies – they’re possibly dropping out because of side-effects or because the drug didn’t work or other relevant reasons.

    7. It doesn’t start from zero so it exaggerates the difference, but also “group A” and “group B” are stupid names – they should say what they actually are. And it goes up to 110 so if it’s meant to be a percentage of people who are unemployed it has a bigger problem than unlabelled axes.

    8. It’s complicated. Depends how awful it is to be false-positive’d, how much early detection is picking up incedentalomas[*] which would never have become problematic, and whether there are any negatives about the test itself including side-effects and screening costs, and how treatable early cancer is compared to late detected cancer, and probably other things I can’t remember now.
    [*] I love that word. I have a fun cyst in my liver in that category.

    9. Really? I’d evaluate a huge bunch of known toupee-wearers … evaluating all hair whether real or fake looking would work in theory but given that the percentage of toupee wearers is probably low you’d waste a ton of effort doing it. Or possibly the statement is false and there are many toupee wearers I was never aware of… Hmmmmm ….

    10. All of the above plus tons more. We need to know what they died of, for a start – being hit by a car on your way home from the vaccination centre would be really crappy but is also not a vaccine-caused problem unless the vaccine caused you to have a stroke while driving home … I’d want to know whether vaccine surveillance databases had indicated any other cases which may be related and I’d want to know whether authorities who were well qualified in epidemiology thought that further study was indicated similarly to how the fluvax and narcolepsy in kids link was explored. In actual life this would probably be another case where I’d come home and google search, focussing on the pro-vaccine skeptics like SAVN that I trust.

    Reply
  7. a) Yeah, but that mug only teaches you the German elements, not proper gentleman’s British elements.
    b) You say “fuck” too much, therefore all statements following are fucking wrong.
    c) Monsanto put you up to this.
    d) Quantum quantum something chaya leaf. Chopra.
    e) Joined-up thinking causes cancer.

    Reply

Go on, derp away...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s