Google Finally Admits That Its Infamous Brainteasers Were Completely Useless for Hiring

July 4, 2013 at 1:15 am 11 comments

Google has found that being great at puzzles doesn’t lead to being a good employee.  They also found that GPA’s aren’t good predictors either.

Nathan Ensmenger could have told them that.  His history The Computer Boys Take Over shows how the relationship between academic mathematics and brainteasers with computer science hiring was mostly an accident.  Human resources people were desperate to find more programmers.  They used brainteasers and mathematics to filter candidates because that’s what the people who started in computing were good at.  Several studies found that those brainteasers and math problems were good predictors of success in academic CS classes — but they didn’t predict success at being a programmer!

How many people have been flunked out of computer science because they couldn’t pass Calculus — and yet knowing calculus doesn’t help with being a programmer at all?!?

You can stop counting how many golfballs will fit in a schoolbus now. Google has admitted that the headscratching questions it once used to quiz job applicants (How many piano tuners are there in the entire world? Why are manhole covers round?) were utterly useless as a predictor of who will be a good employee.”We found that brainteasers are a complete waste of time,” Laszlo Bock, senior vice president of people operations at Google, told the New York Times. “They don’t predict anything. They serve primarily to make the interviewer feel smart.”

via Google Finally Admits That Its Infamous Brainteasers Were Completely Useless for Hiring – Adam Pasick – The Atlantic.

Entry filed under: Uncategorized. Tags: , , .

Google exec to girls: Tech needs you Benefits of Online, Face-to-Face Professional Development Similar, Study Finds

11 Comments Add your own

  • 1. Peter Donaldson  |  July 4, 2013 at 4:36 am

    I’ve always been slightly sceptical of the assertion that Maths and Computer Science were essentially the same thing and that secondary school students would be better served without explicit computer science teaching if maths was taught in a more powerful way. Not that I think that isn’t important, I do, but I don’t think it’s the answer to addressing the issue of exposing everyone to some of the powerful ideas in CS.

    Through reviewing a range of research into success factors for initial cs education and programming it’s clear that other things also correlate including whether students had some formal programming instruction before university or not. However, possibly because the paper is behind a research paywall, I haven’t come across any evidence based argument for why performance in maths and cs appear to be so closely linked.

    My hypothesis is that successful performance in maths and cs requires an extremely persistent mindset and that what a student’s secondary school math grade is mostly telling you is how persistent they are likely to be when faced with a difficult problem or situation. I would also theorise that some aspects of cs might require a higher level of persistence than the level necessary for secondary school mathematics

    If true, that would imply that directly measuring how persistent someone is when faced with the type of problems computer scientists try to solve would give a much better indicator of their chance of success in the longer term. It would also suggest that we could increase success rates in CS by creating experiences that improve students attitude towards, and level of, persistence.

    Unfortunately I’m not a CS education researcher so I’m unsure if this is already an aspect that has been explored in detail or how a robust experiment could be setup to test this hypothesis. It’d be interesting to hear what other people think about it who have a lot more research and possibly teaching experience than me.

    • 2. Don Davis (@gnu_don)  |  July 4, 2013 at 9:15 am

      Ha Peter,
      I couldn’t agree with you more. It’s always seemed a bit awkward requiring Calculus (or higher) as a prerequisite for entry level CS classes. It certainly does make broadening participation difficult. (Running a marathon would be a great prerequisite for the baseball team, but might severely and unnecessarily limit the pool of qualified applicants.)
      As far as persistence, Perkins et al. (1986) identified a CS dichotomy of ‘starters’ and ‘stoppers’ in their research – finding that meaningful persistence (purposeful tinkering) predicted success more than aptitude tests.
      Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2, 37–55.

      • 3. Mark Guzdial  |  July 4, 2013 at 9:31 am

        To take that one step further, Don, there’s Maureen Biggers’ nice paper distinguishing between the “stayers” and the “leavers,” suggesting that those that leave have an overly-constrained definition of programming:
        Maureen Biggers, Anne Brauer, and Tuba Yilmaz. 2008. Student perceptions of computer science: a retention study comparing graduating seniors with cs leavers. In Proceedings of the 39th SIGCSE technical symposium on Computer science education (SIGCSE ’08). ACM, New York, NY, USA, 402-406. DOI=10.1145/1352135.1352274

        Peter, you’re identifying a key idea. Success in mathematics correlates with success in CS classes, perhaps because of persistence as you say, but unlikely because mathematics is causally linked to success in CS. In fact, there are studies showing NO correlation between mathematics and CS learning (Ventura, 2005). Chapter 5 of Michael Caspersen’s dissertation ( is my favorite coverage of the factors that predict success in introductory computing. Unfortunately, Michael is able to find contradicting evidence for just about every predictive relationship — we just don’t know yet what predicts success in computing.

  • 4. Alfred Thompson  |  July 4, 2013 at 7:03 am

    35 years after looking for my first job in software development I can be amused at all the companies who gave me “aptitude tests” to see if I could be a programmer and ignored all of the college course work I had done well in. These days I recommend to students that they have a portfolio – some code they have written – to show people if they don’t have work experience yet.

  • 5. rdm  |  July 4, 2013 at 7:32 am

    It’s probably worth noting here that the label “programmer” is slightly more useful as a job description than the label “english”.

    Like brainteasers, it’s mostly useful because of scarcity.

  • 6. Franklin  |  July 4, 2013 at 11:31 am

    Unless you’re doing scientific computing, calculus is not very useful for programming. In any case, I would expect that logic and algebra are very useful, and that the problem is that everyone is expected to make it through calculus. Furthermore, geometry and trigonometry is useless, again, unless you’re doing robotics or graphics or the like. In short, the average programmer benefits from nothing other than logic, algebra (maybe including category theory, ha), combinatorics, that sort of thing, I’m guessing. K-12 education should stop assuming everyone is going to become a physicist.

    • 7. Erik Engbrecht  |  July 4, 2013 at 5:04 pm

      Calculus (and other forms of mathematics) are useful to programming under two different circumstances:
      1. The mathematics are a core part of the problem domain, or how the problem domain is communicated
      2. The mathematics are a core part of the solution domain

      Of course these frequently aren’t mutually exclusive.

      Scientific computing is certainly an example where 1 and 2 are true, but even if you define scientific computing very broadly it is far from the only example. Many types of machine learning require calculus. Any sort of statistics beyond the most basic level requires understanding calculus.

      A fair amount of mathematical knowledge, often well beyond basic calculus, is necessary to work effectively on programming problems in countless domains.

  • 8. Jeff Erickson  |  July 4, 2013 at 3:47 pm

    But those brainteasers are _also_ utterly useless for predicting success in academic mathematics!

  • 9. Lorin  |  July 6, 2013 at 9:41 pm

    Glad to see the brainteasers are gone, and I share your frustrating on the calculus thing. (Aside: and why is so much of high school mathematics devoted to learning pre-calculus skills that most people will never, ever use? Don’t get me started…).

    I do think the ability to solve Fermi problems (“How many piano tuners…”) has a lot of value, although I would never ask that sort of question in an interview. Estimation plays a huge role in software development, and it’s a skill that we could teach in university, but typically don’t.

  • 10. Mark Miller  |  July 8, 2013 at 6:49 pm

    I read an old article recently about some of these “Google questions,” and most of them sounded like the kind of things people got asked when interviewing at Microsoft. A couple of them sounded like “culture quizzes”:

    – Explain the significance of “dead beef”

    – A man pushed his car to a hotel, and lost his fortune. What happened?

    I fail to see how answers to these questions would be relevant to anything that Google does, beyond banter at lunch.

    In terms of the interview questions I’ve been asked, the best ones in my judgment came from small businesses. They’d either probe my attention to detail, or would test my understanding of mathematics and its application to computing. I don’t say this to say that I necessarily answered them well, but I found them illuminating as to my own capabilities at the time.

  • […] you read this paper, consider Nathan Ensmenger’s assertion that (a) mathematics has been show to predict success in CS classes but not in computing careers […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,052 other followers


Recent Posts

Blog Stats

  • 2,031,112 hits
July 2013

CS Teaching Tips

%d bloggers like this: