Posts tagged ‘educational psychology’
An interesting paper I found reading Annie Murphy Paul’s blog. An Expressed Interest is an answer to a question like “What career do you plan to pursue after College?” A Measured Vocational Interest is measuring an interest in mathematics, and suggesting that the student go into accounting. The former are far more predictive of future careers than the latter. Why are we so bad at predicting what field someone should go into based on their base interests? I’ll bet that it has to do with more things than just interests, like Eccles model of academic achievement (how do people think about this career? can you see yourself in this career?) and values (which are different than interests).
What an interesting paper! (Pun slightly intended.) In this paper from Paul Silvia, he found experimentally that self-efficacy and interest are related on a bell-shaped curve. Too little self-efficacy makes a task seem too daunting and uninteresting. Too much makes the task boring. This is important because we know that self-efficacy is among the most significant factors influencing non-majors success in learning to program. It’s clear that there’s a sweet spot that we’re aiming for.
Once in our Learning Sciences seminar, we all took the Myers-Briggs test on day 1 of the semester, and again at the end. Almost everybody’s score changed. So, why do people still use it as some kind of reliable test of personality?
A test is reliable if it produces the same results from different sources. If you think your leg is broken, you can be more confident when two different radiologists diagnose a fracture. In personality testing, reliability means getting consistent results over time, or similar scores when rated by multiple people who know me well. As my inconsistent scores foreshadowed, the MBTI does poorly on reliability. Research shows “that as many as three-quarters of test takers achieve a different personality type when tested again,” writes Annie Murphy Paul in The Cult of Personality Testing, “and the sixteen distinctive types described by the Myers-Briggs have no scientific basis whatsoever.” In a recent article, Roman Krznaric adds that “if you retake the test after only a five-week gap, there’s around a 50% chance that you will fall into a different personality category.”
Fascinating result: The bystanders have their learning impacted more than the ones who opened up the laptop.
There is a fundamental tension here, and I don’t know how to resolve it. On the one hand, I like it when students have their laptops in class. Many of them are more comfortable taking notes this way than longhand. In the middle of a lecture I might ask someone to look something up that I don’t know off the top of my head.
On the other hand, the potential for distraction is terrible. I’ve walked in the back of the classroom of many of my colleagues and seen that perhaps 50% of the students are on the Web.
Exactly how much standardized testing are school districts subjecting students to these days? A nearly staggering amount, according to a new analysis.
“Testing More, Teaching Less: What America’s Obsession with Student Testing Costs in Money and Lost Instructional Time,” released by the American Federation of Teachers, looks closely at two unnamed medium-sized school districts — one in the Midwest and one in the East — through the prism of their standardized testing calendars.
This article is worth blogging on for two reasons:
First, my colleagues in the UK were stunned when I told them that most tests that students take in US schools are locally invented. ”Doesn’t that lead to alot of wasted effort?” Perhaps so — this report seems to support my claim.
Second, I don’t find that much testing either staggering nor undesirable. Consider the results on the Testing Effect — students learn from testing. 20 hours in an academic year is not too much, if we think about testing as driving learning. We don’t know if these are good or useful tests, or if they are being used in a way that might motivate more learning, so 20 hours isn’t obviously a good thing. But it’s also not obviously a bad thing.
Consider the results of the paper presented by Michael Lee at ICER 2013 this year (and which won the “John Henry Award,” the people’s choice best paper award). They took a video game that required programming (Gidget) and added to it explicit assessments — quizzes that popped up at the end of each level, to ask you questions about what you did. They found that such assessments actually increased engagement and time-on-task. Their participants (both control and experimental) were recruited from Amazon’s Mechanical Turk, so they were paid to complete more levels. Adding assessments led to more levels completed and less time per level — that’s pretty remarkable.
Maybe what we need is not fewer tests, but better and more engaging tests.
The same kind of educational opportunity does not work for all students. In particular, constructionism may not provide enough structure for low achieving students. (See previous discussion about boredom vs. failure.)
Moreover, the researchers found different approaches effective for different types of students: “Usually people say, ‘Yes, autonomy is beneficial. We want to provide students with choices in school,’ This is the case for high achievers, but not low achievers,” Wang said. “Low achievers want more structure, more guidelines.”
Taking a test is better than studying, even if you just guess: We need to flip the flipped classroom
The benefits of testing for learning are fascinating, and the result described below makes me even more impressed with the effect. It suggests even more strongly that the critical feature of learning is trying to understand, trying to generate an answer, even more than reading an answer.
Suppose, for example, that I present you with an English vocabulary word you don’t know and either (1) provide a definition that you read (2) ask you to make up a definition or (3) ask you to choose from among a couple of candidate definitions. In conditions 2 & 3 you obviously must simply guess. (And if you get it wrong I’ll give you corrective feedback.) Will we see a testing effect?
That’s what Rosalind Potts & David Shanks set out to find, and across four experiments the evidence is quite consistent. Yes, there is a testing effect. Subjects better remember the new definitions of English words when they first guess at what the meaning is–no matter how wild the guess.
These results mesh well with a new study from Stanford. They found that the order of events in a “flipped” classroom matters — the problem-solving activity (in the classroom) should come before the reading or videos (at home). The general theme is the same in both sets of studies: problem-solving drives learning, and it’s less true that studying prepares one for problem-solving.
A new study from the Stanford Graduate School of Education flips upside down the notion that students learn best by first independently reading texts or watching online videos before coming to class to engage in hands-on projects. Studying a particular lesson, the Stanford researchers showed that when the order was reversed, students’ performances improved substantially.
Stuart Wray has a remarkable blog that I recommend to CS teachers. He shares his innovations in teaching, and grounds them in his exploration of the literature into the psychology of programming. The quote and link below is an excellent example, where his explanation led to me a paper I’m eager to dive into. Stuart has built an interesting warm-up activity for his class that involves robots. What I’m most intrigued by is his explanation for why it works as it does. The paper that he cites by Jones and Burnett is not one that I’d seen before, but it explores an idea that I’ve been interested in for awhile, ever since I discovered the Spatial Intelligence and Learning Center: Is spatial ability a pre-requisite for learning in computer science? And if so, can we teach it explicitly to improve CS learning?
The game is quite fun and doesn’t take very long to play — usually around a quarter of an hour or less. It’s almost always quite close at the end, because of course it’s a race between the last robot in each team. There’s plenty of opportunity for delaying tactics and clever blocking moves near the exit by the team which is behind, provided they don’t just individually run for the exit as fast as possible.
But turning back to the idea from James Randi, how does this game work? It seems from my experience to be doing something useful, but how does it really work as an opening routine for a programming class? Perhaps first of all, I think it lets me give the impression to the students that the rest of the class might be fun. Lots of students don’t seem to like the idea of programming, so perhaps playing a team game like this at the start of the class surprises them into giving it a second chance.
I think also that there is an element of “sizing the audience up” — it’s a way to see how the students interact with one another, to see who is retiring and who is bold, who is methodical and who is careless. The people who like clever tricks in the game seem often to be the people who like clever tricks in programming. There is also some evidence that facility with mental rotation is correlated with programming ability. (See Spatial ability and learning to program by Sue Jones and Gary Burnett in Human Technology, vol.4(1), May 2008, pp.47-61.) To the extent that this is true, I might be getting a hint about who will have trouble with programming from seeing who has trouble making their robot turn the correct direction.
An interesting study suggesting that role models and how they’re described (in terms of their achievements, or in terms of their struggles) has an interaction with students’ stereotypes about scientists and other professionals in STEM fields. So there are not just cognitive benefits to learning from failure, but there are affective dimensions to focusing on the struggle (including failures) and not just the success.
But when the researchers exposed middle-school girls to women who were feminine and successful in STEM fields, the experience actually diminished the girls’ interest in math, depressed their plans to study math, and reduced their expectations of future success. The women’s “combination of femininity and success seemed particularly unattainable to STEM-disidentified girls,” the authors conclude, adding that “gender-neutral STEM role models,” as well as feminine women who were successful in non-STEM fields, did not have this effect.
Does this mean that we have to give up our most illustrious role models? There is a way to gain inspiration from truly exceptional individuals: attend to their failures as well as their successes. This was demonstrated in a study by Huang-Yao Hong of National Chengchi University in Taiwan and Xiaodong Lin-Siegler of Columbia University.
The researchers gave a group of physics students information about the theories of Galileo Galilei, Issac Newton and Albert Einstein. A second group received readings praising the achievements of these scientists. And a third group was given a text that described the thinkers’ struggles. The students who learned about scientists’ struggles developed less-stereotyped images of scientists, became more interested in science, remembered the material better, and did better at complex open-ended problem-solving tasks related to the lesson—while the students who read the achievement-based text actually developed more stereotypical images of scientists.
I usually really like Annie Murphy Paul’s articles, but this one didn’t work for me. Below are her reasons why TED talk videos work well in learning, with my comments interspersed.
• They gratify our preference for visual learning. Effective presentations treat our visual sense as being integral to learning. This elevation of the image—and the eschewal of text-heavy Power Point presentations—comports well with cognitive scientists’ findings that we understand and remember pictures much better than mere words.
Cognitive scientists like Richard Mayer have found that diagrams and pictures can enhance learning — absolutely. But his work combined diagrams with words (e.g., best combination with diagrams: audio narration, not visual text). This quote seems to suggest that pictures are better than words. For most of STEM, that’s not true. We may have an affinity for visual, but that doesn’t mean that it works better for learning complex material.
• They engage the power of social learning. The robust conversation that videos can inspire, both online and off, recognizes a central principle of adult education: We learn best from other people. In the discussions, debates, and occasional arguments about the content of the talks they see, video-watchers are deepening their own knowledge and understanding.
Wait a minute — isn’t she just saying that TED talks give us something to talk about? TED talks are not themselves inherently social. Isn’t a book discussed in a book club just as effective for “engaging the power of social learning”? What makes TED talks so “social”?
• They enable self-directed, “just-in-time” learning. Because video viewers choose which talks to watch and when to watch them, they’re able to tailor their education to their own needs. Knowledge is easiest to absorb at the moment when we’re ready to apply it.
This was the quote that inspired this blog post. It’s an open question, but here’s my hypothesis. Nobody watches a TED talk for “just-in-time” learning. People watch TED talk for entertainment. ”I am about to go to my school board meeting — I think I’ll watch Sir Ken Robinson to figure out what to say!” ”I need to be able to guess birthdays — isn’t there a TED talk on that?” There are videos that really work for “just-in-time” learning. TED talks aren’t like that.
• They encourage viewers to build on what they already know. Adults are not blank slates: They bring to learning a lifetime of previously acquired information and experience. Effective video instruction build on top of this knowledge, adding and elaborating without dumbing down.
It’s absolutely true that effective instruction builds on top of existing knowledge, which is something that the best teachers know how to do — to figure out what students know and care about, and relate knowledge to that. How does a fixed video build on what viewers (all hundreds of thousands of them) actually know? No, I don’t see how TED talks do that.
The Muller research being described in the below post was discussed here previously, and is related to the predict-before-demo work that Eric Mazur presented at last year’s ICER. The uppermost bit here is that data mining can’t get at this level of abstraction in terms of identifying good teaching. I’m also concerned that data mining can’t help if you lose 80% of your subject pool — you can’t learn about people who aren’t there.
But even granting that you can get sufficiently rich information about the students, there’s another hard problem. Let’s say that, thanks to the upgrade in your big data infinite improbability drive made possible by your new Spacely’s space sprocket, your system is able to flag at least a critical mass of videos taught in the Mueller method as having a bigger educational impact on the students the average educational video by some measure you have identified. Would the machine be able to infer that these videos belong in a common category in terms of the reason for their effectiveness? Would it be able to figure out what Muller did? There are lots of reasons why a video might be more effective than average. And many of those ways are internal to the narrative structure of the video. The machine only knows things like the format of the video, the length, what kind of class it’s in, who the creator is, when it was made, and so on. Other than the external characteristics of the video file, it mostly knows what we tell it about the contents. It has no way for it to inspect the video and deduce that a particular presentation strategy is being used. We are nowhere close to having a machine that is smart enough to do what Muller did and identify a pattern in the narrative of the speaker.
Was anyone else bothered by the argument in this NYtimes blog post? ”MOOCS aren’t effective in terms of completion rates; Duolingo is not a MOOC; Duolingo is effective.” So…what does that tell us about MOOCs?
The paper on Duolingo effectiveness is pretty cool. I think it’s particularly noteworthy that more prior knowledge of Spanish led to less of an effect of Duolingo. I wonder if that’s because Duolingo is essentially using a worked example model, and worked examples do suffer from the expertise reversal effect.
Moreover, there are early indications that the high interactivity and personalized feedback of online education might ultimately offer a learning structure that can’t be matched by the traditional classroom.
Duolingo, a free Web-based language learning system that grew out of a Carnegie Mellon University research project, is not an example of a traditional MOOC. However, the system, which now teaches German, French, Portuguese, Italian, Spanish and English, has roughly one million users and about 100,000 people spend time on the site daily.
The journal article on the research that Klara Benda, Amy Bruckman, and I did finally came out last month the ACM Transactions on Computing Education. The abstract is below. Klara has a background in sociology, and she’s done a great job of blending research from sociology with more traditional education and learning sciences perspectives to explain what happens when working professionals take on-line CS classes. This work has informed our CSLearning4U project significantly, and informs my perspective on MOOCs.
We present the results of an interview study investigating student experiences in two online introductory computer science courses. Our theoretical approach is situated at the intersection of two research traditions: distance and adult education research, which tends to be sociologically oriented, and computer science education research, which has strong connections with pedagogy and psychology. The article reviews contributions from both traditions on student failure in the context of higher education, distance and online education as well as introductory computer science. Our research relies on a combination of the two perspectives, which provides useful results for the field of computer science education in general, as well as its online or distance versions. The interviewed students exhibited great diversity in both socio-demographic and educational background. We identified no profiles that predicted student success or failure. At the same time, we found that expectations about programming resulted in challenges of time-management and communication. The time requirements of programming assignments were unpredictable, often disproportionate to expectations, and clashed with the external commitments of adult professionals. Too little communication was available to access adequate instructor help. On the basis of these findings, we suggest instructional design solutions for adult professionals studying introductory computer science education.
I mentioned in a previous blog post the nice summary article that Audrey Watters wrote (linked below) about Learning to Code trends in educational technology in 2012, when I critiqued Jeff Atwood’s position on not learning to code.
Audrey does an excellent job of describing the big trends in learning to code this last year, from CodeAcademy to Bret Victor and Khan Academy and MOOCs. But the part that I liked the best was where she identified the problem that cool technology and badges won’t solve: culture and pedagogy.
Two organizations — Black Girls Code and CodeNow — did hold successful Kickstarter campaigns this year to help “change the ratio” and give young kids of color and young girls opportunities to learn programming. And the Irish non-profit CoderDojo also ventured state-side in 2012, helping expand afterschool opportunities for kids interested in hacking. The Maker Movement another key ed-tech trend this year is also opening doors for folks to play and experiment with technologies.
And yet, despite all the hype and hullaballoo from online learning startups and their marketing campaigns that now “everyone can learn to code,” its clear there are still plenty of problems with the culture and the pedagogy surrounding computer science education.
We still do need new programming languages whose design is informed by how humans work and learn. We still do need new learning technologies that can help us provide the right learning opportunities for individual student’s needs and can provide access to those who might not otherwise get the opportunity. But those needs are swamped by culture and pedagogy.
What do I mean by culture and pedagogy?
Culture: Betsy diSalvo’s work on Glitch is a great example of considering culture in computing education. I’ve written about her work before — that she engaged a couple dozen African-American teen men in computing, by hiring them to be video game testers, and the majority of those students went on to post-secondary education in computing. I’ve talked with Betsy several times about how and why that worked. The number one reason why it worked: Betsy spent the time to understand the African-American teen men’s values, their culture, what they thought was important. She engaged in an iterative design process with groups of teen men to figure out what would most appeal to them, how she could reframe computing into something that they would engage with. Betsy taught coding — but in a different way, in a different context, with different values, where the way, context, and values were specifically tuned to her audience. Is it worth that effort? Yeah, because it’s about making a computing that appeals to these other audiences.
Pedagogy: A lot of my work these days is about pedagogy. I use peer instruction in my classrooms, and try out worked examples in various ways. In our research, we use subgoal labels to improve our instructional materials. These things really work.
Let me give you an example with graphs that weren’t in Lauren Margelieux’s paper, but are in the talk slides that she made for me. As you may recall, we had two sets of instructional materials: A set of nice videos and text descriptions that Barbara Ericson built, and a similar set with subgoal labels inserted. We found that the subgoal labelled instruction led to better performance (faster and more correct) immediately after instruction, more retention (better performance a week later), and better performance on a transfer task (got more done on a new app that the students had never seen before). But I hadn’t shown you before just how enormous was the gap between the subgoal labelled group and the conventional group on the transfer task.
Part of the transfer task involved defining a variable in App Inventor — don’t just grab a component, but define a variable to represent that component. The subgoal label group did that more often. ALOT more often.
Lauren also noticed that the conventional group tended to “thrash,” to pull out more blocks in App Inventor than they actually needed. The correlation between number of blocks drawn out and correctness was r = -.349 — you are less likely to be correct (by a large amount) if you pull out extra blocks. Here’s the graph of number of blocks pulled out by each group.
These aren’t small differences! These are huge differences from a surprisingly small difference between the instructional materials. Improving our pedagogy could have a huge impact.
I agree with Audrey: Culture and pedagogy are two of the bigger issues in learning to code.
Fascinating question! Bilingual people have some additional executive control. Does learning a programming language give a similar benefit in executive control? The study described below is suggestive but not conclusive. If we could find evidence for it, it would be another benefit of learning to program.
If computer programming languages are languages, then people who spoke one language and could programme to a high standard should be bilingual. Research has suggested that bilingual people perform faster than monolingual people at tasks requiring executive control – that is, tasks involving the ability to pay attention to important information and ignore irrelevant information (for a review of the “robust” evidence for this, see Hilchey & Klein, 2011). So, I set out to find out whether computer programmers were better at these tasks too. It is thought that the bilingual advantage is the result of the effort involved in keeping two languages separate in the brain and deciding which one to use. I noticed that novice computer programmers have difficulty in controlling “transfer” from English to programming languages (e.g. expecting the command “while” to imply continuous checking; see Soloway and Spohrer, 1989), so it seemed plausible that something similar might occur through the learning of programming languages.