Posts tagged ‘educational psychology’
What a cool idea! Rob Moore is building on the subgoal labeling work that we (read: “Lauren”) did, and is using crowd-sourcing techniques to generate the labels.
Subgoal labeling  is a technique known to support learning new knowledge by clustering a group of steps into a higher-level conceptual unit. It has been shown to improve learning by helping learners to form the right mental model. While many learners view video tutorials nowadays, subgoal labels are often not available unless manually provided at production time. This work addresses the challenge of collecting and presenting subgoal labels to a large number of video tutorials. We introduce a mixed-initiative approach to collect subgoal labels in a scalable and efficient manner. The key component of this method is learnersourcing, which channels learners’ activities using the video interface into useful input to the system. The presented method will contribute to the broader availability of subgoal labels in how-to videos.
I met with a prospective PhD student recently, who told me that she’s interested in using big data to inform her design of computing education. She said that she disliked designing something, just crossing her fingers hoping it would work. She and the faculty she’s working with are trying to use big data to inform their design decisions.
That’s a fine approach, but it’s pretty work-intensive. You gather all this data, then you have to figure out what’s relevant, and what it means, and how it influences practice. It’s a very computer science-y way of solving the problem, but it’s rather brute force.
There is a richer data source with much more easily applicable design guidelines: educational psychology literature. Educational psychologists have been thinking about these issues for a long time. They know a lot of things.
We’re finding that we can inform a lot of our design decisions by simply reading the relevant education literature:
- Like our work on subgoal labeling,
- And on worked examples,
- And on lower-cognitive load learning,
- And on peer instruction.
I was recently reading a computer science paper in which the author said that we don’t know much about mathematics education, and that’s because we’ve never had enough data to come up with findings. But there were no references to mathematics education literature. We actually know a lot about mathematics education literature. Too often, I fear that we computer scientists want to invent it all ourselves, as if that was a better approach. Why not just talk to and read the work of really smart people who have devoted their lives to figuring out how to teach better?
What can the teacher do to inculcate interest? What responsibility does the teacher have to sustain interest? If there is a way to teach that can be effective, don’t teachers have a moral obligation to teach that way?
In general, findings from studies of interest suggest that educators can (a) help students sustain attention for tasks even when tasks are challenging—this could mean either providing support so that students can experience a triggered situational interest or feedback that allows them to sustain attention so that they can generate their own curiosity questions; (b) provide opportunities for students to ask curiosity questions; and (c) select or create resources that promote problem solving and strategy generation.
An interesting paper I found reading Annie Murphy Paul’s blog. An Expressed Interest is an answer to a question like “What career do you plan to pursue after College?” A Measured Vocational Interest is measuring an interest in mathematics, and suggesting that the student go into accounting. The former are far more predictive of future careers than the latter. Why are we so bad at predicting what field someone should go into based on their base interests? I’ll bet that it has to do with more things than just interests, like Eccles model of academic achievement (how do people think about this career? can you see yourself in this career?) and values (which are different than interests).
What an interesting paper! (Pun slightly intended.) In this paper from Paul Silvia, he found experimentally that self-efficacy and interest are related on a bell-shaped curve. Too little self-efficacy makes a task seem too daunting and uninteresting. Too much makes the task boring. This is important because we know that self-efficacy is among the most significant factors influencing non-majors success in learning to program. It’s clear that there’s a sweet spot that we’re aiming for.
Once in our Learning Sciences seminar, we all took the Myers-Briggs test on day 1 of the semester, and again at the end. Almost everybody’s score changed. So, why do people still use it as some kind of reliable test of personality?
A test is reliable if it produces the same results from different sources. If you think your leg is broken, you can be more confident when two different radiologists diagnose a fracture. In personality testing, reliability means getting consistent results over time, or similar scores when rated by multiple people who know me well. As my inconsistent scores foreshadowed, the MBTI does poorly on reliability. Research shows “that as many as three-quarters of test takers achieve a different personality type when tested again,” writes Annie Murphy Paul in The Cult of Personality Testing, “and the sixteen distinctive types described by the Myers-Briggs have no scientific basis whatsoever.” In a recent article, Roman Krznaric adds that “if you retake the test after only a five-week gap, there’s around a 50% chance that you will fall into a different personality category.”
Fascinating result: The bystanders have their learning impacted more than the ones who opened up the laptop.
There is a fundamental tension here, and I don’t know how to resolve it. On the one hand, I like it when students have their laptops in class. Many of them are more comfortable taking notes this way than longhand. In the middle of a lecture I might ask someone to look something up that I don’t know off the top of my head.
On the other hand, the potential for distraction is terrible. I’ve walked in the back of the classroom of many of my colleagues and seen that perhaps 50% of the students are on the Web.
Exactly how much standardized testing are school districts subjecting students to these days? A nearly staggering amount, according to a new analysis.
“Testing More, Teaching Less: What America’s Obsession with Student Testing Costs in Money and Lost Instructional Time,” released by the American Federation of Teachers, looks closely at two unnamed medium-sized school districts — one in the Midwest and one in the East — through the prism of their standardized testing calendars.
This article is worth blogging on for two reasons:
First, my colleagues in the UK were stunned when I told them that most tests that students take in US schools are locally invented. ”Doesn’t that lead to alot of wasted effort?” Perhaps so — this report seems to support my claim.
Second, I don’t find that much testing either staggering nor undesirable. Consider the results on the Testing Effect — students learn from testing. 20 hours in an academic year is not too much, if we think about testing as driving learning. We don’t know if these are good or useful tests, or if they are being used in a way that might motivate more learning, so 20 hours isn’t obviously a good thing. But it’s also not obviously a bad thing.
Consider the results of the paper presented by Michael Lee at ICER 2013 this year (and which won the “John Henry Award,” the people’s choice best paper award). They took a video game that required programming (Gidget) and added to it explicit assessments — quizzes that popped up at the end of each level, to ask you questions about what you did. They found that such assessments actually increased engagement and time-on-task. Their participants (both control and experimental) were recruited from Amazon’s Mechanical Turk, so they were paid to complete more levels. Adding assessments led to more levels completed and less time per level — that’s pretty remarkable.
Maybe what we need is not fewer tests, but better and more engaging tests.
The same kind of educational opportunity does not work for all students. In particular, constructionism may not provide enough structure for low achieving students. (See previous discussion about boredom vs. failure.)
Moreover, the researchers found different approaches effective for different types of students: “Usually people say, ‘Yes, autonomy is beneficial. We want to provide students with choices in school,’ This is the case for high achievers, but not low achievers,” Wang said. “Low achievers want more structure, more guidelines.”
Taking a test is better than studying, even if you just guess: We need to flip the flipped classroom
The benefits of testing for learning are fascinating, and the result described below makes me even more impressed with the effect. It suggests even more strongly that the critical feature of learning is trying to understand, trying to generate an answer, even more than reading an answer.
Suppose, for example, that I present you with an English vocabulary word you don’t know and either (1) provide a definition that you read (2) ask you to make up a definition or (3) ask you to choose from among a couple of candidate definitions. In conditions 2 & 3 you obviously must simply guess. (And if you get it wrong I’ll give you corrective feedback.) Will we see a testing effect?
That’s what Rosalind Potts & David Shanks set out to find, and across four experiments the evidence is quite consistent. Yes, there is a testing effect. Subjects better remember the new definitions of English words when they first guess at what the meaning is–no matter how wild the guess.
These results mesh well with a new study from Stanford. They found that the order of events in a “flipped” classroom matters — the problem-solving activity (in the classroom) should come before the reading or videos (at home). The general theme is the same in both sets of studies: problem-solving drives learning, and it’s less true that studying prepares one for problem-solving.
A new study from the Stanford Graduate School of Education flips upside down the notion that students learn best by first independently reading texts or watching online videos before coming to class to engage in hands-on projects. Studying a particular lesson, the Stanford researchers showed that when the order was reversed, students’ performances improved substantially.
Stuart Wray has a remarkable blog that I recommend to CS teachers. He shares his innovations in teaching, and grounds them in his exploration of the literature into the psychology of programming. The quote and link below is an excellent example, where his explanation led to me a paper I’m eager to dive into. Stuart has built an interesting warm-up activity for his class that involves robots. What I’m most intrigued by is his explanation for why it works as it does. The paper that he cites by Jones and Burnett is not one that I’d seen before, but it explores an idea that I’ve been interested in for awhile, ever since I discovered the Spatial Intelligence and Learning Center: Is spatial ability a pre-requisite for learning in computer science? And if so, can we teach it explicitly to improve CS learning?
The game is quite fun and doesn’t take very long to play — usually around a quarter of an hour or less. It’s almost always quite close at the end, because of course it’s a race between the last robot in each team. There’s plenty of opportunity for delaying tactics and clever blocking moves near the exit by the team which is behind, provided they don’t just individually run for the exit as fast as possible.
But turning back to the idea from James Randi, how does this game work? It seems from my experience to be doing something useful, but how does it really work as an opening routine for a programming class? Perhaps first of all, I think it lets me give the impression to the students that the rest of the class might be fun. Lots of students don’t seem to like the idea of programming, so perhaps playing a team game like this at the start of the class surprises them into giving it a second chance.
I think also that there is an element of “sizing the audience up” — it’s a way to see how the students interact with one another, to see who is retiring and who is bold, who is methodical and who is careless. The people who like clever tricks in the game seem often to be the people who like clever tricks in programming. There is also some evidence that facility with mental rotation is correlated with programming ability. (See Spatial ability and learning to program by Sue Jones and Gary Burnett in Human Technology, vol.4(1), May 2008, pp.47-61.) To the extent that this is true, I might be getting a hint about who will have trouble with programming from seeing who has trouble making their robot turn the correct direction.
An interesting study suggesting that role models and how they’re described (in terms of their achievements, or in terms of their struggles) has an interaction with students’ stereotypes about scientists and other professionals in STEM fields. So there are not just cognitive benefits to learning from failure, but there are affective dimensions to focusing on the struggle (including failures) and not just the success.
But when the researchers exposed middle-school girls to women who were feminine and successful in STEM fields, the experience actually diminished the girls’ interest in math, depressed their plans to study math, and reduced their expectations of future success. The women’s “combination of femininity and success seemed particularly unattainable to STEM-disidentified girls,” the authors conclude, adding that “gender-neutral STEM role models,” as well as feminine women who were successful in non-STEM fields, did not have this effect.
Does this mean that we have to give up our most illustrious role models? There is a way to gain inspiration from truly exceptional individuals: attend to their failures as well as their successes. This was demonstrated in a study by Huang-Yao Hong of National Chengchi University in Taiwan and Xiaodong Lin-Siegler of Columbia University.
The researchers gave a group of physics students information about the theories of Galileo Galilei, Issac Newton and Albert Einstein. A second group received readings praising the achievements of these scientists. And a third group was given a text that described the thinkers’ struggles. The students who learned about scientists’ struggles developed less-stereotyped images of scientists, became more interested in science, remembered the material better, and did better at complex open-ended problem-solving tasks related to the lesson—while the students who read the achievement-based text actually developed more stereotypical images of scientists.
I usually really like Annie Murphy Paul’s articles, but this one didn’t work for me. Below are her reasons why TED talk videos work well in learning, with my comments interspersed.
• They gratify our preference for visual learning. Effective presentations treat our visual sense as being integral to learning. This elevation of the image—and the eschewal of text-heavy Power Point presentations—comports well with cognitive scientists’ findings that we understand and remember pictures much better than mere words.
Cognitive scientists like Richard Mayer have found that diagrams and pictures can enhance learning — absolutely. But his work combined diagrams with words (e.g., best combination with diagrams: audio narration, not visual text). This quote seems to suggest that pictures are better than words. For most of STEM, that’s not true. We may have an affinity for visual, but that doesn’t mean that it works better for learning complex material.
• They engage the power of social learning. The robust conversation that videos can inspire, both online and off, recognizes a central principle of adult education: We learn best from other people. In the discussions, debates, and occasional arguments about the content of the talks they see, video-watchers are deepening their own knowledge and understanding.
Wait a minute — isn’t she just saying that TED talks give us something to talk about? TED talks are not themselves inherently social. Isn’t a book discussed in a book club just as effective for “engaging the power of social learning”? What makes TED talks so “social”?
• They enable self-directed, “just-in-time” learning. Because video viewers choose which talks to watch and when to watch them, they’re able to tailor their education to their own needs. Knowledge is easiest to absorb at the moment when we’re ready to apply it.
This was the quote that inspired this blog post. It’s an open question, but here’s my hypothesis. Nobody watches a TED talk for “just-in-time” learning. People watch TED talk for entertainment. ”I am about to go to my school board meeting — I think I’ll watch Sir Ken Robinson to figure out what to say!” ”I need to be able to guess birthdays — isn’t there a TED talk on that?” There are videos that really work for “just-in-time” learning. TED talks aren’t like that.
• They encourage viewers to build on what they already know. Adults are not blank slates: They bring to learning a lifetime of previously acquired information and experience. Effective video instruction build on top of this knowledge, adding and elaborating without dumbing down.
It’s absolutely true that effective instruction builds on top of existing knowledge, which is something that the best teachers know how to do — to figure out what students know and care about, and relate knowledge to that. How does a fixed video build on what viewers (all hundreds of thousands of them) actually know? No, I don’t see how TED talks do that.
The Muller research being described in the below post was discussed here previously, and is related to the predict-before-demo work that Eric Mazur presented at last year’s ICER. The uppermost bit here is that data mining can’t get at this level of abstraction in terms of identifying good teaching. I’m also concerned that data mining can’t help if you lose 80% of your subject pool — you can’t learn about people who aren’t there.
But even granting that you can get sufficiently rich information about the students, there’s another hard problem. Let’s say that, thanks to the upgrade in your big data infinite improbability drive made possible by your new Spacely’s space sprocket, your system is able to flag at least a critical mass of videos taught in the Mueller method as having a bigger educational impact on the students the average educational video by some measure you have identified. Would the machine be able to infer that these videos belong in a common category in terms of the reason for their effectiveness? Would it be able to figure out what Muller did? There are lots of reasons why a video might be more effective than average. And many of those ways are internal to the narrative structure of the video. The machine only knows things like the format of the video, the length, what kind of class it’s in, who the creator is, when it was made, and so on. Other than the external characteristics of the video file, it mostly knows what we tell it about the contents. It has no way for it to inspect the video and deduce that a particular presentation strategy is being used. We are nowhere close to having a machine that is smart enough to do what Muller did and identify a pattern in the narrative of the speaker.
Was anyone else bothered by the argument in this NYtimes blog post? ”MOOCS aren’t effective in terms of completion rates; Duolingo is not a MOOC; Duolingo is effective.” So…what does that tell us about MOOCs?
The paper on Duolingo effectiveness is pretty cool. I think it’s particularly noteworthy that more prior knowledge of Spanish led to less of an effect of Duolingo. I wonder if that’s because Duolingo is essentially using a worked example model, and worked examples do suffer from the expertise reversal effect.
Moreover, there are early indications that the high interactivity and personalized feedback of online education might ultimately offer a learning structure that can’t be matched by the traditional classroom.
Duolingo, a free Web-based language learning system that grew out of a Carnegie Mellon University research project, is not an example of a traditional MOOC. However, the system, which now teaches German, French, Portuguese, Italian, Spanish and English, has roughly one million users and about 100,000 people spend time on the site daily.