Posts tagged ‘ICER’

Growth mindset matters for individual human performance, with a more indirect connection to academic success

One of the most talked-about papers at ICER 2018 was this one, “Fixed versus Growth Mindset Does not Seem to Matter Much: A Prospective Observational Study in Two Late Bachelor level Computer Science Courses.” The claim was that fixed and growth mindset did not have much impact on student course performance.  One of the authors wrote a blog post summarizing the paper.
In my opinion, they got growth/fixed mindset theory wrong.  The mistake is in the first line of the abstract, “Psychology predicts that a student’s mindset—their implicit theory of intelligence—has an effect on their academic performance.”  Growth and fixed mindset have an effect on individual student development. There is an indirect effect on academic performance which is more complex. Grades are not the same as measuring learning. Grades are typically a measure of mastery of concepts.
The presentation of the paper had this amazing graph (picture I took below).  Most students fail in the courses they studied.  Look at the big peaks in the distribution on the left. Those are all the fails.
IMG_0863
In Freakonomics, there’s a chapter on why, if drug dealers make so much money, why do so many of them live with their mothers?  (The chapter is reprinted here.) The answer is that drug dealing (like professional sports or acting) is a “lottery” — many people try and make no money, and very few people get to the top and make lots of money.  All those high school and college football players who are waking up early to pump weights have a growth mindset — they believe that their effort will take them to the NFL.  However, the vast majority are *wrong*. They won’t make it.  There is no apparent connection between growth mindset and success.
That’s how I saw the ICER paper on fixed and growth mindset.  If the outcome variable is academic success, growth mindset isn’t going to always pay-off. Sometimes the deck is stacked against you, and even if you think you can win, you won’t.
However, if the outcome variable is individual development, growth mindset will likely beat fixed mindset.  If you believe you can get better, you might. If you don’t believe you can get better, you won’t. A good outcome variable would be learning gain, measured pre-test to post-test.  In this study, most students had a growth mindset, so they probably wouldn’t have seen much variation (between growth and fixed) even if they measured learning.
The students thought if they worked harder, they could do better. And they probably did all do better (from a learning perspective). They just weren’t going to win in this lottery.
It’s a different question whether a given intervention to improve mindset might lead to improved academic performance.  If you improve learning, and academic performance is reflective of learning, then there should be a connection IF it’s possible to change mindset with an intervention. Duckworth and Dweck have shown that they have successfully intervened to change students’ mindset and consequently improve academic performance, and that work was recently replicated (see post here).  The efforts to intervene on mindset in CS have had mixed success (see my blog post here on that). But it’s one thing to say that fixed vs growth mindset does not seem to matter much (the title of the paper presented at ICER), and another to say that a given mindset intervention did not result in academic performance increase. The first claim is about theory, and the second is about designing interventions with a multi-step causal chain. I don’t buy the former claim, but completely agree that the latter is a complex and interesting issue to explore.

September 7, 2018 at 7:00 am 5 comments

Adaptive Parsons problems, and the role of SES and Gesture in learning computing: ICER 2018 Preview

 

Next week is the 2018 International Computing Education Research Conference in Espoo, Finland. The proceedings are (as of this writing) available here: https://dl.acm.org/citation.cfm?id=3230977. Our group has three papers in the 28 accepted this year.

“Evaluating the efficiency and effectiveness of adaptive Parsons problems” by Barbara Ericson, Jim Foley, and Jochen (“Jeff”) Rick

These are the final studies from Barb Ericson’s dissertation (I blogged about her defense here). In her experiment, she compared four conditions: Students learning through writing code, through fixing code, through solving Parsons problems, and through solving her new adaptive Parsons problems. She had a control group this time (different from her Koli Calling paper) that did turtle graphics between the pre-test and post-test, so that she could be sure that there wasn’t just a testing effect of pre-test followed by a post-test. The bottom line was basically what she predicted: Learning did occur, with no significant difference between treatment groups, but the Parsons problems groups took less time. Our ebooks now include some of her adaptive Parsons problems, so she can compare performance across many students on adaptive and non-adaptive forms of the same problem. She finds that students solve the problems more and with fewer trials on the adaptive problems. So, adaptive Parsons problems lead to the same amount of learning, in less time, with fewer failures. (Failures matter, since self-efficacy is a big deal in computer science education.)

“Socioeconomic status and Computer science achievement: Spatial ability as a mediating variable in a novel model of understanding” by Miranda Parker, Amber Solomon, Brianna Pritchett, David Illingworth, Lauren Margulieux, and Mark Guzdial

(Link to last version I reviewed.)

This study is a response to the paper Steve Cooper presented at ICER 2015 (see blog post here), where they found that spatial reasoning training erased performance differences between higher and lower socioeconomic status (SES) students, while the comparison class had higher-SES students performing better than lower-SES students. Miranda and Amber wanted to test this relationship at a larger scale.

Why should wealthier students do better in CS? The most common reason I’ve heard is that wealthier students have more opportunities to study CS — they have greater access. Sometimes that’s called preparatory privilege.

Miranda and Amber and their team wanted to test whether access is really the right intermediate variable. They gave students at two different Universities four tests:

  • Part of Miranda’s SCS1 to measure performance in CS.
  • A standardized test of SES.
  • A test of spatial reasoning.
  • A survey about the amount of access they had to CS education, e.g., formal classes, code clubs, summer camps, etc.

David and Lauren did the factor analysis and structural equation modeling to compare two hypotheses: Does higher SES lead to greater access which leads to greater success in CS, or does higher SES lead to higher spatial reasoning which leads to greater success in CS? Neither hypothesis accounted for a significant amount of the differences in CS performance, but the spatial reasoning model did better than the access model.

There are some significant limitations of this study. The biggest is that they gathered data at universities. A lot of SES variance just disappears when you look at college students — they tend to be wealthier than average.

Still, the result is important for challenging the prevailing assumption about why wealthier kids do better in CS. More, spatial reasoning is an interesting variable because it’s rather inexpensively taught. It’s expensive to prepare CS teachers and get them into all schools. Steve showed that we can teach spatial reasoning within an existing CS class and reduce SES differences.

“Applying a Gesture Taxonomy to Introductory Computing Concepts” by Amber Solomon, Betsy DiSalvo, Mark Guzdial, and Ben Shapiro

(Link to last version I saw.)

We were a bit surprised (quite pleasantly!) that this paper got into ICER. I love the paper, but it’s different from most ICER papers.

Amber is interested in the role that gestures play in teaching CS. She started this paper from a taxonomy of gestures seen in other STEM classes. She observed a CS classroom and used her observations to provide concrete examples of the gestures seen in other kinds of classes. This isn’t a report of empirical findings. This is a report of using a lens borrowed from another field to look at CS learning and teaching in a new way.

My favorite part of of this paper is when Amber points out what parts of CS gestures don’t really fit in the taxonomy. It’s one thing to point to lines of code – that’s relatively concrete. It’s another thing to “point” to reference data, e.g., when explaining a sort and you gesture at the two elements you’re comparing or swapping. What exactly/concretely are we pointing at? Arrays are neither horizontal nor vertical — that distinction doesn’t really exist in memory. Arrays have no physical representation, but we act (usually) as if they’re laid out horizontally in front of us. What assumptions are we making in order to use gestures in our teaching? And what if students don’t share in those assumptions?

August 10, 2018 at 7:00 am Leave a comment

A Place to Get Feedback and Develop New Ideas: WIPW at ICER 2018

Everybody’s got an idea that they’re sure is great, or could be great with just a bit of development. Similarly, everyone has hit a tricky crossroads in their research and could use a little nudge to get unstuck. The ICER Work in Progress workshop is the place to get feedback and help on that idea, and give feedback and help to others on their cool ideas. I did it a few years ago at the Glasgow ICER and had a wonderful day. You learn a lot, and you get a bunch of new insights about your own idea. As Workshop Leader (and the inventor of the ICER Work in Progress workshop series) Colleen Lewis put it, “You get the chance to borrow the brains of some really awesome people to work on your problem.”

Colleen is the Senior Chair again this year, and I’m the Junior Chair-in-Training.

The workshop is only one day and super-fun. If you’re attending ICER this year, please apply for the Work in Progress workshop! https://icer.hosting.acm.org/icer-2018/work-in-progress/ The application is due June 8 (it’s just a quick Google form).

Let Colleen or me know if you have questions!

May 30, 2018 at 7:00 am 2 comments

ICER 2018 Call for Participation (I’m co-chairing Works in Progress)

Do submit to ICER 2018 in Finland.  I particularly encourage you to join the Works in Progress workshop, for which I’ll be the junior co-chair as I learn the ropes from Colleen Lewis. I was a participant in the Works in Progress workshop in Glasgow and found it fun and useful.

ICER’18 – Call For Participation

The fourteenth annual ACM International Computing Education Research (ICER) Conference aims to gather high-quality contributions to the computing education research discipline. We invite submissions across a variety of categories for research investigating how people of all ages come to understand computational processes and devices, and empirical evaluation of approaches to improve that understanding in formal and informal learning environments.


Research areas of particular interest include:
– discipline based education research (DBER) in computer science (CS), information sciences (IS), and related disciplines
– design-based research, learner-centered design, and evaluation of educational technology supporting computing knowledge or skills development
pedagogical environments fostering computational thinking
learning sciences work in the computing content domain
psychology of programming
learning analytics and educational data mining in CS/IS content areas
learnability/usability of programming languages
informal learning experiences related to programming and software development (all ages), ranging from after-school programs for children, to end-user development communities, to workplace training of computing professionals
measurement instrument development and validation (e.g., concept inventories, attitudes scales, etc) for use in computing disciplines
research on CS/computing teacher thinking and professional development models at all levels
rigorous replication of empirical work to compare with or extend previous empirical research results
systematic literature review on some topic related to computer science education


In addition to standard research paper contributions, we continue our longstanding commitment to fostering discussion and exploring new research areas by offering several ways to engage. These include a doctoral consortium for graduate students just prior to the conference, a work-in-progress workshop for researchers following the conference, and poster and lightning talks. This is in addition to the format of conference sessions, where all research paper presentations include time for discussion among the attendees followed by feedback to the paper presenters.

Submission Categories

ICER provides multiple options for participation, with various levels of discussion and interaction between the presenter and audience. These sessions also support work at various levels, ranging from formative work to polished, complete research results.


Research Papers
Papers are limited to 8 pages, excluding references, double-blind peer reviewed and published in the ACM digital library as part of the conference proceedings. Accepted papers are allotted time for presentation and discussion at the conference


Doctoral Consortium
2 page extended abstract submission required and published in ACM digital library as part of the conference proceedings. Students will present their work to distinguished faculty mentors during an all-day workshop and during the conference in a dedicated poster session.


Lightning Talks and Posters
Abstract (250 words) submission required and made available on conference website, but not published in proceedings. Accepted abstracts for lightning talks will be given a 3-minute time slot for rapid presentation at the conference followed by a discussion period for all attendees. Posters may either accompany a lightning talk or may be proposed separately using the same abstract submission process.


Work in Progress Workshop
This one-day workshop is a venue to get sustained engagement with and feedback about early work in computing education.    White paper submission required but not included in proceedings.


Co-located Workshops
Proposals for pre/post conference workshops of interest to the ICER community (i.e., those that aim to advance computer science education research) are welcomed and encouraged. ICER local arrangements personnel will be available to assist with workshop logistics where possible. If interested, contact the conference chairs for more details by April 10th, 2018: Lauri.Malmi@aalto.fi or Ari.Korhonen@aalto.fi.


For more information about preparation and submission, please visit the page corresponding to the submission type of interest.

Important Deadlines and Dates


Research Papers

30 March, 2018 – – Abstract submission (250 words, mandator)
6 April, 2018 – – Full paper submission 
1 June – – Notification of acceptance 
15 June – -Final camera ready deadline
Other Submission Types
1 May – – Doctoral consortium submissions
8 June – – Lightning talk and Poster proposals
8 June – – Work in progress workshop application

Conference Schedule

Doctoral Consortium, Sunday, August 12, 2018
ICER Conference, Monday, August 13 – Wednesday August 15, 2018
Work in Progress Workshop, Wednesday evening, August 15 – Thursday, August 16, 2018
For more details, see the conference website:
 http://www.icer-conference.org

Conference Co-Chairs
Lauri Malmi, Aalto University, Finland (Lauri.Malmi@aalto.fi)
Ari Korhonen, Aalto University, Finland (Ari.Korhonen@aalto.fi
Robert McCartney, University of Connecticut, USA (robert.mccartney@uconn.edu)
Andrew Petersen, University of Toronto Mississauga, Canada (andrew.petersen@utoronto.ca)


AUTHORS TAKE NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date will be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work.

January 15, 2018 at 7:30 am Leave a comment

Teachers are not the same as students, and the role of tracing: ICER 2017 Preview

The International Computing Education Research conference starts today at the University of Washington in Tacoma. You can find the conference schedule here, and all the proceedings in the ACM Digital Library here. In past years, all the papers have been free for the first couple weeks after the conference, so grab them while they are outside the paywall.

Yesterday was the Doctoral Consortium, which had a significant Georgia Tech presence. My colleague Betsy DiSalvo was one of the discussants. Two of my PhD students were participants:

We have two research papers being presented at ICER this year. Miranda Parker and Kantwon Rogers will be presenting Students and Teachers Use An Online AP CS Principles EBook Differently: Teacher Behavior Consistent with Expert Learners (see paper here) which is from Miranda C. Parker, Kantwon Rogers, Barbara J. Ericson, and me. Miranda and Kantwon studied the ebooks that we've been creating for AP CSP teachers and students (see links here). They're asking a big question: "Can we develop one set of material for both high school teachers and students, or do they need different kinds of materials?" First, they showed that there was statistically significantly different behaviors between teachers and students (e.g. different number of interactions with different types of activities). Then, they tried to explain why there were differences.

We develop a model of teachers as expert learners (e.g., they know more knowledge so they can create more linkages, they know how to learn, they know better how to monitor their learning) and high school students as more novice learners. They dig into the log file data to find evidence consistent with that explanation. For example, students repeatedly try to solve Parsons problems long after they are likely to get it right and learn from it, while teachers move along when they get stuck. Students are more likely to run code and then run it again (with no edits in between) than teachers. At the end of the paper, they offer design suggestions based on this model for how we might develop learning materials designed explicitly for teachers vs. students.

Katie Cunningham will be presenting Using Tracing and Sketching to Solve Programming Problems: Replicating and Extending an Analysis of What Students Draw (see paper here) which is from Kathryn Cunningham, Sarah Blanchard, Barbara Ericson, and me. The big question here is: "Of what use is paper-and-pen based sketching/tracing for CS students?" Several years ago, the Leeds' Working Group (at ITiCSE 2004) did a multi-national study of how students solved complicated problems with iteration, and they collected the students' scrap paper. (You can find a copy of the paper here.) They found (not surprisingly) that students who traced code were far more likely to get the problems right. Barb was doing an experiment for her study of Parsons Problems, and gave scrap paper to students, which Katie and Sarah analyzed.

First, they replicate the Leeds' Working Group study. Those who trace do better on problems where they have to predict the behavior of the code. Already, it's a good result. But then, Katie and Sarah go further. For example, they find it's not always true. If a problem is pretty easy, those who trace are actually more likely to get it wrong, so the correlation goes the other way. And those who start to trace but then give up are even more likely to get it wrong than those who never traced at all.

They also start to ask a tantalizing question: Where did these tracing methods come from? A method is only useful if it gets used — what leads to use? Katie interviewed the two teachers of the class (each taught about half of the 100+ students in the study). Both teachers did tracing in class. Teacher A's method gets used by some students. Teacher B's method gets used by no students! Instead, some students use the method taught by the head Teaching Assistant. Why do some students pick up a tracing method, and why do they adopt the one that they do? Because it's easier to remember? Because it's more likely to lead to a right answer? Because they trust the person who taught it? More to explore on that one.

August 18, 2017 at 7:00 am 1 comment

Call for Nominations to Chair ICER 2019

SIGCSE is changing how they organize ICER.  Posted with Judy Sheard’s permission:

The ACM/SIGCSE International Computing Education Research conference (icer.acm.org) is the premier conference in the world focused on computer science education research, now in its 13th year. The leadership structure has recently been reorganized so that the the individual overseeing the selection of the program (the Program Chair) and the individual overseeing the running of the conference at a particular venue (the Site Chair) are to be held by different individuals.

We are currently seeking nominations for a Site Chair and a Program Chair for ICER 2019, to be held in North America.

Both appointments to Chair are for two years, called the “junior” and “senior” years, respectively. Site Chairs host the conference at their home institution during their senior year. Only one appointment for each role will be made each year, so that in any given year there is a junior and senior Site co-chair and a junior and senior Program co-chair. A nomination committee of the Program and Site chairs for the current year and the SIGCSE Board ICER liaison nominates the ICER Site chair and Program chair to start serving two years from the current year. The SIGCSE Board makes the appointments to both roles.

For both positions, the country of the home institution of each appointee will be rotated geographically by year as has been the tradition for ICER conference chairs, i.e.

  • Year 1: North America
  • Year 2: Europe
  • Year 3: North America
  • Year 4: Australasia

The criteria for appointees:

  • Program co-chair:
    1. Prior attendance at ICER
    2. Prior publication at ICER
    3. Past service on the ICER Program Committee
    4. Research excellence in Computing Education
    5. Collaborative and organizational skills sufficient to work on the Conference Committee and to share oversight of the program selection process.
  • Site chair:
    1. Prior attendance at ICER
    2. Collaborative and organizational skills sufficient to work on the Conference Committee and to oversee all of the local arrangements.
    3. Demonstrated interest in the computing education research community.

To nominate an individual, please include the individual’s CV and a cover letter explaining how the individual meets the criteria for the role. Self-nominations are welcomed. Please send nominations for the Site chair to the 2017 Site Chair, Donald Chinn (dchinn@uw.edu), and nominations for the Program chair to the 2017 Program Chair, Josh Tenenberg (jtenenbg@uw.edu). We also encourage informal expressions of interest to the individuals just mentioned.

March 13, 2017 at 7:00 am Leave a comment

Learning Curves, Given vs Generated Subgoal Labels, Replicating a US study in India, and Frames vs Text: More ICER 2016 Trip Reports

My Blog@CACM post for this month is a trip report on ICER 2016. I recommend Andy Ko’s excellent ICER 2016 trip report for another take on the conference. You can also see the Twitter live feed with hashtag #ICER2016.

I write in the Blog@CACM post about three papers (and reference two others), but I could easily write reports on a dozen more. The findings were that interesting and that well done. I’m going to give four more mini-summaries here, where the results are more confusing or surprising than those I included in the CACM Blog post.

This year was the first time we had a neck-and-neck race for the attendee-selected award, the “John Henry” award. The runner-up was Learning Curve Analysis for Programming: Which Concepts do Students Struggle With? by Kelly Rivers, Erik Harpstead, and Ken Koedinger. Tutoring systems can be used to track errors on knowledge concepts over multiple practice problems. Tutoring systems developers can show these lovely decreasing error curves as students get more practice, which clearly demonstrate learning. Kelly wanted to see if she could do that with open editing of code, not in a tutoring system. She tried to use AST graphs as a sense of programming “concepts,” and measure errors in use of the various constructs. It didn’t work, as Kelly explains in her paper. It was a nice example of an interesting and promising idea that didn’t pan out, but with careful explanation for the next try.

I mentioned in this blog previously that Briana Morrison and Lauren Margulieux had a replication study (see paper here), written with Adrienne Decker using participants from Adrienne’s institution. I hadn’t read the paper when I wrote that first blog post, and I was amazed by their results. Recall that they had this unexpected result where changing contexts for subgoal labeling worked better (i.e., led to better performance) for students than keeping students in the same context. The weird contextual-transfer problems that they’d seen previously went away in the second (follow-on) CS class — see below snap from their slides. The weird result was replicated in the first class at this new institution, so we know it’s not just one strange student population, and now we know that it’s a novice problem. That’s fascinating, but still doesn’t really explain why. Even more interesting was that when the context transfer issues go away, students did better when they were given subgoal labels than when they generated them. That’s not what happens in other fields. Why is CS different? It’s such an interesting trail that they’re exploring!

img_3874

Mike Hewner and Shitanshu Mishra replicated Mike’s dissertation study about how students choose CS as a major, but in Indian institutions rather than in US institutions: When Everyone Knows CS is the Best Major: Decisions about CS in an Indian context. The results that came out of the Grounded Theory analysis were quite different! Mike had found that US students use enjoyment as a proxy for ability — “If I like CS, I must be good at it, so I’ll major in that.” But Indian students already thought CS was the best major. The social pressures were completely different. So, Indian students chose CS — if they had no other plans. CS was the default behavior.

One of the more surprising results was from Thomas W. Price, Neil C.C. Brown, Dragan Lipovac, Tiffany Barnes, and Michael Kölling, Evaluation of a Frame-based Programming Editor. They asked a group of middle school students in a short laboratory study (not the most optimal choice, but an acceptable starting place) to program in Java or in Stride, the new frame-based language and editing environment from the BlueJ/Greenfoot team.  They found no statistically significant differences between the two different languages, in terms of number of objectives completed, student frustration/satisfaction, or amount of time spent on the tasks. Yes, Java students got more syntax errors, but it didn’t seem to have a significant impact on performance or satisfaction. I found that totally unexpected. This is a result that cries out for more exploration and explanation.

There’s a lot more I could say, from Colleen Lewis’s terrific ideas to reduce the impact of CS stereotypes to a promising new method of expert heuristic evaluation of cognitive load.  I recommend reviewing the papers while they’re still free to download.

September 16, 2016 at 7:07 am 4 comments

Older Posts


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 5,317 other followers

Feeds

Recent Posts

Blog Stats

  • 1,554,196 hits
September 2018
M T W T F S S
« Aug    
 12
3456789
10111213141516
17181920212223
24252627282930

CS Teaching Tips