Instruction makes student attitudes on computational modeling worse: Caballero thesis part 3

August 2, 2011 at 10:15 am 31 comments

Note: Danny’s whole thesis is now available on-line.

In Danny Caballero’s first chapter, he makes this claim:

Introductory physics courses can shape how students think about science, how they believe science is done and, perhaps most importantly, can influence if they continue to pursue science or engineering in the future. Students’ attitudes toward learning physics, their beliefs about what it means to learn physics and their sentiments about the connection between physics to the “real world” can play a strong role in their performance in introductory physics courses. This performance can affect their decision to continue studying science or engineering.

Danny is arguing that physics plays a key role in retaining student interest in science and engineering. Computing plays a similar role. Computing is the new workbench for science and engineering, where the most innovative and ground-breaking work is going to happen.  Danny realized that students’ attitudes about computational modeling are important, in terms of (a) student performance and learning in physics and (from above) all of science and engineering learning, and (b) influencing student decisions to continue in science and engineering. What we worry about are students facing programming and saying, “Real scientists and engineers do this?  I hate this!  Time to switch to a management degree!”.

There are validated instruments for measuring student attitudes towards computer science and physics, but not for measuring student attitudes towards computational modeling.  So, Danny built one (which is included in an appendix to his thesis), that he calls “COMPASS” for “Computational Modeling in Physics Attitudinal Student Survey.”  He validated it with experts and with correlations with similar instruments for physics attitudes.  It contains phrases for students to agree-or-disagree with, like:

  • I find that I can use a computer model that I’ve written to solve a related problem.
  • Computer models have little relation to the real world.
  • It is important for me to understand how to express physics concepts in a computer model.
  • To learn how to solve problems with a computer, I only need to see and to memorize examples that are solved using a computer.

Danny gave this instrument to a bunch of experts in computational modeling who generally had similar answers to all the statements, e.g., strongly agreed/strongly disagreed in all the same places. Then he measured student answers in terms of percentage of answers that were “favorable” (agreed with experts) on computational modeling, and the percentage of answers that were “unfavorable” (were different than the experts) on computational modeling.  A student’s answers to COMPASS is then a pair of %favorable and %unfavorable.  He gave this to several cohorts at Georgia Tech and at North Carolina State University, in week 2 (just as the semester started) and in week 15 (as the semester was wrapping up).  The direction of change from week 2 to week 15 was the same for every cohort:

The black square in each indicates the mean.  The answers after instruction shifted to more unfavorable attitudes toward computational modeling.  Instruction led to students being more negative about computational modeling.  Danny did an analysis of where the big shifts were in these answers.  In particular, students after instruction had less personal interest in computational modeling, agreed less with the importance of sense-making (the third bullet above), and agreed more with the importance of rote memorization (last bullet above).

Danny chopped up these data in lots of ways.  Does student grade influence the results?  Gender?  Year in school?  The only thing that really mattered was major.  Computing majors (thankfully!) did recognize more value for computational modeling after instruction.

These results are disappointing.  Teaching students about computational modeling makes them like it less?  Makes them see less value in it?  Across multiple cohorts?!? But from a research perspective, this is an important result.  We can’t fix a problem that we don’t know is there.  Danny has not only identified a problem.  He’s given us a tool to investigate it.

The value of COMPASS is in having a yardstick.  We can use it to see how we can influence these attitudes.  Danny wrote it so that “physics” could be swapped out for “biology” or “chemistry” easily, to measure attitudes towards computational modeling in those disciplines, too.  I’ll bet that this is a useful starting place for many folks interested in measuring computational thinking, too.

“So, Guzdial, you spent a lot of time on these three blog posts?  Why?!?”

I strongly believe that the future of computing education lies in teaching more than just those who are going to be software developers.  Scaffidi, Shaw, and Myers estimate that there are four professionals who program but who are not software developers for every software developer in the US.  We computing educators need to understand how people are coming to computing as a tool for thinking, not just a material for engineering.  We need to figure out how to teach these students, what tools to provide them, and how to measure their attitudes and learning.

Danny’s thesis is important in pursuing this goal.  Each of these three studies is important for computing education research, as well as for physics education research.  Danny has shown that physics students’ learning is different with computational modeling, where their challenges are in computational modeling in Python, and here, how their attitudes about computational modeling are changing. Danny has done a terrific job describing the issues of a non-CS programming community (physics learners) in learning to use computational modeling.

This is an important area of research, not just for computer science, but for STEM more generally.  Computing is critical to all of STEM.  We need to produce STEM graduates who can model use computers and who have positive attitudes about computational modeling.

The challenge for computing education researchers is that Danny’s thesis shows us is we don’t know how to do that yet.  Our tools are wrong (e.g., the VPython errors getting in the way), and our instructional practices are wrong (e.g., such that students are more negative about computational modeling after instruction than before).  We have a long way to go before we can teach all STEM students about how to use computing in a powerful way for thinking.

We need to figure it out.  Computational modeling is critical for success in STEM today.  We will only figure it out by keep trying.  We have to use curricula like Matter and Interactions. We have to figure out the pedagogy.  We have to create new learning tools.  The work to be done is not just for physics educators, but for computer scientists, too.

Finally, I wrote up these blog posts because I don’t think we’ll see work like this in any CS Ed conferences in the near term.  Danny just got a job in Physics at U. Colorado-Boulder.  He’s a physicist.  Why should he try to publish in the SIGCSE Symposium or ICER?  How would that help his tenure case? I wonder if his work could get in.  His results don’t tell us anything about CS1 or helping CS majors become better software developers.  Will reviewers recognize that computational modeling for STEM learning is important for CS Ed, too?  I hope so, and I hope we see work like this in CS Ed forums.  In the meantime, it’s important to find this kind of computing education work in non-CSEd communities and connect to it.  

Entry filed under: Uncategorized. Tags: , , , .

What students get wrong when building computational physics models in Python: Cabellero thesis part 2 H-indices and how academic publishing has changed: Feynman and Einstein just aren’t that impressive anymore

31 Comments Add your own

  • 1. katrinbecker  |  August 2, 2011 at 11:09 am

    Given the experiences I’ve had with what the majority of CS professors/instructors teach, my first reaction is that a chief reason this is happening is that the teachers are driving students away with what and how they are teaching.

    I know there are exceptions (I was one) but most teachers (and most texts) are geared towards the selection of people just like the teachers, and these are, for the most part, *not* the people we want to be attracting.

    I’d love to see some in-depth analysis of people who quit CS degrees before they finish. I’ve had far too many highly talented students tell me they ended up NOT finishing their CS degrees. After taking my 1st year class, they were excited about the possibilities and inspired to discover more, but then after 2-3 years of typical CS drudgery they just couldn’t take it any more. Typical CS drudgery includes: drier than toast theory (with no visible practical application); software engineering that throws people into groups without teaching them how to work in groups, and which seems primarily preoccupied with administration and reports as opposed to solving problems; professors who themselves can no longer write any code (if they ever could); faculty who are focused on that next publication rather than on creating interesting and authentic work for their students.

  • 3. Robert Talbert  |  August 2, 2011 at 12:54 pm

    This has been an excellent series of posts. Thanks for bringing it to us.

    I can’t recall if you mentioned what “instruction” looked like for these classes. Was it all lecture/recitation? Do you think using peer instruction (which has had positive impact on student attitudes and retention) would help in students’ attitudes toward computational modeling?

    Also, I wonder if using a tool that is demonstrably an industry standard, like MATLAB, might help student attitudes. Even if the students don’t like the modeling you could point to the ways the tool is used.

    • 4. Mark Guzdial  |  August 2, 2011 at 1:06 pm

      Thanks, Robert! Physics at Georgia Tech makes heavy use of peer instruction, and there is a mix of lecture plus recitation and discussion. I do think that there are additional pedagogical changes that might be implemented, drawing from CS Ed, to make computational modeling in physics more effective — such as live coding.

      One of the natural places to explore synergy at Georgia Tech is between CS and Physics. All students at Georgia Tech must take CS. We teach three different classes, two in Python and one in MATLAB. Could we introduce VPython in those classes, and make the explicit mapping between the CS topics and physics topics there? What if we used MATLAB instead of VPython, as you suggest? Great directions to explore!

  • 5. katrinbecker  |  August 2, 2011 at 1:21 pm

    Thanks for the link to the paper. I have to wonder if the boys tend not to comment on feelings of belonging and friendliness because doing so might be perceived as ‘whimpy’?

    Most of the reasons for leaving could easily be attributed to what and how students are taught as well as the class or departmental culture.

    CS is not alone here – I think most established disciplines tend to draw people who are like themselves and to dispossess those who aren’t. It takes a conscious effort to counteract this – it may well be a totally natural human tribal thing. Also, there seems to be an unwritten academic golden rule that comes into force when someone becomes an instructor: “I shall do to you what was done to me”. Those who choose to do things differently are the exception. Those who had mentors with uncommon talent for teaching are rarer still.

    After all – the notion of average applies to faculty just like everything else – and nearly half are always going to be below average.

    I think one thing that makes CS different is that the nature of the discipline has changed quite radically in the last decade or two, but the faculty, programs, and courses have not. We need to attract a different kind of person from those who are currently running things. That isn’t going to happen so long as those who are currently running things are allowed to teach what and how they’ve always taught.

    • 6. BKM  |  August 3, 2011 at 3:11 pm

      You are making a lot of assertions without much in the way of specifics. In what way has the field changed that is not being reflected in current teaching? I see almost the opposite – lots of faculty frantically chasing the latest fads – pair programming! Android! social media! cyberforensics! – often without a lot of grounding in the particular area. At SIGCSE and the regional conferences, I see paper after paper describing these courses. Are you seeing differently?

      You also don’t like faculty who can’t code and theory courses. In your opinion, should the major be more applied? Should we be hiring more people with software experience rather than theorists?

      There was an excellent article in the Chronicle this week suggesting that trying to increase STEM majors may be misguided. They noted that only half of STEM majors end up being hired in STEM jobs, and suggest that a fundamental problem is the mismatch between what employers expect and what we are able to teach in 4 years.

      • 7. Mark Guzdial  |  August 3, 2011 at 5:28 pm

        That’s an interesting comment piece. I’m trying to find the study that they refer to. Eric Roberts argues that those STEM hiring data don’t apply to CS (slides 5-6). Most CS graduates are working in CS (71%), yet not all people working in CS have CS degrees (only about 39%). That suggests that we are not over-producing, but under-producing.

        Your comment about SIGCSE is interesting. I did some searching to try to get a sense of how indicative SIGCSE is of CS education overall. According to the CRA, there were 4,758 tenure track and 665 teaching track faculty in research-oriented (PhD-granting) CS programs in the United States. TAURUS (Taulbee for the rest of us) says that there are 1266 CS departments in 4 year institutions in the US (~2700, if we include CE and IS). How large are those departments? I took a stab: Let’s say an average of 3 faculty, for 3,798 faculty in CS departments in 4 year schools. Grand total: 9,221 faculty teaching CS in the US. SIGCSE has never had more than 1200 attendees, and CCSC conferences are much smaller (with significant overlap). That suggests that only about 13% of faculty have ever attended SIGCSE.

        • 8. BKM  |  August 4, 2011 at 9:50 am

          I think the main reason that only 39% of people working in CS have CS degrees is because industry does not put much value on CS degrees. I know that at the companies where I worked, no one cared what the degree was. They cared if people could pass the official programming test. Every company I have ever dealt with gives a programming test because they don’t trust that people actually know what they claim to know on their resumes.

          Speaking of programming tests, I wonder if anyone has done a study on the contents of these things. My students are confronting them. A lot of them are online, so the students often do screenshots and then pass them around. As a result, I have seen some of them, and they look pretty much like what I remember from my industry days. They test on obscure corners of the programming language and APIs rather than anything fundamental. To be fair, a lot of the big financial companies also ask questions about data structures and design patterns. However, my take on these tests is that they tend to stress the obscure, and that they test on material that is more advanced than what a typical graduating senior knows.I think it would be very informative for CS faculty if we had a better idea of the content of these tests, which are ubiquitous here in the NYC region, at least.

      • 9. katrinbecker  |  August 3, 2011 at 6:11 pm

        In many cases, though the language or app may have changed, the rest of the course hasn’t – same examples, same assignments, same dry lectures covering the same topics in the same order (merely adjusted to match the languade or app, but not redesigned to fit the class demographics OR the best features of the tool).

        When I taught Alice a few years ago I got into trouble by the course coordinator for arguing that we should not cover variables in class until much later in the course. You can do a lot in Alice without ever knowing that variables exist, so it makes no sense to cover them at the same point in the course as you would if you were teaching Java, for example. “Just in case” teaching (which is what is normally done in CS) does not foster interest or excitement the same way that “just in time” teaching does.

        It might be worthwhile to ask how many ‘good’ instructors does a student need to ensure success?

        I used to go to the nearest CCSC conference every year and most of the people there were doing interesting things with their students and they were genuinely interested in improving both their own teraching and the learning experience of their students. In the 10 or so years I went regularly, almost NONE of the attendees came from research institutions. My guess is (yup, guilty of unsupported assertions again) that Mark’s estimate for the total number of CS faculty is low, and it may not include Canada, in which case we could easily add another 1-2000 faculty without adding any additional attendees to SIGSCE or CCSC conferences. We’re probably looking at less than 1 in 10 faculty who attend conferences concerned with teaching CS.

        Imagine being an undergrad, and only 1 in 10 of your professors is doing anything interesting, teaching-wise. That’s one professor PER YEAR – hardly enough to inspire a student to stay in a program. The truth of the matter is that that 1 in 10 professor is more likely than not at an institution that actually does care about teaching (as opposed to just saying they care – which they all do). That means that that particular institution may well have several of those 1-in-10 profs. That means that somewhere else, another group of students are going without.

        • 10. gasstationwithoutpumps  |  August 4, 2011 at 1:49 am

          Why do you equate going to a conference with being interested in teaching? Going to a CCSC or SGSCE conference means you have money to burn or are doing researching into education, and has precious little to do with interest in teaching.

          • 11. BKM  |  August 4, 2011 at 9:56 am

            That may be true in large research oriented departments. But in smaller departments, the local CCSC conference may be the only one that people are going to, so attendance tends to be a marker for faculty who are keeping up with trends in CS.

        • 12. Elizabeth Patitsas  |  August 4, 2011 at 11:20 am

          1 in 10 CS profs going out of their way to a teaching conference actually sounds great to me — I highly doubt that any of our related fields (engineering, physics, math, etc) have an equivalent stat anywhere near that good.

      • 14. katrinbecker  |  August 3, 2011 at 6:31 pm

        “You also don’t like faculty who can’t code and theory courses. In your opinion, should the major be more applied? Should we be hiring more people with software experience rather than theorists?”

        A very wise man once said, “CS may be more than programming, but it is not less than programming.”

        This may seem contradictory, but I strongly believe that we should not be catering to industry when we design university programs (certification programs and 2-year colleges are a different story), but at the same time EVERY student has the right to ask these 2 questions of his/her teacher: 1. Why am I doing this? and 2. What is it good for?

        If there’s no good answer, then maybe we should rethink that topic’s place in the program. Now that’s not to say that everything taught must have an immediately obvious, practical application, but the teacher must be able (in all cases) to explain where this fits in and how it can be applied to a) something currently practical; b) something that may turn out to be in the future; c) something that furthers our knowledge of how the world works.

        • 15. gasstationwithoutpumps  |  August 4, 2011 at 1:45 am

          I disagree. It is perfectly acceptable to do something because it is fun, with no utility. I guess my initial desire to be a pure mathematician shows there, though I have become much more applied over the years, and everything I teach now is immediately applicable. (I’m now a bioinformatician.)

          • 16. katrinbecker  |  August 5, 2011 at 11:30 am

            I agree that sometimes it is perfectly fine to do something that’s just fun – these days my specialty is digital games – for entertainment games, fun is the whole point (aside from making money for the developers …).

            However, if it is part of the official course curriculum, and the instructor is the only one in the room who thinks it’s fun (or easy), then something is wrong.

  • […] more from the original source: Instruction makes student attitudes on … – Computing Education Blog This entry was posted in Uncategorized and tagged computing-education, physics-education, […]

  • 18. Elizabeth Patitsas  |  August 2, 2011 at 4:58 pm

    You say “Danny gave this instrument to a bunch of experts in computational modeling” — but did the physics instructors teaching the cohorts in question also get surveyed? I wonder if perhaps the physics instructors teaching those cohorts were presenting the models in a negative fashion?

    • 19. Mark Guzdial  |  August 2, 2011 at 5:17 pm

      Yes, they were surveyed, too.

  • […] Instruction makes student attitudes on computational modeling worse: Caballero thesis part 3 « Comp…. […]

  • 21. Algebra++  |  August 3, 2011 at 1:40 am

    Computational physics was my grad research. Later, while teaching in the early 90s, a phys-ed workshop showed us a book called ‘Spreadsheet Physics” by Misner and Cooney. IMO, their applications mostly missed, but I’ve worked spreadsheets into ALL my classes ever since. The spreadsheet is crude in a certain sense, but has many of the CS principles we’re trying to teach, and is much more accessible and intuitive. The loops that you might put in python or Matlab, aren’t so present in spreadsheet modeling, but then the projectile isn’t looping either.

    • 22. gasstationwithoutpumps  |  August 4, 2011 at 1:47 am

      I find spreadsheet programming very UNintuitive, and damned hard for anyone else to read or debug. Almost everyone I know who uses spreadsheet programs treats them as ritual magic, just copying them without understanding. They may be fine for some purposes, but teaching computational thinking does not seem to be one of them—at least not to me.

      • 23. Rob St. Amant  |  August 4, 2011 at 7:35 am

        Alan Kay shows how it can be done in a Scientific American article from 1984:

        Click to access tr1984001_comp_soft.pdf

      • 24. Algebra++  |  August 4, 2011 at 7:54 am

        I use them starting with my Intro Alg classes. We’re into 2D arrays, in less than 10 min. My basic physics classes do 2D projectile problems (e.g. angles for a target 1800m up, and 2500m downrange. The calc-based physics students work in velocity-dependent air resistance and/or variable gravity.

      • 25. BKM  |  August 4, 2011 at 9:53 am

        I also hate spreadsheets. The interface is just horrendous. However, they are heavily used in industry. The worst thing is that they are used as a substitute for written documents. I have worked with business analysts who write system requirements as Excel spreadsheets so they won’t have to write complete, coherent English sentences.

        • 26. Algebra++  |  August 5, 2011 at 5:01 am

          “…The interface is just horrendous. However, they are heavily used in industry “ See, that’s why to me, it’s the lingua franca alluded to in Mark’s “Post-Basic” blog. The spreadsheet is everywhere, including my smart phone. I can walk into any company, any school and count on access to a spreadsheet. I find the spreadsheet very natural for accomplishing (or at least prototyping) most number-crunching tasks. And – much like Latin – it’s a ‘dead language’. It doesn’t really evolve.

  • […] third blog post in the series, has even more troubling news for us. Caballero developed an instrument to assess the student’s […]

  • […] thesis examining the influence of using computational/programming approaches to teaching physics. This post talks about the finding in the thesis that students attitudes towards computational modelling […]

  • 29. Twitted by ggatin  |  August 13, 2011 at 10:20 pm

    […] This post was Twitted by ggatin […]

  • […] thesis examining the influence of using computational/programming approaches to teaching physics. This post talks about the finding in the thesis that students attitudes towards computational modelling […]

  • […] Danny Caballero has started a blog at Boulder, and a recent post describes a survey he took of upper-division undergraduates in Physics.  They definitely grokked that they need computing in their education. 59 students responded to the survey. Most students were juniors and seniors. That’s because we targeted upper-division courses like Classical Mechanics, E&M and Stat. Mech. […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 10,185 other subscribers


Recent Posts

Blog Stats

  • 2,060,430 hits
August 2011

CS Teaching Tips

%d bloggers like this: