How do we get more, without sacrificing good, while measuring it?

January 27, 2010 at 7:44 am 10 comments

“Without an unrelenting focus on quality—on defining and measuring and ensuring the learning outcomes of students—any effort to increase college-completion rates would be a hollow effort indeed.” So proclaimed Jamie P. Merisotis, president and chief executive of the Lumina Foundation for Education, during the opening plenary of the annual conference of the Association of American Colleges and Universities, held here last week.That statement reflected the tone of the entire conference. The nearly 1,900 presidents, provosts, and faculty members who gathered here generally said that they welcome the White House’s efforts to increase the proportion of Americans who earn college degrees. But they do not want to see those degrees watered down in the process. If colleges are going to provide high-quality educations to millions of additional students, they said, the institutions will need to develop measures of student learning than can assure parents, employers, and taxpayers that no one’s time and money are being wasted.

via Educators Mull How to Motivate Professors to Improve Teaching – Curriculum – The Chronicle of Higher Education.

The tension between wanting to have more students to graduate (to improve competitiveness, for the sake of the students’ future economic success, to reduce inequities) and wanting to maintain standards (what’s the point of more students graduating, if those graduations aren’t worth anything) leads to a need to teach better — to produce more while not decreasing quality.  That’s really hard.  The blurb quote above points to an obvious need to be able to measure the learning.

Thus, we hit a roadblock in Computing. We don’t know how to measure learning in Computing.  Sure, we can ask about learning in any given course, but if you listen to the complaints of teachers (“They learned Scheme — why do they do so poorly in Java?” and “You saw this algorithm in your CS2. Why can’t you do it now?” and “These students know Java and C.  Why can’t they pick up C++ on their own?”) and industry (“We have to hire people who know language X, because it takes too much time to learn X or to retrain our people in X”), you can see that we’re expecting a deeper kind of learning than just a course-at-a-time.  We don’t know how to measure this kind of transfer in Computing. We have no language-independent measures of CS learning.

If we had them, I’d bet we’d be disappointed with the results.  That’s a really high bar that we set for ourselves.

Entry filed under: Uncategorized. Tags: , .

A new medium to engage students: Electronic pop-up books “Who do I have to fight now?”: CS != Apps++

10 Comments Add your own

  • 1. Erik Engbrecht  |  January 27, 2010 at 9:35 am

    Has anyone tried making up a simple language for a test? Implementing a simple interpreted language isn’t that hard. It would have to change frequently, or there would have to be a lot of them. So there would be a lot of work Given a stable abstract syntax you might be able to randomly mutate the concrete syntax within certain constraints.

    Anyway, the idea would be to give students a test in a language that they have never seen before and have no way of studying, but that is filled with concepts from languages that they have seen. That way it measures their conceptual understanding rather than the degree to which they’ve internalized syntax and memorized libraries.

    My observation is that there are people who can pick up a new language sufficiently well in a few days to actually produce something, and there are those who will struggle for weeks or even months.

    • 2. Mark Guzdial  |  January 27, 2010 at 1:21 pm

      In fact, Erik, this is what my student, Allison Tew, is exploring in her thesis. She’s going to try to measure CS1 knowledge by using a pseudocode. But first, you have to show that conceptions AND misconceptions that students make in their “native” (first) language appear the same in the pseudocode. If the errors mutate, then it’s not a good measure. So far, the results are good.

      • 3. Erik Engbrecht  |  January 27, 2010 at 2:16 pm

        Good to hear! When she has some results that can be made public please share.

  • 4. Jason  |  January 27, 2010 at 11:53 am

    If you can understand some languages, you can easily pick up a lot on other languages. The trick is learning a few languages such as C++ in the beginning and branching out from there… you’d be surprised how easy it is to learn other computer languages once you’ve gotten the hang of a few already.

    • 5. Mark Guzdial  |  January 27, 2010 at 1:19 pm

      Actually, Jason, the research results most often show the opposite. Ben Shneiderman documented back in the 80’s that learning a second language was often harder than the first. Anderson and Singley showed how we can predict what aspects of programming will transfer from one language to another, and which do not.

      • 6. Michelle Craig  |  January 27, 2010 at 3:31 pm

        Can you give a specific reference to where Ben Shneiderman published about this? I’m interested in reading more but Shneiderman published an impressive amount during the 80’s and it isn’t obvious to me which of his many papers are relevant here.
        Thank you.

  • 7. Mark Guzdial  |  January 27, 2010 at 3:40 pm

    Google “Shneiderman spiral.” I don’t remember the exact paper, but remember that it was in his series where he talked about his spiral method of learning a new language.

  • 8. Ben Chun  |  January 28, 2010 at 1:19 am

    Just to be fair to CS, we don’t ask these kinds of questions in other disciplines either. There are no topic-independent measures of mathematics learning, for example. Differential equations over here, linear algebra over there. Some do well in one and poorly in another and disappoint teachers who were hoping for deeper than course-at-a-time knowledge.

    Yes, programming languages have some common abstractions that arise from the underlying hardware and they can be seen as different ways to represent or express a general idea of computation. But unless we’re teaching the generalization, why should we think they’re going to learn it or that we can measure it? We don’t expect mathematics students to come up with the epsilon-delta proofs on their own, even having seen calculus. We show that stuff to them. Then we expect them to be able to use and apply that understanding, and then we can measure their learning.

  • 9. Mark Guzdial  |  January 28, 2010 at 9:22 am

    Ben, I think the differences between different math topics is different than between different (programming) languages. You can use different languages for CS1, but still hope to achieve the same learning. Mathematics certainly has achieved this level of measurement. The TIMSS studies are a great example, in that they are reliable and valid, yet cover mathematics knowledge across multiple cultures and (human) languages. There are lots of other reliable and valid tests of mathematical knowledge that allow for comparing different approaches to teaching (say) fractions or algebra or calculus. However, we have no way of comparing teaching Alice or Scratch or Etoys, or Python or Java or MATLAB.

  • 10. Ben Chun  |  January 28, 2010 at 10:18 pm

    You’re right — the analogy breaks down. Maybe I should try a different one? We don’t expect to find a single way to test both students who are learning first-year French and first-year German for their general “language year one” skills. I guess what I want to tease out here is exactly IS the common learning we hope to achieve in CS1, regardless of the choice of Python or Java? I agree with you that it would be useful to have some measure of this but we have to define it first. One approach is to take common tasks and have the students write a solution in whatever language they know. Stuff like: “Find the largest of these numbers,” or, “Put these names in alphabetical order.” Do you think we could make a fair scoring matrix for answers in multiple languages?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,052 other followers


Recent Posts

Blog Stats

  • 2,030,789 hits
January 2010

CS Teaching Tips

%d bloggers like this: