Posts tagged ‘computing education’

A job is a strange outcome measure: Udacity drops money-back guarantee on finding a job

Udacity has dropped a money-back guarantee that they were offering to students in some of their Nanodegree programs. The guarantee (with stipulations and caveats) was that students would find a job after getting the nanodegree, or they would get their money back.

An article in Inside Higher Ed (quoted below and linked here) describes some of the tensions. Other for-profit coding schools offer similar or better guarantees, but others do not. Ryan Craig, quoted below, suggests that Udacity might not have been hitting its targets for job placements. Does that mean that Udacity was doing something wrong?

A job is such a strange outcome measure for any kind of educational program.  I know some techniques for evaluating someone’s knowledge of programming, and I know how to create educational opportunities that might lead to successful evaluation.  There are factors like student attitude and motivation and whether students engage in deliberate practice that are not entirely within my control.  Even then, I’d be willing to say, “I can design a program where the majority of students will achieve this level of proficiency in coding.”  But a job?  Where I can’t control how the students interview, or where they apply, or what the companies are looking for (if they’re looking at all)?

A job is not a well-defined outcome measure for an educational intervention. That may be what the students are seeking, but they are being unrealistic if they think that any school can guarantee them that.

Ryan Craig, managing director of investment company University Ventures, noted that none of the major employers associated with Udacity will publicly commit to hire or interview nanodegree candidates. Craig pointed to a 2017 report from VentureBeat, which stated that of around 10,000 students who had earned nanodegrees since 2014, around 1,000 had found jobs as a result. “A placement rate of around 10 percent should spell the demise of any last-mile training program,” said Craig.

Craig said the effectiveness of Udacity’s job guarantee was likely very limited for students. “Money-back guarantees don’t address the real guarantee that students are seeking: a job,” said Craig.

Daniel Friedman, co-founder of coding school Thinkful, wrote in January 2016 that Udacity’s guarantee was vaguer and weaker than the guarantees offered by his own company and others such as Bloc and Flatiron School. Such guarantees are common at coding schools, though Friedman noted that some schools have had to drop guarantees because they conflicted with state regulations.

April 13, 2018 at 7:00 am 3 comments

High School CS Teacher’s Experience like University CS Teacher’s: “Code Shock”

Jeff Yearout has been teaching for over 25 years, and is just in his second year of teaching CS.  His concerns in his blog echo many of the same ones that I hear from higher-education CS teachers, e.g., dealing with the wide variance of students, and getting all students to engage around code (pseudo or otherwise).

I think one of the hardest things to manage in designing a curriculum is how to dial the difficulty up at a proper pace for the “center mass” of the class skill level. And in this new curriculum from PLTW this particular unit starts out manageable, but suddenly shoots up rapidly, thus the “code shock” mentioned above. I also have the challenge of having a lot of kids in class who simply don’t want to interact in class when, for instance, I’m working through pseudocode on the board.

From “Teaching CS is Hard

April 9, 2018 at 7:00 am 4 comments

Teaching to develop a mental model of program behavior: How do students learn the notional machine

“To understand a program you must become both the machine and the program” – Perlis 1982, cited in Sorva 2013

I’ve been thinking for a few years now about an open research question in computing education. How do students come to understand how programs work? Put in a more technical way, How do students develop their mental model of the language’s notional machine?

I have been thinking about this question in terms of Ashok Goel’s Structure-Behavior-Function (SBF) model of how people think about systems.

  • Structure is the parts of the system — for us in CS, think about the code.
  • Function is what the system does — for us in CS, the requirements or the description of what the program is supposed to do.
  • Behavior is how the structural elements interact to achieve the function. It’s the understanding of the semantic model of the programming language (the notional machine) plus how that plays out in a specific program.

There are studies of students learning notional machines (e.g., Raymond Lister and Juha Sorva are some of the top researchers here). What I don’t know is how it develops and how to help it develop. Lister tells us the stages of development (e.g., of tracing skill). Sorva tells us about theories of how to teach the notional machine, but with little evidence. We have models for how people learn to read and write code (e.g., Elliot Soloway’s plans). But we not have a cognitive model for how they develop a mental model of how the code works.

A Pedagogical Problem That I Faced

I’m teaching Media Computation (n=234) this semester, and students had a disappointing performance on two programming problems on a recent quiz. (We have a 30 minute quiz every other week.) They didn’t really bomb, but an average of 82% on one programming problem (Problem #3 below) and 76% on the second (Problem #4) was lower than I was hoping for. Those are all mostly due to partial credit — only 25 of my 234 students got full credit on Problem #4. Worse yet, we had a “simple” matching problem where we offered four pictures and four programs — which program generated which picture? More than half the students got at least two wrong. The score on the matching problem was 72%, even lower than the programming task problems. My conclusion is that my students can’t yet read code and understand it.

How do I teach my students to understand code?

With my researcher hat on, I don’t have a solid answer. With my teacher hat on, I have to do something. So, I drew on what I know from the research to come up with a best guess solution.

I decided to drop the next two lecture topics from the schedule, to instead re-focus on manipulation of pictures. I know from the learning sciences literature that it’s much better to go deeper than broader. Teaching for mastery is more important than teaching for coverage. Things that students know well are more likely to persist and transfer than things that students are merely familiar with.

I decided to do a live-coded session revisiting the quiz. I had graded a bunch of the programming problems on the quiz. I saw several different strategies for solving those two problems. I had a unique teachable moment here — every student had attempted these two problems. They were primed for the right answer. I decided to solve the problems in a live-coding session (starting from a blank editor, and talking aloud as I wrote the code) in each of the ways that I saw students doing it — three ways for the first problem, four ways for the second problem. While I wrote, I drew pictures to describe the behavior, drawing from Sorva’s visualization approach and the SILC emphasis on sketching rather than diagrams. After writing each program, I tested it on a picture. Along the way, I answered questions and wrote additional examples based on those questions.

This idea is based on Marton’s Variation Theory. You have to vary critical aspects of examples for students to figure out the differences. Janet Kolodner talks about a similar idea when she emphasizes contrasting cases for learning informed by case-based reasoning.  In SBF terms, I was keeping the Function constant, but varying the Structure and Behavior. In Goal-Plan-Code terms, I was achieving the same Goal, but varying the underlying Plan and Code.

Could an exploration of these variations/contrasts help students see how the code changes related to behavior changes?  I don’t actually know how to evaluate the result as a researcher, but as a teacher, I got good response from students.  I’m looking forward to seeing how they do on similar problems on future quizzes.

The rest of this blog post is a static replay of the lecture. I’ll show you the slides I showed, the sketches I made, and the code I wrote (while talking aloud).

Problem clearTopHalf

Solution #1: Iterate through all pixels

def clearTopHalf1(pic):
  h = getHeight(pic)
  for pixel in getPixels(pic):
     y = getY(pixel)
     if y < h/2:

Solution #2: Iterate through half of the pixel indices

def clearTopHalf2(pic):
  all = getPixels(pic)
  for index in range(0,len(all)/2):
    pixel = all[index]

Solution #3: Iterate through all x and y positions in the top half

def clearTopHalf3(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  for y in range(0,h/2):
   for x in range(0,w):
    pixel = getPixel(pic,x,y)

A student asked, “Could we do x first and then y?” Sure!

def clearTopHalf3b(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  for x in range(0,w):
    for y in range(0,h/2):
       pixel = getPixel(pic,x,y)

Pause for Reflection and Discussion

At this point, I asked students to turn to the person next to them and ask, “Which one do you prefer? Which makes the most sense to you?”

I always encourage students to discuss during peer instruction questions. I have never had such an explosion of noise as I did with this invitation. From wandering around the room, what I heard students discussing was, “This is what I did, and this is what I got wrong.”

When we had the whole class discussion, the first and third approaches (all pixels or coordinates) were the preferences. Students who spoke up disliked the index approach — it felt “like there’s too much indirection” (said one student).

Problem copyThirdDown

Solution #1: Iterating through all pixels

def copyAThird1(pic):
  h = getHeight(pic)
  for pixel in getPixels(pic):
    x = getX(pixel)
    y = getY(pixel)
    if y < h/3:

Solution #2: Iterate through first 1/3 of pixels

def copyAThird2(pic):
  all = getPixels(pic)
  h = getHeight(pic)
  for index in range(0,len(all)/3):
    pixel = all[index]
    color = getColor(pixel)
    x = getX(pixel)
    y = getY(pixel)
    targetPixel = getPixel(pic,x,y+(2*h/3))

Solution #3: Iterate through top 1/3 of picture by coordinates

def copyAThird3(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  for x in range(0,w):
   for y in range(0,h/3):
    pixel = getPixel(pic,x,y)
    color = getColor(pixel)
    targetPixel = getPixel(pic,x,y+(2*h/3))

At this point, someone said that they did it by subtracting y from the height. I showed them that this approach mirrors. This is the first incorrect solution that I demonstrated.

def copyAThird3b(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  for x in range(0,w):
    for y in range(0,h/3):
      pixel = getPixel(pic,x,y)
      color = getColor(pixel)
      # Mirrors instead of copies
      targetPixel = getPixel(pic,x,h-y-1)

Solution #4: Iterating through the bottom 1/3 of the picture by x and y coordinates

This was an unusual approach that I saw a few students try: They used nested loops to iterate through the bottom 2/3 of pixel coordinates, and then compute the top 1/3 to copy down. They iterated through the target and computed the source.

def copyAThird4(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  for x in range(0,w):
    for y in range(2*h/3,h):

After I wrote that program, someone asked, “Couldn’t you make an empty picture and copy the top third into the new picture at the top and bottom?” With her guidance, I modified the program above to create a new version, which does exactly as she describes, leaving the middle third blank. So, this was the second incorrect version I wrote to respond to student queries.

def copyAThirdEmpty(pic):
  h = getHeight(pic)
  w = getWidth(pic)
  canvas = makeEmptyPicture(w,h)
  for x in range(0,w):
    for y in range(0,h/3):
      pixel = getPixel(pic,x,y)
      color = getColor(pixel)
      targetPixel = getPixel(canvas,x,y)
      targetPixel = getPixel(canvas,x,y+(2*h/3))

When I asked students a second time which version made the most sense to them, there was a bigger split. Indexing the array continued to be the least preferred one, but students liked both versions with nested loops, and many still preferred the first version.

April 6, 2018 at 7:00 am 14 comments

States requiring CS for all students may be making a mistake: Responding to unfunded mandates

As of this writing, New Jersey and Wyoming are the latest states to require CS for all their students (as described in this article) or to be offered in all their schools (as described in this post and this news article), respectively.  Wyoming has a particularly hard hill to climb.  As measured by involvement in AP exams, there’s just not much there — only 8 students took the AP CS A exam in the whole state last year, and 13 took AP CS Principles.

In 2014, I wrote an article titled “The Danger of Requiring Computer Science in K-12 Schools.”  I still stand by the claim that we should not mandate computer science for US schoolchildren yet. We don’t know how to do it, and we’re unlikely to fund it to do it well.

I can’t find any news articles describing what funding New Jersey and Wyoming are going to put toward the goal of teaching CS across their state.  How do you teach every student CS or teach CS in every school without any increase in funding?

Based on what we’ve seen in other US states, I predict one of three things will happen:

  • States will have to loosen the definition of “computer science,” as happened in South Carolina.  90 classes count for the CS requirement in South Carolina, and only 6 of which have programming in them.  Most of them are about keyboarding skills or application software use. If a state doesn’t fund real CS, something else will have to count as real CS.
  • States will rely heavily on virtual high schools and on-line classes to provide CS class “access” without hiring more CS teachers, as we are seeing in several states. That is particularly concerning since recent studies are showing that remedial students do poorly in on-line classes.
  • Independent CS classes will be deemed too expensive. Instead, the mandate to teach CS to all will lead to integration into math and science classes, which are already funded. School will have changed the reform, again (see Papert’s “Why School Reform is Impossible.”)

Neither of the first two options furthers the goal of having high-quality CS education for all students. The third one may be the best position, if funding doesn’t appear.

April 2, 2018 at 7:00 am 24 comments

How to Study for a CS Exam

I just recently finished Make It Stick (Amazon link) which takes modern learning science and develops recommendations to learners on how to study in order to learn most effectively. As I read it, I was wondering, “What are the implications for how students should study for a computer science exam or quiz?” I did a Web search for “computer science study skills,” and the results were disappointing. Most of the recommendations are wrong (e.g., “take good notes” and “highlight passages in the book”) and were not specific to learning computer science.

So, I wrote up some recommendations for my students. I have not tried to generalize these yet (but would be fun and useful to do so), and I haven’t gone back through Making it Stick to mine it for all the CS recommendations that one might invent from their lessons.

Caveat: these are “evidence-based” as in the learning sciences behind them is pretty solid, but they are not “evidence-based” in the sense of actually testing these practices with learning CS. We know that some learning sciences or educational psychology results from other fields don’t map cleanly to CS (see Briana’s work). I don’t know of research evaluating study practices for learners studying computer science — but it would be great to have some!

I’d be interested to hear how you help your students study computer science!  Please do share your ideas and recommendations in the comments.

How to study for a programming quiz

Here are my suggestions for how to study for Monday’s quiz (with some advice that is useful for all your other CS quizzes).

1. Read the Book. Once

If you haven’t read the relevant chapters, do read them. If you have already, I don’t suggest re-reading them. There are better ways to study than re-reading.

2. Do Your Programming Homework

The Collage Project (and all your other homework) is good preparation for Monday’s Quiz. Finish that up this weekend, and you’ll have reviewed many of the ideas needed for Monday’s Quiz.

3. Make sure you know what you didn’t know

Go over your homework assignments and past quizzes, and make sure that you now know everything you got wrong in the past.

4. Quiz Yourself

The best way to study is always by quizzing yourself. You’ll learn far more from quizzing yourself than from re-reading or reviewing your notes.

Use the problems at the back of the chapter. Get with others in the class and challenge each other by inventing variations on programming problems you have seen before:

  • Rather than create a negative of the whole picture, can you negate just the top half of a picture?
  • Can you move the bottom half of a picture onto the top?
  • Can you mirror a square picture diagonally?
  • Can you put a 100-pixel wide border of grayscale around a picture?

5. Elaborate on each Chapter

At the beginning of each chapter is a list of Learning Objectives — media learning goals and CS learning goals. Write a sentence or two demonstrating that you have learned those objectives. Explain what the goal means, and show that you have achieved those goals. Elaboration helps you to connect ideas so that it’s easier to remember those ideas later.

At the end of each chapter is the Programming Summary that lists the new programming ideas (new functions, new control structures) in that chapter. Make sure that you can use each of those. Just write a program, any program that uses each function at least once.

6. Try things in more than one way

In this class, you know more that one way of doing things.

  • You can get and set colors with setRed, getRed, setBlue, getBlue, setGreen, getGreen. OR getColor, setColor, makeColor. Make sure that you can use these interchangeably. Write increaseRed and decreaseRed using setColor. Write negation and grayscale with both sets.
  • You know several ways of accessing individual pixels.
    • You can use for pixel in getPixels(picture).
    • You can get all the pixels in a list with all = getPixels(picture) then generate the indices for that list with for index in range(0,len(all)): and access each pixel with pixel = all[index].
    • You can generate x and y coordinates with for loops, then get individual pixels with getPixel(picture,x,y).
    • Make sure that you can use all three well. Write chromaKey with all three forms. Create a green border around a picture using all three forms.

Take any program that you have written before, or that we have done in class, and rewrite it another way. This is both a form of elaboration and a form of quizzing.

7. When confused, ask and talk CS

Ask on Piazza. Ask your friends. Come to office hours. When studying, all collaboration is strongly encouraged.

Just talking through things (both what you understand and what you do not understand) with anyone will help you learn.  Explain your programs to someone else.  Summarizing your programs in English is a great way to make sure you understand the programs and to elaborate on your knowledge.

March 30, 2018 at 7:00 am 5 comments

When more information leads to worse performance: Beware throwing in “something fun and totally optional”

Eliane Wiese gave a talk here this last week. She told a story that I found fascinating. It connects to a story I just read about from Kahneman and Tversky. The theme has important implications for the design of software for CS education.

Story One: In Eliane’s dissertation work she explored how to give grounded feedback that would lead students to learn from mistakes. Here (in summary form) is the result of one of her studies.

In some questions, students were shown graphical representations of fractions. In other questions, they were shown some combination of graphical representations and symbolic fractions. In a fourth kind of questions, they’re just shown symbolic fractions. The vertical axis is performance.

The part that I find amazing is the results for condition two and three for fraction addition. Getting more information led to worse performance. Symbolic fractions are so confusing that their appearance depresses performance, even when the graphical information is still there. The students don’t just ignore the fractions. The mere presence of the fractions makes the problem harder for students.

(Original paper available here. Her follow-up/replication study can be found here. Thanks to Eliane for reviewing this post and sending me these links!)

Story Two: I just finished reading The Undoing Project (Amazon link) by Michael Lewis, the story of Daniel Kahneman and Amos Tversky’s amazing collaboration and friendship. One of their experiments is particularly relevant to Eliane’s finding.

You tell people that they’re going to pick a person at random from a pool of 100 people, 70 of whom are engineers and 30 of whom are lawyers. What is the probability that you’re going to get an engineer? Participants in the studies correctly guess 70%. You can change it to lawyers, or change around the ratios, and people solve this problem correctly and easily.

Now you tell them that, from the same pool, they have selected “Dick.”

Dick is a 30 year old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.

Now, what is the probability that Dick is an engineer? Participants say that the probability is 50% — they can’t tell. Notice that the description of Dick offers no additional information to discern if he is an engineer or a lawyer. Yet, people can’t ignore the useless descriptive information. They can’t just rely on the numbers. Getting more information leads to worse performance. People seem to feel a need to use all available information, even if it’s not useful, even if leads to worst performance.

What’s the implication for CS Ed? Our programming languages and professional IDE’s are complex. How about public static void main(String[] args)? How about all the bells and whistles in Eclipse?

When I point these out to teachers, the most common response I get is, “It’s okay. Students just ignore that part.”

I’m not sure that they do, or that they even can. People try to make sense of the information in front of them. We are drawn to create narratives. It is difficult for us to ignore information and make decisions based on only the relevant information. This is particularly hard for novices who don’t understand the relevant information, let alone separate the relevant from the irrelevant.

Before we toss something into our classes, we should pause and consider these stories. Sure, your CS1 students could use a cool new library that lets them do something cool (whatever — robotics, data visualizations, social network analysis) but has a confusing API and almost no documentation. The new library will consume their time and effort to understand. Sure, you might decide to introduce something (maybe list comprehensions or lambda expressions) into your Python code, just as “something fun” and “totally optional.” But students will try to understand it, and might not learn the things you really want them to learn. Sure, you could throw in a quick algorithm animation or use some super cool new debugger, but if your students are already confused, you’ve now just given them yet another representation or interface to make sense of. Think about the fact that the additional/extra/irrelevant information may be distracting your students from what is important. And that might lead to worse performance.

March 23, 2018 at 7:00 am 16 comments

How CS differs from other STEM Disciplines: Varying effects of subgoal labeled expository text in programming, chemistry, and statistics

My colleagues Lauren Margulieux and Richard Catrambone (with Laura M. Schaeffer) have a new journal article out that I find fascinating. Lauren, you might recall, was a student of Richard’s who applied subgoal labeling to programming (see the post about her original ICER paper) and worked with Briana Morrison on several experiments that applied subgoal labeling to textual programming and Parson’s problems (see posts on Lauren’s defense and Briana’s).

In this new paper (see link here), they contrast subgoal labels across three different domains: Chemistry, statistics, and computer science (explicitly, programming).  I’ve been writing lately about how learning programming differs from learning other STEM disciplines (see this post here, for example). So, I was intrigued to see this paper.

The paper contrasts subgoal labeled expository text (e.g., saying explicitly as a heading Compute Average Frequency) and subgoal labeled worked examples (e.g., saying Compute Average Frequency then showing the equation and the values and the computed result).  I’ll jump to the punchline with the table that summarizes the result:

Programming has high complexity.  Students learned best when they had both subgoal labeled text and subgoal labeled worked examples. Either one alone didn’t cut it. In Statistics, subgoal labeled examples are pretty important, but the subgoal labeled text doesn’t help much.  In Chemistry, both the text and the worked examples improve performance, and there’s a benefit to having both.  That’s an argument that Chemistry is more complex than Statistics, but less complex than Programming.

The result is fascinating, for two reasons.  First, it gives us a way to empirically order the complexity of learning in these disciplines. Second, it gives us more reason for using subgoal labels in programming instruction — students just won’t learn as well without it.


March 16, 2018 at 7:00 am 4 comments

Older Posts Newer Posts

Recent Posts

May 2018
« Apr    


Blog Stats

  • 1,510,631 hits

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 5,265 other followers

CS Teaching Tips