Programming and learning CS when legally blind

Since I’ve been using blocks-based languages lately (see my posts on GP and MOHQ), I’ve been thinking more about the challenges of using blocks-based languages, and programming and learning CS more generally, when legally blind.  One of our PhD students in the Human-Centered Computing PhD program is legally blind, and he generously came to visit me and brought with him one of his students who is legally blind and learning programming.

The first and biggest surprise for me was that most (about 85%) legally blind people can actually see. One of the people I worked with can see light/dark (which doesn’t help with programming, but does help him with way-finding and spatial navigation). The other one loves to program in App Inventor using high magnification on her Mac. She’s low-vision and finds the large splotches of color useful in figuring out her code.

The implication, they explained to me, is that some tactile-based affordances for blind people don’t work because low-vision blind people would prefer to use audio and what sight they have, rather than learn a touch-based encoding. I was surprised to learn that most blind people don’t learn Braille because it’s a complicated code, and low vision people would rather magnify the screen than learn the encoding.

Blind programmers who know Braille will often use an audio screen reader along with a Braille reader for a single line of text. It’s easier to scan a line (especially for syntax errors) with Braille than with a screen reader.

The second surprise was about their tools. They showed me Visual Studio and EdSharp, a plain text editor developed by a blind programmer for blind programmers. I asked what features made an editor good for blind programmers. They said, “It works with screen readers.” And really, that’s it. They don’t want specialized tools with non-standard interfaces because of the cognitive load of switching between the standard screen reader interfaces and a novel interface.

I didn’t realize how few tools go to the trouble of accessing the screen reader API’s and providing good mappings from the interface to text. Processing (all platforms) and NetBeans (on Windows) are completely unusable for blind people because they are inaccessible by screen readers. Visual Studio has become a new favorite IDE, not because of any special features, but because it does “it doesn’t crash and I can access it with a screen reader.”

I was particularly interested in the low-vision programmer’s use of App Inventor. We talked about what didn’t work for her and brainstormed what would make it better. One of the tougher parts of block-based languages is that scripts could be anywhere in a 2-D space. It’s hard to scan a 2-D space with a zoomed interface, and there’s no obvious interface for screen-readers. Having blocks snap to a grid would help a lot to make it easier to find scripts for both types of blind programmers.

We talked about how CS classes might be better designed for legally blind students. I was surprised to learn how much they dislike active learning activities in classrooms.  They said that when the whole class breaks into small group discussions, they can’t hear their group.  The definition of the group is by physical proximity, but they discern “close” by “loud.”  They end up listening in to whichever group is loudest around them.  They need a different kind of active learning activity.

Introducing GP: A General Purpose Block Language

GP is a new blocks-based programming language being developed by John Maloney (most well-known for developing Scratch), Jens Mönig (developer of Snap!), and Yoshiki Ohshima (one of the developers of Squeak EToys) in Alan Kay’s group. They are all part of the new partnership between Alan Kay and Y-Combinator Research: HARC (Human Advancement Research Community). GP started in the SAP-funded CDG (Communications Design Group).

GP is not yet released, and there’s not much publicly available on it yet. The GP Team published a paper and poster in the Blocks and Beyond Workshop at last year’s VL/HCC on GP. The best introductory article on GP so-far is on the Scratch Wiki at MIT based on John’s presentation at the Scratch conference last year.

What makes GP remarkable is that it aims to be a general purpose language. John’s vision for GP is to be the language that students might move to after Scratch, with the highest possible ceilings. Think about GP as Python or Smalltalk in blocks — and even more the latter than the former. From the virtual machine (VM) on which it runs to the class browser, GP feels like a blocks-based form of Smalltalk. Because GP is VM-based, it’s portable — there are versions for Mac, Windows, iOS, and even a JavaScript implementation of the VM so that GP runs in the browser.

GP is an exploration of the question, “How far can we go with a blocks-based programming language? Do we have to move students to a textual programming language to let them develop everything from data analyses to real applications?”

GP users can do a lot with GP’s built-in blocks. However, as they grow in mastery, some users may wish to add new blocks to GP (e.g. to manipulate images), or even to extend the GP programming environment itself (e.g. by adding an image editor). GP is designed to be extended in itself using the same blocks language that users already know. However, unlike Smalltalk or Snap!, the GP language itself cannot be extended (e.g. to add a new control structure) without modifying the virtual machine. Keeping the GP language simple and fixed is intended to ease the learning path for beginners.

A brief tour of Smalltalk-like features of GP

When you first start up GP, it looks like Scratch. The blocks palette is different, because it’s covering a larger space of blocks. GP includes blocks for dealing with data (e.g., JSON, comma-separated values), media generation and manipulation, connections to the network and external devices, and the ability to create and coordinate multiple objects.

There are even blocks in there for manipulating pixels in an image and samples in a sound. GP is the first blocks-based language in which I’ve been able to do both sound and pixel Media Computation examples. I built the first version of MediaComp blocks for GP, then John figured out which ones were actually useful and then re-implemented them in GP much more efficiently than what I did.

I’m introducing GP here with the GP Team’s permission in order to show you a prototype ebook I’ve been building the last few months. You can play with GP at http://home.cc.gatech.edu/GPBlocks. This is the browser-based version which is offered with no guarantees — the browser version will likely change dramatically as GP is still being developed, and even the examples in the ebook may break over time. (Note: These browser-based examples are best viewed in Firefox on a desktop or laptop computer; they do not yet work on iOS or Android tablets.)

Here’s a brief series of snapshots to give you a sense of what makes GP so interesting and powerful compared to most other blocks-based languages. In the stage area (upper right-hand corner) right-click (control-click on a Mac) to bring up the stage menu.

The menu options for a workspace and to browse will elicit warm feelings of recognition for Smalltalk and Self programmers. Go ahead and click on the browse menu item.

Scanning the classes along the left hand side you realize that this is a full Smalltalk-like language. All the pieces are there and inspectable. The middle panes show the instance variables in the class (top) and the methods for the class (bottom). The rightmost pane shows the code for the method — in blocks!

One of the big goals of GP is that all of GP is written in GP. Even the lowest levels of GP (e.g., how bitmaps and blocks are constructed) can be manipulated in GP, all in blocks. Those methods are real code and “live.” Change them and you change how GP is working immediately. Right now, that’s super dangerous — there is no “editing” mode. Move a block out of place, and the method is changed at that moment. Beware of re-defining how Integers work! The GP team is currently working to complete this part of GP, allowing the GP programming system to be used to modify itself, like Smalltalk.

The GP team is also exploring the stages between blocks and text. At the top right hand corner of GP is a slider between blocks and text. Switch it to text, and all of GP is presented and usable in a textual form. (There’s even an interesting middle stage between blocks and text.)

I’ve been using GP for about nine months. During the Spring semester, I’ve been using GP with an undergraduate research assistant, David Tran, to build a prototype of a new kind of ebook structure. Play around (muck/MOHQ around) in the GPBlocks MOHQ, and in the next blog post, I’ll explain what it is and what we’re exploring in it.

---

My thanks to the GP team for review and comments on drafts of this post.

ICER 2015 Report: Blocks win–Programming Language Design == UI Design

ICER 2015 at the University of Nebraska, Omaha was fantastic.  Brian Dorn did a terrific job hosting all of us.

The Doctoral Consortium went really well.  We had 20 students from US, Chile, Germany, and UK.  Below is a picture from the “Up against the wall bubble sort” where experienced students went to one side, and newer students went to the other, and the former gave advice to the latter.

 

 

Georgia Tech had even more going on at ICER and RESPECT than I mentioned in my earlier blog posts (like here and here).  The GVU Center did a nice write up about all of us here.  The biggest thrill at ICER for the GT crowd was Briana Morrison receiving the Chairs Award (one of two best paper awards at ICER) for the paper that I blogged about here.  Below is the whole GT contingent at ICER (including chair Brian Dorn, GT alum).

The other best paper award, the peoples’ choice John Henry Award, went to Kristin Searle and Yasmin Kafai (see paper here) about the e-textiles work with American Indians that I blogged about here.  Kristin had so many interesting insights, like the boys in her project telling her that “I don’t own” the projects they made because they felt no ownership over the programming environment they were using.

The quality of the papers was very good (you can see the list of all of them here).  My favorite paper from my review packet was presented Monday morning, Spatial Skills in Introductory Computer Programming.  Steve Cooper and Sheryl Sorby with two undergraduates at Stanford did the study that I’ve been wanting to see for ages (see blog post where I talk about it). Training an experimental group in spatial skills improved performance over a control group.  Surprisingly, SES and race differences disappeared in the experimental group!  This is an important result.

But one session blew me away — it changed how I think about blocks programming.

• The first paper was from Thomas Price and Tiffany Barnes showing that students using blocks were able to achieve programming tasks faster than those using text, but with no difference in learning or attitudes afterwards (paper here).  This was an interesting result, but it was a limited study (short intervention, no pre-test) so it mostly supported a finding from Chris Hundhausen from years previous that graphical, direct-manipulation languages lead to faster start-up than text languages (see paper here).
• David Weintrop presented his remarkable paper with Uri Wilensky (see paper here).  Below is the graph that changed my thinking about blocks.  David carefully developed an isomorphic test in blocks and text, and gave it to the same population.  Students did much better on the blocks-based test. MODALITY MATTERS!  Blocks and text are not equivalent. He did careful analyses at each level of the test. For example, David replicated the result that else clauses in text are really hard for novices (which I talked about here), but students perform much better in blocks-based if-else.

 

 

• Diana Franklin presented their paper describing fourth graders reading Scratch programs (see paper here).  I was expecting a paper on program comprehension — it wasn’t.  Instead, it was a paper about user interfaces, and how the user interface interfered or supported students exploring and coming to understand the program.

I came away from that three papers realizing that blocks programming is likely the best modality to use in elementary school programming, and perhaps even when starting to program in high school, and maybe even for end-user programmers.  But even more important, I realized that Amy Ko’s comments about programming languages as being a powerful and unusable user interface (see her blog post here) is the critical insight about programming today.  David showed us that blocks can dramatically increase readability of programs.  Diana showed us that the user interface dramatically influences the readability of the blocks.  At the novice programming level, blocks-based languages are the most promising direction today, and designing good blocks languages is as much a user interface design problem as it is a programming language design problem.