Iterative and interdisciplinary participatory design sessions: Seeking advice on a research method

July 1, 2019 at 7:00 am 4 comments

Here’s an unusual post for this blog: I’m looking for a research methodology, and I don’t know where to look for it. I’m hoping somebody here will have a suggestion — please do forward this blog post to others you think might have suggestions for me.

We’re running participatory design sessions with teachers — asking teachers to try out programming languages with scaffolded activities, and then tell us about what they’d like for their classroom. I’m collaborating with Tammy Shreiner and Bradford Dykes at Grand Valley State University around having social studies teachers build data visualizations. We’re scouring the book Participatory Design for Learning, and in particular, we’re using Michelle Wilkerson’s chapter (which I’ve read twice now) because it matches the kind of work we’re doing.

Michelle uses a technique called Conjecture Mapping to describe how her teams thinks about the components of a participatory design session. A session has a specific embodiment (things you put into the classroom or session), which you hope will lead to mediating processes (e.g., participants exploring data, people talking about their code, etc.). These are processes which should lead to desired outcomes based on theory. A conjecture map is like a logic model in that it connects your design to what you want to have happen, but a conjecture map is less about measuring outcomes. Rather, it’s more about describing mediating processes, which you theorize will lead to desired outcomes. The mediating process column is really key — it tells you what to look for when you run the design session. If you don’t hear the kind of talk you want and see participant success in the activity, something has gone wrong. Fix it before the next iteration. The paper on this technique is Conjecture Mapping: An Approach to Systematic Educational Design Research by William Sandoval.

So here’s the first problem: We have different set of outcomes for our sessions. They’re interdisciplinary. I want to see the teachers being successful with their programs, my collaborators Tammy Shreiner and Bradford Dyke wants to see them talking about their data, and we all want to see participants relating their data visualizations to their history class (e.g., they shouldn’t be just making pretty pictures, and they should be connecting the visual elements to the historical meaning). Should we put all of these mediating processes and outcomes into one big conjecture map?

We are not satisfied with this combined approach, because we’re going to be iterating on our designs over time. Most participatory design approaches are iterative, but I haven’t seen a way of tracking changes (in embodiment or mediating practices) over time. Right now, we’re working with Vega-Lite and JavaScript. In our next iterations, we’ll likely do different examples with Vega-Lite. Over time, we want to be building prototypes of data visualization languages designed explicitly for social studies educators (task-specific programming languages).

We are concerned about two big problems as we iterate:

  • Missing Out. I don’t want to lose any of our mediating processes. I want to make sure that we continue to see success with programming, engagement with data, and meaningful visualizations.
  • Changing the balance. The easiest trap for our work will be to over-emphasize the programming, and swamp out the data literacy and data visualization processes and outcomes. If our sessions become perceived as primarily a programming activity, we’re moving in the wrong direction. We want the data literacy and data visualization to be primary, with programming as a supporting activity.

The diagram at the bottom may help describe the problem — it’s the sketch that my PhD student Bahare Naimipour and I made while talking through this problem. We need to track multiple disciplinary processes and outcomes over time as we iterate across different embodiments. This isn’t about assessing an intervention or design. This is about gathering input as we design and implement technology prototypes. We want to be moving in the right direction, for the processes and outcomes that we want.

Here’s where I’m asking for help: Where should we be looking for exemplars? Who else is doing iterative, multidisciplinary participatory design sessions? What are good methods for use to use?


Entry filed under: Uncategorized. Tags: , , .

What a CS Ed Letter Writer Needs: Evaluating Impact for Promotion and Tenure in Computing Education Learning to build machine learning applications without code as an example of computing education research

4 Comments Add your own

  • 1. Daniel T Hickey  |  July 1, 2019 at 10:19 am

    I think that Katie Davis at University of Washington has done some very good work with participatory design.

    I also think that conversation analysis is a very good method for capturing the kind of work that people are doing with language in your context.

  • 3. Casper Harteveld  |  July 15, 2019 at 9:07 pm

    Mark, Gail Carmichael pointed me to this blogpost and I thought to give you some thoughts. First, I am part of the Building Systems from Scratch project where we are applying conjecture mapping to evaluate our intervention, which is letting kids build games about climate change topics with Scratch and through this let them learn computational thinking, systems thinking, and climate science knowledge. You can find more about this project here:

    While we are not strictly using participatory design here, we have run professional development sessions where teachers provided input on the curriculum and we iterated on it. We don’t really have yet a publication on this iterative design process that I can point you to.

    Second, as for some of your questions, you do put all these different outcomes into your conjecture map. I don’t see why that would be a problem per se but I may also be misunderstanding the problem you are having. The idea is that you have an intervention and that you try to achieve several outcomes; in our case CT, systems thinking, and climate science knowledge (which are related but still distinct and discipline specific). The idea of conjecture mapping is that you try to systematically reason about what is happening with your intervention without necessarily reducing the complexity of the work you do. And when you are making such a big conjecture map you will find that some of these interdisciplinary outcomes may, in fact, be closely related or be achieved through similar embodiments. If that’s the case, it may be an embodiment you want to focus on. There’s only so much you can do in a project. Then you have a good “conjecture” to work with.

    Hope this helps.

    • 4. Mark Guzdial  |  July 16, 2019 at 5:07 am

      Thanks, Casper — super interesting. How do you deal with tracking change in embodiments over time? How do you make sure that you don’t stray from your original goals?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 9,030 other followers


Recent Posts

Blog Stats

  • 1,994,871 hits
July 2019

CS Teaching Tips

%d bloggers like this: