What a CS Ed Letter Writer Needs: Evaluating Impact for Promotion and Tenure in Computing Education

June 24, 2019 at 7:00 am 2 comments

I’ve been asked, “When I’m writing a tenure or promotion letter for someone who works in CS education, what should I say?” I’m motivated to finally answer, in response to an excellent post by Andy Ko, On the academic quantified self. I recommend it highly, and suggest you go read that before this post.

Andy’s post is on how to present his scholarly self. His key question is “How can senior faculty like myself model scholarly selves rather than quantified selves?” He critiques his own biographic paragraph, which contains phrases like “is the author of over 80 peer-reviewed publications, 11 receiving best paper awards and 3 receiving most influential paper awards.” He restructures it to emphasize the narrative of his research, with sentences like this:

His most recent investigations have conceptualized the skills involved in programming, theorizing about the interplay between rigorous knowledge of programming language semantics, strategies for addressing the range of problems that arise in programming, and self-regulation skills for managing the selection, execution, and abandonment of strategies; these are impacting how programming is learned and taught.

Andy is the program chair at the University of Washington’s School of Information. He writes as a role model for how to present oneself in academia — not just numbers, but a narrative about knowledge-building.

I have a slightly different perspective. I am frequently a letter writer for promotion or tenure (and often both). I don’t get to set the criteria — those are set by the institution. The challenge gets harder when the criteria were clearly written for traditional Computer Science Scholarship of Discovery (versus the other forms of scholarship described by Boyer such Scholarship of Application or Integration), but the candidate specializes in computing education researcher or is teaching-track faculty.

The criterion that most departments agree on for academic success is impact. So there’s the question: How do we evaluate impact of academic work in computing education?

As a letter writer, I need a combination of both of Andy’s biographical paragraphs, but the latter is more valuable for me. Statistics like “80 peer-reviewed publications, 11 receiving best paper awards and 3 receiving most influential paper awards” tells me about perceptions of quality by the reviewers. Peer review (for papers and grants) and paper awards are really important for third year review and sometimes for tenure, to make the argument that the candidate is doing good work and is on a promising trajectory.

A letter writer should not just cite the numbers. The promotion and tenure committees are looking for judgment based on the letter writers’ expertise. Construct a narrative. Make an argument.

An argument for impact has to be about realized potential. Andy’s second paragraph tells me where to look for that impact. Phrases like “these are impacting how programming is learned and taught” inform me where to look for evidence. I want to see that this work is actually changing learning and teaching practices — by someone other than the candidate.

If the candidate is in computing education research, then some of the traditional measures of Scholarship of Discovery still work. One important form of impact is on other researchers. Candidates can help me as a letter writer when they can show in the narrative of their research statement how other researchers and other projects are building on their work. I once was reviewing a candidate in the US who showed that a whole funding program in another country referenced and built upon their work. Indirectly, that candidate impacted every research project that that program funded — that’s amazing impact, but hard to measure. As Andy says, you have to spell out the narrative.

As much as we dislike bean-counting, an H-index (and similar metrics) does provide evidence that other researchers are building on the work of the candidate. It’s not the only measure. It’s just a number, and it has to be put in context with judgment informed by the letter writers’ expertise.

If a candidate is only focused on teaching, I usually turn away the request to write the letter.  I have some research interest in how to measure high-quality teaching (e.g., how to measure CS PCK), but I don’t know how to evaluate the practice of teaching computing.

If the candidate is (a) tenure-track in computing education or (b) teaching track and aims to influence others’ practice, the argument for impact may require some non-traditional measures. Some that I’ve used in my letters:

  • If a candidate can find evidence that even one other instructor adopted curriculum or teaching practices invented by the candidate, that’s impact. That means somebody else looked at the candidate’s work, saw the value in it, and adopted it. Links to syllabi, letters from instructors or schools, and even textbooks that incorporate the candidate’s work (even if not cited directly) are all good forms of evidence.
  • One of the reasons I get asked to write letters is that I’m still active in computing education. I can give evidence of impact from my personal experience. Researchers influence the research discourse, even before it shows up in the research literature. The discourse happens in hallways of conferences, in social media, and in workshops and seminars like Dagstuhl. This is inherently a biased form of evidence — I can’t be everywhere and hear everything. I might not notice everything that gets discussed. An institution only gets my evidence if they ask me. That bias is one reason why any case for promotion and tenure asks for several letters.
  • Sometimes, there is impact by influence and association. I have written a supportive letter for candidate who had not published a lot, but had been critical in the success of several other people. The candidate’s co-authors and co-investigators on projects had become influential leaders in computing education. I knew from talking to those co-authors that the candidate had been a leader on the projects. The candidate had launched significant projects and advanced the careers of others. That’s an important form of impact.
  • It’s hard to use success of students as an indicator of candidate’s impact. How much did the candidate influence the success of those students? Letters from the students can be helpful, but it’s still hard to make that kind of case. If a candidate works with terrific students, the candidate does not have to make much impact, and the students will still be successful. How do you argue for the value added by the candidate? If a whole class dramatically improves in performance or retention due to the efforts of a candidate — that’s a positive and measurable form of impact.
  • I’m a big fan of using Boyer’s Scholarship of Integration and Application in letters. If a candidate is one of the first to integrate two areas of research, or to apply a new method, or to build a curriculum or tool that meets a unique need, that is a potential form of impact. I still like to see evidence that the work itself had influence (e.g., was adopted by someone else, or changed student demographics, or changed practice of others).

We need to write letters that advance computing education candidates. Other countries are further than the US in recognizing computing education contributions (see post on that theme here). We need to learn how to tell the stories of impact in computing education, in order to advance the candidates doing that kind of work.

(Thanks to Andy Ko and Shriram Krishnamurthi who gave me feedback on earlier forms of this post.)

Entry filed under: Uncategorized. Tags: , , .

An Ebook for Java AP CS Review: Guest Blog Post from Barbara Ericson Iterative and interdisciplinary participatory design sessions: Seeking advice on a research method

2 Comments Add your own

  • 1. Briana Morrison  |  June 24, 2019 at 11:03 pm

    I’ll put a plug in here – you can demonstrate that others have viewed your material and thought it of substantial quality by submitting to EngageCSEdu (http:\engage-csedu.org). We treat introductory programming assignments, materials like a journal submission. Materials are reviewed by 2 CS instructors and 1 Social Science researcher to ensure the material is engaging for all and does not promote sterotypes or marginalize any group. Having vetted, published materials on your CV is certainly evidence of good work, and having download statistics shows a measurable quantity of potential use.

    Reply
  • 2. Mehran Sahami  |  June 27, 2019 at 2:35 am

    Mark, thanks for taking the time to put together this thoughtful post. I agree with pretty much all the points you bring up. It’s a very important issue for fostering growth (and respect) in the CS Ed community.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackback this post  |  Subscribe to the comments via RSS Feed


Enter your email address to follow this blog and receive notifications of new posts by email.

Join 6,273 other followers

Feeds

Recent Posts

Blog Stats

  • 1,664,024 hits
June 2019
M T W T F S S
« May   Jul »
 12
3456789
10111213141516
17181920212223
24252627282930

CS Teaching Tips


%d bloggers like this: