BDSI – A New Validated Assessment for Basic Data Structures: Guest Blog Post from Leo Porter and colleagues
February 24, 2020 at 7:00 am Leave a comment
Leo Porter, Michael Clancy, Cynthia Lee, Soohyun Nam Liao, Cynthia Taylor, Kevin C. Webb, and Daniel Zingaro have developed a new concept inventory that they are making available to instructors and researchers. They have written this guest blog post to describe their new instrument and explain why you should use it. I’m grateful for their contribution!
We recently published a Concept Inventory for Basic Data Structures at ICER 2019 [1] and hope it will be of use to you in your classes and/or research.
The BDSI is a validated instrument to measure student knowledge of Basic Data Structure Concepts [1]. To validate the BDSI, we engaged faculty at a diverse set of institutions to decide on topics, help with question design, and ensure the questions are valued by instructors. We also conducted over one hundred interviews with students in order to identify common misconceptions and to ensure students properly interpret the questions. Lastly, we ran pilots of the instrument at seven different institutions and performed a statistical evaluation of the instrument to ensure the questions are properly interpreted and discriminate between students’ abilities well.
What Our Assessment Measures
The BDSI measures student performance on Basic Data Structure concepts commonly found in a CS2 course. To arrive at the topics and content of the exam, we worked with fifteen faculty at thirteen different institutions to ensure broad applicability. The resulting topics on the CI include: Interfaces, Array-Based Lists, Linked-Lists, and Binary Search Trees. If you are curious about the learning goals or want more details on the process we used in arriving at these goals, please see our SIGCSE 2018 publication [2].
Why Validated Assessments are Great for Instructors
Suppose you want to know how well your students understand various topics in your CS2 course. How could you figure out how much your students are learning relative to other schools? You could, perhaps, get a final exam from another school and use it in your class to compare results, but invariably, the final exam may not be a good fit. Moreover, you may find flaws in some of the questions and wonder if students interpret them properly. Instead, you can use a validated assessment. The advantage of using a validated assessment is there is general agreement that it is measuring what you want to measure and it accurately measures student thinking. As such, you can compare your findings to results from other schools who have used the instrument to determine if your students are learning particular topics better or worse than cohorts and similar institutions.
Why Validated Assessments are Great for Researchers
As CS researchers, we often experiment with new ways to teach courses. For example, many people use Media Computation or Peer Instruction (PI), two complementary pedagogical approaches developed over the past several decades. It’s important to establish whether these changes are helping our students. Do more students pass? Do fewer students withdraw? Do more students continue studying CS? Does it boost outcomes for under-represented groups? Answering these questions using a variety of courses can give us insight into whether what we do corresponds with our expectations.
One important question is: using our new approach, do students learn more than before? Unfortunately, answering this is complicated by the lack of standardized, validated assessments. If students score 5% higher on an exam when studying with PI vs. not studying with PI, all we know is that PI students did better on that exam. But exams are designed by one instructor, for one course at one institution, not for the purposes of cross-institution, cross-cohort comparisons. They are not validated. They do not take into account the perspectives of other CS experts. When students answer a question on an exam correctly, we assume that it’s because they know the material; when they answer incorrectly, we assume it’s because they don’t know the material. But we don’t know: maybe the exam contains incidental cues that subtly influence how students respond.
A Concept Inventory (CI) solves these problems. Its rigorous design process leads to an assessment that can be used across schools and cohorts, and can be used to validly compare teaching approaches.
How to Obtain the BDSI
The BDSI is available via the google group. If you’re interested in using it, please join the group and add a post with your name, institution, and how you plan to use the BDSI.
How to Use the BDSI
The BDSI is designed to be given as a post-test after students have completed the covered material. Because the BDSI was validated as a full instrument, it is important to use the entire assessment, and not alter or remove any of the questions. We ask that instructors not make copies of the assessment available to students after giving the BDSI, to try to avoid the questions becoming public. We likewise recommend giving participation credit, but not correctness credit, to students for taking the BDSI, to avoid incentivizing cheating. We have found giving the BDSI as part of a final review session, collecting the assessment from students, and then going over the answers to be a successful methodology for having students take it.
Want to Learn More?
If you’re interested in learning more about how to build a CI, please come to our talk at SIGCSE 2020 (from 3:45-4:10pm on Thursday, March 12th) or read our paper [3]. If you are interested in learning more about how to use validated assessments, please come to our Birds of a Feather session on “Using Validated Assessments to Learn About Your Students” at SIGCSE 2020 (5:30-6:20pm on Thursday, March 12th) or our tutorial on using the BDSI at CCSC-SW 2020 (March 20-21).
References:
[3] Cynthia Taylor, Michael Clancy, Kevin C. Webb, Daniel Zingaro, Cynthia Lee, and Leo Porter. 2020. The Practical Details of Building a CS Concept Inventory. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE ’20).
Entry filed under: Uncategorized. Tags: assessment, computing education research, evaluation.
Computing Ed Research - Guzdial's Take 
Trackback this post | Subscribe to the comments via RSS Feed