Collaborative Research: Transforming Computer Science Education Research Through Use of Appropriate Empirical Research Methods: Mentoring and Tutorials

Funding: National Science Foundation [$406,557]
Principal Investigators: Jeffrey Carver (University of Alabama), Sarah Heckman, Mark Sherriff (University of Virginia)

The computer science education (CSEd) research community consists of a large group of passionate CS educators who often contribute to other disciplines of CS research. There has been a trend in other disciplines toward more rigorous and empirical evaluation of various hypotheses. However, many of the practices that we apply to demonstrate rigor in our discipline research are ignored or actively avoided when performing research in CSEd. This suggests that CSEd is "theory scarce" because most publications are not research and do not provide the evidence or replication required for meta-analysis and theory building . An increase in empiricism in CSEd research will move the field from "scholarly teaching" to the "scholarship of teaching and learning" (SoTL) providing the foundation for meta-analysis and the generation of theories about teaching and learning in computer science. We propose the creation of training workshops and tutorials to educate the educators about appropriate research design and evaluation of educational interventions. The creation of laboratory packages, "research-in-a-box," will support sound evaluation and replication leading to meta-analysis and theory building in the CSEd community.


  • A. Al-Zubidy, J. Carver, S. Heckman, M. Sherriff, "A (Updated) Review of Empiricism at the SIGCSE Technical Symposium," 2016 SIGCSE Technical Symposium, Memphis, TN, March 2-5, 2016, p. 120-125. (Acceptance Rate: 35.4%)

Research Triangle Peer Teaching Fellows: Scalable Evidence-Based Peer Teaching for Improving CS Capacity and Diversity

Funding: Google Computer Science Capacity Award
Principal Investigators: Jeff Forbes (Duke University), Ketan Mayer-Patel (UNC Chapel Hill), & Kristy Boyer (University of Florida),

The project seeks to increase undergraduate retention and diversity in introductory programming courses. We are interested in creating a scalable, effective, and evidence-based peer teacher training program at NC State, Duke University, and UNC Chapel-Hill.


  • M. Vellukunnel, P. Buffum, K. E. Boyer, J. Forbes, S. Heckman, K. Mayer-Patel, "Deconstructing the Discussion Forum: Student Questions and Computer Science Learning," SIGCSE 2017, pp. 603-608. (Acceptance Rate: 30%)
  • A. Smith, K. E. Boyer, J. Forbes, S. Heckman, K. Mayer-Patel, "My Digital Hand: A Tool for Scaling Up One-to-One Peer Teaching in Support of Computer Science Learning," SIGCSE 2017, pp. 549-554. (Acceptance Rate: 30%)

Incorporating Software Engineering Best Practices into CSC216


Students in CSC216, a second semester programming course at NC State, struggle with the transition from a small integrated lecture-lab classroom to a large lecture class. While I use active learning the classroom, the exercises are small (e.g., write a single method) and do not demonstrate how the piece might fit into a larger solution. Student evaluations have consistently requested additional time to practice programming larger solutions in class. I revised a linear data structure unit to use in-class laboratory assignments. Before class, students watch short videos about the topic and they engage with the topic during class time on small teams. While working on the in-class labs, students are encouraged to follow software engineering best practices like test driven development. I have several research questions:

  • Do in-class labs on linear data structures in CSC216 increase student learning, engagement, and efficacy on linear data structures when compared with standard active learning lectures?
  • Does requiring good software engineering practices during in-class labs promote better software engineering practices during development of independent or team projects?
  • Do in-class labs in CSC216 reduce the number of students repeating the course?
  • Do in-class labs in CSC216 increase student success in follow on courses CSC230 and CSC326?

Current findings show no major difference on student learning between in-class labs and standard active learning lectures. Pass rates are lower than what we might expect to see from active learning classes [Freeman2014], but are higher than traditional lecture pass rates [Freeman2014] and the pass rates of CS1 courses from literature [Bennedsen2007, Watson2014].

The DELTA Course Redesign grant is supporting work to move CSC216 to a lab-based course. The in-class labs will be transitions into an affiliated Peer Teaching Fellow run lab of 20-24 students starting in Fall 2016.


  • S. Heckman, "An Empirical Study of In-Class Laboratories on Student Learning of Linear Data Structures," International Computing Education Research Conference, 2015, pp. 217-225.
  • S. Heckman, "A Continuous Integration Framework for Promoting Software Engineering Best Practices," Poster: International Computer Education Research Conference, 2015.
  • S. Heckman, "An Investigation of In-class Labs on Student Learning of Linear Data Structures," Poster: NC State 2015 Teaching and Learning Symposium, 2015.

Expressive and Scalable Notifications from Program Analysis Tools

Agency/Program: NSF
Duration: October 2012 - December 2016
Principle Investigators: Emerson Murphy-Hill and Sarah Heckman

A wide variety of program analysis tools have been created to help software developers do their jobs, yet the output of these tools are often difficult to understand and vary significantly from tool to tool. As a result, software developers may waste time trying to interpret the output of these tools, instead of making their software more capable and reliable. This proposal suggests a broad investigation of several types of program analysis tools, with the end goal being an improved understanding of how program analysis tools can inform developers in the most expressive and uniform way possible. Once this goal is reached, we can create program analysis tools that enable developers to make tremendous strides towards more correct, more reliable, and more on-time software systems.

The resources, below, are used in CSC216: Programming Concepts - Java, a second semester programming course for CSC majors. The resources are related directly to exercises and materials that were added to CSC216 as part of this grant. Access the PPTX and actual Google forms available upon request. Please note that some of the slides are modified from Reges and Stepp's Building Java Programs website. See the available supplements.

Lecture Exercises Code
Code Coverage and Static Analysis

Exercise 05.01
Exercise 05.02
Exercise 05.03
Exercise 05.04
Exercise 05.05

Linked Lists

Exercise 11.03
Exercise 11.06
Inspections Exercise 12.01
Exercise 12.02
Exercise 12.03
Iterators and Inner Classes Exercise 13.03
Recursion Exercise 18.03

Software Engineering Materials and Labs
Providing exercises related to program analysis tools can help students reason about the notifications that they receive in terms of recently covered lecture materials. Students should also use the tools in practice and resolve notifications as they arise in their work. In CSC216, students are introduced to program analysis tools in three guided projects and continue to use the tools during the lab activities for the course.


  • B. Johnson, R. Pandita, J. Smith, D. Ford, S. Elder, E Murphy-Hill, S. Heckman, C. Sadowski, "A Cross-Tool Communication Study on Program Analysis Tool Notifications," Foundations of Software Engineering, Seattle, WA, USA, November 13-18, 2016, pp. 73-84.
  • Brittany Johnson, Rahul Pandita, Emerson Murphy-Hill, Sarah Heckman, "Bespoke Tools: Adapted to the Concepts Developers Know," 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, NIER Track, Bergamo, Italy, August 30-September 4, 2015, p. 878-881.

Jenkins: Automated Grading while Reinforcing Software Engineering Best Practices

Automated grading systems provide students with feedback early and often. Use of automated grading systems and other software engineering tooling could reinforce software engineering best practices when developing software. We have developed a custom configuration of Jenkins, a continuous integration server, to automatically grade assignments in CSC216, CSC230, and CSC326 while reinforcing software engineering best practices of test driven development, continuous integration, version control, code coverage, and static analysis.


  • S. Heckman, J. King, M. Winters, "Automating Software Engineering Best Practices Using an Open Source Continuous Integration Framework," Poster: 2015 SIGCSE Technical Symposium, 2015, p. 677.

CPATH II: Incorporating Communication Outcomes into the Computer Science Curriculum

Agency/Program: NSF
Duration: October 2009 - September 2012
Role: Senior Personal

Summary: (from grant)
"In partnership with industry and faculty from across the country, this project will develop a transformative approach to developing the communication abilities (writing, speaking, teaming, and reading) of Computer Science and Software Engineering students. We will integrate communication instruction and activities throughout the curriculum in ways that enhance rather than replace their learning technical content and that supports development of computational thinking abilities of the students. We will implement the approach at two institutions. By creating concepts and resources that can be adapted by all CS and SE programs, this project also has the potential to increase higher education's ability nationwide to meet industry need for CS and SE graduates with much better communication abilities than, on average, is the case today. In addition, by using the concepts and resources developed in this project, CS and SE programs will be able to increase their graduates' mastery of technical content and computational thinking."


  • Paul V. Anderson, Sarah Heckman, Mladen Vouk, David Wright, Michael Carter, Janet E. Burge, and Gerald C. Gannod, "CS/SE Instructors Can Improve Student Writing without Reducing Cass Time Devoted to Technical Content: Experimental Results," Joint Software Engineering Education and Training (JSEET), p. 455-464.
  • Michael Carter, Robert Fornaro, Sarah Heckman, Margaret Heil, "Creating a Progression of Writing, Speaking, & Teaming Learning Outcomes in Undergraduate Computer Science/Software Engineering Curricula," World Engineering Education Forum (WEEF), Buenos Aires, Argentina, October 15-18, 2012.
  • Michael Carter, Robert Fornaro, Sarah Heckman, and Margaret Heil, "Developing a Learning Progression that Integrates Communication into an Undergraduate CS/SE Curriculum," NCSU Technical Report, TR-2012-7, May 25, 2012.

Predicting Actionable Static Analysis Alerts


[IBM PhD Fellowship] August 2006 - May 2009
[CACC Grant] July 1, 2005 - June 30, 2006

Automated static analysis (ASA) can identify potential source code anomalies like null pointers. These anomalies could lead to field failures. Each alert or notification from an ASA tool requires inspection by a developer to determine if the alert is important enough to fix, which is called an actionable alert. Automated identification of actionable alerts generated by ASA could reduce the inspection overhead. Actionable alert identification techniques (AAIT) supplement ASA by using ASA alerts and other information about the software under analysis, called artifact characteristics, to prioritize or classify alerts.

We proposed an AAIT that utilizes machine learning on the history of a software development project to classify alerts as actionable or unactionable. Using the benchmark, FAULTBENCH, we completed a comparative evaluation between our AAIT and other AAIT from literature and found that for the subject programs our AAIT tended to better classify actionable an unactionable alerts when focusing on the goal of reducing false positives.

Future work will do the following:

  1. Expand FAULTBENCH to allow for a deeper analysis of various goals for AAIT;
  2. Further analyze the artifact characteristics and machine learning algorithms to determine which are most predictive; and
  3. Investigate how active usage of static analysis through discrete fixes and filterings might adjust the predictiveness of AAIT models.

Main Publications:

  • Sarah Heckman and Laurie Williams,  “A Systematic Literature Review of Actionable Alert Identification Techniques for Automated Static Code Analysis,” Information and Software Technology, vol. 53, no. 4, April 2011, pp. 363-387.
  • Sarah Heckman and Laurie Williams, "A Comparative Evaluation of Static Analysis Actionable Alert Identification Techniques," 9th International Conference on Predictive Models in Software Engineering (PROMISE), Baltimore, Maryland, USA, October 9, 2013, Article No. 4.
  • Sarah Heckman and Laurie Williams, "A Model Building Process for Identifying Actionable Static Analysis Alerts," 2nd IEEE International Conference on Software Testing, Verification, and Validation (ICST), Denver, CO, USA, April 1-4, 2009, pp. 161-170.
  • Sarah Heckman and Laurie Williams, "On Establishing a Benchmark for Evaluating Static Analysis Alert Prioritization and Classification Techniques," Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM), Kaiserslautern, Germany, October 9-10, 2008, pp. 41-50.
  • Sarah Smith Heckman, A Systematic Model Building Process for Predicting Actionable Static Analysis Alerts, Dissertation, Computer Science, North Carolina State University, 2009.

Google Apps as an Enhanced Classroom Response System

Agency/Program: DELTA IDEA Grant
Duration: September 2011 - June 2012
Principle Investigators: Ed Gehringer and Sarah Heckman

NC State now provides Google Apps for Education to all students, faculty, and staff.  This allows for the use of Google forms as a classroom response system. Faculty can pose questions to students during class using Google forms. Students can answer, and the instructor can look at these answers to find out how well the students understand the material. Instructors can modify their lectures based on student’s understanding of recently covered topics. As a classroom response system, Google forms are quite usable today. But further innovation would make them much more attractive to instructors and students.


  • S. Heckman and E. Gehringer “Google Forms as an Enhanced Classroom Response System,” Abstract: International Society for the Scholarship of Teaching and Learning (ISSOTL ’13).



[IBM Eclipse Innovation Grant] January 2004 - December 2005


OpenSeminar is a web-based, expert-maintained repository of links to openly available course resources on the Internet. Professors may then customize the contents to present materials to their students. I have been working on several OpenSeminars: Software Engineering, Open Source, and Memory Systems.


  • Michael Rappa, Sarah E. Smith*, Alex Yacoub, and Laurie Williams, “OpenSeminar: A Web-Based Collaboration Tool for Open Educational Resources,” Proceedings of the 1st International Conference on Collaborative Computing: Networking, Applications, and Worksharing (CollaborateCon 2005), San Jose , CA , 2005.
  • Michael Rappa, Sarah E. Smith*, and Alex Yacoub, “Open Course Resources as Part of the OpenSeminar in Software Engineering,” 19th Conference on Software Education and Training (CSEE&T 2006), Turtle Bay, Hawaii, 2006, pp. 187-189.
  • Laurie Williams, Sarah E. Smith*, Michael Rappa, “Resources for Agile Software Development in the Software Engineering Course,” 18th Conference on Software Engineering Education and Training (CSEE&T 2005), pp. 236-238.