The Programming Exercise Markup Language: A Teacher-Oriented Format for Describing Auto-graded Assignments

Automated programming assignment grading tools have become integral to CS courses at introductory as well as advanced levels. However a lot of these tools have their own custom approaches to setting up assignments and describing how solutions should be tested, requiring instructors to make a significant learning investment to begin using a new tool.

In addition, differences between tools mean that initial investment must be repeated when switching tools or adding a new one. Worse still, tool-specific strategies further reduce the ability of educators to share and reuse their assignments.

As a solution to this problem, we describe our experiences working with PEML, the Programming Exercise Markup Language, which provides an easy to use, instructor friendly approach for writing programming assignments. Unlike tool-oriented data interchange formats, PEML is designed to provide a human friendly authoring format that has been developed to be intuitive, expressive and not be a technological or notational barrier to instructors.

We describe the design of PEML and also discuss its implementation as a programming library, a web application, and a microservice that provides full parsing and rendering capabilities for easy integration into any tools or scripting libraries. We also describe the integration of PEML into two automated testing and grading tools used at Virginia Tech by the CS department: Code Workout and Web-CAT. We then describe our experiences using PEML to describe a full range of programming assignments, laboratory exercises, and small coding questions of varying complexity in demonstrating the practicality of the notation.

We evaluate the feasibility of PEML using this encoding exercise as well as the effect of its integration into the aforementioned automated grading tools. We finally present a framework for integrating PEML into existing grading tools and then draw our conclusions as well as list down avenues PEML can be expanded into in the future.