Grading MATLAB assignments automatically

Hi Everybody,

I had a kind of an odd question, as part of my job, I am required to grade 80 students MATLAB functions and scripts. Being the optimizing engineering student I hope to resemble, I thought there could be an easier way to do this than by examining all their code by hand and running each test automatically. Parsing certain sections of code would be really easy, but one of my questions I have been pondering is in regards to input.

I was wondering if there would be a way to have a script (GradingScript) run a different script (StudentScript) and have the GradingScript automatically provide arguments for the standard input prompts (response=input('Enter response: '); )

I know there are many ways around this, but I was curious if there was any way to actually do that.

Also, if you have any other ideas of how to grade introductory matlab assignments automatically, that would be awesome!

Thanks, Shaun

[Matt Fig]

There is an old NG thread discussing the dangers of doing this and strategies to avoid them. I can't remember the conclusion but Walter was part of the conversation. A search would probably find it.

[Scott Rowe]

I do this routinely and published a guide on the approach (LINK: A Matlab Assignment Framework For Engineering Education That Automates Grading).

Answer 1

[Jason Ross]

When I took programming classes in the past, the professor would specify what the function/method/CLI was supposed to look like and then use that to check results using a framework of their design. A big theme was that you should be able to program to a specified API without knowing who was calling you and give the correct result.

There would be sample inputs and outputs, for example "bar is 10 with the following four inputs", e.g.

bar = foo(1,2,4,5)

So your automation would just need to put your test case driver and the student's code in the same place and call them, or modify the path, etc. I also agree with Daniel that avoiding input is generally a better way to do programming. If they want to run tests interactively, that's fine, but it's far easier to do things like

bar = foo(0,0,0,0)
bar = foo(1,3,4,5)
bar = foo(10,10,10,10)
bar = foo(-4,5,0,1)

and just check that the output is correct rather than entering it manually every time. It also opens the door to using other input methods (GUI, the interactive prompt, some other program, an instrument, etc)

It would probably be a good idea to run a diff of each file versus all the others and catch the especially lazy cheaters.

I also agree with Jan that it's important to critique and review the code. Design review is a very real part of engineering, and they should get used to having other people review their code for aspects that are difficult or impossible to check automatically -- maintainability, readability, ease of debugging and so on.

Answer 2

[Cris LaPierre]

It's been a while since this question was asked, but for those reading who are interested in the topic, take a look at MATLAB Grader , a new product recently released by MathWorks.

Answer 3

[Walter Roberson]

If you are using Linux or OS-X, and there is no graphics, you should be able to run with no command line and redirect input from your test file.

matlab -nodesktop -r 'try; ./StudentScript; catch;end;quit'

[Shaun VanWeelden]

I would definitely not have thought of that, great suggestion, the only thing is we use almost exclusively windows computers.

[Walter Roberson]

I think you should be able to do this for MS Windows as well; the -r flag would be the same but the -nodesktop might have to change.

Answer 4

[Santosh Kasula]

I think what you are looking for is Cody. Cody is a puzzle solving game that challenges your skills and helps you expand your knowledge about MATLAB programming. When creating a problem, you are required to provide a test suite, so that each solution submitted to that problem gets evaluated automatically. Cody could address some of your auto-grading issues. Read more about Cody here.

[Shaun VanWeelden]

I am quite familiar with Cody, but ideally I would replicate something more or less similar to Cody, but personalized for the current university lesson plans we have now. Thanks for the input though!

Answer 5

[Daniel Shub]

I would not do this for a number of reasons.

Things like response=input('Enter response: '); are going to make automation hard. I would think they would be better off learning to write functions with well defined input arguments? This would make automation easier.

Second, sloppy/malicious coding could seriously damage your computer. Imagine the student who does adds

!rm -rf *.*

to the start of his script.

[Shaun VanWeelden]

You know, I hadn't even thought about that, good call!

Answer 6

[Jan]

Especially the keyword Cody should warn you, that every automatic grading system can be cheated. And if you have tought students, it will be cheated.

Some famous papers about coding theory have been based on the idea of checking automatically, if the code provided by students, contains infinite loops.

Inspecting the code of students is time consuming and sometimes tedious. But it is the ideal method to teach them to write clean code.

[Shaun VanWeelden]

That is a great point, it is pretty hard to automatically grade style and readability haha

[Jan]

But of course anybody should teach students to write clean code. I've seen too many students in my university and in the Matlab forums, who have learned their numeric lessons, but write such ugly code that it cannot be used for real world projects. And when they should apply their piece of work in a greater framework, e.g. as PhD in a workgroup of the university, they claim that their programming skills are high, because no professor or supervisor had ever critized the code or mentioned any potential improvement. In consequence I think, if the quality of the code cannot be graded automatically, automatical grading is not sufficient.

[Shaun VanWeelden]

I would agree, I think I am shifting more towards automatically graded practice sessions, and then still grading by hand the actual assignments. That is some really good feedback and things that definitely need to be considered.