December 7, 2009

Testing generated code

Tags: Java, Technical

What is the accepted way to test generated source code? A recent work project required me to generate some Java source code from small definition files as part of a larger framework. The plan was for other project teams to take the framework and use the generator to create source which would become part of their project. I wanted to write unit tests for the source code. The problem was how to write a test for code that didn’t exist at the time the test was written because it hadn’t been generated yet! I came up with a few options:

  • Do a text comparison between generated code and the previously created known text output of the generation process. If they are exactly the same, the test passes. This tests the generation process if correct but doesn’t test the logic of the generated code. If the generated code has syntactic errors, compilers or IDEs will pick this up quite fast and the developers in the other project will complain quickly, but subtle logic could go unchecked for ages.
  • Write a test that generates a class file from known inputs then compiles it programmactically with the Java Compiler API then loads the class into the JVM and runs tests against it. This would be better than the previous idea, as it tests the functionality of the result. This was my first choice solution. However, getting the Java Compiler API to work can be a bit hard. There was a paucity of documentation on this and in the end I couldn’t get it to work within time constraints.
  • Generate the test at the same time as the generated code and pass them both to the other project. The other project would then test the generated code as if it was their code. This felt wrong to me. No other generations systems do this - it is like passing the buck. If there is an error in the generated code then the generator needs to be fixed not the generated class!
  • A 2-step process - test the functionality and generation process separately. First, create a class that should be the same as a generated class given known inputs (probably by generating it!) and write tests that work against that class. Thus the problem of testing not yet existing code is solved by using some code generated earlier. This tests the generated code’s functionality as long as the generating process doesn’t change. So the second part is to test the generation process hasn’t changed using the first test in this list. Add another test to generate the code and check that it is exactly the same text as the class the rest of the test is written against. If the generation output changes then you have to modify the pre-generated class, and thus change the functionality test if required. Easy.