December 7, 2009
Testing generated code
What is the accepted way to test generated source code? A recent work
project required me to generate some Java source code from small
definition files as part of a larger framework. The plan was for other
project teams to take the framework and use the generator to create
source which would become part of their project. I wanted to write unit
tests for the source code. The problem was how to write a test for code
that didn’t exist at the time the test was written because it hadn’t
been generated yet! I came up with a few options:
- Do a text comparison between generated code and the previously
created known text output of the generation process. If they are
exactly the same, the test passes. This tests the generation process
if correct but doesn’t test the logic of the generated code. If the
generated code has syntactic errors, compilers or IDEs will pick
this up quite fast and the developers in the other project will
complain quickly, but subtle logic could go unchecked for ages.
- Write a test that generates a class file from known inputs then
compiles it programmactically with the Java Compiler
then loads the class into the JVM and runs tests against it. This
would be better than the previous idea, as it tests the
functionality of the result. This was my first choice solution.
However, getting the Java Compiler API to work can be a bit hard.
There was a paucity of documentation on this and in the end I
couldn’t get it to work within time constraints.
- Generate the test at the same time as the generated code and pass
them both to the other project. The other project would then test
the generated code as if it was their code. This felt wrong to me.
No other generations systems do this - it is like passing the buck.
If there is an error in the generated code then the generator needs
to be fixed not the generated class!
- A 2-step process - test the functionality and generation process
separately. First, create a class that should be the same as a
generated class given known inputs (probably by generating it!) and
write tests that work against that class. Thus the problem of
testing not yet existing code is solved by using some code generated
earlier. This tests the generated code’s functionality as long as
the generating process doesn’t change. So the second part is to test
the generation process hasn’t changed using the first test in this
list. Add another test to generate the code and check that it is
exactly the same text as the class the rest of the test is written
against. If the generation output changes then you have to modify
the pre-generated class, and thus change the functionality test if