Letting AI write the test data

The most laborious part of writing tests is deciding what inputs to supply. Aim for coverage and the numbers explode; narrow the set and things slip through.

This work generates test data automatically using generative adversarial networks (GAN). A GAN trains two sides against each other — one producing plausible data, the other telling real from fake — and is best known for generating images. Here it is applied to generating test inputs.

Trained on past test data, it produces large quantities of plausible inputs, including combinations a person is unlikely to think of, which is what leads to finding defects.

The work has since developed towards diffusion models, and it marks the starting point of our use of generative AI in software testing.