Automatic unit testing and continuous integration

[Ivan_C]

Hello guys !

I would like to know if you have some experience about automatic unit testing. One of my clients is working on a continuous integration system to create the installers of their plug-ins when someone update the git repository, and we would like also to include some kind of automatic testing, both for the code and for the plug-ins themselves, to test a few things with tools like PluginVal or directly in some DAWs to check if everything is working properly. We don't know how to start there yet, so any suggestion of tools, report from your experience, or any source of information about the topic would be much appreciated. They use Gitlab for both git + CI.

Thanks in advance !

Ivan.

[BertKoor]

I do CI/CD daily, but with Java projects and with Jenkins. I've used GitLab, but only as a repo server, before it had the CI capabilities. The principles are the same though, regardless of underlying technology.

First the horses mouth (as they say) : https://about.gitlab.com/product/contin ... tegration/
That should get you started.

It's good practice to create unit tests for the lowest classes, covering their base functionality. Should be possible with C++. Probably you can let msbuild do the work, based on your VisualStudio project file.

True integration tests are harder, especially in a DAW, because you must run the tests headless on your build server. Also for a plugin, it's hard to assert the DAW and plugin do the right things and dance well together. GUIs are notoriously hard to test in CI-CD without either writing lots of code or making brittle tests that fail as soon as the slightest implementation detail has changed.

[hugoderwolf]

Basically you "just" need to make sure you can build and run from a command line, usually you'd have to write a couple of short scripts. Any testing would be run on command line as well. And of course you need a node that is capable of compiling this. So if you have any dependencies that have to be installed, you either need to set up a build node (probably a VM) or install that stuff on the ones that already exist.

As a start, having some unit tests and probably auval/pluginval running would be quite nice (but even just building on windows and mac simultaneously, if applicable, would be a start). The real power comes if you really have the complete release pipeline running, including testing on various supported platforms. Then you can very quickly spot if you develop on mac and break the code for the windows build. Usually you'd notice things like that much too late. Great for teams that work on different platforms!

[DJ Warmonger]

Unit testing for DSP and state machines is easy. It's integration tests with multiple DAWs what is difficult and probably can't be handled out-of-the-box.
One thing you can do, I suppose, is to unit-test VST interfaces to make sure they conform to official Steinberg specification. Reading these in detail will already get you far

You can try one of (free?) utilities to diagnose VSTs and use them in CI script, too. Unfortunatelly don't have examples at hand.

[Ivan_C]

Thanks for all the answers !

Currently, the CI allow us to compile the plug-ins in VST/VST3/AU/AAX for Windows + mac OS, to integrate the PACE protection system, and to generate the installers. I do run some tests manually from times to times using AUVal and PluginVal, and the first thing we are going to do is probably to put them in the CI as well. I was wondering if you have some experience with automated testing using directly the DAWs. I guess I'll have to include some DSP unit tests as well