Chapter 6. 10h00 - nfsim: Untested Code is Buggy Code

This was a very interesting talk in which Rusty emphasized the need for more vigorous testing of kernel subsystems - in this case the netfilter subsystem. As he did two years ago when he gave the keynote address for OLS2003, he encouraged the audience to take a serious look at, and start using, various pieces of software used for debugging code. Both runtime and static analysis checkers were suggested such as:

• valgrind

• talloc

• findstatic.pl and

• gcov

The rest of his talk demonstrated and described the nfsim project. The nfsim project is a set of tools that are used to test the kernel's netfilter code. Basically the netfilter subsystem is taken from the kernel itself, compiled in a sterile environment, and then tests can be written which throw packets at the subsystem and record the results. It's a very interesting idea, and the netfilter package is lucky for the existence of the TAP interface which allows this to happen. It's too bad more kernel subsystems couldn't be sliced out from the kernel and tested in this way.

Rusty wanted to get developers thinking about all the decision points in their code, such as:

ret = vmalloc (...);
if (ret == 0) {
    A ();
}
else {
    B ();
}
       

void *
vmalloc (...) {
    ...
    if (should_i_fail (__func__))
        return NULL;
    ptr = malloc (...);
    ...
}
       

Additionally, in order to make this framework work he uses the fork() function in such a way that the child always runs the expected failure scenario and the parent always runs the expected success path.

One fabulous piece of wisdom Rusty imparted was to always double-check the results of failure tests. He pointed out that just because we expect a test to fail, when the failure occurs there's a tendency to assume it failed correctly in the manner in which it was supposed to fail. However it could so happen that the reason it failed has nothing to do with what we were expecting to fail but rather was to do with something completely unrelated to the thing we were expecting to fail in the first place. Therefore, double-check the results of those expected-to-fail tests to ensure that they failed for the reason we expected them too. As an example, let's say we want to test the scenario whereby a file write fails due to lack of disk space. So we setup some environment with little space left and run our test which opens a file and tries to write to it. The test fails. But it could have failed for any number of reasons such as no write permission, or an invalid path, not necessarily because the write failed to complete. So if this check isn't verified a developer might think this failure is gracefully handled when in fact it was never tested.