Introducing jenvtest — Kubernetes API Server Integration Tests Made Easy for Java

We recently made the first release of jenvtest, under the umbrella of Java Operator SDK.

In this post I will explain the motivation behind this library, how it works, its benefits but also the compromises that it brings on the table.

What is jenvtest?

jenvtest is very similar to envtest for Golang. In fact is was motivated by the go counterpart and in the background even it uses some facilities created for envtest.

It is a library that makes it easy to implement integration tests against Kubernetes API Server. In the background it runs the API Server locally and manages all related components for you. However, at the end the usage is as easy as applying @EnableKubeAPIServer annotation to an unit test:

@EnableKubeAPIServer
class JUnitExtensionTest {

    @Test
    void testCommunication() {
        // use any Kubernetes client to communicate with the server
        var client = new KubernetesClientBuilder().build();
        client.resource(configMap()).createOrReplace();
        var cm = client.resource(configMap()).get();

        assertThat(cm).isNotNull();
    }

    private ConfigMap configMap() {
        return new ConfigMapBuilder()
                .withMetadata(new ObjectMetaBuilder()
                        .withName("test1")
                        .withNamespace("default")
                        .build())
                .withData(Map.of("key", "data"))
                .build();
    }

}

It will start the K8S API server, and update the Kubernetes config file at ~/.kube/config. From that point any client is able to communicate with the running server. Including kubectl or the Fabric8 Kubernetes Client in the example above.

How Does It Work

In the background a lot is going on:

1. Automatically downloads all the required binaries (if not present already) to ~/.jenvtest/k8s directory for the target platform (Linux, Mac or Windows). This includes the API Server, Etcd and kubectl.
2. Generates all the certificates required, such as certificates for API Server and for the client. Client’s target group is set to system:masters in the client certificate.
3. Runs etcd and Kubernetes API Server binaries.
4. Updates the mentioned kube config file, with the relevant information. So any client can be used from now on to communicate with the API Server.
5. Cleans etcd data on stop. So next time it will be started with clean state.

Note, that it does not start other parts of the control plane. It is not able to start pods, there are no nodes, but it is possible to do all the operations against the API Server that is possible on a real cluster.

The Nice Parts

The nice aspect about this approach is that it can be seamlessly used in a unit tests. There is no need to run Minikube or Kind, it is not even required to have docker installed on the machine. But still all (well most of, see below) the features of API Server works, it is possible to test controllers or any communication as it was a real Kubernetes deployment running.

It makes it trivial to test also Dynamic Admission Controllers (Mutation and Validation hooks) and conversion hooks locally. Although this is certainly possible with Minikube and/or Kind, the configuration might be different for different platforms. Since here is no virtual network involved, it is trivial to make a call to the service running locally.

The not so Nice Parts

One of the not so nice aspects is the startup time, now it takes little more than three seconds until the Kubernetes API is started and fully functional. Well, this is not great, not terrible — we also might be able to make this faster in the future. However in case it is started and stopped multiple times within a test suit, the overall execution duration of the test suit might increase significantly, depending on the setup.

Another problem is that garbage collection is implemented by a controller in Kubernetes, therefore cascading delete with owner references will not work. Similarly deleting namespaces is not working in this setup, but this is easy to fix just by creating a new namespace for every test, as it is automatically done by junit exension in JOSDK. More in depth description of these problems can be found in envtest docs.

Summary

Although this approach has some limitations it has some very nice benefits too. Conceptually sits between a mocked approach described here and the approach that runs Minikube or Kind locally. It probably depends case by case which approach to use. In future we will work on improvement of this library, will provide more configuration options and other enhancements.

And as always feedback is more than welcome!