Fabric8 supports integration testing of Apps using Kubernetes via fabric8-arquillian which provides an extension for Arquillian.
Using this extension you can easily:
fabric8-arquillian uses:
The following code provides a sample system test:
import io.fabric8.kubernetes.client.KubernetesClient;
import org.jboss.arquillian.junit.Arquillian;
import org.jboss.arquillian.test.api.ArquillianResource;
import org.junit.Test;
import org.junit.runner.RunWith;
import static io.fabric8.kubernetes.assertions.Assertions.assertThat;
@RunWith(Arquillian.class)
public class KubernetesIntegrationKT {
@ArquillianResource
KubernetesClient client;
@Test
public void testRunningPodStaysUp() throws Exception {
assertThat(client).deployments().pods().isPodReadyForPeriod();
}
}
This will assert that the current project's Deployment, ReplicaSet or ReplicationController creates at least one pod; that it becomes Ready within a time period (30 seconds by default), then that the pod keeps being Ready for a period (defaults to 10 seconds).
This may seem a fairly simple test case; but it catches most errors with the image or Deployment being invalid or failing to start; the pod starting then failing due to some configuration issue etc. However you can then add extra custom assertions and code after this initial assertion!
If your application uses liveness checks (which are used by default with Spring Boot apps) then this test also asserts that the liveness checks keep valid for the period too. So if your application fails to connect to a database or your Camel route fails to start a route or whatever; then the test fails!
This means to improve your system tests you can just improve your liveness checks; which also helps Kubernetes manage your production environment too!
Often if you want to test, say, a messaging client you need to also provision a message broker. You can configure various settings for this in your src/test/resources/arquillian.xml file but the easiest thing is to refer to directories or maven dependencies for those dependent JSON/YAML files for the kubernetes resources you want to deploy to your test namespace first before you actual app gets deployed and tested.
Here is an example which deploys a Fabric8 Message Broker, store this in src/test/resources/arquillian.xml in your project:
<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://jboss.org/schema/arquillian"
xsi:schemaLocation="http://jboss.org/schema/arquillian
http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
<extension qualifier="kubernetes">
<property name="env.dependencies">mvn:io.fabric8.ipaas.packages/messaging/${fabric8-ipaas.version}/json/kubernetes</property>
</extension>
</arquillian>
Note to refer to version properties in your XML you need to make sure you enabled filtering in your pom.xml
<build>
<testResources>
<testResource>
<directory>src/test/resources</directory>
<filtering>true</filtering>
</testResource>
</testResources>
...
The kubernetes configuration is applied once per test suite. This means that the environment is getting created once per test suite and then multiple test cases are run against that environment. To encapsulate everything that has been applied to kubernetes as part of the current test suite, the notion of "session" is used.
In order to distinguish which pods, services and replication controllers have been created as part of the testing session, anything that is created using the arquillian extension, will be created inside a unique namespace per session. This namespace is also going to be used to cleanup everything after the end of the suite or upon termination of the process.
The Session is also made available to the test cases as an arquillian resource (see below).
If a test fails you may want to look inside the Kubernetes namespace took at the failed pods. By default the namespace gets garbage collected immediately.
To delay this you can use these environment variables
A typical integration test, would need to apply the kubernetes configuration, wait for everything to start up and then assert that that expected pods and services are available and of course test the actual services.
To do that, the test needs to have access to information like:
Each of the items above is made available to the test as an arquillian resource.
To obtain a reference to the KubernetesClient:
@ArquillianResource
io.fabric8.kubernetes.api.KubernetesClient client;
To obtain the list of all services created in the current session:
@ArquillianResource
io.fabric8.kubernetes.api.model.ServiceList sessionServices;
To obtain a reference to a particular service created in the current session:
@Id("my-service-id")
@ArquillianResource
io.fabric8.kubernetes.api.model.Service myService;
To obtain the list of all replication controllers created in the current session:
@ArquillianResource
io.fabric8.kubernetes.api.model.ReplicationControllerList sessionControllers;
To obtain a refernce to a particular replication controller created in the current session:
@Id("my-controller-id")
@ArquillianResource
io.fabric8.kubernetes.api.model.ReplicationController myController;
To obtain the list of all pods created in the current session:
@ArquillianResource
io.fabric8.kubernetes.api.model.PodList sessionPods;
To obtain the Session:
@ArquillianResource
io.fabric8.arquillian.kubernetes.Session mySession;
Any configuration option can be provided as an environment variable, system property or arquillian property. For example:
<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://jboss.org/schema/arquillian"
xsi:schemaLocation="http://jboss.org/schema/arquillian
http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
<extension qualifier="kubernetes">
<property name="masterUrl">true</property>
</extension>
</arquillian>
The only required configuration option is url to the kubernetes master. Since in most cases this will be specified as KUBERNETES_MASTER env variable, there is no need to specify it again. This means that the arquillian.xml configuration file is completely optional.
Supported options:
masterUrl: The url to the kubernetes master.configFileName: If a url hasn't been explicitly specified, the configFileName can be used for discovery of the configuration in the classpath.configUrl: The url to the kubernetes configuration to be tested.connectToServices: Whether or not an attempt is made to connect to a service port; failing the test if it can't be connected. This is disabled by default since its likely a service ClusterIP / port cannot be opened by the JUnit test case (and may require authentication)env.dependencies: A space separated list of directories, files or urls to kubernetes configurations that are required to be applied before the current one.waitForServiceConnection: Wait until a network connection to all applied services is possible.serviceConnectionTimeout: The connection timeout for each attempt to "connect to the service".waitForServices: Explicitly specify which services to wait. If this option is ommitted or empty all services will be waited.timeout: The total amount of time for to wait for pods and service to be ready.pollInterval: The interval between polling the status of pods and services.If dependencies have not been explicitly specified, the extension will find all test scoped dependencies and search inside those artifacts for kubernetes.json resources. Then it will treat those as dependencies and apply them along with the configuration.
One thing to be careful of is that adding test dependencies on kubernetes.json will add transitive dependencies to your test project; so you may want to add exclusions as this example shows.
There are lots of examples in the quickstarts.
Here is an example Arquillian Fabric8 integration test that tests that an AMQ broker, producer and consumer startup and properly produce and consume messages correctly.
In particular here is the code that does JMX assertions via jolokia on the containers that are created by the integration test.
When running an fabric8-arquillian integration test then the environment variable KUBERNETES_MASTER needs to be specified to point to the kubernetes environment in which to create the containers and services.
Also you may want to set the KUBERNETES_TRUST_CERTIFICATES variable to allow connection to kubernetes without a client certificatE:
export KUBERNETES_TRUST_CERTIFICATES=true
export KUBERNETES_MASTER=http://localhost:8443
Note that this can be any kubernetes environment (a kubernetes installation, OpenShift, RHEL Atomic or GKE). Also note that different integration tests can be running at the same time on the same kubernetes environment.
If you find that you are getting exceptions and output of the form:
Waiting for: SomeException....
Waiting for: SomeException....
and if the exception isn't clear from the message, you could define this environment variable to get an additional full stack trace (which is very noisy but can be handy):
export FABRIC8_VERBOSE_ASSERT=true
The following libraries are provided to help you create concise assertions using assertj based DSL:
To be able to use this library add this to your Apache Maven based project add this into your pom.xml
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>fabric8-arquillian</artifactId>
<version>2.2.101</version>
<scope>test</scope>
</dependency>