Automatically launch docker-compose from Gradle for dev and integration tests

As applications get more complex with many moving parts, integration tests against the full-stack are becoming more critical. In this article, I’ll show how to use docker-compose in a Spring Boot application with Neo4j backend to aid both the development and the integration tests.

This blog post assumes that you already know what integration tests are, how Gradle works. Spring Boot and Neo4j are just example components for the project. The code for this post is available on Github.

Creating fresh Spring Boot application.

Before we start with integration tests, let’s create a simple project. In this project, we have a RESTful API that manages one resource. We call this resource Person, and it has only two properties (id and name).

I used Spring Initializer to generate the skeleton project. You can get your copy from here. Since this blog post is not focused on building Spring Boot applications, you can download the needed skeleton from Github or develop your own code. I also built some unit tests for the project, but this is out of scope for this blog post.

Building Integration Tests

Our unit tests all passed, but does our project really work? we haven’t tested it against the backend yet!

To build integration tests, we need four things:

1. Write the tests
2. Configure Gradle to run them
3. Configure Gradle to launch the backend automatically for the integration tests.
4. Run the tests

Step 1: Writing the integration tests

I won’t get into the details of how to write integration tests. The integration tests are places at src/integration. But you can copy the code from Github. Two important things to note here: first, we clean up the DB after each test, and second, we configure the connection to Neo4j in src/integration/application.yml. The configurations look like this:

spring:
  neo4j:
    uri: "bolt://localhost:8687"
    authentication:
      username: neo4j
      password: password

Step 2: Configure Gradle to the integration tests

By default, Gradle won’t recognize the new source directory for the integration tests. In this step we add the new integration tests directory to the sourceSets of Gradle, configure the new source set, and define a task to run the integration tests.

a. Defining new sourceSet in build.gradle:

sourceSets {
    integrationTest {
        java {
            srcDir "src/integration/java"
            compileClasspath += main.output + test.output
            runtimeClasspath += main.output + test.output
        }
        resources.srcDir file("src/integration/resources")
    }
}

b. Configuring the integration sourceSetin build.gradle:

We configure the integration tests source to inherit the configurations from implementation and testImplementation. Hence, all the dependencies will be available in the integration tests.

configurations {
    compileOnly {
        extendsFrom annotationProcessor
    }
    integrationTestImplementation.extendsFrom implementation
    integrationTestImplementation.extendsFrom testImplementation
    integrationTestRuntime.extendsFrom testRuntime
}

c. Defining an integration tests task in build.gradle:

Finally, we define a new task called integrationTest to run the integration tests. In this task, we use the JUnit platform to drive the tests and specify that integration tests should run after unit tests.

task integrationTest(type: Test) {
    group 'Test'
    useJUnitPlatform()
    mustRunAfter test
    testClassesDirs = sourceSets.integrationTest.output.classesDirs
    classpath = sourceSets.integrationTest.runtimeClasspath
}

Step 3: Integrating docker-compose

Our integration tests rely on the Neo4j instance to be available. Then we configure Gradle to run it before every test.

a. Defining a docker-compose file in src/integration/resources/docker-compose.yml:

Note the specified exposed ports and credentials configured for Neo4j in the docker-compose file.

version: '3'
services:
  neo4j:
    image: neo4j:4.2.5
    hostname: neo4j
    container_name: neo4j
    ports:
      - "8474:7474"
      - "8687:7687"
    environment:
      NEO4J_AUTH: neo4j/password
      NEO4J_dbms_memory_heap_max__size: 256M
      NEO4J_dbms_logs_debug_level: DEBUG

b. Configuring Gradle to run docker-compose for the integration tests:

For this purpose, I used com.avast.gradle.docker-compose plugin. It’s easy to configure. Just add it to the plugins in build.gradle:

plugins {
    id 'org.springframework.boot' version '2.4.4'
    id 'io.spring.dependency-management' version '1.0.11.RELEASE'
    id 'java'
    id 'idea'
    id 'com.avast.gradle.docker-compose' version "0.14.2"
}

And then configure it to run for the integration tests:

dockerCompose {
    integration {
        // Define the docker compose file for integration testing
        useComposeFiles = ["${buildDir}/resources/integrationTest/docker-compose.yml"]
        isRequiredBy(integrationTest)
    }
}

Step 4: Running the integration tests

Running the integration tests is as simple as ./gradlew integrationTest. Gradle will automatically launch the needed Neo4j instance and tearing it down once the tests are finished.

You can aslo manually start/shutdown the containers with ./gradlew integrationComposeUp or ./gradlew integrationComposeDown.

Running docker-compose for dev

In the fast development cycle, I would like to run the application locally and experiment with different approaches before I commit the code. But managing external dependencies is challenging. For my project, I applied a similar approach to the integration tests. I created a docker-compose file in src/main/resources/docker-compose.yml that ran on a different port than the integration tests. The docker-compose file is as follows:

version: '3'
services:
  neo4j:
    image: neo4j:4.2.5
    hostname: neo4j
    container_name: neo4j
    ports:
      - "7474:7474"
      - "7687:7687"
    environment:
      NEO4J_AUTH: neo4j/password
      NEO4J_dbms_memory_heap_max__size: 256M
      NEO4J_dbms_logs_debug_level: DEBUG

Then extended the dockerCompose config in build.gradle to:

dockerCompose {
    dev {
        // Specify that bootRun depends on dockerCompose
        useComposeFiles = ["${buildDir}/resources/main/docker-compose.yml"]
        isRequiredBy(bootRun)
    }
    integration {
        // Define the docker compose file for integration testing
        useComposeFiles = ["${buildDir}/resources/integrationTest/docker-compose.yml"]
        isRequiredBy(integrationTest)
    }
}

Now, i can run it manually with ./gradlew devComposeUp and ./gradlew devComposeDown.

Limitations and room for improvements

1. I fixed the expose ports for Neo4j. This is not good practice! Especially if your CI/CD system is running multiple tests on different branches at the same time. The gradle-docker-compose-plugin can randomly assign ports to avoid conflict, then you can read them as environment variables in your tests. I was just lazy to do so for this blogpost 🙈
2. Testcontainers provides an alternative approach to this method and integrates more into your java tests. I’ve not explored that project in detail yet.