Fast, Reliable Integration Tests with Spring Boot and Flyway

For integration tests to be effective, they must be reliable and fast. If tests take too long to run, nobody will run them. If tests are unreliable, everybody will ignore them.

To be reliable, tests may not result in false positives or negatives. Furthermore, they should not be flaky. That can be achieved by isolating the tests from each other, for example, by supplying a dedicated test database to each test with the help of Testcontainers. Additionally, tests should start from a known, consistent good state. That can be achieved by resetting the database to a known, good state before every test. Unfortunately, all that additional work makes tests slower.

In this post, we will look at various ways to write reliable integration tests in Spring Boot with the help of Flyway and Testcontainers. And then, we will look at which one is the fastest.

A working implementation of all approaches with the full source code can be found in my sample project on GitHub.

Setting the Stage

If we follow the Spring Boot documentation about writing tests with Testcontainers, we probably end up with something like the following test class to test an AuthorRepository that gives us access to book authors stored in the database:

@JdbcTest
@ContextConfiguration(
	initializers = AuthorRepositoryTest.class,
	classes = AuthorRepository.class
)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Commit
@Testcontainers
public class AuthorRepositoryTest
	implements ApplicationContextInitializer<ConfigurableApplicationContext>
{
	@Containers
	static final PostgreSQLContainer postgres = new PostgreSQLContainer("postgres:15.1");

	@Autowired
	AuthorRepository authorRepository;

	@Override
	public void initialize(@NotNull ConfigurableApplicationContext applicationContext) {
		TestPropertyValues
			.of(
				"spring.datasource.url=" + postgres.getJdbcUrl(),
				"spring.datasource.password=" + postgres.getPassword(),
				"spring.datasource.username=" + postgres.getUsername(),
				"spring.flyway.clean-disabled=false"
			)
			.applyTo(applicationContext);
	}

	@Test
	void add_insertsAuthor() {
		var author = new Author(AuthorId.from("82f4870d-f2e5-4a9f-a2b2-b297f66733a0"), "Brian Goetz");

		this.authorRepository.add(author);

		assertThat(this.authorRepository.findAll())
			.extracting(Author::name)
			.containsExactly("Bert Bates", "Brian Goetz", "Joshua Bloch", "Kathy Sierra", "Trisha Gee");
	}
}

Spring Boot automatically initialises the database before the first test method is run but does nothing after that. Every other test after add_insertsAuthor() has to somehow deal with another author’s presence (or absence, in the case of a test failure) in the database. That makes tests brittle.

One alternative could be that tests clean up after themselves. But that needs work and is problematic if we want to diagnose a test failure. Would it not be much better if the database was automatically brought into a consistent state before every test?

Spring Boot offers at least three different ways to do this that actually work:

• Using a custom FlywayMigrationStrategy.
• Invoking Flyway from a TestExecutionListener.
• Eschewing Flyway and using @Sql with a SQL script instead.

There are ways that do not work properly, too. For example, using a JUnit setup method (a method annotated with @BeforeEach in JUnit 5) to invoke Flyway makes @Sql unusable. The reason is that the TestExecutionListener that processes @Sql annotations runs before @BeforeEach. Thus, the database would still be empty when @Sql is applied.

The Options, Explained

FlywayMigrationStrategy

When Spring Boot starts, its autoconfiguration for Flyway invokes the registered FlywayMigrationStrategy to apply all pending migrations. By supplying your own FlywayMigrationStrategy, you can alter that behaviour and invoke clean() before migrate() and thereby reset the test database:

@Bean
public FlywayMigrationStrategy flywayMigrationStrategy() {
	return flyway -> {
		flyway.clean();
		flyway.migrate();
	};
}

As mentioned before, the FlywayMigrationStrategy is only applied when the application starts. To make Spring apply it again before each test, you have to trigger a context refresh by annotating the test class with @DirtiesContext(classMode = AFTER_EACH_TEST_METHOD).

This approach requires the least amount of code but is also the slowest. @DirtiesContext causes Spring to rebuild its context. That is an expensive operation, especially if you declare many beans or use multiple starters.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

Flyway with TestExecutionListener

TestExecutionListener is an API provided by Spring to react to test-execution events. For example, it allows running custom code before (or after) the execution of each test method. Therefore, it can serve as a replacement for FlywayMigrationStrategy without the need for a context refresh.

public class CleanDatabaseTestExecutionListener
	implements TestExecutionListener, Ordered
{
	@Override
	public void beforeTestMethod(@NotNull TestContext testContext) {
		var flyway = testContext.getApplicationContext().getBean(Flyway.class);
		flyway.clean();
		flyway.migrate();
	}

	@Override
	public int getOrder() {
		// Ensures that this TestExecutionListener is run before
		// SqlScriptExecutionTestListener which handles @Sql.
		return Ordered.HIGHEST_PRECEDENCE;
	}
}

Register the TestExecutionListener by annotating your test classes with @TestExecutionListeners:

@TestExecutionListeners(
	value = CleanDatabaseTestExecutionListener.class,
	mergeMode = MERGE_WITH_DEFAULTS
)

The last step is to register a FlywayMigrationStrategy that does nothing. Otherwise, the autoconfiguration would apply the migrations and the TestExecutionListener would apply them a second time before the first test runs.

@Bean
public FlywayMigrationStrategy flywayMigrationStrategy() {
	return flyway -> { /* Do nothing. */ };
}

While this approach requires significantly more code than FlywayMigrationStrategy, it is considerably faster, too, because we could get rid of the expensive context refresh after each test.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

@Sql (Almost) Without Flyway

The previous approaches suffer from a common problem: They run Flyway before every test and have to apply each and every migration over and over again to bring the test database into a consistent state. That is fine if a project only has a few migrations. But as soon as there are more than a couple of migrations, their execution time starts adding up. I pulled up an old project with around 50 migrations and measured how long they take to run: between 1 and 2 seconds in total. There are about 600 test methods across all integration tests. Therefore, a single build spends between 10 and 20 CPU minutes applying migrations alone – what a waste.

The solution is to remove Flyway from the integration tests:

1. Apply all migrations on an empty database, for example, with the help of Flyway’s plug-ins for Gradle and Maven.

2. Dump the database to a SQL script with pg_dump, mysqldump, and so on. Specify the necessary options so that the generated script includes the commands to drop all database objects if they already exist. That ensures that the database does not contain any leftovers from previous test runs.

3. Use the SQL script in the integration tests to reinitialise the database before every test instead of Flyway:

   @JdbcTest(properties = "spring.flyway.enabled=false")
   @Sql(scripts = "/db.sql", config = @SqlConfig(transactionMode = ISOLATED))
   // Other annotations
   public class AuthorRepositoryTest {
   	// Tests
   }
   

This approach significantly cuts down on the total runtime because it creates all database objects once per test without replaying the entire migration history.

Unfortunately, this speed-up comes at a cost: The SQL script needs to be checked into source control. And someone needs to keep the database dump in sync with the Flyway migrations. However, it is possible to avoid the maintenance burden by automating the generation of the database dump. Please see Keeping the Growing Number of Flyway Migrations in Check for how to do this.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

Performance

I did some measurements to give you a very rough idea of how the various approaches perform relative to each other:

The numbers are indicative at best because they highly depend on the specifics of your project and the hardware. For example, if your application defines a lot of beans or uses many starters, the FlywayMigrationStrategy will perform worse because the context refresh takes even longer. Therefore, the gap between it and the other two methods widens. If you have many Flyway migrations, @Sql without Flyway will be even faster.

Alternatives

Rollback

One popular method to keep the test database in a consistent state is to automatically rollback the transaction after each test. It is even the default behaviour of the testing support in Spring Framework.

I believe it is not wise to rely on rollback because it might hide errors. Database systems do not necessarily check every constraint after every statement but might defer it until later in the transaction. One example is PostgreSQL’s behaviour with ON DELETE NO ACTION.

Another drawback of testing with rollback is that it might hinder or even make it impossible to test complex scenarios. For example, in batch processing it is common that the transaction is committed after each batch has been processed. How are you supposed to test that only the last (failing) batch is discarded with rollback testing?

Baseline Migrations

Flyway Teams Edition, a paid version of Flyway, offers Baseline Migrations ¹. In a nutshell, you can define a separate SQL script that brings a database to the state of a specific version (the baseline version), skipping all migrations up to and including the baseline version.

If you keep only a handful of migrations around, using Flyway with baseline migrations should perform similarly to @Sql without Flyway. However, someone has to manually create the baseline migration. And you need Flyway Teams Edition.