The goal of this article is teach you how to implement integrations tests in applications that use bigquery. The reason of that is that I need to implement this recently and I would like register my knowledge.

After finish this article you are prepared to:

• Implementing tests with Tests containers
• Make basic operations with Google Bigquery

I assumed you know how to connect with bigquery using java, although if don't you can try see this class.

So, let's go!

Dependencies

For this article you can need create a basic maven/gradle Spring Boot project and add the bellow dependencies:

implementation 'com.google.cloud:google-cloud-bigquery'
testImplementation 'org.testcontainers:junit-jupiter'
testImplementation 'org.testcontainers:gcloud'

The first is used to connect on Google Bigquery and the lasts two is used to provide containers for tests programmatically.

Insert data on Bigquery dataset

Before the tests (sorry, it isn't TDD hahaha) we need to create code for test! So I'll create a service for insert and list data of the bigquery dataset. It's too easy!

@Service
@RequiredArgsConstructor
public class BigQueryService {
    String tableName = "teste";

    private final BigQuery bigquery;
    private final BigQueryProperties properties;

    public InsertAllResponse insertData(Map<String, String> data) {
        var tableId = TableId.of(properties.getProjectId(), properties.getDataset(), tableName);
        var contentForInsert = InsertAllRequest
                .newBuilder(tableId)
                .addRow(data).build();

        return bigquery.insertAll(contentForInsert);
    }

    public List<HashMap<String, String>> getData() throws InterruptedException {
        var fullTableName = String.format("%s.%s.%s", properties.getProjectId(), properties.getDataset(), tableName);
        var query = "SELECT * FROM `" + fullTableName + "` LIMIT 100";

        var results = bigquery.query(QueryJobConfiguration.of(query));
        var schemaResult = results.getSchema();

        return results.streamValues().map(fieldValues -> {
            var map = new HashMap<String, String>();
            IntStream.range(0, fieldValues.size()).forEach(index -> {
                var columnName = schemaResult.getFields().get(index).getName();
                map.put(columnName, fieldValues.get(index).getStringValue());
            });
            return map;
        }).toList();
    }
}

Bigquery container

Now I'll start to create my integration test for my service BigQueryService. Basically I need to instantiate a test container with to bigquery in my class test and set up my dataset and my table in my local bigquery before each execution. For create the bigquery container you must declare:

@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class BigQueryIntegrationTest {
    @Container
    static BigQueryEmulatorContainer bigQueryContainer = new BigQueryEmulatorContainer("ghcr.io/goccy/bigquery-emulator:0.4.3");
}

The attribute bigQueryContainer was decelerated as static with the annotation @Container for the container starting one time for all tests. If you want a new container for each test you must remove the static declaration. The type of them is BigQueryEmulatorContainer, a class of org.testcontainers:gcloud dependency.

Furthermore, I need to replace my configuration properties for my service get a connection with my fake bigquery. I can do this using the annotation @DynamicPropertySource and replacing my properties values as bellow:

@DynamicPropertySource
static void setProperties(DynamicPropertyRegistry registry) {
    registry.add("gcp.bigquery.project-id", bigQueryContainer::getProjectId);
    registry.add("gcp.bigquery.host", bigQueryContainer::getEmulatorHttpEndpoint);
    registry.add("gcp.bigquery.dataset", () -> "local-bigquery");
}

These properties are custom and mapped for me in this class.