Spring Boot

How to instrument Spring Boot with OpenTelemetry

The OpenTelemetry Java agent with byte code instrumentation can cover most of your needs when instrumenting Spring Boot applications.

Alternatively, the OpenTelemetry Spring Boot starter can help you in the following cases:

• Spring Boot Native image applications for which the OpenTelemetry Java agent does not work
• Startup overhead of the OpenTelemetry Java agent exceeds your requirements
• OpenTelemetry Java agent might not work if your application already uses another Java monitoring agent
• You can use the Spring Boot configuration files (application.properties, application.yml) to configure the OpenTelemetry Spring Boot starter which doesn’t work with the OpenTelemetry Java agent

The OpenTelemetry Java agent has more automatic instrumentation features than the OpenTelemetry starter.

You can use OpenTelemetry instrumentations libraries to complete the automatic instrumentation of the Spring Boot starter.

OpenTelemetry Spring Boot starter

Compatibility

The OpenTelemetry Spring Boot starter works with Spring Boot 2.0 and 3.0, and Spring Boot native image applications. The opentelemetry-java-examples/spring-native repository contains an example of a Spring Boot Native image application instrumented using the OpenTelemetry Spring Boot starter.

Dependency management

A Bill of Material (BOM) ensures that versions of dependencies (including transitive ones) are aligned.

Importing the opentelemetry-bom and opentelemetry-instrumentation-bom-alpha BOMs when using the OpenTelemetry starter is important to ensure version alignment across all OpenTelemetry dependencies.

The following example shows how to import both BOMs using Maven:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-bom</artifactId>
            <version>1.38.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
            <version>2.4.0-alpha</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

With Gradle and Spring Boot, you have two ways to import a BOM.

You can use the Gradle’s native BOM support by adding dependencies:

import org.springframework.boot.gradle.plugin.SpringBootPlugin

plugins {
  id("java")
  id("org.springframework.boot") version "3.2.O"
}

dependencies {
  implementation(platform(SpringBootPlugin.BOM_COORDINATES))
  implementation(platform("io.opentelemetry:opentelemetry-bom:1.38.0"))
  implementation(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.4.0-alpha"))
}

The other way with Gradle is to use the io.spring.dependency-management plugin and to import the BOMs in dependencyManagement:

plugins {
  id("java")
  id("org.springframework.boot") version "3.2.O"
  id("io.spring.dependency-management") version "1.1.0"
}

dependencyManagement {
  imports {
    mavenBom("io.opentelemetry:opentelemetry-bom:1.38.0")
    mavenBom("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.4.0-alpha")
  }
}

OpenTelemetry Starter dependency

Add the dependency given below to enable the OpenTelemetry starter.

The OpenTelemetry starter uses OpenTelemetry Spring Boot autoconfiguration.

• Maven (pom.xml)
• Gradle (gradle.build)

<dependencies>
  <dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-spring-boot-starter</artifactId>
  </dependency>
</dependencies>

dependencies {
	implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")
}

Configuration

This spring starter supports configuration metadata, which means that you can see and autocomplete all available properties in your IDE.

General configuration

The OpenTelemetry Starter supports all the SDK Autoconfiguration (since 2.2.0).

You can update the configuration with properties in the application.properties or the application.yaml file, or with environment variables.

application.properties example:

otel.propagators=tracecontext,b3
otel.resource.attributes.deployment.environment=dev
otel.resource.attributes.service.name=cart
otel.resource.attributes.service.namespace=shop

application.yaml example:

otel:
  propagators:
    - tracecontext
    - b3
  resource:
    attributes:
      deployment.environment: dev
      service:
        name: cart
        namespace: shop

Environment variables example:

export OTEL_PROPAGATORS="tracecontext,b3"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=dev,service.name=cart,service.namespace=shop"

Overriding Resource Attributes

As usual in Spring Boot, you can override properties in the application.properties and application.yaml files with environment variables.

For example, you can set or override the deployment.environment resource attribute (not changing service.name or service.namespace) by setting the standard OTEL_RESOURCE_ATTRIBUTES environment variable:

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod"

Alternatively, you can use the OTEL_RESOURCE_ATTRIBUTES_DEPLOYMENT_ENVIRONMENT environment variable to set or override a single resource attribute:

export OTEL_RESOURCE_ATTRIBUTES_DEPLOYMENT_ENVIRONMENT="prod"

The second option supports SpEL expressions.

Note that DEPLOYMENT_ENVIRONMENT gets converted to deployment.environment by Spring Boot’s Relaxed Binding.

Disable the OpenTelemetry Starter

Programmatic configuration

You can use the AutoConfigurationCustomizerProvider for programmatic configuration. Programmatic configuration is recommended for advanced use cases, which are not configurable using properties.

Exclude actuator endpoints from tracing

As an example, you can customize the sampler to exclude health check endpoints from tracing:

• Maven (pom.xml)
• Gradle (gradle.build)

<dependencies>
  <dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-samplers</artifactId>
    <version>1.33.0-alpha</version>
  </dependency>
</dependencies>

dependencies {
  implementation("io.opentelemetry.contrib:opentelemetry-samplers:1.33.0-alpha")
}

import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.contrib.sampler.RuleBasedRoutingSampler;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.semconv.SemanticAttributes;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class OpenTelemetryConfig {

  @Bean
  public AutoConfigurationCustomizerProvider otelCustomizer() {
    return p ->
        p.addSamplerCustomizer(
            (fallback, config) ->
                RuleBasedRoutingSampler.builder(SpanKind.SERVER, fallback)
                    .drop(SemanticAttributes.URL_PATH, "^/actuator")
                    .build());
  }
}

Configure the exporter programmatically

You can also configure OTLP exporters programmatically. This configuration replaces the default OTLP exporter and adds a custom header to the requests.

import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import java.util.Collections;
import java.util.Map;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class OpenTelemetryConfig {

  @Bean
  public AutoConfigurationCustomizerProvider otelCustomizer() {
    return p ->
        p.addSpanExporterCustomizer(
            (exporter, config) -> {
              if (exporter instanceof OtlpHttpSpanExporter) {
                return ((OtlpHttpSpanExporter) exporter)
                    .toBuilder().setHeaders(this::headers).build();
              }
              return exporter;
            });
  }

  private Map<String, String> headers() {
    return Collections.singletonMap("Authorization", "Bearer " + refreshToken());
  }

  private String refreshToken() {
    // e.g. read the token from a kubernetes secret
    return "token";
  }
}

Resource Providers

The OpenTelemetry Starter includes the same resource providers as the Java agent:

• Common resource providers
• Resource providers that are disabled by default

In addition, the OpenTelemetry Starter includes the following Spring Boot specific resource providers:

Distribution Resource Provider

FQN: io.opentelemetry.instrumentation.spring.autoconfigure.resources.DistroVersionResourceProvider

Spring Resource Provider

FQN: io.opentelemetry.instrumentation.spring.autoconfigure.resources.SpringResourceProvider

Service name

Using these resource providers, the service name is determined by the following precedence rules, in accordance with the OpenTelemetry specification:

1. otel.service.name spring property or OTEL_SERVICE_NAME environment variable (highest precedence)
2. service.name in otel.resource.attributes system/spring property or OTEL_RESOURCE_ATTRIBUTES environment variable
3. spring.application.name spring property
4. build-info.properties
5. Implementation-Title from META-INF/MANIFEST.MF
6. The default value is unknown_service:java (lowest precedence)

Use the following snippet in your pom.xml file to generate the build-info.properties file:

• Maven (pom.xml)
• Gradle (gradle.build)

<build>
    <finalName>${project.artifactId}</finalName>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <executions>
                <execution>
                    <goals>
                        <goal>build-info</goal>
                        <goal>repackage</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

springBoot {
  buildInfo {
  }
}

Automatic instrumentation

Automatic instrumentation is available for several frameworks:

JDBC Instrumentation

Logback

You can enable experimental features with system properties to capture attributes :

Alternatively, you can enable these features by adding the OpenTelemetry Logback appender in your logback.xml or logback-spring.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
            </pattern>
        </encoder>
    </appender>
    <appender name="OpenTelemetry"
        class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <captureExperimentalAttributes>false</captureExperimentalAttributes>
        <captureCodeAttributes>true</captureCodeAttributes>
        <captureMarkerAttribute>true</captureMarkerAttribute>
        <captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
        <captureLoggerContext>true</captureLoggerContext>
        <captureMdcAttributes>*</captureMdcAttributes>
    </appender>
    <root level="INFO">
        <appender-ref ref="console"/>
        <appender-ref ref="OpenTelemetry"/>
    </root>
</configuration>

Spring Web Autoconfiguration

Provides autoconfiguration for the RestTemplate trace interceptor defined in opentelemetry-spring-web-3.1. This autoconfiguration instruments all requests sent using Spring RestTemplate beans by applying a RestTemplate bean post processor. This feature is supported for spring web versions 3.1+. To learn more about the OpenTelemetry RestTemplate interceptor, see opentelemetry-spring-web-3.1.

The following ways of creating a RestTemplate are supported:

@Bean
public RestTemplate restTemplate() {
    return new RestTemplate();
}

public MyService(RestTemplateBuilder restTemplateBuilder) {
    this.restTemplate = restTemplateBuilder.build();
}

The following ways of creating a RestClient are supported:

@Bean
public RestClient restClient() {
    return RestClient.create();
}

public MyService(RestClient.Builder restClientBuilder) {
    this.restClient = restClientBuilder.build();
}

Spring Web MVC Autoconfiguration

This feature autoconfigures instrumentation for Spring WebMVC controllers by adding a telemetry producing servlet Filter bean to the application context. The filter decorates the request execution with a server span, propagating the incoming tracing context if received in the HTTP request. To learn more about the OpenTelemetry Spring WebMVC instrumentation, see the opentelemetry-spring-webmvc-5.3 instrumentation library.

Spring WebFlux Autoconfiguration

Provides autoconfigurations for the OpenTelemetry WebClient ExchangeFilter defined in opentelemetry-spring-webflux-5.3. This autoconfiguration instruments all outgoing HTTP requests sent using Spring’s WebClient and WebClient Builder beans by applying a bean post processor. This feature is supported for spring webflux versions 5.0+. For details, see opentelemetry-spring-webflux-5.3.

The following ways of creating a WebClient are supported:

@Bean
public WebClient webClient() {
    return WebClient.create();
}

public MyService(WebClient.Builder webClientBuilder) {
    this.webClient = webClientBuilder.build();
}

Kafka Instrumentation

Provides autoconfiguration for the Kafka client instrumentation.

Micrometer Instrumentation

Provides autoconfiguration for the Micrometer to OpenTelemetry bridge.

MongoDB Instrumentation

Provides autoconfiguration for the MongoDB client instrumentation.

R2DBC Instrumentation

Provides autoconfiguration for the OpenTelemetry R2DBC instrumentation.

Additional Instrumentations

Log4j2 Instrumentation

You have to add the OpenTelemetry appender to your log4j2.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
    <Appenders>
        <OpenTelemetry name="OpenTelemetryAppender"/>
    </Appenders>
    <Loggers>
        <Root>
            <AppenderRef ref="OpenTelemetryAppender" level="All"/>
        </Root>
    </Loggers>
</Configuration>

You can find more configuration options for the OpenTelemetry appender in the Log4j instrumentation library.

Instrumentation Annotations

This feature uses spring-aop to wrap methods annotated with @WithSpan in a span. The arguments to the method can be captured as attributed on the created span by annotating the method parameters with @SpanAttribute.

Note: this annotation can only be applied to bean methods managed by the spring application context. To learn more about aspect weaving in spring, see spring-aop.

import org.springframework.stereotype.Component;

import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;

/**
 * Test WithSpan
 */
@Component
public class TracedClass {

    @WithSpan
    public void tracedMethod() {
    }

    @WithSpan(value="span name")
    public void tracedMethodWithName() {
        Span currentSpan = Span.current();
        currentSpan.addEvent("ADD EVENT TO tracedMethodWithName SPAN");
        currentSpan.setAttribute("isTestAttribute", true);
    }

    @WithSpan(kind = SpanKind.CLIENT)
    public void tracedClientSpan() {
    }

    public void tracedMethodWithAttribute(@SpanAttribute("attributeName") String parameter) {
    }
}

OpenTelemetry instrumentations libraries

You can configure other instrumentations with OpenTelemetry instrumentations libraries.

Other solutions

Instead of using the OpenTelemetry Spring starter, you can use the OpenTelemetry autoconfiguration features with an annotation or the Zipkin starter.

Spring support

Autoconfiguration is natively supported by Spring Boot applications. To enable these features in “vanilla” use @EnableOpenTelemetry to complete a component scan of this package.

import io.opentelemetry.instrumentation.spring.autoconfigure.EnableOpenTelemetry;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableOpenTelemetry
public class OpenTelemetryConfig {}

Zipkin starter

OpenTelemetry Zipkin Exporter Starter is a starter package that includes the opentelemetry-api, opentelemetry-sdk, opentelemetry-extension-annotations, opentelemetry-logging-exporter, opentelemetry-spring-boot-autoconfigurations and spring framework starters required to setup distributed tracing. It also provides the opentelemetry-exporters-zipkin artifact and corresponding exporter autoconfiguration. Check out opentelemetry-spring-boot-autoconfigure for the list of supported libraries and features.

If an exporter is present in the classpath during runtime and a spring bean of the exporter is missing from the spring application context, an exporter bean is initialized and added to a simple span processor in the active tracer provider. Check out the implementation here.

• Maven (pom.xml)
• Gradle (gradle.build)

<dependencies>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
    <version>1.38.0</version>
  </dependency>
</dependencies>

dependencies {
  implementation("io.opentelemetry:opentelemetry-exporter-zipkin:1.38.0")
}

Configurations