Java app tracing with OpenTelemetry

This guide demonstrates how to instrument your Java applications with OpenTelemetry and send traces to the TrueFoundry backend. We’ll cover adding the required dependencies, initializing the tracer, automatic HTTP instrumentation, adding custom attributes and spans, and configuring sampling and debugging options. By the end, you’ll have a clear blueprint for integrating OpenTelemetry tracing into your Java services using Spring Boot.

Installation

To start, add the OpenTelemetry SDK and necessary instrumentation dependencies to your project. If you’re using Maven, add these to your pom.xml:

<dependencies>
    <!-- Spring Boot Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    
    <!-- OpenTelemetry SDK -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
        <version>1.49.0</version>
    </dependency>
    
    <!-- OTLP Exporter -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
        <version>1.49.0</version>
    </dependency>
    
    <!-- OpenTelemetry API -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.49.0</version>
    </dependency>
    
    <!-- Spring Boot Actuator -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    
    <!-- Micrometer Tracing Bridge for OpenTelemetry (Auto-instrumentation) -->
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing-bridge-otel</artifactId>
    </dependency>
</dependencies>

If you’re using Gradle, add these to your build.gradle:

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'io.opentelemetry:opentelemetry-sdk:1.49.0'
    implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.49.0'
    implementation 'io.opentelemetry:opentelemetry-api:1.49.0'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'io.micrometer:micrometer-tracing-bridge-otel'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

Initializing OpenTelemetry

Next, initialize the OpenTelemetry SDK in your Spring Boot application. Create a configuration class to set up the Tracer Provider and OTLP exporter to send traces to TrueFoundry Backend.

package com.example.spring_boot;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

@Configuration
public class OpenTelemetryConfig {

    @Bean
    public OpenTelemetry openTelemetry() {
        // Set up headers for authentication and project identification
        Map<String, String> headers = new HashMap<>();
        headers.put("Authorization", "Bearer <Enter your api key here>");
        headers.put("TFY-Tracing-Project", "<Enter your tracing project fqn>");
        
        // Set up OTLP HTTP exporter
        OtlpHttpSpanExporter otlpExporter = OtlpHttpSpanExporter.builder()
            .setEndpoint("<enter_your_api_endpoint>/v1/traces")
            .setHeaders(() -> headers)
            .build();

        // Set up tracer provider
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
            .build();

        // Set up propagators
        ContextPropagators propagators = ContextPropagators.create(W3CTraceContextPropagator.getInstance());

        // Build and return OpenTelemetry SDK
        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setPropagators(propagators)
            .buildAndRegisterGlobal();
    }
    
    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("truefoundry.com/open-telemetry/demo/java/");
    }
}

Automatic HTTP Instrumentation

Now that the OpenTelemetry SDK is set up, let’s instrument the HTTP server to automatically trace incoming requests. Spring Boot with OpenTelemetry automatically instruments HTTP requests when properly configured.

Add this to your application.properties or application.yml:

# Spring Boot application name
spring.application.name=order-service

# Micrometer tracing bridge for OpenTelemetry (enables auto-instrumentation)
io.micrometer:micrometer-tracing-bridge-otel

At this point, all incoming HTTP requests are being traced automatically.

package com.example.spring_boot;

import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api/orders")
public class OrderController {

    @GetMapping("/{id}")
    public String getOrder(@PathVariable String id) {
        return "Order ID: " + id;
    }
}

Context propagation for outgoing requests

To fully benefit from distributed tracing, you should also propagate trace context in your outgoing HTTP requests. This helps downstream services recognize that their requests are part of a larger distributed trace.

Configure your HTTP client to automatically inject trace context into outgoing requests using OpenTelemetry’s instrumentation:

This functionality is not incorporated into the Complete Application example to reduce complexity

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import org.springframework.web.client.RestTemplate;

@Service
public class ExternalServiceClient {
    
    private final RestTemplate restTemplate;
    private final Tracer tracer;
    
    public ExternalServiceClient(RestTemplate restTemplate, OpenTelemetry openTelemetry) {
        this.restTemplate = restTemplate;
        this.tracer = openTelemetry.getTracer("order-service");
    }
    
    public String makeRequest(String url) {
        Span span = tracer.spanBuilder("external.request")
            .setAttribute("http.url", url)
            .startSpan();
            
        try (var scope = span.makeCurrent()) {
            return restTemplate.getForObject(url, String.class);
        } finally {
            span.end();
        }
    }
}

Adding Attributes to Spans

Automatic instrumentation captures basic request information, but you can add custom data to your traces using attributes. Attributes are key-value pairs that provide additional context about your operations. For example, in order service, you might add order.id to make traces more useful.

package com.example.spring_boot;

import org.springframework.web.bind.annotation.*;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.OpenTelemetry;

@RestController
@RequestMapping("/api/orders")
public class OrderController {

    private final OrderService orderService;

    public OrderController(OrderService orderService, OpenTelemetry openTelemetry) {
        this.orderService = orderService;
    }
    
    @GetMapping("/{id}")
    public String getOrder(@PathVariable String id) {
        // Add attributes to the current span
        Span span = Span.current();
        span.setAttribute("order.id", id);
        
        // Fetch order details
        String orderDetails = orderService.fetchOrder(id);
        
        return orderDetails;
    }
}

Creating Custom Spans

Automatic instrumentation captures HTTP requests and external calls, but it doesn’t track your application’s internal logic. For important operations, you can manually create spans to trace specific parts of your code. A span represents a unit of work, and creating sub-spans helps you see detailed timing and context for key processes.

For example, if a request triggers a complex function or external call that isn’t automatically captured, you can create a span to trace that specific operation. Manual instrumentation fills these gaps by letting you track what happens inside your application, not just at the edges.

package com.example.spring_boot;

import java.util.Arrays;
import org.springframework.stereotype.Service;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

@Service
public class OrderService {

    private final Tracer tracer;

    public OrderService(Tracer tracer) {
        this.tracer = tracer;
    }
    
    public String fetchOrder(String id) {
        Span span = tracer.spanBuilder("db.query")
            .setAttribute("order.id", id)
            .setAttribute("order.items", Arrays.asList("item1", "item2", "item3").toString())
            .startSpan();
            
        try (Scope scope = span.makeCurrent()) {
            // Simulate database query
            Thread.sleep(100);
            return String.format("order.id: %s, items: %s", id, Arrays.asList("item1", "item2", "item3"));
        } catch (InterruptedException e) {
            span.recordException(e);
            throw new RuntimeException(e);
        } finally {
            span.end();
        }
    }
}

Complete Application Example

Below is a comprehensive example that demonstrates all the OpenTelemetry concepts we’ve covered.

This application creates a Spring Boot order service that sets up OpenTelemetry tracing with proper configuration, automatically instruments HTTP requests, creates custom spans for database operations, and adds custom attributes to provide order-specific context.

package com.example.spring_boot;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

import java.util.HashMap;
import java.util.Map;

@SpringBootApplication
public class OrderServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(OrderServiceApplication.class, args);
    }

    @Bean
    public OpenTelemetry openTelemetry() {
        // Set up headers for authentication and project identification
        Map<String, String> headers = new HashMap<>();
        headers.put("Authorization", "Bearer <Enter your api key here>");
        headers.put("TFY-Tracing-Project", "<Enter your tracing project fqn>");
        
        // Set up OTLP HTTP exporter
        OtlpHttpSpanExporter otlpExporter = OtlpHttpSpanExporter.builder()
            .setEndpoint("<enter_your_api_endpoint>/v1/traces")
            .setHeaders(() -> headers)
            .build();

        // Set up tracer provider
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
            .build();

        // Set up propagators
        ContextPropagators propagators = ContextPropagators.create(W3CTraceContextPropagator.getInstance());

        // Build and return OpenTelemetry SDK
        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setPropagators(propagators)
            .buildAndRegisterGlobal();
    }
    
    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("truefoundry.com/open-telemetry/demo/java/");
    }
}

package com.example.spring_boot;

import org.springframework.web.bind.annotation.*;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.OpenTelemetry;

@RestController
@RequestMapping("/api/orders")
public class OrderController {

    private final OrderService orderService;

    public OrderController(OrderService orderService, OpenTelemetry openTelemetry) {
        this.orderService = orderService;
    }

    @GetMapping("/{id}")
    public String getOrder(@PathVariable String id) {
        // Add attributes to the current span
        Span span = Span.current();
        span.setAttribute("order.id", id);

        // Fetch order details
        String orderDetails = orderService.fetchOrder(id);
        
        return orderDetails;
    }
}

package com.example.spring_boot;

import java.util.Arrays;
import org.springframework.stereotype.Service;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

@Service
public class OrderService {

    private final Tracer tracer;

    public OrderService(Tracer tracer) {
        this.tracer = tracer;
    }

    public String fetchOrder(String id) {
        Span span = tracer.spanBuilder("db.query")
            .setAttribute("order.id", id)
            .setAttribute("order.items", Arrays.asList("item1", "item2", "item3").toString())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            // Simulate database query
            Thread.sleep(100);
            return String.format("order.id: %s, items: %s", id, Arrays.asList("item1", "item2", "item3"));
        } catch (InterruptedException e) {
            span.recordException(e);
            throw new RuntimeException(e);
        } finally {
            span.end();
        }
    }
}

Advanced Configuration

Sampling

Tracing sampling is a crucial technique for managing the volume of trace data in production environments. By default, OpenTelemetry Java traces every request, which works well for debugging or development but can become expensive and noisy in high-traffic production systems.

Sampling helps in several ways: it reduces noise in traces, helping you focus on important traces while maintaining visibility into your system. It also helps with cost management in terms of storage, processing, and network bandwidth, making it essential for production deployments.

Sampling Strategies

OpenTelemetry supports several built-in samplers, but in practice, two cover most use cases:

1. TraceIdRatioBased Sampler

Samples a fixed percentage of root traces. This sampler makes sampling decisions independently for each trace.

@Bean
public OpenTelemetry openTelemetry() {
    // ... existing setup code ...

    // Configure sampling for production
    Sampler sampler = Sampler.traceIdRatioBased(0.1);

    SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
        .setResource(resource)
        .setSampler(sampler)
        .build();

    // ... rest of setup ...
}

Pros: Simple to configure, predictable sampling rate, deterministic behavior

Cons: May create partial traces if child spans are sampled differently, especially when spans are spread across multiple microservices or services with different sampling configurations.

2. ParentBased Sampler (Recommended)

Samples a fixed percentage of root traces and ensures that child spans follow the parent’s sampling decision, maintaining complete trace integrity.

@Bean
public OpenTelemetry openTelemetry() {
    // ... existing setup code ...

    // Configure sampling for production
    Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.1));

    SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
        .setResource(resource)
        .setSampler(sampler)
        .build();

    // ... rest of setup ...
}

Pros: Maintains trace integrity, prevents partial traces, ensures complete trace visibility when sampled

Cons: Slightly more complex configuration, but worth the additional setup for production environments

Troubleshooting

Partial Traces

If you see partial traces (missing spans in the middle of a trace), ensure you’re using ParentBased sampler:

// ❌ May create partial traces across services
Sampler sampler = Sampler.traceIdRatioBased(0.1);

// ✅ Maintains trace integrity across all services
Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.1));

Too Much Data

If you’re still collecting too much data or experiencing high costs, reduce the sampling rate:

// Reduce from 10% to 5% for high-traffic environments
Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.05));

// For very high traffic, consider 1% sampling
Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.01));

Too Little Data

If you need more visibility for debugging or monitoring, increase the sampling rate:

// Increase from 10% to 25% for better visibility
Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.25));

// For critical debugging, consider 50% or higher
Sampler sampler = Sampler.parentBased(Sampler.traceIdRatioBased(0.5));

Local Debugging

For local development and debugging, you can use the console exporter to see traces in your terminal:

@Bean
public OpenTelemetry openTelemetry() {
    // ... existing setup code ...

    // Console exporter for local debugging
    ConsoleSpanExporter consoleExporter = ConsoleSpanExporter.create();

    SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
        .addSpanProcessor(BatchSpanProcessor.builder(consoleExporter).build())
        .build();

    // ... rest of setup ...
}

Using Instrumentation Libraries

OpenTelemetry provides instrumentation libraries for many popular frameworks and libraries in the Java ecosystem. These are drop-in packages that automatically generate spans and metrics for operations in those libraries, so you don’t have to instrument everything manually. When you are using third-party libraries or frameworks, you should take advantage of these to save time and ensure consistency.

For Spring Boot applications, the io.micrometer:micrometer-tracing-bridge-otel dependency provides automatic instrumentation for HTTP requests, database operations, and other common Spring Boot components. This bridge connects Micrometer’s tracing capabilities with OpenTelemetry, enabling seamless integration with the OpenTelemetry ecosystem.

OpenTelemetry’s Java Instrumentation repository contains many such instrumentation packages for popular Java libraries and frameworks.

LLM Deployment

LLM Finetuning

Prompt Management

LLM Tracing

Java app tracing with OpenTelemetry

Installation

Initializing OpenTelemetry

Automatic HTTP Instrumentation

Context propagation for outgoing requests

Adding Attributes to Spans

Creating Custom Spans

Complete Application Example

Advanced Configuration

Sampling

Sampling Strategies

Troubleshooting

Local Debugging

Using Instrumentation Libraries

LLM Deployment

LLM Finetuning

Prompt Management

LLM Tracing

​Installation

​Initializing OpenTelemetry

​Automatic HTTP Instrumentation

​Context propagation for outgoing requests

​Adding Attributes to Spans

​Creating Custom Spans

​Complete Application Example

​Advanced Configuration

​Sampling

​Sampling Strategies

​Troubleshooting

​Local Debugging

​Using Instrumentation Libraries

Installation

Initializing OpenTelemetry

Automatic HTTP Instrumentation

Context propagation for outgoing requests

Adding Attributes to Spans

Creating Custom Spans

Complete Application Example

Advanced Configuration

Sampling

Sampling Strategies

Troubleshooting

Local Debugging

Using Instrumentation Libraries