This guide demonstrates how to instrument your Go applications with OpenTelemetry and send traces to the TrueFoundry backend. We’ll cover installing the required packages, initializing the tracer, automatic HTTP instrumentation, adding custom attributes and spans, and configuring sampling and debugging options. By the end, you’ll have a clear blueprint for integrating OpenTelemetry tracing into your Go services.
To start, add the OpenTelemetry SDK and necessary instrumentation packages to your Go module. You can install them with the go get command:
Copy
Ask AI
go get go.opentelemetry.io/otelgo get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpgo get go.opentelemetry.io/otel/exporters/stdout/stdouttracego get go.opentelemetry.io/otel/sdkgo get go.opentelemetry.io/otel/trace
Next, initialize the OpenTelemetry SDK in your application.
This involves setting up a Tracer Provider (which manages tracers and spans) and oltp exporter to send the traces to TrueFoundry Backend.
Copy
Ask AI
package mainimport ( "context" "errors" "fmt" "os" "time" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/trace")// setupOTelSDK bootstraps the OpenTelemetry pipeline.// If it does not return an error, make sure to call shutdown for proper cleanup.func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) { // set up env variables os.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "<enter_your_api_endpoint>") os.Setenv("OTEL_EXPORTER_OTLP_HEADERS", fmt.Sprintf("Authorization=Bearer %s,TFY-Tracing-Project=%s", "<Enter your api key here>", "<Enter your tracing project fqn>")) // Set up propagator. otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{}, )) // set up exporter exporter, err := otlptracehttp.New(context.Background()) if err != nil { return nil, err } // set up tracerProvider tracerProvider := trace.NewTracerProvider( trace.WithBatcher(exporter), ) otel.SetTracerProvider(tracerProvider) return tracerProvider.Shutdown, nil}
Now that the OpenTelemetry SDK is set up, let’s instrument the HTTP server to automatically trace incoming requests. OpenTelemetry provides middleware for popular frameworks; for Go’s built-in net/http, we use the otelhttp handler.At this point, all incoming HTTP requests are being traced automatically.
Copy
Ask AI
package mainimport ( "net/http" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp")func main() { handler := func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("hello world")) } // Adding instrumentation to the handler otelHandler := otelhttp.NewHandler(http.HandlerFunc(handler), "http.server") http.ListenAndServe(":8080", otelHandler)}
To fully benefit from distributed tracing, you should also propagate trace context in your outgoing HTTP requests. This helps downstream services recognize that their requests are part of a larger distributed trace.Configure your HTTP client to automatically inject trace context into outgoing requests using OpenTelemetry’s otelhttp instrumentation:
This functionality is not incorporated into the Complete Application example to reduce complexity
Automatic instrumentation captures basic request information, but you can add custom data to your traces using attributes.
Attributes are key-value pairs that provide additional context about your operations.
For example, in order service, you might add order.id to make traces more useful.
Automatic instrumentation captures HTTP requests and external calls, but it doesn’t track your application’s internal logic. For important operations, you can manually create spans to trace specific parts of your code. A span represents a unit of work, and creating sub-spans helps you see detailed timing and context for key processes.For example, if a request triggers a complex function or external call that isn’t automatically captured, you can create a span to trace that specific operation. Manual instrumentation fills these gaps by letting you track what happens inside your application, not just at the edges.
Below is a comprehensive example that demonstrates all the OpenTelemetry concepts we’ve covered.This application creates an order service HTTP server that sets up OpenTelemetry tracing with proper configuration, automatically instruments HTTP requests using otelhttp, creates custom spans for database operations, and adds custom attributes to provide order-specific context.
Copy
Ask AI
package mainimport ( "context" "errors" "fmt" "net/http" "os" "time" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/exporters/stdout/stdouttrace" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/trace" oteltrace "go.opentelemetry.io/otel/trace")// setupOTelSDK bootstraps the OpenTelemetry pipeline.// If it does not return an error, make sure to call shutdown for proper cleanup.func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) { // set up env variables os.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "<enter_your_api_endpoint>") os.Setenv("OTEL_EXPORTER_OTLP_HEADERS", fmt.Sprintf("Authorization=Bearer %s,TFY-Tracing-Project=%s", "<Enter your api key here>", "<Enter your tracing project fqn>")) // Set up propagator. otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{}, )) // set up exporter exporter, err := otlptracehttp.New(context.Background()) if err != nil { return nil, err } tracerProvider := trace.NewTracerProvider( trace.WithBatcher(exporter), ) otel.SetTracerProvider(tracerProvider) return tracerProvider.Shutdown, nil}var tracer = otel.Tracer("truefoundry.com/open-telemetry/demo/golang/")func main() { // Set up OpenTelemetry. otelShutdown, err := setupOTelSDK(context.Background()) if err != nil { return } defer func() { err = errors.Join(err, otelShutdown(context.Background())) }() mockFetchOrder := func(ctx context.Context, id string) string { _, span := tracer.Start(ctx, "db.query") defer span.End() span.SetAttributes(attribute.StringSlice("order.items", []string{"item1", "item2", "item3"})) // this will be added to the span span.SetAttributes(attribute.String("order.id", id)) // this will be added to the span return fmt.Sprintf("order.id: %s, items: %v", id, []string{"item1", "item2", "item3"}) } // This is a workaround to add attributes to the span. handler := func(w http.ResponseWriter, r *http.Request) { orderId := r.URL.Query().Get("id") if orderId == "" { orderId = "123" } // Update existing span attributes span := oteltrace.SpanFromContext(r.Context()) span.SetAttributes(attribute.String("order.id", orderId)) // this will be added to the span order := mockFetchOrder(r.Context(), orderId) w.Write([]byte(order)) } otelHandler := otelhttp.NewHandler(http.HandlerFunc(handler), "order-service") http.ListenAndServe(":8080", otelHandler)}
Tracing sampling is a crucial technique for managing the volume of trace data in production environments.
By default, OpenTelemetry Go traces every request, which works well for debugging or development but can become expensive and noisy in high-traffic production systems.Sampling helps in several ways: it reduces noise in traces, helping you focus on important traces while maintaining visibility into your system.
It also helps with cost management in terms of storage, processing, and network bandwidth, making it essential for production deployments.
OpenTelemetry supports several built-in samplers, but in practice, two cover most use cases:1. TraceIdRatioBased SamplerSamples a fixed percentage of root traces. This sampler makes sampling decisions independently for each trace.
Pros: Simple to configure, predictable sampling rate, deterministic behaviorCons: May create partial traces if child spans are sampled differently, especially when spans are spread across multiple microservices or services with different sampling configurations.2. ParentBased Sampler (Recommended)Samples a fixed percentage of root traces and ensures that child spans follow the parent’s sampling decision, maintaining complete trace integrity.
Pros: Maintains trace integrity, prevents partial traces, ensures complete trace visibility when sampledCons: Slightly more complex configuration, but worth the additional setup for production environments
Partial TracesIf you see partial traces (missing spans in the middle of a trace), ensure you’re using ParentBased sampler:
Copy
Ask AI
// ❌ May create partial traces across servicessampler = trace.TraceIDRatioBased(0.1)// ✅ Maintains trace integrity across all servicessampler := trace.ParentBased(trace.TraceIDRatioBased(0.1))
Too Much DataIf you’re still collecting too much data or experiencing high costs, reduce the sampling rate:
Copy
Ask AI
// Reduce from 10% to 5% for high-traffic environmentssampler := trace.ParentBased(trace.TraceIDRatioBased(0.05))// For very high traffic, consider 1% samplingsampler := trace.ParentBased(trace.TraceIDRatioBased(0.01))
Too Little DataIf you need more visibility for debugging or monitoring, increase the sampling rate:
Copy
Ask AI
// Increase from 10% to 25% for better visibilitysampler := trace.ParentBased(trace.TraceIDRatioBased(0.25))// For critical debugging, consider 50% or highersampler := trace.ParentBased(trace.TraceIDRatioBased(0.5))
For local development and debugging, you can use the console exporter to see traces in your terminal:
Update setupOTelSDK function on create a console exporter and inject into newTracerProvider
OpenTelemetry provides instrumentation libraries for many popular frameworks and libraries in the Go ecosystem . These are drop-in packages that automatically generate spans and metrics for operations in those libraries, so you don’t have to instrument everything manually. When you are using third-party libraries or frameworks, you should take advantage of these to save time and ensure consistency.For example, if you are using a web framework like Gin, there is an official instrumentation package otelgin that can create spans for each HTTP request handled by Gin.OpenTelemetry’s Go Contrib repository contains many such instrumentation packages.