Monitoring

Vertx OpenTracing

Vert.x integrates with OpenTracing thanks to the Jaeger client.

You can configure Vert.x to use the Jaeger client configured via Environment

Vertx vertx = Vertx.vertx(new VertxOptions()
  .setTracingOptions(
    new OpenTracingOptions()
  )
);

You can also pass a custom Tracer allowing for greater control over the configuration.

Vertx vertx = Vertx.builder()
  .withTracer(new OpenTracingTracerFactory(tracer))
  .build();

Tracing policy

The tracing policy defines the behavior of a component when tracing is enabled:

PROPAGATE: the component reports a span in the active trace
ALWAYS: the component reports a span in the active trace or creates a new active trace
IGNORE: the component will not be involved in any trace.

The tracing policy is usually configured in the component options.

HTTP tracing

The Vert.x HTTP server and client reports span around HTTP requests:

operationName: the HTTP method
tags
http.method: the HTTP method
http.url: the request URL
http.status_code: the HTTP status code

The default HTTP server tracing policy is ALWAYS, you can configure the policy with setTracingPolicy

HttpServer server = vertx.createHttpServer(new HttpServerOptions()
  .setTracingPolicy(TracingPolicy.IGNORE)
);

The default HTTP client tracing policy is PROPAGATE, you can configure the policy with setTracingPolicy

HttpClient client = vertx.createHttpClient(new HttpClientOptions()
  .setTracingPolicy(TracingPolicy.IGNORE)
);

To initiate a trace for a client call, you need to create it first and make Vert.x aware of it by using OpenTracingUtil.setSpan:

Span span = tracer.buildSpan("my-operation")
  .withTag("some-key", "some-value")
  .start();
OpenTracingUtil.setSpan(span);
// Do something, e.g. client request
span.finish();

In an HTTP scenario between two Vert.x services, a span will be created client-side, then the trace context will be propagated server-side and another span will be added to the trace.

EventBus tracing

The Vert.x EventBus reports spans around message exchanges.

The default sending policy is PROPAGATE, you can configure the policy with setTracingPolicy.

DeliveryOptions options = new DeliveryOptions().setTracingPolicy(TracingPolicy.ALWAYS);
vertx.eventBus().send("the-address", "foo", options);

Obtain current Span

Vert.x stores current Span object in local context. To obtain it, use method OpenTracingUtil.getSpan().

This method will work only on Vert.x threads (instances of VertxThread). Obtaining from non-Vert.x thread doesn’t work by design, method will return null.

Coroutines support

There is no direct support for coroutines, but it can be achieved with minimal instrumentation.

There are several steps to achieve this.

Use CoroutineVerticle.
Convert every route handler you have to a coroutine.
Use CoroutineContext to store Tracer and current Span object

Example code:

class TracedVerticle(private val tracer: Tracer): CoroutineVerticle() {
   override suspend fun start() {
       val router = Router.router(vertx)

       router.route("/hello1")
           .method(HttpMethod.GET)
           .coroutineHandler { ctx ->                          // (1)
               launch { println("Hello to Console") }
               ctx.end("Hello from coroutine handler")
           }

       router.route("/hello2")
           .method(HttpMethod.GET)
           .coroutineHandler(::nonSuspendHandler)              // (2)

       vertx.createHttpServer()
           .requestHandler(router)
           .listen(8080)
           .await()
   }

   private fun nonSuspendHandler(ctx: RoutingContext) {
       ctx.end("Hello from usual handler")
   }

   private fun Route.coroutineHandler(handler: Handler<RoutingContext>): Route = // (3)
       this.coroutineHandler(handler::handle)

   private fun Route.coroutineHandler(                                           // (4)
       handler: suspend (RoutingContext) -> (Unit)
   ): Route = handler { ctx ->
       val span: Span = OpenTracingUtil.getSpan()                                // (5)
       launch(ctx.vertx().dispatcher() + SpanElement(tracer, span)) {            // (6)
           val spanElem = coroutineContext[SpanElement]                          // (7)
           if (spanElem == null) {
               handler(ctx)
           } else {
               val span = spanElem.span
               val tracer = spanElem.tracer
               val childSpan = span                                                // (8)
               try {
                   withContext(SpanElement(tracer, childSpan)) { handler(ctx) }    // (9)
               } finally {
                   // childSpan.finish()                                           // (10)
               }
           }
           // OR create a helper method for further reuse
           withContextTraced(coroutineContext) {
               try {
                   handler(ctx)
               } catch (t: Throwable) {
                   ctx.fail(t)
               }
           }
       }
   }
}

Creates a coroutine handler with coroutineHandler extension method.
Creates usual async handler, which is then wrapped to a coroutine.
Extension method to convert Handler<RoutingContext> to suspendable function.
Extension method which creates and launches a coroutine on Vert.x EventLoop.
Get current Span from Vert.x local context (populated automatically).
Create a wrapper coroutine, add current Span to CoroutineContext.
Retrieve a Span from coroutine context.
Either reuse span or create a new span with tracer.buildSpan("").asChildOf(span).start().
Put a new Span to a context.
Finish childSpan, if you created a new one.

Helper code, your implementation may vary:

/**
* Keeps references to a tracer and current Span inside CoroutineContext
*/
class SpanElement(val tracer: Tracer, val span: Span) :
   ThreadContextElement<Scope>,
   AbstractCoroutineContextElement(SpanElement) {

   companion object Key : CoroutineContext.Key<SpanElement>

   /**
   *  Will close current [Scope] after continuation's pause.
   */
   override fun restoreThreadContext(context: CoroutineContext, oldState: Scope) {
       oldState.close()
   }

   /**
   * Will create a new [Scope] after each continuation's resume, scope is activated with provided [span] instance.
   */
   override fun updateThreadContext(context: CoroutineContext): Scope {
       return tracer.activateSpan(span)
   }
}

/**
* Advanced helper method with a few options, also shows how to use MDCContext to pass a Span to a logger.
*/
suspend fun <T> withContextTraced(
   context: CoroutineContext,
   reuseParentSpan: Boolean = true,
   block: suspend CoroutineScope.() -> T
): T {
   return coroutineScope {
       val spanElem = this.coroutineContext[SpanElement]

       if (spanElem == null) {
           logger.warn { "Calling 'withTracer', but no span found in context" }
           withContext(context, block)
       } else {
           val childSpan = if (reuseParentSpan) spanElem.span
           else spanElem.tracer.buildSpan("").asChildOf(spanElem.span).start()

           try {
               val mdcSpan = mapOf(MDC_SPAN_KEY to childSpan.toString())
               withContext(context + SpanElement(spanElem.tracer, childSpan) + MDCContext(mdcSpan), block)
           } finally {
               if (!reuseParentSpan) childSpan.finish()
           }
       }
   }
}
private const val MDC_SPAN_KEY = "request.span.id"

OpenTelemetry

Metrics using Dropwizard