Vert.x Micrometer Metrics
This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI). It uses Micrometer for managing metrics and reporting to several backends.
Features
-
Vert.x core tools monitoring: TCP/HTTP client and servers,
DatagramSocket
,EventBus
and pools -
User defined metrics through Micrometer
-
Reporting to any backend supported by Micrometer
-
Built-in options for InfluxDB, Prometheus and JMX reporting.
InfluxDB
Prerequisites
Follow the instructions to get InfluxDb up and running.
Getting started
The modules vertx-micrometer-metrics and micrometer-registry-influx must be present in the classpath.
Maven users should add this to their project POM file:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-micrometer-metrics</artifactId>
<version>4.5.10</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-influx</artifactId>
<version>${micrometer.version}</version>
</dependency>
And Gradle users, to their build file:
compile 'io.vertx:vertx-micrometer-metrics:4.5.10'
compile 'io.micrometer:micrometer-registry-influx:${micrometer.version}'
Configuration examples
Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options.
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true))
.setEnabled(true)));
Using a specific URI
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)
.setUri("http://influxdb.example.com:8888"))
.setEnabled(true)));
Connecting to InfluxDB V2
Connecting to InfluxDB V2 requires that at least the org
property is set.
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)
.setOrg("my-org"))
.setEnabled(true)));
Using a specific database name or bucket
To configure a specific database name:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)
.setDb("sales-department"))
.setEnabled(true)));
For InfluxDB V2, you can use setBucket
as well.
With authentication
Authentication can be configured with a username and a password:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)
.setUserName("username")
.setPassword("password"))
.setEnabled(true)));
Or it can be configured with a token:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)
.setToken(authToken))
.setEnabled(true)));
Prometheus
Prerequisites
Follow the instructions to get Prometheus up and running.
Getting started
The modules vertx-micrometer-metrics and micrometer-registry-prometheus must be present in the classpath. You may also probably need vertx-web, to expose the metrics.
Maven users should add this to their project POM file:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-micrometer-metrics</artifactId>
<version>4.5.10</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>${micrometer.version}</version>
</dependency>
And Gradle users, to their build file:
compile 'io.vertx:vertx-micrometer-metrics:4.5.10'
compile 'io.micrometer:micrometer-registry-prometheus:${micrometer.version}'
Configuration examples
Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.setEnabled(true)));
Using an embedded HTTP server with custom endpoint
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true)
.setStartEmbeddedServer(true)
.setEmbeddedServerOptions(new HttpServerOptions().setPort(8080))
.setEmbeddedServerEndpoint("/metrics/vertx"))
.setEnabled(true)));
If the embedded server endpoint is not specified, it defaults to /metrics.
Binding metrics to an existing Vert.x Web router
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.setEnabled(true)));
// Later on, creating a router
Router router = Router.router(vertx);
router.route("/metrics").handler(PrometheusScrapingHandler.create());
vertx.createHttpServer().requestHandler(router).listen(8080);
JMX
Getting started
The modules vertx-micrometer-metrics and micrometer-registry-jmx must be present in the classpath.
Maven users should add this to their project POM file:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-micrometer-metrics</artifactId>
<version>4.5.10</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-jmx</artifactId>
<version>${micrometer.version}</version>
</dependency>
And Gradle users, to their build file:
compile 'io.vertx:vertx-micrometer-metrics:4.5.10'
compile 'io.micrometer:micrometer-registry-jmx:${micrometer.version}'
Configuration examples
Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setJmxMetricsOptions(new VertxJmxMetricsOptions().setEnabled(true))
.setEnabled(true)));
With step and domain
In Micrometer, step
refers to the reporting period, in seconds. domain
is the JMX domain under which
MBeans are registered.
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setJmxMetricsOptions(new VertxJmxMetricsOptions().setEnabled(true)
.setStep(5)
.setDomain("my.metrics.domain"))
.setEnabled(true)));
Other backends or combinations
Even if not all backends supported by Micrometer are implemented in Vert.x options, it is still possible to create any Micrometer registry and pass it to Vert.x.
The list of available backends includes Graphite, Ganglia, Atlas, etc. It also enables the Micrometer Composite Registry in order to report the same metrics to multiple backends.
In this example, metrics are reported both for JMX and Graphite:
CompositeMeterRegistry myRegistry = new CompositeMeterRegistry();
myRegistry.add(new JmxMeterRegistry(s -> null, Clock.SYSTEM));
myRegistry.add(new GraphiteMeterRegistry(s -> null, Clock.SYSTEM));
Vertx vertx = Vertx.builder()
.with(new VertxOptions()
.setMetricsOptions(new MicrometerMetricsOptions()
.setEnabled(true)))
.withMetrics(new MicrometerMetricsFactory(myRegistry))
.build();
Advanced usage
Please refer to MicrometerMetricsOptions
for an exhaustive list of options.
Averages and quantiles in Prometheus
By default, when using the Prometheus registry, histogram-kind metrics will not contain averages or quantile stats.
Averages don’t come out of the box, but they are typically computed at query time,
with promql
. Example, for HTTP client response time average during the last 5 minutes:
rate(vertx_http_client_response_time_seconds_sum[5m])
/
rate(vertx_http_client_response_time_seconds_count[5m])
To compute quantiles, there are two options available. The first is to activate quantile stats globally
and make them usable for Prometheus function histogram_quantile
:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true)
.setPublishQuantiles(true))
.setEnabled(true)));
And then, for example the promql
query for the HTTP client response time, 99th percentile over the last 5 minutes:
histogram_quantile(0.99, sum(rate(vertx_http_client_response_time_seconds_bucket[5m])) by (le))
The advantage of this option is that it can be leveraged in promql
, aggregable across dimensions.
The downside is that it creates a lot of time-series for stats under the hood.
The second option is to create limited stats, non-aggregable across dimensions. It requires to access directly the Micrometer / Prometheus registry:
PrometheusMeterRegistry registry = (PrometheusMeterRegistry) BackendRegistries.getDefaultNow();
registry.config().meterFilter(
new MeterFilter() {
@Override
public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
return DistributionStatisticConfig.builder()
.percentiles(0.95, 0.99)
.build()
.merge(config);
}
});
See also, more on histograms and percentiles:
-
from Micrometer doc
-
from Prometheus doc
Furthermore, you can check some full working examples. They come along with few instructions to set up with Prometheus and view dashboards in Grafana.
Disable some metric domains
Restricting the Vert.x modules being monitored can be done using
disabledMetricsCategories
.
For a full list of domains, see MetricsDomain
User-defined metrics
The Micrometer registries are accessible, in order to create new metrics or fetch the existing ones. By default, a unique registry is used and will be shared across the Vert.x instances of the JVM:
MeterRegistry registry = BackendRegistries.getDefaultNow();
It is also possible to have separate registries per Vertx instance, by giving a registry name in metrics options. Then it can be retrieved specifically:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setInfluxDbOptions(new VertxInfluxDbOptions().setEnabled(true)) // or VertxPrometheusOptions
.setRegistryName("my registry")
.setEnabled(true)));
// Later on:
MeterRegistry registry = BackendRegistries.getNow("my registry");
As an example, here is a custom timer that will track the execution time of a piece of code that is regularly called:
MeterRegistry registry = BackendRegistries.getDefaultNow();
Timer timer = Timer
.builder("my.timer")
.description("a description of what this timer does")
.register(registry);
vertx.setPeriodic(1000, l -> {
timer.record(() -> {
// Running here some operation to monitor
});
});
For more examples, documentation about the Micrometer registry and how to create metrics, check Micrometer doc.
Reusing an existing registry
It is possible to reuse an existing Micrometer registry (or CollectorRegistry
from the Prometheus Java client),
and inject it into the Vert.x instance using a VertxBuilder
:
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
// You could also reuse an existing registry from the Prometheus Java client:
CollectorRegistry prometheusClientRegistry = new CollectorRegistry();
registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT, prometheusClientRegistry, Clock.SYSTEM);
// It's reused in MicrometerMetricsOptions.
// Prometheus options configured here, such as "setPublishQuantiles(true)", will affect the whole registry.
Vertx.builder()
.with(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true)
.setPublishQuantiles(true))
.setEnabled(true)))
.withMetrics(new MicrometerMetricsFactory(registry))
.build();
Important
|
Vert.x Micrometer Metrics registers meter filters for label matching.
Since Micrometer 1.13, a warning is logged when a To avoid this warning, make sure to register your own meters only after Vert.x Micrometer Metrics has been initialized. |
JVM or other instrumentations
Since plain access to Micrometer registries is provided, it is possible to leverage the Micrometer API. For instance, to instrument the JVM:
MeterRegistry registry = BackendRegistries.getDefaultNow();
new ClassLoaderMetrics().bindTo(registry);
new JvmMemoryMetrics().bindTo(registry);
new JvmGcMetrics().bindTo(registry);
new ProcessorMetrics().bindTo(registry);
new JvmThreadMetrics().bindTo(registry);
From Micrometer documentation.
Metric names
Each metric that Vert.x provides can be renamed through the metrics options, using
MetricsNaming
and setMetricsNaming
.
The default metric names were changed in Vert.x 4 to better align with backend conventions, but it is
still possible to retrieve the names used in Vert.x 3.x for compatibility:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.setMetricsNaming(MetricsNaming.v3Names())
.setEnabled(true)));
Labels and matchers
Vert.x Micrometer Metrics defines a set of labels (aka tags or fields) that are used to provide dimensionality to a metric. For instance, metrics related to event bus messages have an address label, which allows then to query time-series for a specific event bus address, or compare time-series per address, or perform any kind of aggregation that the query API allows.
While setting up metrics options, you can specify which labels you want to enable or not:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.setLabels(EnumSet.of(Label.REMOTE, Label.LOCAL, Label.HTTP_CODE, Label.HTTP_PATH))
.setEnabled(true)));
The full list of labels is detailed here: Label
.
Warning
|
Enabling labels may result in a high cardinality in values, which can cause troubles on the metrics backend and affect performances. So it must be used with care. In general, it is fine to enable labels when the set of possible values is bounded. |
For that reason, labels enabled by default are restricted to the ones with known bounded values.
It is possible to interact with labels further than just enabling/disabling. There are two ways for that:
Using Matchers
Match
objects can be used to filter or rename some label value
by matching it with either an exact string or a regular expression (the former being more efficient).
Here is an example to restrict HTTP server metrics to those with label local=localhost:8080 only:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.addLabelMatch(new Match()
// Restrict HTTP server metrics to those with label "local=localhost:8080" only
.setDomain(MetricsDomain.HTTP_SERVER)
.setLabel("local")
.setValue("localhost:8080"))
.setEnabled(true)));
When an alias is specified in the Match, it will be used to rename value instead of filtering.
Matchers are especially useful to control labelling through configuration as they are set via
MicrometerMetricsOptions
.
Using Micrometer’s MeterFilter
Micrometer’s MeterFilter API can be accessed directly in order to define rules on labels. Compared to Matchers, it offers more features in manipulating the labels, but cannot be defined from configuration. So both have their advantages.
Here is an example to replace the actual path
label of HTTP requests with a generic form using regex:
MeterRegistry registry = BackendRegistries.getDefaultNow();
Pattern pattern = Pattern.compile("/foo/bar/.*");
registry.config().meterFilter(
MeterFilter.replaceTagValues(Label.HTTP_PATH.toString(), actualPath -> {
Matcher m = pattern.matcher(actualPath);
if (m.matches()) {
return "/foo/bar/:id";
}
return actualPath;
}, ""));
Note
|
Matchers use MeterFilters under the hood. |
Custom tags provider
You can define a function that generates additional tags (or labels) for HTTP server or client metrics.
Such function takes an HttpRequest
object as a parameter, and returns
an Iterable of Tag
.
As an example, here is how to map the x-user header to a custom label user in both server and client metrics:
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
new MicrometerMetricsOptions()
.setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
.setServerRequestTagsProvider(req -> {
String user = req.headers().get("x-user");
return Collections.singletonList(Tag.of("user", user));
})
.setClientRequestTagsProvider(req -> {
String user = req.headers().get("x-user");
return Collections.singletonList(Tag.of("user", user));
})
.setEnabled(true)));
Snapshots
A MetricsService
can be created out of a Measured
object
in order to take a snapshot of its related metrics and measurements.
The snapshot is returned as a JsonObject
.
A well known Measured object is simply Vertx
:
MetricsService metricsService = MetricsService.create(vertx);
JsonObject metrics = metricsService.getMetricsSnapshot();
System.out.println(metrics);
Other components, such as an EventBus
or a HttpServer
are
measurable:
HttpServer server = vertx.createHttpServer();
MetricsService metricsService = MetricsService.create(server);
JsonObject metrics = metricsService.getMetricsSnapshot();
System.out.println(metrics);
Finally, it is possible to filter the returned metrics from their base names:
MetricsService metricsService = MetricsService.create(vertx);
// Client + server
JsonObject metrics = metricsService.getMetricsSnapshot("vertx.http");
System.out.println(metrics);
Vert.x core metrics
This section lists all the metrics generated by monitoring the Vert.x core tools.
Note
|
The metric backends may have different conventions or rules for naming metrics. The names described below are the default ones used in Vert.x 4, using underscore separators. The actual names may vary depending on the metrics backend. |
TCP Client
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Counter |
Number of bytes received from the remote host. |
|
|
Counter |
Number of bytes sent to the remote host. |
|
|
Gauge |
Number of connections to the remote host currently opened. |
|
|
Counter |
Number of errors. |
HTTP Client
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Counter |
Number of bytes received from the remote host. |
|
|
Counter |
Number of bytes sent to the remote host. |
|
|
Gauge |
Number of connections to the remote host currently opened. |
|
|
Counter |
Number of errors. |
|
|
Timer |
Time spent in queue before being processed, in seconds. |
|
|
Gauge |
Number of pending elements in queue. |
|
|
Gauge |
Number of requests being processed, waiting for a response. |
|
|
Counter |
Number of requests sent. |
|
|
Summary |
Size in bytes of the requests. |
|
|
Timer |
Response time in seconds. |
|
|
Counter |
Number of received responses. |
|
|
Summary |
Size in bytes of the responses. |
|
|
Gauge |
Number of websockets currently opened. |
TCP Server
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Counter |
Number of bytes received by the Net Server. |
|
|
Counter |
Number of bytes sent by the Net Server. |
|
|
Gauge |
Number of opened connections to the Net Server. |
|
|
Counter |
Number of errors. |
HTTP Server
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Counter |
Number of bytes received by the HTTP Server. |
|
|
Counter |
Number of bytes sent by the HTTP Server. |
|
|
Gauge |
Number of opened connections to the HTTP Server. |
|
|
Counter |
Number of errors. |
|
|
Gauge |
Number of requests being processed. |
|
|
Counter |
Number of processed requests. |
|
|
Counter |
Number of request resets. |
|
|
Summary |
Size in bytes of the requests. |
|
|
Timer |
Request processing time in seconds. |
|
|
Summary |
Size in bytes of the responses. |
|
|
Gauge |
Number of websockets currently opened. |
Datagram sockets
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Summary |
Total number of bytes received on the |
|
(none) |
Summary |
Total number of bytes sent to the remote host. |
|
|
Counter |
Total number of errors. |
Event Bus
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Summary |
Total number of bytes received while reading messages from event bus cluster peers. |
|
|
Summary |
Total number of bytes sent while sending messages to event bus cluster peers. |
|
|
Gauge |
Number of event bus handlers in use. |
|
|
Gauge |
Number of messages not processed yet. One message published will count for |
|
|
Counter |
Number of processed messages. |
|
|
Counter |
Number of messages published (publish / subscribe). |
|
|
Counter |
Number of discarded messages (e.g. still pending messages while handler is unregistered, or overflowing messages). |
|
|
Counter |
Number of messages sent (point-to-point). |
|
|
Counter |
Number of messages received. |
|
|
Counter |
Number of messages delivered to handlers. |
|
|
Counter |
Number of message reply failures. |
Worker pool metrics
The Vert.x worker pool expose metrics as defined by the pool metrics.
The type of pool (pool_type
) is worker.
Vert.x creates two worker pools upfront, worker-thread and internal-blocking.
Generic metrics
Pool metrics
Vert.x clients, other than the core HTTP clients, may expose pool metrics.
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Timer |
Time spent in queue before being processed, in seconds. |
|
|
Gauge |
Number of pending elements in queue. |
|
|
Timer |
Time using a resource (i.e. processing time for worker pools). |
|
|
Gauge |
Number of resources used. |
|
|
Counter |
Number of elements done with the resource (i.e. total number of tasks executed for worker pools). |
|
|
Gauge |
Pool usage ratio, only present if maximum pool size could be determined. |
Client metrics
Vert.x clients, other than the core HTTP / Net clients, may implement a standard set of client metrics.
Such client metrics are named after a "client type" identifier, displayed as $TYPE
in the table below.
The meaning of the namespace
label is left to the discretion of the client implementation.
Metric name | Labels | Type | Description |
---|---|---|---|
|
|
Gauge |
Number of pending elements in queue. |
|
|
Timer |
Time spent in queue before being processed, in seconds. |
|
|
Gauge |
Number of elements being processed. |
|
|
Timer |
Processing time, from request start to response end, in seconds. |
|
|
Counter |
Total number of resets. |
Database clients metrics
The following clients expose client metrics as well as optionally pool metrics.
-
Vert.x SQL client
-
Vert.x Redis client
-
…
The pool metrics carry a pool_type
, e.g. the SQL client uses the sql
type
-
vertx.pool.queue.pending
withpool_type
=sql -
vertx.pool.queue.time
withpool_type
=sql -
vertx.pool.usage
withpool_type
=sql -
vertx.pool.ratio
pool_type
=sql -
vertx.pool.completed
pool_type
=sql -
vertx.pool.in.use
pool_type
=sql
The client metrics type appears in the metrics name
-
vertx.sql.processing.pending
-
vertx.sql.processing.time
In order to distinguish pools, you can set a pool name on the PoolOptions
, exposed as a label
-
vertx.pool.queue.pending
withpool_type
=sql andpool_name
=the-pool-name -
vertx.pool.queue.time
withpool_type
=sql andpool_name
=the-pool-name -
vertx.pool.usage
withpool_type
=sql andpool_name
=the-pool-name -
vertx.pool.ratio
pool_type
=sql andpool_name
=the-pool-name -
vertx.pool.completed
pool_type
=sql andpool_name
=the-pool-name -
vertx.pool.in.use
pool_type
=sql andpool_name
=the-pool-name
Likewise, client metrics use a namespace with distinguish between them
-
vertx.sql.processing.pending
withclient_namespace
=the-client-name
-
vertx.sql.processing.time
withclient_namespace
=the-client-name
Micrometer must be configured to expose these labels
micrometerMetricsOptions.addLabels(Label.POOL_NAME);
micrometerMetricsOptions.addLabels(Label.NAMESPACE);