Native Prometheus integration with NGINX

Today we are officially open-sourcing our own Prometheus statistics exporter module for NGINX!

Introduction

NGINX has been the de-facto choice for a reverse proxy server for quite some time now. At Categorica, we too use NGINX for this purpose, along with more traditional web-content serving. Similarly, we rely on Prometheus to collect and Grafana to aggregate and display statistics on our active deployments.

Metrics collection

As it stands, there’s only really one choice for statistics & metrics collection with NGINX: NGINX Prometheus Exporter, a fully featured Prometheus exporter. When it came to setting up our own metrics collection, there was something that just felt slightly off:

Why does the metrics endpoint itself run its own web-server?

In practise, this isn’t a big problem. Whilst it’s possible to deploy it as a systemd service (and hence just a regular binary), it is more typical to deploy NGINX Prometheus Exporter via Docker’s side-car containers; an additional complexity that doesn’t feel necessary. It does however mean that metrics collection itself evades metrics collection, à la “who watches the watchmen”.

In short, this idealogical indignation did not survive the free time we had on evenings and weekends, and the resulting NGINX plugin quickly became our preferred mechanism for statistics collection. As of today, we’ve decided to share this.

Usage

You’ll need to be comfortable with the command line, and having a basic understanding of the GNU “configure, make, make install” dance won’t hurt, but it’s not strictly necessary. Building NGINX from source is recommended, to ensure the required features are compiled in (as your package manager’s idea of NGINX may vary from ours), although again, this is not required.

Prerequisites

1. Even if you’re not building NGINX from source, you’ll need the NGINX sources to compile the module. The supplied Makefile can do this step for you.
2. If you’re not building NGINX from source, NGINX must have been compiled with --with-compat and --with-http_stub_status_module.
3. A C99 compiler (e.g. Clang).
4. PCRE, OpenSSL, and zlib libraries & headers.

Additionally, these steps presume that you are building from within a Unix-like environment.

Building NGINX

Skip this if you already have an appropriate version of NGINX installed. From a terminal

cd path-to-nginx-source;

# we'll use the path to the NGINX source whilst building the plugin.
export NGINX_SRC=`pwd`;

./configure --with-compat --with-http_stub_status_module;

make -j;

sudo make install;

This will build & install NGINX, typically with the /usr/local prefix. For more information on building NGINX, you can check out their documentation.

Building the plugin

If you’ve already got an appropriate version of NGINX installed, then the supplied Makefile will determine what version it is (by invoking nginx -v), and download the appropriate NGINX sources required to build the plugin.

If this behaviour is not desired, then you can set the path to the NGINX sources with the environment variable - NGINX_SRC - which will skip the version check & source download.

Afterward, building the plugin is done with

make module

sudo make install

This will build the plugin, and install it to the appropriate plugin location.

More information can be found on our Github repository’s README.md.

Running NGINX with the plugin

Once the plugin has been built, some configuration is needed to describe the route which statistics should be made available on.

load_module modules/ngx_http_stats_prom_module.so;

http {
  # Optional: set shared memory size (default 1m)
  # prometheus_metrics_shm_size 2m;

  server {
    # standard NGINX server directives apply here

    listen 9100;
    location /metrics {
      prometheus_metrics;
    }
  }
}

This configuration exports statistics suitable for Prometheus ingestion on http://hostname:9100/metrics. Presuming that the nginx instance is running locally, you should be able to query the metrics endpoint, via curl:

curl http://localhost:9100/metrics

with sample response:

# HELP nginx_up Whether the nginx process is up
# TYPE nginx_up gauge
nginx_up 1

# HELP nginx_process_pid PID of the worker process (serving metrics)
# TYPE nginx_process_pid gauge
nginx_process_pid 2985885

# HELP nginx_http_requests_total Total number of HTTP requests
# TYPE nginx_http_requests_total counter
nginx_http_requests_total 434488

# HELP nginx_http_responses_total HTTP responses by status class
# TYPE nginx_http_responses_total counter
nginx_http_responses_total{status="1xx"} 0
nginx_http_responses_total{status="2xx"} 42598
nginx_http_responses_total{status="3xx"} 771
nginx_http_responses_total{status="4xx"} 77
nginx_http_responses_total{status="5xx"} 1

# etc.

Exported statistics

Once running, you should be able to view metrics for NGINX on the route specified in your configuration.

Once started, the available metrics are:

Metric	Type	Labels
nginx_up	gauge
nginx_process_pid	gauge
nginx_http_requests_total	counter
nginx_http_responses_total	counter	status (1xx–5xx)
nginx_http_requests_by_method_total	counter	method (GET, HEAD, POST, PUT, DELETE, OPTIONS, PATCH, other)
nginx_http_request_bytes_total	counter
nginx_http_response_bytes_total	counter
nginx_http_version_total	counter	version (1.0, 1.1, 2.0, 3.0)
nginx_http_request_duration_seconds	histogram	(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10)
nginx_connections_active	gauge
nginx_connections_reading	gauge
nginx_connections_writing	gauge
nginx_connections_waiting	gauge
nginx_connections_accepted_total	counter
nginx_connections_handled_total	counter

These are available as soon as the process launches, and are measured from process start.

In addition, there are basic metrics for each server or upstream directive defined in your nginx.conf file. These will appear after the first received request on that prefix.

Metric	Type	Labels
<prefix>_requests_total	counter	server / upstream
<prefix>_response_total	counter	server / upstream, status (1xx–5xx)
<prefix>_request_bytes_total	counter	server / upstream
<prefix>_response_bytes_total	counter	server / upstream

It is worth noting that, at most, 512 server or upstream directives are supported. Prefix lengths greater than 255 characters are ignored. Finally, only 1 megabyte of memory is allocated for statistics. If this is exhausted, statistics collection will be silently skipped. This can be configured in your NGINX configuration with prometheus_metrics_shm_size.

Limitations

nginx-stats-prometheus is not a drop-in replacement for nginx-prometheus-exporter. The key differences are:

1. Significantly fewer statistics are collected.
2. The plugin is tied to the version of NGINX that it is built against. Updating NGINX typically requires that the plugin be re-built.
3. Significantly less battle tested. Obviously our stats collection is far less mature and used.

Licence & Contributing

We’ve decided to release nginx-stats-prometheus as MIT licenced. We are open to pull requests, so feel free to submit one. We only ask that a human has at least reviewed the patch-set before submitting to us.

We currently offer no support for nginx-stats-prometheus and are unable to guarantee help with any issues that may arise from using it.

Conclusion

Perhaps you want to run NGINX without containerisation, are looking to simplify your deployment’s configuration, or just prefer the idea of running statistics collection from a plugin, give nginx-stats-prometheus a try!