Send Your Application Metrics to Prometheus and Grafana

This guide shows you how to have the platform collect your application's own
Prometheus metrics, find them in Grafana, and work out why they have stopped
arriving.

Prerequisites

• A dedicated observability stack (your own Grafana and Prometheus) in the
  environment. Without one, application metrics have nowhere to go. See
  Observability for how to request it.
• An application that serves metrics in the Prometheus text format over HTTP,
  usually on a path such as /metrics. The platform collects what your
  application exposes. It cannot create metrics your code does not emit.
• Permission to edit the application Component's configuration in SVIEW.

What your application must expose

• An HTTP endpoint that returns metrics in the Prometheus text format. Most
  languages have a client library or framework module for this, for example
  Micrometer with the Prometheus registry for Java, or prometheus_client for
  Python.
• A fixed port and path, for example 9100 and /metrics. The port can be
  the same as your application port or a separate one.
• Plain HTTP inside the cluster, unless you also set the scheme (see
  Optional settings below).

Check it locally first: requesting the endpoint should return lines such as:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027

Turn on scraping for a Component

Scraping is switched on per Component with the metrics block in its Helm
values. It adds the annotations that tell the platform collector to scrape
every pod of that Component.

1. Open the application Component in SVIEW.

2. Add a metrics block to its configuration, with the port and path your
   application uses:

   YAML

   metrics:
     enabled: true
     port: 8080
     path: /metrics

3. Click Save. The Component redeploys, and collection starts once the new
   pods are running.

The setting is per Component and per Environment. Enabling it in staging does
not enable it in production: repeat the change in each Environment, or copy the
block across when you promote the configuration.

Optional settings

For anything the metrics block does not cover, add pod annotations under
podAnnotations in the same Helm values:

podAnnotations:
  k8s.grafana.com/metrics.scrapeInterval: "15s"
  k8s.grafana.com/metrics.scheme: "https"
  k8s.grafana.com/job: "orders-api"

• metrics.scrapeInterval — how often to scrape. The default is 30 seconds.
• metrics.scheme — http or https.
• job — overrides the job label, which is useful when you want a stable
  name across Environments.

Do not repeat keys that the metrics block already sets (scrape,
metrics.path, metrics.portNumber, instance) under podAnnotations.

Platform-provided Components

Some Components from the Catalog already expose metrics and are scraped with no
extra setup. Kafka clusters deployed from the Catalog, for example, publish
broker metrics that appear in your Prometheus once the observability stack is
in place.

Network policies

The collector runs in the observability stack's own namespace and reaches your
pods across namespaces. Under the default
Network Policies posture, cross-namespace traffic is
denied, so the metrics port may need an explicit allow rule. If scraping is
enabled, the endpoint answers from inside the pod, and the metrics still do not
appear in Grafana, a network policy is the most likely cause.

To have the collector allowed, raise a request with the namespace, the Component,
and the metrics port. Ask for it in every Environment where you enable
scraping. A rule added in staging does not carry over to production.

Find your metrics in Grafana

1. Open your Grafana and go to Explore.

2. Select the Prometheus data source.

3. Query a metric name your application exposes, filtered to your namespace:

   Text

   http_requests_total{namespace="your-namespace"}

To see everything collected from one Component, filter on its instance
label and list the metric names:

count by (__name__) ({namespace="your-namespace", instance="your-component"})

Data appears within one or two scrape intervals of the pods starting. Once the
query returns results, build panels and dashboards on it as usual.

If your environment runs on Google Cloud and your application sends metrics
through the OpenTelemetry Collector instead, those metrics live in a different
data source. See
Grafana — Querying Google Managed Prometheus.

Why metrics stop arriving

When metrics that used to be there disappear, or appear in one Environment but
not another, check these in order:

1. The port or path changed. A new release that moves the metrics endpoint
   leaves the metrics block pointing at the old one. Update port and path
   to match.
2. Scraping is not enabled in this Environment. Compare the Component's
   Helm values between the Environment that works and the one that does not.
   The metrics block is often present in one and missing in the other.
3. A network policy blocks the collector. This is the usual cause when the
   configuration matches but one Environment has no data. See
   Network policies above.
4. The pods are not running. Pods that are pending, crash-looping or
   restarting are not scraped for as long as they are down. Check the
   Component's status in SVIEW.
5. A label changed. Dashboards and alerts filter on labels. A new
   Component name changes instance, a new job annotation changes job, and
   a renamed metric in your code changes the series itself. The data is still
   collected, but your old queries no longer match it. Run the count by (__name__) query above to see what is arriving now.

Still having issues?

Raise a request with the Environment, the namespace, the Component name, the
metrics port and path, and the query that returns nothing.

Related articles

• Observability — the dedicated stack and Grafana access
• Grafana — Querying Google Managed Prometheus
• Network Policies — requesting cross-namespace access
• Components and Deployments — changing a Component's configuration