Ship Your Kubernetes Service Metrics and Logs to External Prometheus and Loki with Sidecars

Here is a situation many developers run into. Your application already exposes a Prometheus /metrics endpoint and writes useful logs. Somewhere, there is a central Prometheus and a central Loki, owned by other team. But your metrics never show up in Prometheus, and your logs never reach Loki, because the central systems have no configuration pointing at your namespace, and you do not have the access to add one. You cannot edit cluster-wide scrape configs, you cannot install a ServiceMonitor if the operator is locked down, and filing a ticket every time you add a metric is painful.

There is a clean way around this that lives entirely inside the one thing you do control: your own Deployment manifest. You add small sidecar containers to your pod. One runs Prometheus in agent mode to scrape your app and push metrics out with remote_write. Another runs Promtail, reads your log file from a shared volume, and pushes the lines to Loki. No cluster-admin, no operator, no central config change required.

This tutorial is for application developers and DevOps engineers who deploy to a shared Kubernetes cluster they do not own, and who want their service metrics and logs to land in monitoring systems they cannot configure directly. By the end you will have both sidecars running and shipping data to remote endpoints, plus the knowledge to secure and tune them.

How the Sidecar Approach Works

Prometheus normally works by pulling: a central server reaches out and scrapes each target’s /metrics endpoint on a schedule. That model is great when whoever runs Prometheus can list your targets. It falls apart when they cannot, or will not, configure your namespace.

The sidecar flips the responsibility. Instead of waiting for the central server to come to you, you bring tiny collectors along with your app and have them forward data outward.

A few concepts make this click:

• A sidecar is an extra container that runs in the same pod as your app. Because all containers in a pod share one network namespace, a sidecar can reach your app at localhost. Because they can also share volumes, one container can write a file and another can read it.
• Prometheus agent mode is a stripped-down way to run Prometheus. It scrapes targets and forwards samples, but it does not store data long-term, run queries, or evaluate alerts. That makes it light enough to sit next to every pod.
• remote_write is the protocol Prometheus uses to push scraped samples to another Prometheus-compatible endpoint over HTTP. This is how your metrics reach the central server.
• Promtail is the log shipping agent for Loki. It tails log files and pushes the lines to a Loki endpoint with labels attached.
• A shared volume is the bridge for logs. Your app writes its log file to an emptyDir volume mounted in both containers, and the Promtail sidecar reads that same file.

So the flow is two parallel pipelines inside one pod. For metrics: your app exposes /metrics, the agent scrapes localhost:<port>/metrics, and remote_writes to https://prometheus.example.com/api/v1/write. For logs: your app writes to /var/log/app/app.log on the shared volume, Promtail tails that file, and pushes to https://loki.example.com/loki/api/v1/push. The central team only has to give you the two URLs and credentials.

If you want the bigger picture of how these systems are built and queried, I covered setting up Prometheus and Grafana and Loki and Promtail log aggregation separately. This guide assumes those servers already exist and are owned by someone else.

Prerequisites

• kubectl access to the namespace where your app runs. Check with kubectl auth can-i create configmap and kubectl auth can-i patch deployment.
• An application already exposing a Prometheus-format /metrics endpoint on a known port. We will use 8080.
• The remote_write URL for metrics and the Loki push URL for logs, from your platform team.
• Any credentials those endpoints need, usually basic-auth or a bearer token.
• Your app able to write logs to a file path you choose, or willing to do so. More on this in the logs section.
• Basic comfort editing Kubernetes YAML.

Throughout, replace myteam, the image name, the port, and the remote URLs with your real values.

Step 1: Confirm Your App Exposes Metrics

Never wire up shipping before you have verified there is something to ship. Port-forward to a running pod and curl the endpoint:

kubectl -n myteam port-forward deploy/checkout 8080:8080

In a second terminal:

curl -s http://localhost:8080/metrics | head -n 20

You should see Prometheus exposition format, lines like:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1834
process_resident_memory_bytes 5.1273728e+07

If you get a connection refused or an HTML page instead, fix that first. The agent can only forward what the app exposes.

Step 2: Store Credentials in Secrets

Do not hardcode credentials in a ConfigMap, because ConfigMaps are stored in plain text and easy to leak. Put them in Secrets. Assuming basic-auth for both endpoints:

kubectl -n myteam create secret generic prom-remote-write \
  --from-literal=username='checkout-team' \
  --from-literal=password='REPLACE_WITH_PROM_TOKEN'

kubectl -n myteam create secret generic loki-push \
  --from-literal=username='checkout-team' \
  --from-literal=password='REPLACE_WITH_LOKI_TOKEN'

Verify they exist:

kubectl -n myteam get secret prom-remote-write loki-push

If your endpoints use bearer tokens instead, store a single token key and adjust the config in the following steps.

Step 3: Write the Prometheus Agent Configuration

The agent needs a config describing what to scrape and where to push it. Save this as prom-agent-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prom-agent-config
  namespace: myteam
data:
  agent.yml: |
    global:
      scrape_interval: 30s
      external_labels:
        cluster: shared-prod
        namespace: myteam
        service: checkout

    scrape_configs:
      - job_name: checkout
        static_configs:
          - targets: ["localhost:8080"]

    remote_write:
      - url: https://prometheus.example.com/api/v1/write
        basic_auth:
          username_file: /etc/prom-creds/username
          password_file: /etc/prom-creds/password

A walk through the important parts:

• scrape_interval: 30s controls how often the agent scrapes your app. Thirty seconds balances freshness against load and storage cost on the remote side.
• external_labels are attached to every sample the agent ships. These are critical. Because many teams may remote_write into the same central Prometheus, labels like service, namespace, and cluster are how you tell your metrics apart from everyone else’s. Agree on the naming with whoever owns Prometheus.
• static_configs.targets: ["localhost:8080"] works precisely because the sidecar shares the pod’s network with your app.
• username_file and password_file read credentials from files mounted from the Secret, so the secret never appears in the config.

If your endpoint uses a bearer token, replace the basic_auth block with:

        authorization:
          type: Bearer
          credentials_file: /etc/prom-creds/token

Apply it:

kubectl apply -f prom-agent-config.yaml

Step 4: Write the Promtail Configuration for Logs

For logs we use the shared-volume pattern. Your app writes its log file to a directory, and Promtail tails it from the same directory. Save this as promtail-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: myteam
data:
  promtail.yml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /run/promtail/positions.yaml

    clients:
      - url: https://loki.example.com/loki/api/v1/push
        basic_auth:
          username_file: /etc/loki-creds/username
          password_file: /etc/loki-creds/password

    scrape_configs:
      - job_name: checkout-logs
        static_configs:
          - targets: ["localhost"]
            labels:
              service: checkout
              namespace: myteam
              cluster: shared-prod
              __path__: /var/log/app/*.log

The parts that matter:

• positions.filename is where Promtail records how far it has read in each file, so a restart does not re-send everything. It lives on a small writable volume.
• clients.url ends in /loki/api/v1/push, the Loki ingestion path. A bare hostname will 404.
• The labels block is the log equivalent of external_labels. The same service, namespace, and cluster values let you correlate a log line with the matching metric in Grafana.
• __path__: /var/log/app/*.log tells Promtail which files to tail. This path must match where your app writes, on the shared volume.

A note on logging to a file. Many apps log to stdout by default, which is the standard Kubernetes pattern. For the shared-volume approach the app must write to a file inside /var/log/app. If your app supports a file transport, point it there. If you would rather keep logging to stdout, that is a different pattern where the sidecar reads the container runtime’s log files, which usually needs host-path access you may not have. The shared-volume file approach keeps everything inside your pod and needs no special permissions, which is exactly why it suits a developer with limited access.

Apply it:

kubectl apply -f promtail-config.yaml

Step 5: Add Both Sidecars to Your Deployment

Now merge the sidecars, the ConfigMap volumes, the Secret volumes, and the shared log volume into your existing Deployment. Adapt this to your real spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: myteam
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/checkout:v2.3.1
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: app-logs
              mountPath: /var/log/app

        - name: prom-agent
          image: prom/prometheus:v2.53.0
          args:
            - "--config.file=/etc/prom/agent.yml"
            - "--enable-feature=agent"
            - "--storage.agent.path=/data"
          volumeMounts:
            - name: agent-config
              mountPath: /etc/prom
            - name: agent-creds
              mountPath: /etc/prom-creds
              readOnly: true
            - name: agent-data
              mountPath: /data
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { cpu: 100m, memory: 128Mi }

        - name: promtail
          image: grafana/promtail:3.0.0
          args:
            - "-config.file=/etc/promtail/promtail.yml"
          volumeMounts:
            - name: promtail-config
              mountPath: /etc/promtail
            - name: loki-creds
              mountPath: /etc/loki-creds
              readOnly: true
            - name: app-logs
              mountPath: /var/log/app
              readOnly: true
            - name: promtail-positions
              mountPath: /run/promtail
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { cpu: 100m, memory: 128Mi }

      volumes:
        - name: app-logs
          emptyDir: {}
        - name: agent-config
          configMap:
            name: prom-agent-config
        - name: agent-creds
          secret:
            secretName: prom-remote-write
        - name: agent-data
          emptyDir: {}
        - name: promtail-config
          configMap:
            name: promtail-config
        - name: loki-creds
          secret:
            secretName: loki-push
        - name: promtail-positions
          emptyDir: {}

The pieces that matter:

• --enable-feature=agent turns regular Prometheus into the lightweight agent. Without it you get a full Prometheus that keeps a local TSDB and tries to serve queries, which is wasteful and will likely OOM the sidecar.
• The app-logs emptyDir is the shared bridge. The app mounts it read-write at /var/log/app and writes its log file there. Promtail mounts the same volume read-only at the same path and tails it. This is the heart of the shared-storage log pattern.
• --storage.agent.path=/data and the promtail-positions mount give each shipper a small scratch area to buffer data and track progress, so a brief network blip does not lose samples or re-send logs.
• The resources blocks keep both sidecars small so they never compete with your actual application.

Apply and watch the rollout:

kubectl apply -f deployment.yaml
kubectl -n myteam rollout status deploy/checkout

Step 6: Verify Both Pipelines Are Flowing

Check the metrics agent first:

kubectl -n myteam logs deploy/checkout -c prom-agent

A healthy log shows Server is ready and a remote write WAL watcher starting, with no repeated errors. Two failures are common and loud: HTTP 401 or 403 on remote_write means wrong credentials, and connection refused means the agent cannot reach your app on localhost:8080.

Now check Promtail:

kubectl -n myteam logs deploy/checkout -c promtail

Look for it detecting your file and successfully pushing batches. If you see error sending batch with a 4xx, re-check the Loki URL and credentials. If it never finds a file, confirm your app is actually writing to /var/log/app/*.log.

Finally, confirm the data arrived on the other side, if you have query access:

up{service="checkout"}

in Prometheus should return 1, and a LogQL query in Grafana like:

{service="checkout"}

should return your recent log lines. Because both pipelines use the same service, namespace, and cluster labels, you can pivot between a metric spike and the matching logs in one dashboard. If you have no query access, clean sidecar logs and 2xx responses are your confirmation.

Common Mistakes and Troubleshooting

Forgetting consistent labels. Without distinguishing labels, your metrics blend with every other team’s and your logs are unfindable. Always set service, namespace, and cluster on both shippers, and keep them identical so correlation works.

Running full Prometheus instead of agent mode. Omitting --enable-feature=agent makes the container hold a local TSDB and use far more memory than provisioned, ending in an OOMKilled sidecar. The flag is mandatory.

App logs to stdout, Promtail reads a file. This is the most common log mistake here. If the app prints to stdout but Promtail is told to read /var/log/app/*.log, nothing ships. Either configure the app to write that file, or switch to a stdout-reading pattern, which needs host access you may not have.

Credentials in a ConfigMap. ConfigMaps are plain text and often land in git. Always use Secrets with the *_file config options so the value is mounted, not embedded.

Cardinality explosions. High-cardinality labels, like a unique user ID per metric series or per log stream, flood the central systems fast. Keep label values bounded.

Wrong endpoint paths. Metrics must end in /api/v1/write, logs in /loki/api/v1/push. A bare hostname returns a 404 that is easy to misread as an auth error.

Best Practices

• Trim what you send. Drop noisy metrics before they leave the pod with metric_relabel_configs, and avoid shipping debug-level logs in production. Less data means lower cost and faster queries for everyone using the central store.

• Match the team’s resolution. Scraping every 5 seconds quadruples the volume versus 20 seconds for little benefit. Use the interval your team actually dashboards on.

• Always set requests and limits on sidecars. It guarantees scheduling and prevents a shipper from starving your app.

• Secure the transport. Use https for both endpoints so credentials and data are never sent in clear text. For a private CA, mount the CA cert and reference it under tls_config.

• One agent and one Promtail per pod. They cover the whole pod over localhost and the shared volume. If your pod runs two apps, add both ports to the scrape targets and both log paths to __path__ rather than running more sidecars.

• Structure your logs. Promtail ships whatever your app writes, so JSON logs with consistent fields are far more queryable in Loki than raw text. I covered exactly that in structured logging in Node.js with Winston.

Conclusion

You started with a service whose metrics were invisible to the central Prometheus and whose logs never reached Loki, plus a hard constraint that you cannot touch the cluster’s monitoring configuration. By adding two small sidecars to your own Deployment, you turned the model around. A Prometheus agent scrapes your app over localhost and remote_writes the samples, while a Promtail sidecar reads your log file from a shared emptyDir volume and pushes the lines to Loki. Everything happens inside the one resource you fully control.

The pattern is small but powerful. The whole solution is two ConfigMaps, two Secrets, a shared volume, and two extra containers in your pod. It needs nothing from the platform team except endpoint URLs and credentials, and it scales naturally because every pod carries its own lightweight shippers.

From here, two next steps are worth taking. First, agree on a label convention with whoever owns Prometheus and Loki so your data slots cleanly into their dashboards and alerts. Second, learn to read the destinations yourself by exploring Prometheus and Grafana and Loki and Promtail, so that once your metrics and logs arrive, you can actually act on them.