Using Prometheus to Monitor Kubernetes Clusters Running Cloud Foundry

Cloud Foundry sponsored this post.

Telemetry lies at the heart of every application that needs to be optimized for performance. In addition, every abstraction layer built over the cloud infrastructure running these applications introduces a new set of parameters that need to be observed for potential failures.

Containers made application monitoring slightly more complex than before. Then, the introduction of Kubernetes brought another layer of abstraction and inadvertently monitoring challenges became an order of magnitude more complex. Using Kubernetes implies that there are new parameters to monitor and troubleshoot. If you’re also working with microservices, a new set of application monitoring challenges present themselves.

Every Platform-as-a-Service (PaaS) tool comes with the promise of “focus on your code.” In addition, consider that PaaS tools typically come with proprietary components and black box environments as part of the whole build-and-deploy tool chain. Does an engineer totally lose control over the ability to optimize performance?

Let’s turn specifically to Cloud Foundry and its promise: Cloud Foundry aims to provide a simplified PaaS experience for developers who aim to deploy their microservices to Kubernetes. 

PaaS, Kubernetes, plus microservices — sounds like a recipe for monitoring disaster doesn’t it?

Not quite! Cloud Foundry (CF) is open source and in turn, uses other popular open source frameworks to fulfill its promise of a simplified developer experience. Using all these open source components allows unprecedented transparency into the PaaS framework. Specifically, CF-for-K8s employs other open source components — such as FluentD, Istio, Envoy, Eirini, etc — and blends the developer-friendly Cloud Foundry API with the power of Kubernetes.

CF-for-K8s makes use of Prometheus for scraping metrics and firing events. There are many reasons the CF community decided to make use of Prometheus, but chiefly because of its popularity among the DevOps community and that it is open source software. Prometheus is also robust and extensible, providing great coverage of metrics, and has a tested way of installing as containers. To top it all, when paired with complementary technologies such as Grafana, OpenTracing, Zipkin or Jaeger, engineers choosing Cloud Foundry get a complete monitoring stack.

A common question is, will CF-for-K8s with Prometheus install with Helm? While it is possible to install Prometheus using Helm charts, it requires a lot of configuration to make it work with the internal namespaces, network policies, and security certs. Therefore, installing Prometheus on a CF-for-K8s instance using Helm is not the recommended path.

Instead, there is a simpler way to set up Prometheus when using CF-for-K8s. When installing the Cloud Foundry abstraction over Kubernetes, you can include config files that also include Prometheus pods as part of the CF API deploy.

Let’s switch to tutorial mode for a bit and learn how to install Prometheus as part of a CF-for-K8s installation.

1. First, clone the GitHub repo cf-k8s-prometheus.

2. Switch to the cloned directory. Inside this repo you will find the required config files.

3. Next, use the Carvel (formerly k14s) tool ytt to create the templates that will be used for deployment. All the required tools, including Prometheus, will be included as declarative syntax in the YAML file that is generated.

4. We will then deploy the application using the kapp tool. This will install CF-for-K8s, along with Prometheus on the Kubernetes cluster.

5. Typically, users are required to patch the Istio Ingress gateway with a domain or a static IP, so it can be accessed from external networks.

There are a couple of additional steps that you need to take to login and create target spaces within the Cloud Foundry installation. That is slightly out of scope for this article (details available here).

The Prometheus deployment can be confirmed by getting information about all the pods running on the cf-system namespace on the Kubernetes cluster.

6. To demonstrate that Prometheus is fully functional, you can deploy a test application that is configured with some telemetry. In this particular case, we use an application written in Go that is available on GitHub.

7. The manifest.yml file, which is in the TLD of the application, includes the following metadata:

This metadata is used by Prometheus to set up a route and local port to which the raw metrics data is forwarded. Deploy the app by using the standard command:

8. Once the app is deployed successfully, we can get their status and pod names within the cf-workloads namespace. Copy the name of the pod –

– and forward the traffic from the local port to the pod that runs Prometheus.

9. Confirmation of the port forwarding being set up successfully will be shown on the terminal.

10. Opening the route in a browser (in this case localhost:2112/metrics) will display all the metrics that are being collected from the application running on the pod.

👁 Image

11. Alternatively, you can also curl the endpoint right from the terminal. The result is the same.

The Cloud Foundry community is hard at work to provide support for a full-fledged monitoring stack for CF users. Support is currently available for Grafana, which can be used to visualize data captured by Prometheus. It is a work in progress and many improvements are in the pipeline.

Some of the feature requests we have received so far:

• How does CF integrate with existing monitoring stacks?
• Will there be support for distributed tracing?
• Does CF work with ZipKin and Jaeger?

There are many more open questions related to monitoring tools that need answers, which we will provide in the near future. Meanwhile, you’re invited to join the Cloud Foundry community on Slack to discuss all aspects of operating Cloud Foundry on Kubernetes. You can find issues pertaining to monitoring and observability on the #logging-and-monitoring channel.