DevOps Matrix

Continuous Monitoring

"Continuous monitoring is the process and technology used to detect compliance and risk issues associated with an organization's financial and operational environment. The financial and operational environment consists of people, processes, and systems working together to support efficient and effective operations. Controls are put in place to address risks within these components. Through continuous monitoring of the operations and controls, weak or poorly designed or implemented controls can be corrected or replaced – thus enhancing the organization's operational risk profile. Investors, governments, the public and other stakeholders continue to increase their demands for more effective corporate governance and business transparency."

-- Wikipedia

The ultimate goal of implementing Continuous Monitoring is to improve the company’s Risk Profile and gain confidence from the investors. However, as a technical person, I don’t intend to bore you to death about Risk Management and or details on how to contribute to the company’s portfolio. I am not an expert in that field either. This section is intended to describe how to monitor items that are related to application development and support.

The target beneficiaries are the Developers and Technical support. Finance is also as beneficiary by reducing the operational costs like the AWS/GCP Cloud billing statements. To achieve the “Company Portfolio” level benefits, we must address the Continuous Monitoring in our own backyard, the Delivery and Operations Team.

There is no complete Continuous Monitoring framework in the DevOps world, so I invented one! This framework consists of existing DevOps practices such as RED & USE Methods.

I invented the "STRATUM DIAGRAM" to clearly define different types and layers of monitoring. Adopted from the word Stratum, which is used by geologists to describe a bed or layer of sedimentary rock that is visually distinguishable from adjacent beds or layers. Each layer provides tell-tales about earth's past events.

For DevOps monitoring purposes, each layer corresponds to a network layer. The elongated blocks (ex. blue) represents the monitoring-scope. Each monitoring scope will have specific metrics and a purpose. By combining the multiple monitoring scopes, this will allow us to pinpoint where issues occur.

Blue monitoring-scope

1. Check the end-to-end connections with the best effort to include all the involved components and network layers.
2. Sample Assertions

- A sanity test that retrieves data from the database and checks its value. This data should be public and no authentication should be required.

- Sanity Automated test to read from a page content. The source of the content is an S3 bucket.

Orange monitoring-scope

1. Check the application's basic business functionalities.
2. Sample Assertions

- A sanity test that retrieves data from the database and checks its value.

- Sanity automated tests that retrieve data from other microservices that are also connected to their corresponding databases

Green monitoring-scope

1. Resource monitoring like CPU & RAM usage.
2. Requested vs Actual.Pod restarts acceptable counts, etc.

Technical Implementations: Traceability

Kiali is an observability console for Istio with configuration capabilities like canary traffic adjustments. It helps you to understand the structure of your microservices architecture and also provides the health of your servers. Kiali provides detailed metrics, and a basic Grafana integration is available for advanced queries.

I took the GIF image from my Demo App. It shows the transaction flows between Microservices. Thru Kiali, it is a lot easier to track the flow of transactions compare with the traditional approach that is via console logs.

Kiali is more than just for traceability. You can control the flow of traffic between two versions of the same application. For example, you can use it to deploy a canary version. See the staff-service and the staff-service-risky-version modules (right most). I configured kiali to direct 90% of the traffic to staff-service, a stable version. Then 10% to staff-service-risky-version, the latest version. For some use cases, it is beneficial to try the application first with a small portion of users before making a full release.

Server Resource Utilization

For example, you have Metrics-Server or Prometheus and Grafana installed and configured to monitor your server and have gathered enough data. What will you do with the data? What data will you gather?

Data from CPU and Memory utilization reports must be regular inputs in the way we develop our applications or manage CPU/RAM allocations or autoscalling configurations.

For example, in Dockerfile, Kubernetes pods, and ECS tasks definitions. Initially, we guess the CPU and RAM allocations. Once data becomes available, the configuration of the CPU/RAM values must be tweaked for optimal operations.

// TODO: Add Grafana / Prometheus Graphics example of CPU and Memory usage.

//TODO Jaeger

• Jaeger is a far more advance of implementing traceability.
• Jaeger can allow us to trace a transaction of a single user.
• The full implementation of Jaeger requires modification of source code or should be considered in the Design Patterns followed by the application. It is important that the implementor of Jaeger has (Java or other languages) coding and architectural skills.