Summary

An organization running a cloud-native architecture on AWS, with stringent targets of 99.999% uptime and sub-100 millisecond response times, lacked the real-time visibility and centralized monitoring needed to proactively detect and resolve incidents across its microservices and APIs. Bajaj Tech.AI built a comprehensive monitoring and alerting solution using Grafana, Prometheus, OpenSearch, and Thanos on Amazon EKS, integrated with ITSM tools for automated incident management. The result: 70% faster issue resolution, a 55% reduction in mean time to resolve (MTTR), a 15% increase in customer satisfaction, and a 25% reduction in security vulnerabilities.

Business Challenge

With 40-50% of businesses striving to meet demanding KPI targets for uptime and response times, the pressure for advanced monitoring and alerting capabilities was significant and companies that fail to maintain infrastructure reliability risk damaging customer trust and losing their competitive edge.

The organization's cloud-native architecture, hosted on AWS, consisted of numerous microservices and APIs, each with its own performance requirements and interdependencies. With several teams sharing the same lower environments, managing infrastructure performance became a complex task. The lack of real-time visibility and centralized monitoring made it difficult to proactively detect and resolve incidents, leading to delayed response times and impacting overall service availability. The company also faced the challenge of integrating monitoring and alerting systems with existing IT Service Management (ITSM) tools to streamline ticketing and incident management.

Stringent performance targets 99.999% uptime and sub-100 millisecond response times added pressure on teams to maintain optimal service levels. The organization's DevSecOps approach also required continuous monitoring to ensure compliance and security across all deployments, further complicating the monitoring strategy.

At 99.999% uptime, there's essentially no room for a slow manual incident response the target itself demanded a monitoring architecture built for speed and automation from the start.

Solution Approach

An Infrastructure Monitoring and Alerting solution was developed using a suite of open-source tools: Grafana, Prometheus, OpenSearch, and Thanos, all deployed on Amazon EKS. The solution provided comprehensive monitoring, log analysis, and alerting capabilities, integrated seamlessly with call and SMS notification services for real-time alerts. The team began by categorizing the infrastructure landscape and establishing standard monitoring and alerting policies for each category.

• Grafana for visualization and dashboards: Grafana visualized metrics and performance data through customizable dashboards monitoring key infrastructure KPIs, CPU utilization, memory consumption, API response status codes, and latency giving teams real-time insight for data-driven decisions.
• Prometheus for metrics collection and alerting: Prometheus collected and stored time-series data from various AWS services and APIs. Alert thresholds were defined based on business-critical metrics, with Prometheus Alertmanager configured to trigger notifications to a custom Python application that created ITSM tickets and sent SMS and call notifications whenever thresholds were breached.
• OpenSearch for log aggregation and analysis: OpenSearch aggregated and indexed logs from all applications and microservice components, providing a centralized log repository that facilitated faster debugging and root cause analysis during incidents with logs also visualized and alerted on through Grafana for faster error response.
• Thanos for scalable metric storage: Thanos extended Prometheus by enabling long-term storage of monitoring data, giving the organization a highly available, scalable solution for storing and querying historical metrics crucial for audit, compliance, and performance trend analysis.
• ITSM integration for automated incident management: The monitoring system integrated with ITSM tools like GLPI and ManageEngine, enabling automated ticket creation for P1 and P2 alerts, ensuring incidents were promptly logged and assigned to the appropriate teams.
• DevSecOps compliance monitoring: The solution was embedded into the DevSecOps pipeline, enabling continuous monitoring and compliance checks during each deployment phase, ensuring security and performance standards were met throughout the development lifecycle.

Building entirely on open-source tools kept the solution flexible and cost-efficient, while the ITSM integration ensured that a detected anomaly turned into an assigned, trackable incident not just an alert sitting in a dashboard.

Business Impact & Results

• Enhanced visibility and faster issue resolution: Real-time dashboards and centralized log management provided a unified view of the entire infrastructure, API calls, and microservices health, enabling teams to detect and resolve issues 70% faster than before.
• Improved uptime and customer experience: By proactively identifying performance bottlenecks and anomalies, the organization consistently achieved its uptime target of 99.999%, resulting in a 15% increase in customer satisfaction and a higher Net Promoter Score.
• Optimized incident management: Integration with ITSM tools and automated ticketing streamlined incident management, reducing mean time to resolve (MTTR) by 55%.
• Scalable and compliant monitoring: The use of Thanos for long-term metric storage ensured the solution could scale as the business grew, while maintaining compliance with industry regulations.
• SeamlessDevSecOps implementation: Embedding the monitoring system into the DevSecOps pipeline reduced security vulnerabilities by 25% and ensured continuous compliance throughout all development stages.

These gains reinforced each other: faster detection led to faster resolution, which supported the uptime target, which in turn drove the customer satisfaction improvement all traceable back to the same underlying monitoring investment.

Key Takeaways

• A 99.999% uptime target effectively requires automated, real-time monitoring manual processes can't keep pace with that level of reliability
• Combining Grafana, Prometheus, OpenSearch, and Thanos gives comprehensive coverage across metrics, logs, and long-term storage using entirely open-source tools
• ITSM integration is what turns a detected anomaly into an assigned, trackable incident instead of an ignored alert
• Embedding monitoring into the DevSecOps pipeline catches security and compliance issues during deployment, not after an incident
• This kind of monitoring architecture reflects the same discipline behind Bajaj Tech.AI's API and infrastructure monitoring blog and SRE practice

Conclusion

This implementation of a comprehensive monitoring system using open-source tools facilitated improved operational efficiency, high availability, and stringent KPI targets while delivering exceptional customer experiences, a 70% faster issue resolution, 55% reduction in MTTR, 15% increase in customer satisfaction, and 25% reduction in security vulnerabilities, all from the same monitoring investment. The solution's scalability, flexibility, and integration capabilities have enabled the organization to adapt to the changing demands of the digital landscape and maintain a competitive edge.

This kind of cloud monitoring discipline pairs naturally with Bajaj Tech.AI's cloud cost optimization and disaster recovery work — together forming a complete picture of resilient, cost-efficient infrastructure.

Looking to build monitoring that meets stringent uptime and compliance targets? Connect with our experts to explore the right approach for your organization.