Scaling Kubernetes Observability: Aptos Labs & groundcover

About Aptos Labs

Aptos Labs is a blockchain technology company building a scalable, secure, and user-friendly Layer 1 blockchain. Launched in 2022 with a vision to enable mainstream Web3 adoption, Aptos Labs provides infrastructure and developer tools that power a wide range of applications, from DeFi and NFTs to gaming and enterprise solutions. Being a modern company in the Web3 age and a heavy performance oriented focus, they are built on top of Kubernetes and Cloud Run functions in GCP, with infrastructure spread across multiple sites globally, to support high availability and minimize latency.

The Problem: Infrastructure Monitoring Challenges in a High-Performance Blockchain.

Aptos Labs operates a high-performance Layer 1 blockchain, making infrastructure monitoring critical to ensuring availability, low latency, and seamless performance. Given the decentralized nature of the network, production environments demand rigorous real-time observability to track node health, transaction propagation, and consensus stability.

Beyond production, Aptos runs extensive staging environments that simulate blockchain behavior before new features go live. Some of these environments are open to real users, adding complexity by mimicking real-world conditions. This hybrid staging approach increases the load on platform engineers, who must monitor numerous interconnected components delivering real-time functionality. Ensuring visibility across both production and pre-production environments is essential to maintaining system reliability and performance while continuously shipping innovations.

As a result, they rely on many metrics to monitor the health of their platform in various stages and deep log analytics to detect complex behaviors and create proactive alerts on anything that skews from the expected performance baseline.

Aptos Labs was relying on a hybrid observability stack based on:

• VictoriaMetics Cloud for all infrastructure metrics
• GCP managed Prometheus service for additional custom metrics
• LogScale for log management and analytics
• Grafana Cloud for dashboarding and synthetic monitoring

Their existing stack created multiple challenges:

• Metric stack was unreliable: Relying on metrics for troubleshooting and alerting was critical to the Aptos platform team. With multiple downtimes, performance issues, and inability to get clarity on the status of the VictoriaMetrics Cloud platform, the Aptos team was considering moving to a self-hosted version of their metric stack for enhanced control and governance.‍
• Distributed observability backend made troubleshooting complicated:  High availability is a crucial requirement of observability for Aptos Labs. The team was using multiple single instances of VictoriaMetrics for their metrics hosting on VictoriaMetrics Cloud. They wanted to consolidate this distributed deployment into a VictoriaMetrics Cluster deployment to be able to easily visualize metrics from different regions in one place.‍
• Metrics stack wasn’t scalable in a cost-efficient way:  VictoriaMetric Cloud’s pricing model didn’t allow the team to build out their metrics stack the way they wanted. With the different pricing offered by groundcover they could get more value from their metrics stack in a fully managed experience.

"groundcover has helped us a lot in many troubleshooting incidents… We automatically gained deep visibility into what’s happening in our Kubernetes cluster, at the infrastructure level but also all the way down to the individual workload level.”

- Christian Theilemann, Staff Software Engineer, Aptos Labs

Why groundcover?

• Automated Kubernetes observability: The native monitoring of their K8s stack, including visibility at the network level, infrastructure events, and a deployment-level grouping of all insights, allowed the team to get better visibility into their stack. It helped the team build a more comprehensive and coherent story about what is happening in their environment.
• From raw logs and metrics to deep, actionable insights: Throughout the platform, groundcover offers the ability to correlate and contextualize data with logs, metrics, infrastructure events and traces. This was a major improvement from the typical, fixed metrics dashboards and raw logs searches. The ability to view events and issues on a timeline, getting a workload-centric view, and correlating the infrastructure and application layers into one view, sometimes made all the difference and helped investigations.
• Much more value for the same price: groundcover’s unique data-agnostic pricing model allowed the team to get the best of both worlds - a full-fledged observability backend using a technology stack they believe in and hosted under their control, and the ability to scale queries and the amount of data stored without it inflicting unpredictable costs.

“Whenever we suspect there’s an issue with our infrastructure we open groundcover and figure out what is actually going on. ”

- Christian Theilemann, Staff Software Engineer, Aptos Labs

The Impact

Aptos Labs was able to migrate from their legacy metrics stack, replacing their VictoriaMetrics Cloud and GCP’s Managed Prometheus with groundcover.

1. Much faster troubleshooting: groundcover is now the team’s go-to platform whenever there is a complex, deep infrastructure issue. It has reduced time to detection and remediation of most incidents, and improved the team’s ability to predict and preemptively tackle potential future issues.
2. Improved reliability: groundcover’s fully managed BYOC backend allowed better reliability and transparency into the health of Aptos Labs’ monitoring stack. The team can self-monitor the health of the BYOC infrastructure and rely on groundcover’s built-in support for high availability and scaling of the VictoriaMetrics and ClickHouse databases.
3. More intuitive user experience: Querying metrics across multiple separated backends of VictoriaMetrics made consolidated troubleshooting too complex for the Aptos team. Now, with groundcover’s unified BYOC backend, all the data is located in one place, helping to streamline processes. Additionally, groundcover’s user interface (UI), often used for live investigations, provided a much more intuitive way to view their Kubernetes stack.