Near Real-Time Data Architecture on Kubernetes
The client is an innovative company in the cognitive security sector that needs to give its end users immediate, in-depth visibility into security events. In a market where timely information is critical, the ability to offer dynamic, up-to-date dashboards becomes a key competitive advantage, especially when dealing with C-level executives who expect advanced reporting.
The existing reporting infrastructure (as-is), although operational, had clear architectural limits. First, the system offered little customization for the growing and specific requests of end customers. Given this starting point, processing highly variable data flows, ranging from 5 to 200 transactions per second (TPS), was out of the question. In addition, the system could not keep dashboards updated in real time under stress (a near real-time requirement). Finally, given the nature of the customers, the new solution had to install identically and reliably both on-premises (for customers with strict security policies) and in the cloud as a SaaS service (with no deployment constraints).
Hybrid by Design
An open source, cloud-native reporting platform that scales dynamically to handle complex transactional flows. With automation based on Helm charts, the architecture delivers a very fast time to market and supports both on-premises and SaaS delivery models, while maintaining the highest standards of security and governance.
To address these challenges, Miriade designed and implemented, in about 5 months (covering analysis, development, and testing), a near real-time platform built on Kubernetes infrastructure.
At the heart of the solution is a fully automated architecture based on Helm charts, which governs the deployment of the entire stack. This engineering approach automatically generates the ingestion pipelines, the underlying data structures, and the baseline dashboards.
The Hybrid by Design approach ensures flexible deployment (on-premises or cloud), while native integration with modern authentication protocols such as SAML and OIDC provides the access governance and security that enterprise environments require.
To ensure scalability, performance, and vendor independence, the solution was built by integrating the best open source layers.
- Kubernetes and Helm charts for container orchestration and fully automated deployment in hybrid environments.
- Apache Superset, chosen as the engine for data visualization and dashboarding, giving end users maximum flexibility.
- PostgreSQL with the Timescale extension, implemented for highly efficient storage of historical time-series data.
- Apache Kafka and Kafka Connect, used for resilient data queue management and asynchronous real-time ingestion.
- Redis, integrated as the caching layer, essential for maintaining performance during traffic peaks.