Making Seamless Edge Computing, Seamless

Seamless edge computing components logs is a challenge. Why? Because there are a number of pertinent actions I have to take in order to make it happen. 

I need a lightweight client.

I need an authentication system that can verify that client (the “collector”).

I need to be able to enrich the client data by transferring the relevant parameters.

Now that I have an aggregator that can keep all of this information stored, I need to be able to see it all displayed in a centralized place. 

And then I have the actual queries that are meant to enable me to make this orchestra play in harmony, which can get messy. 

Most of these steps can be made simpler by utilizing various OSS tools available at your disposal. Let’s take a look at how we can utilize these well-known tools and discover some lesser-known ones that can help make your implementation seamless. 

Aggregate with Loki

Grafana Loki is an open source log aggregation system developed by Grafana Labs, designed for use with Prometheus and other monitoring solutions. It is designed to provide easy access to log data across multiple sources, and to provide a unified view of the data. Loki is highly scalable and provides powerful features to help make sense of log data. Grafana Loki offers a number of advantages over traditional log aggregation solutions. Here are some of the main benefits of using Grafana Loki:

1. ‍Scalability: Loki is designed to scale up to hundreds of millions of log lines, making it an ideal solution for large organizations.
2. ‍Ease of Use: Loki is designed to be easy to use, and can be deployed in minutes. It integrates with existing logging solutions, and allows users to quickly search and analyze logs.
3. ‍Cost-Efficiency: Loki is an open source solution, and there are no licensing costs associated with it. This makes it an attractive option for users looking for a cost-effective solution.
4. ‍Flexible Data Model: Loki allows users to store logs in a variety of formats, including JSON, protobufs, and other formats. This makes it easy to integrate with existing solutions.

Collect with Fluent Bit

Fluent-Bit is an open source data forwarding and collection software designed for edge computing. It enables you to collect and forward data from multiple sources to a centralized data store. It can also be used to collect and forward logs from various sources, such as containers, servers, and IoT devices. The Fluent-Bit data pipeline is fast, reliable, and secure. One of the main benefits of Fluent-Bit is its ability to reduce latency. It can quickly collect and forward data, which helps to reduce the amount of time needed to move data from the edge to the central data store. This helps to reduce the time it takes to analyze data, as well as the time it takes to respond to events. Fluent-Bit also provides improved security. It encrypts data as it moves from the edge to the central data store, helping to protect sensitive areas.

Query with Grafana

Grafana is a popular open source dashboard and visualization tool that allows users to easily monitor their data and visualize it in a meaningful way. As Grafana continues to grow in popularity, so do the number of data sources available for it. One such data source is Loki, an open source, horizontally-scalable, multi-tenant log aggregation system. Loki provides a powerful and cost-effective solution for ingesting, querying, and analyzing logs in Grafana. Loki is optimized for Grafana's LogQL query language, which makes it easier to explore and analyze log data. Loki is built on a horizontally-scalable, multi-tenant architecture, which means it can scale to handle large volumes of log data from multiple sources. It has the ability to store logs in a highly compressible format, which helps reduce storage costs. Additionally, it supports dynamic label-based filtering, which allows users to quickly filter out irrelevant logs and focus on the ones that are important.

Multi Tenancy Challenges

In a multi-tenant environment, each user or team must be able to access its own set of data without interfering with other tenants. To enable this, Loki uses its own label system to identify each tenant and its associated data. This allows Loki to easily distinguish between tenants and ensure that only the appropriate data is accessed. Unfortunately, this label system also limits the efficiency of multi-tenancy queries. The query API request defines the tenants. Specify multiple tenants in the query request HTTP header _{'X-Scope-OrgID'} by separating the tenant IDs with the pipe character (|). This can be inefficient and time consuming, as the query must be updated each time a new tenant is added or removed.

In addition, manual tenant configuration can lead to security risks. This is because it is difficult to ensure that only authorized tenants are able to access the edge device, as the manual configuration must be updated each time a new tenant is added or removed. This can leave the edge device vulnerable to malicious actors. Finally, manual tenant configuration can limit the scalability of an edge computing system. This is because manual configuration requires users to manually specify which tenants are allowed to send their data to the edge device. As the number of tenants grows, this manual configuration process can become increasingly difficult to manage. Overall, manual tenant configuration on Fluent Bit can be a time-consuming and error-prone process, and can lead to security risks and scalability issues. As such, organizations should consider using an automated tenant configuration solution to ensure that their edge computing system remains secure and scalable.

Shift Left Tenants Management with API-GW

Kong API Gateway is one of the most popular API management platforms, providing enterprise-grade security, scalability, and reliability. It offers an array of features including key authentication and transformer plugins, which can be used to improve the security, performance, and flexibility of your API endpoints. Key Authentication: Key authentication is a method of authentication that uses a unique key to identify and authorize API requests. This method is used to ensure that only authorized requests can access the API. With Kong, you can use API keys to authenticate requests and ensure that only those with valid credentials can access the API.

Specific use case is to use the consumer username attribute in the key authentication plugin as the tenant ID which is accessible over the _{‘X-Consumer-Username’} header.

Now our Fluent-Bit output configuration will require only an API key to authenticate and identify itself.

Next step will be to use the transformer plugin to rename the header to Loki’s tenant header _{‘X-Scope-OrgID’.}

Tenant to Label with Loki-Proxy 

When it comes to multi-tenant queries, Loki requires a static HTTP header with all tenants ids separated with pipe character, limiting the use of dynamic tenant ids and therefore its usability in real-life use cases.

This is where the ”Loki-proxy” comes in - a simple HTTP proxy server writing in Go, which can aggregate push requests to a single tenant, appending the original tenant id value as a label, circumventing the static multi-tenant mechanism offered by Loki.

Using the ”Loki-proxy” requires zero code changes in the client side, allowing teams to instantly forward traffic through the proxy, maintaining all previous functionality of the Loki endpoint.

If you're looking to find a good solution to seamless edge computing, check out our code, it may just be what you're after: https://github.com/groundcover-com/loki-proxy

Stepping up the ease of multi-tenancy scalability

Multi tenancy is a core building block in building an enterprise grade solution that needs to be implemented with great care, this can be challenging in dynamic edge environments.

The above solution aims to solve core issues in implementing multi-tenant architecture that is secure - by shifting left tenant assignment, and allowing for scalability - by making new tenants discovered automatically.

We hope our solution in general, and the open source loki-proxy specifically, can help the community implement multi-tenancy that fits their solutions the best.