What are OpenTelemetry Logs? Key Concepts, Benefits, and Best Practices

Logs are a key source of observability insights, and OpenTelemetry is a framework that can help you collect logs (alongside other sources of insight, namely metrics and traces) to observe the state of your applications and infrastructure. In fact, not only does OpenTelemetry provide functionality for collecting logs, but it also offers support for processing and exporting logs. And it supports multiple types of logs.

All of the above make OpenTelemetry a powerful log management solution. That said, OpenTelemetry logging is also complex, and it can take some time to wrap your head around exactly how the various OpenTelemetry logging components and concepts fit together.

To provide direction, we’ve prepared the following guide to OpenTelemetry logs. Below, you’ll learn how OpenTelemetry manages logs, which types of logs it supports, how the OpenTelemetry log data model works, and how to get the most out of OpenTelemetry logging.

What are OpenTelemetry logs?

OpenTelemetry logs are log files that you can collect and manage using OpenTelemetry.

To understand fully what that means, let’s step back and talk about OpenTelemetry for a moment. OpenTelemetry is an observability framework designed to standardize and simplify the process of collecting observability data – specifically logs, metrics, and traces, which constitute the so-called “three pillars” of observability.

Since applications and programming languages generate logs in varying ways, OpenTelemetry’s approach to logging is to make it possible to collect and manage logs in their native format. In other words, you don’t have to create special logs, or format your logs in a particular way, for OpenTelemetry to be able to work with them. OpenTelemetry can simply go in and grab logs based on the existing log formats that your infrastructure and applications are already generating, and then process them as needed.

OpenTelemetry logs vs. traditional logging systems

That said, OpenTelemetry logging is more than just another log aggregation or log collection system. It provides additional capabilities that distinguish it from traditional logging systems ((such as Log4j, SLF4J, or Python’s logging module):

• Agnostic logging: OpenTelemetry can work with virtually any type of log file, including both structured and unstructured logs. Traditional logging tools sometimes only support certain types of logs or apps written in certain languages.
• Native processors: OpenTelemetry provides built-in processors that can transform or restructure log data, which can make it easier to interpret.
• Holistic observability: Whereas most traditional logging systems are designed for log aggregation and management alone, OpenTelemetry supports logs in addition to metrics and traces, which enables comprehensive observability.

Types of OpenTelemetry logs

At a high level, OpenTelemetry logs can be broken into three basic categories.

1. System logs and infrastructure logs

In OpenTelemetry, system logs and infrastructure logs are log files generated by the core infrastructure or platform that hosts applications, as opposed to applications themselves. Examples include operating system logs, logs produced by cloud infrastructure services, and network logs.

2. First-party application logs

First-party application logs are OpenTelemetry logs produced by applications that a business builds and deploys itself. These logs typically provide a deep level of visibility into application performance and health because the business can include any data it wants within those logs files, since it controls the applications (and therefore the types of log data that its apps generate).

3. Third-party application logs

Third-party application logs are those made available by apps developed by an external vendor. Collecting these logs via OpenTelemetry is important because even if you don’t fully manage a third-party app, you’ll still want to know if it’s experiencing performance issues that may impact workflows inside your organization that depend on the app. However, because you don’t control third-party apps, you typically have to rely on external APIs or software services to provide access to the log data.

Key components and concepts within OpenTelemetry logging

To enable an agnostic, flexible approach to logging, OpenTelemetry defines several distinct logging concepts and supplies software components to support them.

Log Record

OpenTelemetry defines a Log Record as any event that is recorded in a log file and that includes:

1. At least one top-level field, such as a timestamp or trace ID related to the event.
2. Resource values and attributes.

For instance, here is an example of a Log Record in JSON format:

{
  "timestamp": "2024-12-28T15:20:00Z",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "severityText": "INFO",
  "severityNumber": 9,
  "body": "User login successful",
  "attributes": {
	"http.method": "POST",
	"http.url": "https://example.com/api/login",
	"http.status_code": 200,
	"user.id": "12345",
	"user.ip": "192.168.1.100",
	"session.id": "a1b2c3d4e5f6"
  },
  "resource": {
	"service.name": "auth-service",
	"service.version": "1.2.3",
	"host.name": "auth-server-01",
	"cloud.provider": "aws",
	"cloud.region": "us-west-2"
  }
}

Key values within this file include:

• timestamp: The time the log was recorded.
• traceId: Correlates the log with a specific trace.
• spanId: Correlates the log with a specific span within the trace.
• severityText & severityNumber: Represents the log's severity level (e.g., INFO, ERROR).
• body: The main message or description of the log event.
• attributes: Key-value pairs providing additional context, such as HTTP method, user ID, and session ID.
• resource: Metadata about the application or environment, such as the service name, version, and cloud provider.

Log Record Exporter

The log Record Exporter is an OpenTelemetry component that sends log data to a specified OpenTelemetry consumer. Typically, this is an OpenTelemetry Collector (meaning the part of OpenTelemetry that imports observability data), but you can also use the Log Record Exporter to send log data to another compatible backend, as well as to standard input/output.

Logs Bridge API and related components

The OpenTelemetry Logs Bridge API is a mechanism that can integrate existing logging frameworks with OpenTelemetry's logging pipeline. It’s called a “bridge” because it bridges traditional application logging tools with OpenTelemetry, allowing the former to generate OpenTelemetry-compliant logs.

In addition to the Logs Bridge API itself, OpenTelemetry also provides a Log Appender, Logger, and Logger Provider, which help integrate external logging frameworks with the Log Bridge API.

Typically, you wouldn’t use any of these components unless you’re developing logging libraries. If you just want to collect and process logs using OpenTelemetry, you only need to understand Log Records and the Log Record Exporter.

Benefits of the OpenTelemetry log data model

To be sure, OpenTelemetry isn’t the only way to collect log data. Plenty of other tools and frameworks exist for pulling log files from applications and infrastructure.

Compared to other existing logging solutions, however, the OpenTelemetry log data model offers several unique benefits:

• Structured and consistent format: While OpenTelemetry is flexible enough to work with most types of logs, the Log Record concept makes it possible to structure log data in a consistent way. In turn, it helps parse and interpret data, since it’s easier to make sense of logs (or analyze them using automated observability tools) when their data is formatted consistently.
• Contextual insights: By organizing information based on top-level fields and resource attributes, the OpenTelemetry log data model helps ensure that log events include meaningful context. It doesn’t just tell you that something happened; it includes the context to determine where, when, and (in many cases) how an event occurred.
• Integration with traces and metrics: As we mentioned, OpenTelemetry not only collects logs, but also logs and traces. In this way, it provides end-to-end visibility into complex systems and allows you to correlate log data with other sources of insight.
• Interoperability with open standards: OpenTelemetry can import almost any type of log into any observability tool that supports the OpenTelemetry standards. As a result, OpenTelemetry provides a consistent approach to logging that you can keep in place even if your observability toolset changes, or you have to collect logs from new types of applications or infrastructure.
• Scalability for distributed systems: Because OpenTelemetry can collect logs from multiple sources, it scales well in distributed systems (like Kubernetes) where log data tends to be scattered across various locations. You don’t need to have all of your logs in a central location to work with them in OpenTelemetry.

How to collect log data using OpenTelemetry Collector

To illustrate how OpenTelemetry logs work in practice, let’s look at a basic example of collecting log data using OpenTelemetry Collector.

Step 1: Install Collector

The first step is to install the Collector. Detailed instructions are available from the OpenTelemetry documentation. In Kubernetes, you can typically install the Collector with:

kubectl apply -f
https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector/v0.116.1/examples/k8s/otel-config.yaml

Step 2: Configure the Collector

Next, create a configuration file for OpenTelemetry Collector that defines the following components:

• filelog receiver: This tells the Collector to read logs from a file.
• exporters: To send the logs to a backend (e.g., OTLP, Elasticsearch, or a logging system).

You can also optionally define processors if you want to transform the log file.

Here’s an example configuration:

receivers:
  filelog:
	include: ["/path/to/application.log"]  # Path to the log file
	start_at: beginning               	# Read logs from the beginning
	include_file_path: true           	# Include file path in log attributes
	include_file_name: true           	# Include file name in log attributes
	operators:
  	- type: regex_parser
    	regex: '^(?P<timestamp>[^\s]+) (?P<level>[^\s]+) (?P<message>.+)$'
    	timestamp:
      	parse_from: timestamp
      	layout: '%Y-%m-%dT%H:%M:%S' 	# Adjust format to match your log's timestamp
      	layout_type: strptime
    	severity:
      	parse_from: level

processors:
  batch: {}

exporters:
  otlp:
	endpoint: "http://localhost:4317" 	# Example: Sending to an OTLP endpoint

service:
  pipelines:
	logs:
  	receivers: [filelog]
  	processors: [batch]
  	exporters: [otlp]

Step 3: Run the Collector

Then, simply run the collector, being sure to point it to your configuration file:

otelcol --config=/path/to/otel-collector-config.yaml

The Collector will now automatically begin collecting logs from the location you specified.

Kubernetes OpenTelemetry logging example

Now that we’ve walked through a generic OpenTelemetry logging example, let’s look at a real-world example of using OpenTelemetry to collect Kubernetes logs.

Step 1: Install Collector

First, install the Collector. In this example, we’ll install it using Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring --create-namespace

Step 2: Configure Collector

To configure the Collector for Kubernetes, create a file like the following:

receivers:
  k8s_events:
	auth_type: serviceAccount # Use service account for authentication
	namespaces: [default] 	# Collect events only from the "default" namespace 

exporters:
  otlp:
	endpoint: "http://logging-backend:4317"  # Replace with your OTLP endpoint
  logging:
	log_level: debug

service:
  pipelines:
	logs:
  	receivers: [k8s_events]
  	exporters: [otlp, logging]

Here, we’ve configured the Collector to collect Kubernetes events from the default namespace.

Step 3: Deploy the configuration

We’ll deploy the configuration as a ConfigMap with:

kubectl create configmap otel-collector-config \
  --from-file=otel-collector-config.yaml \
  --namespace=monitoring

Step 4: Configure RBAC

For this setup to work, we need to create a Role and RoleBinding to allow the OpenTelemetry Collector to access Kubernetes events. You can do this using the following YAML:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: monitoring
  name: otel-collector-role
rules:
  - apiGroups: [""]
	resources: ["events"]
	verbs: ["list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: monitoring
  name: otel-collector-rolebinding
subjects:
  - kind: ServiceAccount
	name: otel-collector
	namespace: monitoring
roleRef:
  kind: Role
  name: otel-collector-role
  apiGroup: rbac.authorization.k8s.io

Remember to deploy the RBAC configuration with:

kubectl apply -f /path/to/k8s-events-rbac.yaml

Step 5: Run the Collector

Finally, create a Deployment for the Collector based on the ConfigMap and service account we defined above:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  replicas: 1
  selector:
	matchLabels:
  	app: otel-collector
  template:
	metadata:
  	labels:
    	app: otel-collector
	spec:
  	serviceAccountName: otel-collector
  	containers:
    	- name: otel-collector
      	image: otel/opentelemetry-collector:latest
      	volumeMounts:
        	- name: config-volume
          	mountPath: /etc/otel-collector-config
          	readOnly: true
      	args: ["--config=/etc/otel-collector-config/otel-collector-config.yaml"]
  	volumes:
    	- name: config-volume
      	configMap:
        	name: otel-collector-config

Apply the Deployment to run the Collector:

kubectl apply -f /path/to/otel-collector-deployment.yaml

Common challenges in OpenTelemetry logging

While OpenTelemetry is a powerful logging solution, it also has some potential drawbacks;

• Integration complexity: Implementing OpenTelemetry logging requires an additional step that may be more complex than deploying native existing logging libraries (meaning logging libraries provided for a specific programming language) for a given application. The upside is that once you have OpenTelemetry logging set up, you can use it to manage logs across all of your apps and infrastructure. But if you only need to collect logs from a handful of applications, it may be simpler to use existing logging libraries instead of configuring OpenTelemetry.
• Performance impact: While the resources required to collect an individual log via OpenTelemetry are minimal, they can add up if you’re collecting hundreds or thousands of logs. At scale, OpenTelemetry logging may reduce the resources available to your applications and, by extension, reduce their performance.
• Varying levels of support: Some native logging libraries offer better support for OpenTelemetry than others. As a result, the amount of effort required to import log data using OpenTelemetry, and the number of transformations you need to perform to structure the data in the way you want, can vary depending on the programming languages and logging libraries used within your applications.

OpenTelemetry logging best practices

To get the most from OpenTelemetry logging, consider the following best practices:

Include context with traces and spans

Logs should include trace and span identifiers (via traceId and spanId). This allows you to correlate logs with traces to achieve a unified view of distributed system behavior.

Use structured logging

Rather than writing plain-text logs, design applications (where possible) to use structured log formats such as JSON. Although OpenTelemetry is capable of collecting unstructured logs as well, structured logs are preferable because they are machine-readable, easier to parse, and more effective for downstream processing. Structured logging also allows you to include attributes like severity, timestamp, and resource information, making analysis and querying faster and more accurate.

Use resource attributes

Attach resource attributes to logs, such as service name, environment (e.g., production or staging), and region. These attributes help identify the source of the logs in complex environments with multiple services or regions. OpenTelemetry’s resource API allows you to define these attributes globally to reduce redundancy and ensure consistency.

Control log verbosity and volume 

Managing log verbosity helps prevent you from overwhelming your logging infrastructure, using too much CPU, or incurring high storage costs. Also, consider sampling to limit the number of logs collected during high-traffic periods and ensure critical logs are always captured. OpenTelemetry processors, like tail_sampling, allow for fine-grained control over which logs are exported based on predefined conditions.

Integrating OpenTelemetry logging with groundcover

As an OpenTelemetry-compliant observability solution, groundcover fully supports the ingestion of OpenTelemetry logs.

In most cases, you don’t actually need to do anything to collect logs using groundcover because groundcover collects them automatically for you. But if you want details on how to integrate OpenTelemetry-compliant logging with groundcover, check out our documentation.

We’d be remiss if we didn’t mention that groundcover can also use eBPF to glean observability insights. With eBPF, you get hyper-efficient data collection that requires minimal configuration – so it can lead to better performance and deeper insights in many cases.

That said, if you want to ingest logs in groundcover using OpenTelemetry, you’re welcome to do that, too.