What Are the Four Golden Signals?

What Are the Four Golden Signals?

The four golden signals are latency, traffic, errors, and saturation. They provide a simple way to understand whether a service is responsive, how much demand it is handling, whether requests are failing, and whether the system is approaching its capacity limits.

The framework comes from Google's Site Reliability Engineering guidance, which recommends focusing on these four signals when you need a small but meaningful set of metrics for a user-facing system. The idea is not that these are the only metrics worth collecting. They give you a reliable starting point for answering a much more practical question: Is the service healthy from the user's point of view?

Four Golden Signals at a Glance

These signals become much more useful when you look at them together. A latency spike means something different when traffic has doubled and CPU is nearly exhausted than when traffic is normal and the system has plenty of spare capacity.

Where Did the Four Golden Signals Come From?

The four golden signals were popularized by Google's Site Reliability Engineering book in its chapter on monitoring distributed systems. The recommendation is straightforward: if you can monitor only four things in a user-facing system, focus on latency, traffic, errors, and saturation.

That simplicity is the reason the framework has remained useful. Modern systems may expose thousands of metrics, but having more metrics does not necessarily make it easier to determine whether users are experiencing a problem.

Golden signals give teams a common place to start. Once one of these signals looks unusual, engineers can move deeper into traces, logs, infrastructure metrics, database activity, deployment events, and other telemetry to understand the cause.

1. Latency

Latency measures how long it takes a system to complete a request or operation. For a web application, this might be the time between receiving an HTTP request and returning the response. For a database, it could be query duration, while a message-processing system might measure how long a message takes to process.

Latency matters because users notice it immediately. A service can be technically available and return successful responses while still providing a poor experience because every request takes several seconds.

Example

Imagine a checkout API normally responds in 180 milliseconds but suddenly starts taking 1.8 seconds. The requests may still return HTTP 200 responses, so a basic uptime check would consider the service healthy, but users would clearly notice the slowdown.

That is why latency is one of the first signals worth checking when an application "feels slow."

Don't Rely Only on Average Latency

Average response time can hide the requests that users struggle with most. If most requests complete in 100 milliseconds but a smaller percentage take several seconds, the average can still look perfectly reasonable.

Percentiles usually provide a better picture:

• p50 represents the median request.
• p95 shows the response time experienced by 95% of requests.
• p99 exposes the slower tail of the distribution.

Looking at p95 or p99 latency can reveal performance problems that averages hide. Google's SRE guidance also emphasizes looking beyond averages because a small percentage of very slow requests can have a significant impact on users.

Separate Successful and Failed Request Latency

Successful and failed requests should not always be mixed into the same latency measurement. A request that immediately returns an HTTP 500 error may have extremely low latency, but that does not mean the service is performing well.

For example, suppose normal requests take 500 milliseconds, but a broken dependency causes thousands of requests to fail in 20 milliseconds. If you mix them together, average latency could actually improve during the outage.

Tracking latency by response status or outcome avoids that problem. Google specifically recommends distinguishing successful request latency from failed request latency for this reason.

Common Latency Metrics

Useful latency measurements can include:

• HTTP request duration
• API response time
• Database query duration
• RPC latency
• Queue processing time
• External API latency
• Page load time
• Serverless function duration

The right measurement depends on what the service actually does. The most useful latency metric is usually one that closely represents the experience of the caller or end user.

2. Traffic

Traffic measures how much demand is being placed on a system. For an API, traffic might be requests per second. For a database, it could be queries per second, while a streaming platform might measure messages, records, or concurrent connections.

Unlike errors or latency, high traffic is not automatically a problem. Traffic mainly gives context to the other golden signals and helps explain why system behavior changed.

Example

Suppose API latency increases from 200 milliseconds to 800 milliseconds. If traffic increased fivefold at the same time, capacity may be the first thing worth investigating.

Now imagine the same latency increase occurs while traffic remains unchanged. That points toward a different set of possibilities, such as a slow database query, external dependency, deployment, or configuration change.

Without traffic data, both situations can look identical.

What Should You Use as a Traffic Metric?

Traffic should represent meaningful work rather than simply measuring activity at a low infrastructure level.

Common examples include:

Google describes traffic as a system-specific measurement of demand, so there is no single universal traffic metric. The right measurement should reflect how users or downstream systems actually consume the service.

3. Errors

Errors measure how often the system fails to perform the work users expect. They may appear as HTTP 500 responses, failed RPC calls, exceptions, timeouts, unsuccessful transactions, or other clearly identifiable failures.

Error monitoring becomes more useful when expressed as a rate rather than just a raw number. One hundred errors can be disastrous for a service processing 200 requests, but almost insignificant for one processing several million.

A simple error rate can be thought of as:

Error rate = Failed requests ÷ Total requests

If 500 out of 100,000 requests fail, the error rate is 0.5%.

Not Every Error Looks Like an Error

Some failures are easy to identify. An HTTP 500 response or an exception clearly indicates that something went wrong.

Other failures are less obvious. Google's SRE guidance describes errors as including explicit, implicit, and policy-based failures.

Explicit Errors

These are failures the system clearly reports. Examples include HTTP 500 errors, failed RPC calls, unhandled exceptions, rejected database transactions, or failed background jobs.

They are usually the easiest errors to measure because the application or protocol already provides a clear success or failure state.

Implicit Errors

An application can return a technically successful response while still providing the wrong result. For example, a search API could return HTTP 200 while showing an empty result because one of its backend services failed.

From the protocol's perspective, the request succeeded. From the user's perspective, it did not.

Policy-Based Errors

A request can also technically succeed but violate an expected service objective. If your service promises that requests will complete within one second, a successful request taking six seconds may still count as a failure for SLO purposes.

This is why error monitoring should reflect what users consider successful, not just what the HTTP status code says.

Common Error Metrics

Teams commonly monitor:

• HTTP 5xx rate
• Failed request percentage
• Exception rate
• Timeout rate
• Failed transactions
• RPC failures
• Database errors
• Job failures
• Dependency failures
• SLO violations

Error rate is one of the clearest indicators of user impact. When it increases, the next step is usually to correlate those failures with traces, logs, deployments, dependencies, and infrastructure changes.

4. Saturation

Saturation measures how close a system or resource is to its practical capacity limit. It answers a question the other golden signals cannot: how much room does the system have left before performance begins to degrade?

CPU and memory are common saturation metrics, but saturation is much broader than host utilization. The real goal is to identify the resource that limits the service and understand how close it is to exhaustion.

That resource could be:

• CPU
• Memory
• Disk capacity
• Disk I/O
• Network bandwidth
• Thread pools
• Database connections
• File descriptors
• Worker pools
• Queue capacity
• Kubernetes resources
• API rate limits

A service can have plenty of available CPU and still be saturated because its database connection pool is full. This is why saturation should be tied to the actual bottleneck rather than whichever infrastructure metric is easiest to graph.

Google describes saturation as how "full" the service is and recommends focusing on its most constrained resource. It also notes that many systems start degrading before reaching 100% utilization.

Example of Saturation

Imagine an API with a database connection pool that allows 100 simultaneous connections. During normal operation, it uses 30 to 40 connections.

Traffic increases and connection usage reaches 95. Requests begin waiting for an available connection, causing latency to rise even though CPU remains at only 45%.

In this case, CPU is not the useful saturation signal. Database connection pool utilization is.

Saturation Can Warn You Before Failure

Saturation is often a leading indicator of trouble. A system may still be serving requests successfully even though a resource is approaching its limit.

For example, disk capacity increasing steadily toward 100% gives you time to act before the disk actually fills. Queue depth growing faster than workers can process messages can similarly warn that the system is falling behind.

Rising tail latency can also indicate that a system is approaching saturation even before an obvious resource reaches its configured limit. This relationship between latency and saturation is highlighted in Google's original SRE discussion as well as more recent interpretations of the framework.

How the Four Golden Signals Work Together

Each golden signal provides useful information on its own, but the framework becomes much more valuable when the signals are viewed together. One signal usually tells you that something changed, while the others help provide context around why.

These patterns are starting points, not automatic root-cause diagnoses. The golden signals tell you where to look next, while traces, logs, metrics, events, and other telemetry provide the evidence needed to confirm what happened.

A Practical Example

Suppose customers suddenly report that an ecommerce checkout page is slow.

Your golden signals show:

• Latency: p95 checkout latency increased from 400 ms to 3.2 seconds.
• Traffic: Request volume is roughly normal.
• Errors: Error rate remains below 1%.
• Saturation: Database connection pool utilization has increased from 55% to 98%.

That immediately narrows the investigation. Since traffic did not increase significantly, the slowdown probably isn't simply caused by a sudden traffic spike.

You can now inspect database traces, queries, logs, or recent deployments. A trace might reveal a query holding connections much longer than usual, leading to connection pool saturation and higher application latency.

Without the four golden signals, you might start with hundreds of unrelated infrastructure metrics. With them, you already have a useful direction.

How Do Golden Signals Apply to Different Systems?

The framework stays the same, but the actual measurements should change depending on the system.

The important part is not forcing every system into identical metrics. You want each golden signal to represent the same underlying question while using measurements that make sense for that workload.

Four Golden Signals vs. RED vs. USE

Golden signals are not the only framework for deciding what to monitor. RED and USE are also widely used, and each approaches the system from a slightly different direction.

Golden Signals vs. RED

RED stands for Rate, Errors, and Duration. Rate corresponds closely to traffic, while duration maps to latency, which means RED covers three of the four golden signals.

The main difference is saturation. RED concentrates on request-driven service behavior, whereas the golden signals add a view of whether the service is approaching a resource or capacity limit. Grafana similarly describes RED as especially applicable to services and the four golden signals as a closely related model that also includes saturation.

Golden Signals vs. USE

USE stands for Utilization, Saturation, and Errors. It is generally applied to infrastructure resources such as CPU, disks, memory, and network devices.

Golden signals start closer to the service and user experience, while USE starts with the underlying resources. In practice, they complement each other. Golden signals can tell you that users are experiencing high latency, while USE-style resource metrics can help explain whether an infrastructure bottleneck is responsible.

Golden Signals and SLIs/SLOs

Golden signals and service level objectives often overlap because both focus on measurable service behavior. Latency and error rate, in particular, are frequently used when defining service level indicators.

For example, an API may define an SLO such as:

99.9% of requests should succeed during a 30-day period.

Another might state:

95% of requests should complete within 300 milliseconds.

Golden signals provide the telemetry needed to measure these objectives, but they do not replace SLIs or SLOs. The golden signals are a monitoring framework, while SLIs and SLOs define specific reliability measurements and targets.

How Does OpenTelemetry Help with Golden Signals?

OpenTelemetry can collect much of the telemetry required to calculate golden signals without locking application instrumentation to one observability backend.

Distributed traces and HTTP metrics can provide request duration, request counts, status codes, and service attributes. Infrastructure and application metrics can provide additional saturation information such as CPU utilization, memory usage, queue depth, or connection counts.

A common flow might look like:

Application → OpenTelemetry instrumentation → OpenTelemetry Collector → Observability platform

Once that telemetry is collected, an observability platform can calculate and visualize latency, traffic, error rate, and saturation alongside traces and logs.

This correlation is important. Golden signals tell you that a problem exists, while the surrounding telemetry helps explain what caused it.

Common Mistakes When Monitoring Golden Signals

The framework itself is simple, but the way teams measure the signals can make the resulting dashboards much less useful.

Watching Average Latency Only

Average latency can make a service appear healthy while a meaningful percentage of users are experiencing very slow requests. Looking at latency distributions and percentiles such as p95 and p99 gives a clearer picture of tail performance.

You should also separate successful and failed request latency where appropriate. Otherwise, fast failures can make latency numbers look artificially good during an outage.

Using Raw Error Counts Without Traffic

An error count has very little context on its own. One thousand errors might represent a severe outage for a small application or a tiny percentage of requests for a very large service.

Error rate usually provides a more meaningful view because it compares failures with total demand.

Treating Traffic as a Problem by Itself

Higher traffic is not necessarily unhealthy. In many systems, it is exactly what the business wants.

Traffic becomes useful when it is correlated with latency, errors, and saturation. If traffic doubles and every other signal remains healthy, the system may simply be handling growth successfully.

Assuming Saturation Means CPU

CPU usage is only one possible saturation signal. Database connections, queues, network bandwidth, memory, disk I/O, thread pools, or external service limits may constrain an application long before CPU becomes a concern.

The better question is: What resource would prevent this service from handling the next request?

That is usually the resource worth monitoring for saturation.

Creating the Same Threshold for Every Service

A 500 ms latency may be completely acceptable for one workload and unacceptable for another. Similarly, 80% resource utilization can be normal for one system and a warning sign for another.

Golden signals provide categories, not universal thresholds. Thresholds should reflect each service's normal behavior, architecture, capacity model, and reliability objectives.

Are the Four Golden Signals Enough?

The four golden signals are a strong starting point, but they are not intended to replace every other type of monitoring. They help teams quickly understand the health of a service, especially from the user's perspective.

Once a problem appears, you still need deeper telemetry to explain the cause. Distributed traces can reveal where latency originates, logs can show application errors, infrastructure metrics can expose resource constraints, and deployment events can explain when behavior changed.

You may also need domain-specific measurements such as database replication lag, Kafka consumer lag, cache hit rate, business transaction failures, frontend Core Web Vitals, or AI token usage.

The value of the golden signals is focus. Instead of putting 100 metrics on the first dashboard, start with four questions:

• Is the service slow?
• How much demand is it receiving?
• Are requests failing?
• Is it running out of capacity?

If those answers show a problem, then go deeper.

The Bottom Line

The four golden signals are latency, traffic, errors, and saturation. Together, they provide a compact view of service performance, demand, reliability, and capacity.

Latency tells you whether requests are slow. Traffic shows how much work the system is handling. Errors reveal whether that work is succeeding, while saturation shows whether the system still has enough capacity to keep up.

The framework works because these signals provide context for one another. A latency spike alone tells you that users are waiting longer, but latency combined with traffic, error, and saturation data gives you a much better idea of what to investigate next.