What is etcd

The first problem an etcd cluster solves is “who has the final say over critical state.” Many distributed systems need to store a small but highly critical amount of state — such as service discovery, configuration, leases, leader information, Kubernetes API objects, and control-plane state. This data isn’t necessarily large in volume, but once multiple nodes see inconsistent values, the system can make conflicting decisions.

So an etcd cluster needs to provide several things:

  • An authoritative state confirmed by a majority of nodes, rather than each node maintaining its own potentially conflicting copy;
  • A global update ordering, so that concurrent writes, transactions, and lease changes take effect along a single timeline;
  • Continued service when some nodes fail, as long as the remaining nodes can still form a quorum;
  • Rejection of minority partitions attempting to commit new state independently during a network partition, preventing two control planes from each believing they are correct;
  • Mechanisms like revision, watch, and lease that allow upper-layer components to coordinate based on state changes.

This is also why etcd is better suited as a distributed coordination and control-plane state store rather than a high-volume business database. Its core pursuit is not “every node can write locally however it wants,” but rather “all nodes ultimately agree on the same commit ordering.”

From an implementation standpoint, etcd is a replicated state machine: clients see a key-value API, while internally the cluster maintains a Raft log with entries committed in order. Every operation that changes state — such as Put, Delete, transactions, or lease changes — first enters the Raft log; only after the log has been confirmed and committed by a majority of nodes is it applied to the local MVCC store.

Therefore, when thinking about an etcd cluster, it’s best not to start by thinking of it as “multiple nodes each writing to their own database.” Instead, think of it first as a Raft group. The majority can continue advancing state; the minority can keep their processes alive, but cannot independently commit new cluster state externally. This is also why etcd is typically classified as a CP system: under network partitions, it prioritizes consistency and partition tolerance; if a partition cannot reach quorum, it would rather become unwritable than produce a new state that might conflict.

Cluster Architecture

Let’s first look at what layers compose the etcd cluster, then examine what components exist inside a single member.

The Three-Layer View of Cluster Architecture

Viewed from the outside in, an etcd cluster can be divided into three layers.

The first layer is the client access layer. Clients access any member via gRPC/HTTP API. Read requests can be handled by the local node, or they can go through ReadIndex for linearizability; if a write request hits a follower, the follower forwards it to the leader or asks the client to retry with the leader.

The second layer is the consensus replication layer. Each member internally has a Raft node responsible for leader election, log replication, advancing the commit index, membership changes, and snapshot synchronization. Raft does not directly understand key-value semantics — it is only responsible for establishing the same ordering of a series of log entries across a majority of nodes.

The third layer is the state machine and storage layer. After etcdserver receives committed logs from Raft, it applies these logs to the MVCC store. The MVCC store then uses a local backend to persist to disk; in the current etcd source code, this backend is based on bbolt.

Three-layer view of etcd cluster architecture

This can be simplified into the following chain:

  • client API
  • etcdserver
  • raft.Node
  • WAL / snapshot / transport
  • apply loop
  • MVCC store
  • bbolt backend
  • subsystems working around the state machine: watch, lease, auth, etc.

There is one easily confused point here: bbolt is a single member’s local persistence engine, while Raft is the replication and consistency protocol among multiple members. bbolt itself is not responsible for distributed consistency; Raft does not directly store key-value data either. The two work together in etcd to form a structure where “each node has local state while the cluster as a whole remains consistent.”

What’s Inside a Single Node

An etcd member is not just a single database file. It includes at least the following categories of key components.

  • etcdserver: Accepts the API externally, and internally connects Raft, storage, lease, watch, auth, alarm, maintenance, and other modules.
  • Raft node: Maintains the current term, role, leader, vote state, log progress, and commit index, and delivers content to be persisted, sent, and applied to the upper layer via the Ready interface.
  • WAL: Saves Raft HardState and log entries. When a node restarts, it must first restore Raft state from WAL and snapshots.
  • Snapshot: When the log grows too long, snapshots compress historical state. New nodes or nodes that have fallen too far behind may also catch up via snapshots.
  • MVCC store: Maintains multi-version data for keys, revisions, compact revisions, and indexes.
  • backend: The local persistence implementation for the MVCC store; in the current source code it opens a local database file via go.etcd.io/bbolt.
  • watch/lease: Watch relies on revisions to push changes; lease handles TTL, expiration, and the lifecycle of bound keys.

This layering is also visible in the source code structure: raftNode contains applyc, readStateC, ticker, and transport-related fields; the MVCC store holds backend.Backend, kvindex, lease.Lessor, and currentRev; the backend directly opens the bbolt database.

Request Paths

Mapping the architecture to request processing, there are two paths: writes and reads.

How Write Requests Flow

The write path can be divided into two phases: first establishing ordering in Raft, then executing in the state machine.

After a client initiates a write request, if the request reaches the leader, the leader packages it as a proposal, appends it to its own Raft log, and replicates it to followers via AppendEntries. Followers first persist the log entry upon receiving it, then reply to the leader. When the leader sees that a log entry has been replicated to a majority, it advances the commit index.

Only after committing does this log entry enter the apply phase. The etcdserver apply loop reads committed entries and applies the key-value operations in them to the MVCC store. After application, the local revision advances, watchers can observe the change based on the revision, and the client can receive the write result.

The key boundary here is: a log being “replicated to a node” does not mean it is “committed,” and a log being “committed” does not mean it has been “applied to the local state machine.” Raft is responsible for the former; etcdserver’s apply loop is responsible for the latter. In the source code, raftNode.start reads Ready from Raft, first saves HardState and Entries, then sends committed entries into the apply channel, and finally calls Advance to tell Raft that this batch of Ready has been processed.

How Read Requests Guarantee Consistency

etcd read requests have different consistency options. The default semantics lean toward linearizability; if the caller explicitly selects a serializable read, it can read from local state for lower latency, but may read stale data that the current node hasn’t yet caught up to from the leader. A linearizable read must confirm that “all writes completed before this read are visible.”

The Raft layer provides the ReadIndex mechanism. Simply put, the leader confirms it is still the leader through the current quorum and obtains a read index; when the local state machine has applied up to at least this index, it is safe to handle this read. etcd’s read loop sends ReadIndex and waits for the corresponding read state before allowing a linearizable read to proceed.

This echoes the write path: just as writes depend on majority commitment, linearizable reads also cannot simply trust the leader identity in local memory. They need to confirm the current leader is still recognized by the majority and that local apply progress has caught up to a sufficiently recent position.

The Leader Election Process

Leader election can be understood through four questions: trigger, campaigning, success criteria, and log freshness.

When Does Election Begin

Raft has three main roles: follower, candidate, and leader. Normally, the leader periodically sends heartbeats or AppendEntries, and each time a follower receives these messages from the leader, it resets its own election elapsed.

If a follower doesn’t receive a leader message within one election timeout, it concludes the current leader may be unreachable and triggers a local MsgHup to enter the election process. etcd’s default tick is 100ms and the default election timeout is 1000ms; Raft internally also randomizes the actual trigger point to somewhere in [electionTimeout, 2*electionTimeout-1] to prevent multiple nodes from always initiating elections at the same moment.

The leader also checks quorum liveness. etcd sets CheckQuorum: true when starting Raft. If the leader cannot observe majority activity within one election timeout, it proactively steps down to follower, preventing a former leader that has lost the majority from long believing it is still valid.

How Candidates Campaign for Votes

When a candidate initiates a campaign, it sends vote requests to all voters. The request includes not just the term but also the candidate’s last log position: last log term and last log index.

Nodes receiving vote requests make roughly three types of judgments:

  • Can they still vote?: They haven’t voted in this term yet, or they’re re-voting for the same candidate.
  • Are they still within the leader lease?: If they just heard from the current leader and CheckQuorum is enabled, they will ignore vote requests with higher terms, neither updating the term nor granting a vote.
  • Is the candidate’s log fresh enough?: They can only grant a vote if the candidate’s log is at least as up-to-date as their own.

If the candidate receives approval from a majority, it becomes the leader and immediately broadcasts an append, taking on log replication responsibilities. If it receives rejection from the majority, the election fails; if there is no result, it waits for the next election timeout.

Success Criteria for Leader Election

Quorum means the majority. In a Raft configuration with N voters, the majority size is floor(N/2) + 1. For example:

  • 1 node: quorum is 1;
  • 3 nodes: quorum is 2;
  • 5 nodes: quorum is 3;
  • 7 nodes: quorum is 4.

Raft uses quorum to solve two problems.

One is leader election: a candidate must receive approval votes from quorum to become the leader. The other is committing: the leader’s log must be replicated to quorum before a log entry is considered committed. Because any two majorities must have an intersection, a new leader cannot bypass already-committed logs. This is one of the cores of Raft’s safety.

etcd’s membership changes involve a joint quorum, but for a regular stable configuration, the majority understanding suffices. The majority vote judgment in source code is also len(c)/2 + 1.

How Log Freshness Is Determined

“Log freshness” is not about comparing log entry counts, nor wall clock time — it’s about comparing the (term, index) of the last log entry.

The rules are very simple:

  • If the candidate’s last log entry has a higher term, it is more up-to-date;
  • If the last log entry has the same term, the one with a greater or equal index is more up-to-date;
  • If the term is lower, it cannot be considered more up-to-date even if the index is larger.

This rule is critical. Raft’s safety requires that a new leader contain at least all committed logs. Since committed logs must exist in some majority, and a candidate must also obtain majority votes, checking log freshness during voting prevents nodes with stale logs from becoming the leader.

Election Stability

Beyond basic elections, etcd also needs to handle stability issues such as partitioned nodes, stale leaders, and frequent elections.

Why PreVote Matters

etcd enables PreVote by default. Its purpose is not to change Raft’s safety properties, but to reduce disruption from partitioned nodes returning to a cluster with a stable leader.

Without PreVote, a long-partitioned node might continuously increment its own term. When it reconnects to the cluster, it sends vote requests with a higher term; other nodes seeing the higher term might first step down to follower, causing the originally healthy leader to be interrupted.

With PreVote, before a node actually increments its local term, it first sends a MsgPreVote. PreVote uses term + 1 but does not immediately modify the local term. Only after it first obtains pre-votes from a majority does it enter a real election. This way, an isolated node that cannot reach quorum cannot disrupt the main cluster by incrementing its term.

etcd’s default configuration has PreVote: true, and the command-line parameter description explicitly states: enabling PreVote is to prevent a node that was previously isolated by a network partition from disrupting the cluster when it rejoins.

How CheckQuorum Handles Stale Leaders

PreVote mainly limits “isolated candidates disrupting the cluster”; CheckQuorum limits “stale leaders continuing to be confident in a minority partition.”

When the leader is still in the majority partition, most followers continuously receive its heartbeats or appends and their election elapsed keeps being reset. If another node mistakenly believes the leader is gone due to network issues and sends Vote or PreVote to these followers, the requests will be ignored because the leader lease has not yet expired.

When the leader loses the majority, the situation reverses. The leader cannot observe quorum liveness, and MsgCheckQuorum causes it to step down to follower. If the majority partition also cannot hear the old leader, it will initiate a new election after the election timeout and elect a new leader.

So etcd doesn’t avoid chaos by “locking all nodes from elections,” but rather through constraints in two directions: followers within the majority don’t easily abandon a live leader; a stale leader that has lost the majority cannot long retain leader status.

What Happens During a Network Partition

Looking at a three-node cluster in two typical scenarios makes this clearer.

If the leader and one follower are in the majority partition, while the other follower is isolated: the majority can still accept writes, and the leader continues sending heartbeats. The isolated follower may PreVote after timing out, but it cannot reach quorum, so it will not become leader. When it reconnects, it replicates missing logs from the leader.

If the leader is alone in the minority partition, while the other two followers are in the majority partition: the old leader steps down because it cannot reach quorum with CheckQuorum enabled; followers in the majority partition elect a new leader after timing out. The partition containing the old leader cannot commit new writes — even if it briefly hasn’t realized the problem, writes cannot pass the quorum threshold.

If the system has had no new writes for a long time, a partitioned candidate’s log may be as fresh as other nodes’. In this case, the election result depends on two conditions: whether it can contact quorum, and whether other voters are still within the leader lease. As long as the main cluster can still hear the leader, its vote request will be ignored; only when the majority genuinely cannot hear the leader and the election timeout has expired will a new election begin.

Can Election Storms Occur

etcd cannot guarantee that frequent elections will never happen. Network jitter, disk stalls, long GC pauses, CPU saturation, and misconfigured timeout parameters can all cause followers to mistakenly believe the leader has disappeared, triggering elections.

However, etcd’s Raft implementation has several layers of buffering to reduce the probability of election storms.

  • The heartbeat timeout is significantly shorter than the election timeout, so followers normally continue receiving leader messages before the timeout.
  • The election timeout is randomized to prevent all followers from campaigning simultaneously.
  • PreVote keeps nodes that can’t reach quorum in the pre-vote phase without directly raising the term.
  • CheckQuorum causes a leader that has lost the majority to step down, and also causes followers that just heard from the leader to reject vote requests within the lease period.
  • The log freshness check prevents candidates with stale logs from becoming the leader.

Therefore, frequent elections are usually not “normal behavior” in Raft, but rather a sign that cluster operating conditions have gone wrong. When investigating, the primary focus should be on network latency and packet loss, disk fsync latency, CPU throttling, GC pauses, node load, and whether heartbeat/election parameters match the actual environment.

Conclusion

etcd’s cluster architecture can be summarized in one sentence: each member locally stores state using WAL, snapshots, MVCC store, and bbolt; multiple members replicate logs using Raft and determine commit ordering.

The leader election logic can also be summarized in one sentence: only a node whose log is fresh enough, is not blocked by a leader lease, and can obtain quorum can become the leader.

These two sentences correspond to etcd’s core boundaries. bbolt handles single-machine persistence, MVCC handles the versioned semantics of key-values, and Raft handles ordering, consistency, and failover among multiple nodes. Leader election is not about selecting the “most powerful” node, but about ensuring that all subsequent writes continue advancing along the same log that has been approved by the majority.

Source Code Index

This article is based on a local source code checkout at 8f8587672abb5704d8134b88a1e43059e9440578. Below are several key locations for further reference.