When people first encounter service mesh, it often feels like an advanced but complicated infrastructure component: Istio, Envoy, sidecars, mTLS, traffic management, observability, and a pile of other terms arrive all at once.
But if we look at service mesh only through the tools, we can easily misunderstand it. Service mesh did not appear for the sake of sophistication, and it is not the starting point of microservice architecture. It is better understood as an infrastructure layer that emerged after years of enterprise architecture evolution, as the industry tried to answer one recurring question: after we split a system into more and more services, how should those services communicate and be governed?
To understand service mesh, it is useful to start with the architectures that came before it.
1. The Monolith Era: Simple Does Not Mean Backward
Early enterprise applications and internet applications often started as monoliths.
One codebase, one deployable package, one database. Users, orders, payments, admin backends, and marketing campaigns might all live inside the same application. Modules called each other through in-process function calls. Data consistency was handled by local database transactions. Troubleshooting usually meant looking at one application and one database.
Common practices in this stage included:
| Topic | Common Practices |
|---|---|
| Application structure | MVC, layered architecture, Service Layer, DAO / Repository |
| Database | Relational database, local ACID transactions, primary-replica replication, read/write splitting |
| Deployment | Application servers, deployable packages, fixed release windows |
| Collaboration | CVS, SVN, centralized repositories, trunk, branches, tags |
| Release rhythm | Large releases, centralized testing, coordinated deployment |
One easily overlooked point is that during the era when monoliths were dominant, many teams also used centralized version control systems such as CVS and SVN. Teams worked around a central trunk. Multiple features were integrated together, tested over a longer period, and released in one batch. This matched the architecture of the time: one large application, one main release branch, one deployable package, and one release window.
So a monolith is not automatically backward. In the early stage of a business, it is often the most reasonable choice. It is fast to build, easy to debug, clear in transaction boundaries, and simple to deploy. The problems usually appear after scale grows: the codebase becomes larger, teams block each other, release cycles get longer, and a problem in one module can affect the whole system.
The industry then began to look for new ways to split systems.
2. From N-Tier Architecture to SOA: The Predecessors of Service Orientation
Before microservices, the industry tried many forms of decomposition.
First came three-tier or N-tier architecture: web layer, business layer, data access layer, and database layer. This improved code organization and made responsibilities clearer, but the deployment unit was usually still one large application.
Then came distributed object and enterprise component technologies such as CORBA, DCOM, RMI, and EJB. They tried to make remote calls feel like local object calls. The industry soon learned an important lesson:
A remote call cannot pretend to be a local call.
Networks fail. They time out. Latency fluctuates. Serialization has cost. Versions become incompatible. This lesson later shaped both microservices and service mesh: once a system becomes distributed, failure must be designed into it.
After that came SOA, or Service-Oriented Architecture.
SOA already contained many ideas that resemble microservices: exposing capabilities as services, communicating through contracts, composing services, and reusing enterprise capabilities. Common technologies in that era included SOAP, WSDL, XML Schema, ESB, and BPEL.
WSDL was not an architecture. It was a service interface description language. Its role was somewhat similar to a .proto file in gRPC: it described the operations a service exposed, the input and output messages, the endpoint, and the binding. The difference is that WSDL belonged to the SOAP / SOA era and was usually based on XML and the heavier WS-* specifications. gRPC is a modern RPC framework, usually based on HTTP/2 and Protobuf, and is lighter and more suitable for high-performance internal service calls.
The problem with SOA was not that service orientation was wrong. The problem was that many implementations became too heavy. ESB in particular could easily become an enterprise hub where protocol conversion, message routing, process orchestration, and governance rules were all placed into one central bus. In the end, it could become another large monolith.
This is one reason why microservices later emphasized “smart endpoints and dumb pipes”: services should own their business capabilities, while the communication layer should stay as simple as possible. In Microservices, Martin Fowler also describes business capability orientation, independent deployment, infrastructure automation, and design for failure as important characteristics of microservices.
3. Microservice Architecture: Not Just Smaller Services, But Aligned Boundaries
Microservices became popular not simply because people wanted smaller applications, but because the engineering environment changed.
Git, GitHub/GitLab, pull requests, CI/CD, containers, Kubernetes, cloud platforms, and observability systems gradually matured. Only then did teams gain the ability to manage many independently deployed units. The move from monoliths to microservices also came with a shift in collaboration: from centralized, large-batch, low-frequency releases toward distributed, small-batch, high-frequency automated delivery.
But microservices do not mean “the smaller the service, the better.” More accurately:
Microservice architecture tries to align business boundaries, team boundaries, code boundaries, data boundaries, and deployment boundaries.
Several bodies of theory matter here.
The first is DDD, especially the idea of Bounded Context. Service boundaries should not be split by technical layers or by database tables. They should be split by business capabilities and business semantics. Microsoft’s microservice modeling guide also recommends using domain analysis, bounded contexts, entities, aggregates, and services to identify microservice boundaries: Use domain analysis to model microservices.
The second is Conway’s Law. System architecture reflects the communication structure of the organization that builds it. If five teams must coordinate every time one service changes, that is not a good service boundary. A stable team that owns one business capability end to end, including development, deployment, monitoring, and incident response, is much closer to the ideal shape of microservices. Martin Fowler also explains Conway’s Law clearly here: Conway’s Law.
The third is distributed systems theory. Microservices turn in-process calls into network calls, so we must accept timeouts, retries, circuit breakers, rate limiting, idempotency, compensation, and eventual consistency as normal design concerns. Cross-service transactions cannot simply reuse the local ACID mindset of a monolith. They often require sagas, outbox patterns, events, and compensating actions.
The fourth is cloud native practice and 12-Factor thinking. Microservices need externalized configuration, stateless processes, standardized logs, environment parity, fast startup, graceful shutdown, and automated deployment. Otherwise, more services simply mean more operational pain. The Twelve-Factor App summarizes many of these principles.
So the maturity of a microservice architecture is not measured by the number of services. It is measured by whether these questions are clear:
| Question | Good Microservice Practice |
|---|---|
| Service boundary | Organized around business capabilities and bounded contexts |
| Data ownership | Each service owns its core data |
| Team responsibility | Each service has a clear owner |
| Release model | Services can be tested, deployed, and rolled back independently |
| Failure handling | Network failure is expected; calls have timeouts and fallback paths |
| Observability | Metrics, logs, traces, and alerts are in place |
| Consistency | Eventual consistency is accepted and designed for |
Many failed microservice systems are actually distributed monoliths: services are split, but databases are shared, models are shared, releases are coupled, call chains are long, and team boundaries are unclear. Such systems do not gain the benefits of microservices. They only inherit the complexity of distributed systems.
4. Microservice Frameworks and Service Mesh: One Inside the Application, One Outside It
After microservices became popular, the industry first relied heavily on microservice frameworks.
Frameworks such as Spring Cloud, Dubbo, and go-kit provide capabilities inside application code: service discovery, RPC/HTTP clients, load balancing, timeouts, retries, circuit breakers, configuration management, tracing, rate limiting, and degradation.
Their key characteristic is that the capability is embedded in the application. Developers use SDKs, annotations, configuration, or client libraries to access these governance features.
Service mesh moves part of this communication governance out of application code and into the infrastructure layer. A common implementation places a proxy next to each service, such as an Envoy sidecar. The business service communicates with the local proxy, while service discovery, load balancing, retries, mTLS, access policy, and metrics collection are handled by the proxy. The control plane distributes policy, and the data plane handles traffic.
The relationship is not simple replacement. It is layering:
| Dimension | Microservice Framework | Service Mesh |
|---|---|---|
| Location | Inside application code | Outside application code |
| Integration | SDKs, frameworks, annotations, configuration | Proxies, sidecars, node-level agents |
| Multi-language consistency | Harder, often one stack per language | Stronger, unified infrastructure governance |
| Complexity | Application-side complexity | Platform-side complexity |
| Typical capabilities | RPC, configuration, discovery, circuit breaking | mTLS, traffic governance, policy, telemetry |
| Suitable scenarios | Unified language stack, medium service scale | Multiple languages, large scale, complex governance |
Service mesh does not replace application frameworks. Domain modeling, interface definition, dependency injection, business logic, serialization, and data access still belong to application frameworks. Service mesh mainly handles cross-cutting service-to-service communication concerns.
In one sentence:
A microservice framework is like every car installing its own navigation and safety system; service mesh is like the road system providing traffic lights, speed limits, monitoring, and access rules.
5. Why Did Service Mesh Appear?
When a system has only a few services, it is still manageable for each service to handle timeouts, retries, logging, TLS, and monitoring on its own.
But when a system grows to dozens or hundreds of services, implemented by different teams, in different languages, and with different frameworks, the problem grows quickly.
Every service must deal with service discovery, load balancing, timeouts, retries, circuit breaking, rate limiting, canary release, access control, tracing, mTLS, logs, and metrics. If these capabilities are scattered across business code and language-specific SDKs, versions are hard to align, policies are hard to govern, and incidents are hard to debug.
The core motivation of service mesh is:
Move “how services communicate with each other” out of business code and into a unified infrastructure layer.
Service mesh mainly governs east-west traffic: service-to-service traffic inside the system. API gateways focus more on north-south traffic: external requests entering the system.
But service mesh is not free. It brings additional operational complexity, performance overhead, debugging cost, control-plane reliability requirements, and learning cost. For small systems with only a few services and simple call relationships, service mesh is usually unnecessary. Kubernetes Services, Ingress/Gateway, application frameworks, and basic monitoring may already be enough.
Service mesh is suitable for communication governance at scale. It does not solve the problem of designing good microservice boundaries. If service boundaries are wrong, data consistency is chaotic, or team responsibility is unclear, service mesh cannot save the architecture.
6. Open Source Service Mesh Practices: Several Main Routes
As of June 2026, the open source service mesh ecosystem can roughly be grouped into several routes.
The first route is Istio + Envoy: the full-featured platform route.
Istio is the most typical service mesh implementation. The control plane manages configuration and policy, while Envoy usually serves as the data plane. Istio covers mTLS, traffic governance, canary releases, traffic mirroring, timeouts, retries, authorization policy, telemetry, and multi-cluster scenarios. It now supports both traditional sidecar mode and ambient mode. The official documentation lists sidecar, ambient, and proxyless as data plane modes: Istio dataplane modes.
The second route is Linkerd: the lightweight and secure-by-default route.
Linkerd emphasizes simplicity, lower operational burden, Kubernetes-native behavior, and default mTLS. It is suitable for teams that first want to solve service-to-service encryption, basic observability, and reliability without adopting a heavy governance platform all at once. Linkerd’s documentation states that it automatically enables mTLS for TCP traffic between meshed pods: Linkerd automatic mTLS.
The third route is Cilium Service Mesh: the eBPF and sidecar-free route.
Cilium started as a Kubernetes networking, security, and observability solution, and later expanded into service mesh. It emphasizes the eBPF datapath, network policy, Hubble observability, and an optional sidecar-free service mesh model. For teams already using Cilium as their CNI, consolidating networking, security, observability, and mesh capabilities into one infrastructure layer can be attractive. See Cilium Service Mesh.
The fourth route is Kuma / Kong Mesh: the multi-environment route.
Kuma is based on Envoy and emphasizes unified governance across Kubernetes, VMs, multiple zones, and multiple meshes. CNCF describes Kuma as a multi-zone service mesh for containers, Kubernetes, and VMs: Kuma CNCF project.
The fifth category is Consul Connect / Consul Service Mesh.
Consul has long been common in service discovery, configuration, and mixed VM + Kubernetes environments. However, HashiCorp announced in 2023 that future versions would use the Business Source License, so it should no longer be treated as a traditional open source default choice. See HashiCorp adopts Business Source License.
Some projects also require attention because their status has changed. Open Service Mesh has entered the CNCF Archived state, and Service Mesh Interface has also been archived. New projects should not treat them as mainstream candidates. See Open Service Mesh CNCF and CNCF archives SMI.
Another important trend is Gateway API / GAMMA. In the past, each mesh had its own CRDs, such as Istio’s VirtualService and DestinationRule. Kubernetes Gateway API is now trying to provide a more unified API for ingress and mesh traffic management, while GAMMA focuses specifically on service mesh scenarios. See Gateway API for Service Mesh.
7. When Do You Need Service Mesh?
The decision to use service mesh should not begin with “which component is popular.” It should begin with the problem.
If the system has only a few services, uses a unified technology stack, has simple call relationships, does not require mandatory service-to-service encryption, and does not need complex canary or traffic governance, then service mesh is usually not the first priority. At this stage, it is more valuable to make service boundaries clear and to build solid basics: automated deployment, logs, metrics, timeouts, retries, and alerts.
Service mesh becomes more relevant when service-to-service communication governance becomes a systemic problem.
For example:
| Signal | Meaning |
|---|---|
| Many services | Each service handling communication governance alone becomes too costly |
| Diverse technology stacks | Governance capabilities are hard to unify across languages and frameworks |
| Higher service-to-service security requirements | Identity, encrypted communication, and access control need standardization |
| Complex canary rules | Traffic needs to be controlled by version, weight, or request attributes |
| Difficult call-chain debugging | Metrics, logs, and traces need to be unified |
| More cross-team collaboration | The platform needs to provide consistent communication capabilities |
| Multi-cluster or hybrid environments | Discovery, connectivity, and policy governance become more complex |
So service mesh is not the default configuration for microservices. It is a governance choice after microservices reach a certain scale.
It is suitable for solving “how services communicate, how they stay secure, how they are observed, and how traffic is controlled.” It does not redesign domain boundaries for you, and it does not fix incorrect service decomposition. If the system is already a distributed monolith, service mesh can only give that distributed monolith more unified traffic governance. It cannot automatically turn it into a good microservice architecture.
A more practical order might be:
Make each service clear.
Make service boundaries clear.
Make deployment and observability solid.
Finally, when service-to-service communication governance becomes repetitive and systemic, consider moving those capabilities down into service mesh.
8. In the AI Era, Will Microservice Architecture Still Be the Default Answer?
If past architecture evolution was about answering “how can people and teams collaborate better to deliver software,” then the AI era changes the question.
AI will certainly help us write code. But its deeper architectural impact may not be “writing faster.” It may be changing the cost model of architectural iteration.
In the past, once a system became large, teams often moved toward microservices: split repositories, split services, split databases, and split deployment units. This brought team autonomy, release independence, and fault isolation, but it also introduced network calls, distributed transactions, tracing, version compatibility, traffic governance, and operational complexity.
Many microservice boundaries are not only technical boundaries. They are also organizational boundaries. One team owns one business capability; one service carries one relatively independent context. This is consistent with Conway’s Law: the communication structure of an organization shapes the structure of the system it builds.
But AI may change this premise.
When AI can help developers understand large codebases, generate tests, analyze dependencies, locate failures, explain legacy code, and assist with refactoring, the business and code area that one person can cover becomes larger. Modules that previously required several people to maintain may in the future be handled by smaller teams, or even by some “super individuals.”
Here, a “super individual” does not mean one person replaces everyone. It means that with AI assistance, one person can understand a larger context, handle more cross-module problems, and maintain a longer business chain. When an individual’s capability radius expands, the system may no longer need to be split into so many fine-grained services just to fit team boundaries.
This will affect the number of microservices.
Some services that were split out in the past because human collaboration was expensive may be pulled back together. Systems may move from over-microservice designs back to coarser-grained services, or even back to modular monoliths. If one person or a small team can reliably understand and maintain a larger business context, splitting every small capability into an independent service can become excessive.
This does not mean the old big ball of mud will return. A more likely shape is a more restrained architecture: keep clear module boundaries in code, but do not rush to split everything into many deployment units. Only extract a module into a service when it truly needs independent scaling, independent release, fault isolation, or a dedicated owning team.
So the AI era may not be an era of more microservices. It may be an era of fewer services with clearer boundaries.
The real question is no longer “monolith or microservices,” but:
Is this boundary still reasonable?
Does this service really need independent deployment?
Was this module split out only because past collaboration cost was high?
After AI expands the maintenance radius of individuals and small teams, does this service still need to exist independently?
Can the system remain modular, testable, releasable, and observable without being split into a large number of services?
From this perspective, AI may make modular monoliths attractive again. When tools can better understand large codebases, check module boundaries, assist refactoring, and generate tests, a monolith does not have to mean chaos. The real issue is not whether the system is a monolith. The real issue is whether it has clear boundaries.
Perhaps future architecture evolution will not move in one simple direction. It will not simply return to the old monolith, and it will not keep splitting services without restraint. With AI, platform engineering, and automated governance, system boundaries may become more observable, more verifiable, and easier to adjust.
In other words:
The architectural theme of the AI era may not be microservices replacing monoliths. It may be turning architectural boundaries from one-time decisions into an engineering process that can be continuously evaluated and adjusted.