Symptoms of a distributed monolith, and how we fixed our service boundaries

A few years ago I worked on splitting an order platform into services. We started with a Rails monolith and, over eighteen months, ended up with eleven services: orders, cart, pricing, inventory, customers, payments, shipping, notifications, promotions, tax and a reporting service.

On paper it looked like a textbook microservices architecture. In practice we had made things worse. Deploys were slower than before. Incidents spread across services. Developers needed three repositories open to make one change. We had built a distributed monolith: the operational cost of many services, with the coupling of one codebase.

This post describes the symptoms we saw, how we measured them, and what we changed.

Symptom 1: lock-step deploys

What we saw. Most features required changes to two or three services, deployed in a specific order. Release notes read like "deploy pricing 4.12, then cart 7.3, then orders 9.1; do not deploy orders first". Rollbacks had to be coordinated the same way.

How we measured it. We looked at six months of deploys and counted how often a pull request in one service was linked to pull requests in others and released within the same day. For the core services (orders, cart, pricing, promotions) it was 64 percent. Independent deployability, the main promise of services, was mostly fiction.

What it indicates. Services that must change together belong together, or their interface is wrong. Usually the interface leaked internal details, so internal changes became interface changes.

Symptom 2: a shared database

What we saw. Seven services connected to the same PostgreSQL database. Each "owned" some tables, by convention, but several read each other's tables directly, because it was quicker than adding an API. The reporting service read everyone's tables.

A schema migration in the orders service once renamed a column that the shipping service queried. Shipping broke in production. Nobody had a list of which service read which column.

How we measured it. We enabled log_statement briefly in staging and parsed table access by database user, one user per service:

-- from pg_stat_statements, grouped by role and table name extracted from the query
role         | tables_read_owned_by_others
-------------+---------------------------------------------
shipping     | orders, order_lines, customers
promotions   | orders, cart_items, customers, products
reporting    | (all 41 tables)

What it indicates. A shared database is a shared, undocumented API. Every table another service reads is part of your contract, whether you meant it or not.

Symptom 3: chatty synchronous calls

What we saw. Rendering the cart page called cart, which called pricing for each item, which called promotions, which called customers to check eligibility. A cart with 12 items made over 50 internal HTTP calls. Latency added up, and an outage in customers (a service the cart page should barely depend on) took the cart down.

How we measured it. Distributed tracing. For the main user journeys we counted spans per request and service hops on the critical path:

What it indicates. Data needed together is split across services that each own a fragment, so every request reassembles it over the network.

Symptom 4: shared libraries with domain logic

What we saw. A commons library used by every service contained the Order model, price calculation helpers and validation rules. Changing how a discount was rounded meant releasing a new library version and redeploying every service that used it, in the right order.

What it indicates. Shared domain code is coupling hidden in a dependency. Sharing infrastructure code (logging, HTTP clients, metrics) is fine. Sharing the meaning of the business is not.

What was actually wrong

We had split by technical noun rather than by business capability. "Pricing", "promotions" and "tax" were separate services, but all three were parts of one question: "what does this cart cost?". Every change to how carts are priced touched all three. Meanwhile "orders" covered everything from cart conversion to returns, which changed for unrelated reasons.

Good boundaries follow how the business changes. Things that change together, for the same reason, decided by the same people, should be inside one boundary. That is the idea behind bounded contexts in domain-driven design, but you do not need the vocabulary to apply it: look at your change history.

What we changed

1. Merged services that always changed together

Pricing, promotions and tax became one pricing service with one database. Its API answers one question: "price this cart", returning line prices, discounts and taxes. The cart service calls it once per request, not once per item. (A separate post covers merging services in more detail.)

2. Gave each service its own data, and moved readers to APIs or events

Each service got its own database schema with its own credentials, and other services lost access to it. For the cross-service reads we found:

• Shipping reading orders: replaced by an OrderPlaced event containing what shipping needs (address, items, weight). Shipping stores its own copy.
• Promotions reading customers: replaced by a customer segment attribute published in CustomerUpdated events, cached locally.
• Reporting reading everything: replaced by a data pipeline that streams changes from each service's database into a warehouse. Reporting queries the warehouse.

This is more data duplication, deliberately. Each copy has one owner (the publishing service) and one refresh path (events).

3. Replaced call chains with data the caller already has

Where a service needed reference data from another on every request, it now subscribes to changes and keeps a local read model. The cart service no longer calls customers at all; eligibility data arrives with the pricing response.

4. Shrunk the shared library to infrastructure

commons lost every domain class. Services that need an order representation define their own, containing only the fields they use, mapped from events or API responses.

5. Made versioning explicit

Service APIs are versioned, and changes are additive. Contract tests between consumers and providers run in CI, so a provider cannot ship a change that breaks a consumer's recorded expectations.

Results after a year

A checklist to find your own distributed monolith

• Do most features need coordinated deploys of several services?
• Does any service read another service's tables?
• Does one user request fan out into dozens of internal calls?
• Does a shared library contain business rules?
• Can you deploy a service on a Friday without asking another team?
• Is there an up-to-date list of who consumes each API and event?

Two or more "yes" answers to the first four, or "no" to the last two, are worth investigating.

Summary

• A distributed monolith has the costs of services and the coupling of a monolith.
• Measure coupling: coordinated deploys, cross-service table access, call depth per request.
• Draw boundaries around things that change together for the same business reason.
• Give each service its own data. Share facts through events and APIs, not tables or domain libraries.
• Fewer, better-bounded services beat many fragments.

References: Martin Fowler, Microservices, Martin Fowler, Bounded Context, PostgreSQL pg_stat_statements.