# Part 3: Building a Cluster That Manages Itself

By the end of Part 2, the tombstone math was **understood**. Device-A's write frequency was a **fixed cost**. Device-B's TTL had been **cut 3×**. Device-C's reconnect dump had a **calendared expiry date**.

None of that made the cluster *static*. It made **the cluster *cyclical*** — and the next few weeks were about building the tooling to read that cycle, and discovering a second tombstone problem we hadn't even been watching.

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/00d88d6b-13ee-455e-a289-5c4bafdc0404.svg align="center")

* * *

## The ts\_partitions Subplot

The partition index table which our platform uses to track **which monthly time buckets exist for each device and metric key** — is tiny compared to the main telemetry table. About 4 MB per node. We hadn't been monitoring it at all.

It surfaced when node-2 silently accumulated **33% droppable tombstones** in `ts_partitions`, caught only during a manual `tablestats` inspection. Not by any alert, not by any dashboard. We simply happened to look.

The tombstones came from device keys being added and removed over each device's lifetime. Every key removal leaves a tombstone in the partition index. Device-B, active since October 2023, had accumulated **1,889 tombstones** in this table from 2.5 years of key lifecycle changes sensors added, sensors retired, keys renamed.

`ts_partitions` uses `LeveledCompactionStrategy`, appropriate for a small, read-heavy lookup table. LCS cleans up efficiently once tombstones become droppable but only after they age past `gc_grace_seconds`, same as everywhere else.

We added `ts_partitions` to the monitoring script immediately after this. It now reports alongside `ts_data` in every Slack message, six-hourly, no exceptions.

> **The lesson here isn't about this specific table.** It's that "the main table is healthy" and "the cluster is healthy" are different claims. Every supporting table — indexes, lookup tables, materialized views — has its own tombstone lifecycle and deserves the same monitoring discipline as the table everyone's actually querying.

### The Nightly Repair Interaction

After we started compacting `ts_partitions` regularly, we noticed something stranger: a **synchronized spike** across all three nodes — the ratio jumping from 0% to roughly 19% within 6 hours, identically on every node.

The synchronization was the tell. Random accumulation from independent device activity does not produce identical numbers across three separate machines at the same moment. Something coordinated was happening.

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/dbf1d7e5-c7ab-480b-b1cb-dbd65c566f53.svg align="center")

The cause: our nightly incremental repair runs at 23:30. Repair triggers LCS to recompact the repaired token ranges in `ts_partitions`. When it does, it consolidates tombstone metadata that had been spread across multiple SSTables — making tombstones that were previously invisible to `tablestats` suddenly visible, all at once, on all three nodes, because repair touches all three nodes together.

This is **compaction surfacing tombstones, not creating them.** The spike resolves naturally within 24–48 hours as LCS finishes its cleanup pass. It isn't a problem. But it looked alarming the first few times, before the pattern was understood — which is exactly why it's worth writing down.

### A Single-Node Experiment

We tested lowering `gc_grace_seconds` on `ts_partitions` specifically to 86400 (1 day), reasoning that LCS keeps this table almost continuously repaired, making the default 2-day window unnecessarily conservative for *this table* — even though we'd decided against the same change for the main telemetry table.

One operational detail matters here: `ALTER TABLE` propagates cluster-wide via gossip. There's no per-node schema. To run a genuine single-node experiment, you apply the change and then immediately revert the other nodes to hold them as a control group:

With node-2 and node-3 holding the old setting, the Slack reports made the comparison visible directly

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/3f958923-fd92-46a2-889c-92706f50b57b.svg align="center")

node-1 showed lower, more stable `ts_partitions` ratios following each nightly repair spike, while the other two nodes continued the original pattern. A controlled experiment, run entirely through schema changes and a monitoring dashboard that already existed.

* * *

## Building the Monitor

A Slack webhook **script running every 6 hours** turned out to be more useful than we expected going in. The piece that made it actionable rather than merely informational was the **delta column** — the change in tombstone ratio since the previous report.

```text
NODE SSTABLES TS_DATA TS_PARTITIONS CHANGE  
node-1 5 0.00000 0.00000 —  
node-2 4 0.00000 0.00000 ▼-11.030%  
node-3 4 0.00000 0.00000 ▼-10.775%
```

The ▼ symbols are **TWCS auto-drop events** — entire expired SSTables discarded wholesale. A large ▼ means the cluster cleaned itself up without anyone touching a terminal. A sustained ▲ above 1.5% across multiple consecutive reports is the **signal to go looking for a new high-frequency device**, the same way we found device-A and device-C in Part 2.

The mechanism behind the delta column is simple: a **state file** storing the previous run's tombstone ratios per node, diffed against the current run. Without it, the six-hourly numbers are just numbers. With it, they have direction and **direction is what turns a report into a decision**.

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/e282d997-58d2-42cc-a61a-135c64e3d2a0.svg align="center")

### Intervention Thresholds That Emerged From Operation

These weren't designed up front. They came from watching the reports for several weeks and noticing which patterns actually warranted action versus which were noise

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/1429e6df-ed67-4bc5-a0a1-d1d1e617498a.png align="center")

The last row is the one that's easy to skip building, and the most valuable once it exists. **Log-based alerting catches things the ratio-based monitor structurally can't see until hours later**.

* * *

## The Self-Managing Cluster

The first time TWCS dropped an entire expired SSTable automatically **no manual compact, no intervention, nobody watching**, the Slack report showed this:

> node-1 9 0.00000 0.00000 ▼-9.517%

Four SSTables gone. 9.5 percentage points of droppable tombstones eliminated in a single compaction cycle. The `unsafe_aggressive_sstable_expiration` flag **doing precisely** what it was configured to do back in Part 1: identifying an entire 30-day window where every row had passed both its TTL and `gc_grace_seconds`, and **deleting the file wholesale** rather than scanning it row by row.

Node-2 and node-3 followed within 24 hours, each on its own schedule, each without anyone running a command.

The self-managing cycle now reads like this: **tombstones accumulate from device writes**, TWCS recognizes when a **time window has fully expired**, the **whole SSTable drops**, the ratio **falls sharply**, accumulation **begins again**. In the Slack reports this shows up as a **sawtooth** — a slow climb, a sharp drop, repeat, on a timescale measured in weeks.

This is the part worth sitting with: **the sawtooth is not a problem to eliminate.** It's the **expected behavior** of a correctly configured time-series compaction strategy carrying a write load that includes fixed-frequency industrial sensors. A permanently flat 0% line would require either no writes (not a useful database) or infinite compaction resources (not a real cluster). The goal from the start was never a flat line. It was **making the cleanup automatic, predictable, and bounded** and watching that first `-9.517%` drop happen with nobody at the keyboard was the moment that goal was confirmed met.

### The Repair Log as Evidence

At one point during this process we seriously considered reducing `gc_grace_seconds` cluster-wide from 2 days to 1, reasoning that it would shrink the post-compaction rebound window described in Part 2. The repair log turned out to be the decisive piece of evidence against it:

> Repair command #25 finished in 2 minutes 34 seconds
> 
> node-2: Repair command #24 finished in 1 minute 18 seconds
> 
> node-3: Repair command #24 finished in 9 seconds
> 
> Overall Duration: 00:04:12

Four minutes twelve seconds for a full incremental repair across the entire cluster. Node-3 finished in 9 seconds because it was already 99.84% repaired going in. This told us our **cluster consistently repairs within a tiny fraction** of the existing 2-day grace window — call it a **47-hour safety buffer** between "repair completes" and "grace window closes."

Reducing `gc_grace_seconds` to 1 day would cut that buffer to roughly 22 hours. On a cluster where one node has a documented history of **49 aborted** compactions tied to **memory pressure**, any extended maintenance event, **disk pressure** incident, or **OOM kill** that keeps a node offline for **more than 22 hours** becomes a **genuine data-consistency risk**: deleted rows reappearing because a node missed the repair that would have propagated their tombstones. The rebound pattern from Part 2 is visually annoying but **costs nothing operationally**. A narrower safety margin would cost something real if it ever got tested.

> Measure your repair duration before touching `gc_grace_seconds` in either direction. The number it gives you is the floor your grace period needs to clear comfortably — not just on average, but on your worst realistic day.

* * *

## What Remains Open

Two changes stayed on the list, **deliberately deferred** rather than forgotten:

**TWCS window size.** Cassandra's own documentation recommends roughly 20–30 active compaction windows for a given TTL. At our 90-day TTL, that points to a 7-day window — about 13 windows, smaller compaction jobs, less memory pressure on nodes that are already tight on RAM, and more frequent auto-drops instead of fewer larger ones. We **kept 30-day windows** because the cluster is stable today and the change means **another full round of compaction** work to re-bucket existing data. The documentation's idealized 3-day window is very likely too fine-grained for 2-core hardware regardless. 7 days is the **more realistic target for *this* cluster**, not the textbook one

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/e8a406b3-fc11-4cdb-b5cf-5a036e797e89.png align="center")

**Heap increase.** Running Cassandra on a **2 GB JVM heap** with 7.7 GB of total RAM, on a node managing **36 GB of SSTable data**, is **tight** by any standard. The 49 compaction aborts on node-1. The ones that drove a meaningful share of the diagnostic work across both this post and the last, trace directly to memory pressure during large merge operations. Going from 2G to 4G would very likely eliminate that failure mode outright. We deferred it because it requires **another rolling restart** and we'd already done several in close succession. It remains, honestly, the single highest-leverage infrastructure change still on the table.

* * *

## Where the Series Lands

Three nodes. 3.9 billion writes. 2 cores and 7.7 GB of RAM each. A read latency that went from **7.99 ms to 1.90 ms and stayed there**. A tombstone ratio that no longer sits at a fixed number, because the right number for a cluster like this was never zero — it's a **bounded, self-correcting cycle** with a monitoring system reading it every six hours and a documented set of **thresholds for when a human needs to step in**.

None of the individual pieces here were exotic. **TWCS over STCS** for time-series data is well-documented. `gc_grace_seconds` math is in the Cassandra docs. A **Slack webhook** is a cron job and a curl command. What made this work was less any single decision and more the **discipline of checking** `compactionhistory` before assuming an abort, **querying TTLs** before assuming misconfiguration, and measuring repair duration before touching a safety setting.

The principle that carried the whole series: **make the change you understand before the change you're still learning.**

![](https://cdn.hashnode.com/uploads/covers/69e47735ee84f66e94219a17/1057205d-0bdd-49c8-9d7e-262eea17a77a.svg align="center")

The cluster runs itself now. **Understanding *why* it runs itself** is what made that possible.