The paper surveys legal precedent around software, especially in medicine, and argues that the rise of AI weakens one of the assumptions that has historically protected software vendors from malpractice-style liability. That assumption is that human professionals remain the real decision makers, while software merely assists them.

For ordinary clinical decision-support systems, that line may seem plausible. A doctor reviews the output, applies professional judgment, and accepts or rejects the recommendation. But with narrow AI systems that develop deep expertise in a domain, the relationship becomes less clean. If a system produces a recommendation that a human reviewer cannot meaningfully audit, then “human oversight” can become more procedural than real.

That is the paper’s strongest point. AI makes it harder to rely on the fiction that a human independently reviewed the basis for a software recommendation.

Where I agree

I agree with the paper’s central concern. Technology companies and software professionals should not be immune from responsibility simply because the harm is mediated through code.

Software now helps decide medical treatment, credit access, hiring, public benefits, fraud detection, insurance pricing, and many other areas where mistakes can alter a person’s life. When systems operate at that level of consequence, the industry needs a more serious account of duty, care, review, and accountability.

The paper is also right to look at professional malpractice as a useful comparison. Older professions have already built ideas around competence, fiduciary responsibility, licensing, insurance, standards of care, and disciplinary consequences. Software does not need to copy that structure blindly, but it should learn from it.

Where the analogy weakens

The main weakness is that the doctor-software-engineer analogy does not map cleanly.

A doctor usually owns the professional act of diagnosis or treatment. Even when the doctor consults peers, the final professional decision is attributable to that doctor. Software is rarely produced that way.

A deployed AI product is usually the result of many decisions:

• Product defines the user problem and desired behavior.
• Data teams choose or prepare datasets.
• Engineers implement the system.
• Model teams tune behavior.
• Designers shape how users interpret the output.
• Legal and compliance teams approve risk language.
• Executives decide whether the product ships.

If harm occurs, it may be impossible or unfair to assign responsibility only to the individual engineer who wrote part of the implementation. The engineer may have seen the risk but lacked authority to block release. Or the risk may have emerged from a product decision, data limitation, or business constraint outside the engineer’s control.

That does not mean nobody is responsible. It means software responsibility has to be organizational, not merely individual.

What the paper leaves open

The paper is valuable as a survey and warning, but it does less to explain how responsibility should actually be assigned inside software organizations.

Several questions need more work:

• What would a software “standard of care” look like for AI systems?
• Which duties should belong to engineers, product leaders, data scientists, executives, and companies?
• Should licensing apply to all software engineers, or only to those working in high-risk domains?
• How should certifications, professional societies, or bodies such as IEEE shape enforceable practice?
• How do existing frameworks such as GDPR or human-rights declarations become operational inside product development?

The strongest version of this argument would move from analogy to mechanism. It would define what competent AI development requires, what evidence must be produced before deployment, and who is accountable when that evidence is ignored.

Bottom line

“Software Malpractice in the Age of AI” is worth reading because it names a real gap: AI systems are becoming professionally consequential without the professional accountability structure that older high-stakes domains developed over time.

But the solution cannot simply be “treat software engineers like doctors.” Software is too collaborative, too organizational, and too entangled with product and business incentives for that analogy to carry the whole burden.

AI malpractice, if the term is to mean anything useful, has to attach responsibility to the system of production. That includes the engineer, but it also includes the company that creates the incentives, approves the risks, and profits from the deployment.

Reference

Danny Tobey, “Software Malpractice in the Age of AI: A Guide for the Wary Tech Company”, AIES 2018.

Miner Town: Awakening is a dystopian science fiction novel that blends post-apocalyptic survival, coming-of-age, and rebellion against an oppressive system. The world is imaginative and immediately engaging. Miner Town feels harsh, believable, and internally consistent, and the contrast between the mining settlement and the pristine Trinity cities provides a compelling foundation for the story.

The strongest aspect of the novel is its cast. Gage, Camilla, Otto, and Ben each represent different strengths, and their relationships drive much of the narrative. Rather than relying solely on action, the story spends time developing why these characters become who they are.

My main problem is with the prose. There is a vivid metaphor or philosophical reflection in every paragraph. While individually well written, their cumulative effect occasionally slows the narrative and reduces the impact of the strongest moments. The book would hit harder if it were plainer most of the time. Related issue: several pieces of exposition get delivered twice by different characters. Cutting those would help the pace.

Themes of sacrifice and exploitation are handled well, mostly because the author trusts the setting to carry them instead of having someone announce them.

An ambitious debut. If you liked The Hunger Games, you will recognize the genre. I will definitely like to read the next one.

Rating: 4.5/5

Find the book on Amazon.

The first question is what the team still verifies by hand. Every manual verification habit is a candidate skill file. The content is nothing more than the steps a reviewer already performs before trusting a change: what to start, what to click, what to compare, which tests to run. The habit feels too obvious to document, which is exactly why it never gets documented. A useful test for the finished file is whether a new team member could follow it without asking anyone.

The next question is how the team decides that a piece of work is done. If the honest answer is that an experienced person looks at it and says so, the missing artifact is a measurable exit criterion. A worthwhile exercise is to take one recent change that was accepted on judgement and restate the acceptance in checkable terms: a response time under a stated number, a set of tests that must pass, a score from an audit tool. Not every kind of done converts into a measurement, but more of them convert than the first attempt suggests, and each conversion is one less decision that has to be relitigated per change.

Then there are the tasks people do at the same time every day. A morning routine of reading a channel, sweeping a queue, or checking a dashboard is a schedule that exists only in someone’s habits. Writing it down as an actual schedule is the small step. The more valuable step is to ask what change in the world the routine is standing in for, and whether the system where that change happens can announce it. A webhook or a queue alert replaces a guess about timing with a fact about the source.

The last question concerns streams: bug reports, support tickets, dependency alerts, anything that arrives continuously and gets triaged by a person. This is where a proactive routine could stand, and it is also where the diagnostic has to be most careful. The routine needs a scoped goal, a budget, a rate limit, a shutoff, and an owner. The owner is the deciding item. If no one can be named who is responsible when the routine acts wrongly, the routine is not ready to build, whatever the tooling makes possible.

The output of the exercise is a short list: the checks nobody has written down, the criteria that live in one person’s judgement, the schedules that exist as habits, and the streams with no owner for their automation. A team does not need to adopt all four loops, and most should not. It needs to know which artifact on that list to write first.

But the file underneath is not a grid. A text file is a one-dimensional sequence: a flat run of characters, h e l l o \n w o r l d, where the newline is a character that means “start drawing on the next row.” The buffer that holds the document in memory is the same, a sequence you index with a single number.

So every editor lives with a permanent translation problem. This post looks at how a few of them solve it: a from-scratch Rust editor I’m building (textr), the two it is modeled on (gedit and Emacs), and VS Code for a fourth point of comparison.

• The impedance mismatch
• Where it comes up in textr
• Why not just pick one and be done?
• How gedit does it: iterators over a tree, cursor-as-mark
• How Emacs does it: point is a number, everything else is derived
• How VS Code does it: a piece tree
• Side by side
• The choice follows the data structure and the dominant operation
• Notes
• References

The impedance mismatch

Two coordinate systems, describing the same text:

• Humans and screens think in 2D: (line, column).
• Files and buffers store 1D: a single offset into a sequence.

Figure 1 — The same caret, two coordinate systems. (line 1, column 1) on the left is flat index 4 on the right. The mapping between them is the editor’s job.

Bridging those two systems correctly and cheaply is one of the basic jobs of a text editor. Get it wrong and the cursor lands mid-character in a UTF-8 sequence, or an insert meant for one place writes three lines away.

Where it comes up in textr

This shows up in textr’s View, the cursor model.¹ The cursor is stored the way a user thinks about it:

struct View { line: usize, column: usize, goal_column: usize }

That is a 2D coordinate. But the buffer, a ropey rope, has a strictly 1D editing API:

doc.insert(char_idx, text);   // one index
doc.remove(char_idx..end);    // one index

insert doesn’t know what a “line” is. To insert a character where the cursor is, I have to convert (line, column) into a single flat index. That conversion is one function:

pub fn cursor_char_idx(&self, doc: &Document) -> usize {
    doc.line_to_char(self.line) + self.column
}

line_to_char(line) gives the flat index where that line starts; add column and you have the caret’s absolute position in the whole document. It’s tempting to think column already is the index — and on line 0 it is, because line 0 starts at 0. But look at "ab\ncd" with the caret on the d:

             index:  0   1   2   3   4
             char:   a   b   \n  c   d
                                     ^
   (line 1, column 1)  ->  line_to_char(1) + 1  =  3 + 1  =  4

column is 1, but the flat index is 4. column is “where I am on this line”; the flat index is “where I am in the file.” The rope only understands the second one. (The reverse exists too, char_to_line(idx), for when you have a flat position, like a search hit or a mouse click, and need to find the right row.)

Both conversions are cheap on a rope: ropey documents line_to_char and char_to_line as O(log n) each, and the + column is O(1), so moving a cursor between the two coordinate systems runs in O(log n) whichever way it goes.²

Why not just pick one and be done?

The harder question is not how to convert but which representation is the source of truth, and how it survives edits. There are a handful of strategies:

1. Store 2D, derive 1D on demand (textr’s choice). Keep (line, column) canonical; compute the flat index whenever the buffer needs it. Never store the index — a stored index goes stale the instant text changes ahead of it. Pro: vertical movement is native (Up/Down, and the “goal column” that remembers your column as you glide over short lines, are 2D ideas). Con: every edit must update the 2D cursor by hand, and 2D positions need clamping.
2. Store 1D, derive 2D on demand (Emacs’s choice). Keep a single integer offset canonical; compute line and column only to display or move by rows. Pro: dead simple — a position is one number. Con: line and column now cost a computation you mitigate with caches.
3. Store both, keep them in sync. Fast reads either way, but now two things must be updated on every edit or they drift.
4. Hand out position objects. An “iterator” that internally carries both representations. Cheap to read either way — but typically invalidated by edits, so you can’t hold one across a modification.
5. Persistent marks. The subtle problem with any raw offset: if I remember “position 487” and someone inserts 10 characters at the top of the file, 487 now points somewhere wrong. A mark is a position the buffer itself keeps updated as text moves around it.

The choice is not free-floating. It falls out of what the buffer is made of:

Figure 2 — The storage structure decides which coordinate is cheap, which in turn pushes the design toward 1D-canonical or 2D-canonical.³

Strategies 4 and 5 are not hypothetical. gedit uses both: iterators for cheap reads and marks for persistence. Emacs uses markers (strategy 5) alongside its 1D point.

How gedit does it: iterators over a tree, cursor-as-mark

gedit is built on GTK’s GtkTextView/GtkSourceView, backed by a GtkTextBuffer.⁴ Internally that buffer isn’t a flat array — it’s a tree (a B-tree) that indexes lines and character/byte offsets, so it can answer “where does line N start?” and “what line is offset K on?” in roughly O(log n). Both directions of our conversion are cheap by construction.

You never touch raw offsets directly. Instead you work with GtkTextIter — a small stack-allocated struct representing a position, obtained from either coordinate system:

GtkTextIter iter;
gtk_text_buffer_get_iter_at_line_offset(buffer, &iter, line, column); // from 2D
gtk_text_buffer_get_iter_at_offset(buffer, &iter, char_offset);       // from 1D

And crucially, an iter carries both representations at once — you can ask it for its line (gtk_text_iter_get_line), its column (gtk_text_iter_get_line_offset), or its absolute char offset (gtk_text_iter_get_offset). The 2D↔1D conversion is baked into the iterator, powered by that tree. GTK’s answer to “which is the source of truth?” is essentially “neither — the tree makes both cheap.”

The catch: a GtkTextIter is transient — invalidated the moment the buffer is modified. So you don’t store iters; you re-fetch them.

For positions that must persist across edits, GTK gives you GtkTextMark, a named position the buffer maintains automatically. gedit’s cursor is a mark named "insert" (the selection’s other end is "selection_bound"). To render the caret, gedit gets an iter at the "insert" mark and reads its line and column; to type a character, it inserts at that iter. The persistence problem (strategy 5) and the conversion problem (strategy 4) are handled by two separate abstractions.

Figure 3 — gedit with this post’s draft open. The status bar reads “Ln 5, Col 12”, the 2D face of the "insert" mark.

How Emacs does it: point is a number, everything else is derived

Emacs comes at it from the opposite end. Its buffer is a classic gap buffer — one big array of characters with a movable gap where edits happen, which makes insertion and deletion at the cursor very cheap.⁵

Position in Emacs is point: a single integer, the character offset of the caret (1-based — point-min is 1; positions sit between characters). Point is 1D and canonical. Almost every primitive takes or returns a buffer position as a plain integer.

Line and column are not stored — they’re computed when asked:

(line-number-at-pos)   ; scan / caches to count newlines before point
(current-column)       ; scan back to line start, honoring tab-width and char widths

current-column shows that “column” is subtler than a character count: Emacs makes a tab advance to the next tab stop and accounts for wide characters, so the visual column and the character column differ. (textr, for now, uses character columns, a documented simplification.)⁶

Computing line numbers means scanning for newlines — O(distance) — so Emacs keeps caches to amortize it (line-number lookup on a multi-megabyte buffer is a real, historically-tuned concern). For persistence, Emacs has markers — objects that hold a position and are automatically nudged as text is inserted or deleted before them, with an insertion type that decides whether a marker sticks or advances when text lands exactly on it. Same idea as GTK’s marks, different name.

Emacs also separates character positions from byte positions (position-bytes) in multibyte buffers, so ordinary code can index by character and never split a multibyte sequence, the same reason textr indexes by char, never by byte.

Figure 4 — Emacs with the same file open. The mode line reads (12,4), the line and column derived from point.

How VS Code does it: a piece tree

VS Code reaches a similar place to textr from a different direction.⁷ Its buffer is a piece tree: a piece table whose pieces hang off a balanced red-black tree, where each node caches the text length and line-break count of its subtree, so a lookup by line or by offset walks the tree in roughly O(log n) instead of scanning. Its canonical position is 2D, like textr’s — the editor keeps the cursor as a Position of lineNumber and column, both 1-based, and the flat offset is derived on demand through getOffsetAt and its inverse getPositionAt. The reason for keeping 2D canonical differs from textr’s, though: the whole editor API is written in terms of Position, so 2D is the natural currency for the interface, not because vertical movement is the hardest job.

One detail sets VS Code apart from the other three. It measures column and the offset in UTF-16 code units, not characters, so an emoji outside the Basic Multilingual Plane counts as two columns and a grapheme built from combining marks spans several units. textr, gedit, and Emacs all index by char (Unicode scalar values); VS Code indexes by UTF-16 unit, and the two do not agree on where a given column falls.

Side by side

The choice follows the data structure and the dominant operation

There is no universally correct answer, only trade-offs that fall out of two things: what the buffer is good at, and which operation you do most.

• Emacs’s gap buffer makes offsets cheap and edits-at-point cheap, so 1D-canonical is the simplest choice; it pays for line and column with scans and caches.
• gedit’s tree makes both directions cheap, so it can afford to hide the whole question behind iterators and lean on marks for persistence.
• textr’s rope also makes both directions cheap (O(log n) either way), so the choice was free. I picked 2D-canonical because the View’s busiest and hardest job is vertical movement with a goal column, which is a 2D idea. The representation matches the operation it does most.
• VS Code’s piece tree makes both directions cheap like gedit’s, and it too keeps a 2D Position canonical — but for a different reason than textr: its whole editor API speaks in positions, so 2D is a matter of interface rather than of any single dominant operation.

One point is easy to miss: a raw offset is only correct for one moment. Once the text changes ahead of it, an old index points to the wrong place. That is why converting on demand (textr) or letting the buffer maintain the position (marks and markers) is safer than storing the number and reusing it. textr avoids the problem today because it has a single cursor that updates itself; when it grows multiple cursors or collaborative editing, it will want marks too, for the same reason every editor eventually adopts them.

Notes

• GtkTextIter is a value, not a handle. It’s valid only until the next buffer mutation; treat it as a snapshot and re-fetch after any edit. Persistent positions are GtkTextMarks, not stored iters.
• Marker insertion type decides the tie-break: when text is inserted exactly at a marker, does the marker stay put or advance past the new text? Emacs lets you choose per-marker; it’s the kind of detail that quietly determines whether your saved position feels “before” or “after” an edit.
• Character vs byte positions are different numbers in any multibyte buffer. textr indexes by char throughout and never does byte math (ropey also exposes byte-index methods; textr uses only the char ones) — which is why inserting a precomposed 'é' (U+00E9: one char, two UTF-8 bytes) advances the cursor by exactly one column. A decomposed 'é' (e + a combining accent) is two chars and would advance by two — so this assumes NFC-normalized input; char indexing is not grapheme indexing.
• ropey’s phantom trailing line. A rope for "a\nb\n" reports three lines — "a\n", "b\n", and a final empty "". The caret may rest on that empty line but no further; textr’s line_len_chars helper strips the trailing \n so “end of line” lands before it, not after.
• “Column” is not “display column.” This post (and textr, today) uses character columns. A real editor’s visible column has to account for tab stops and wide/zero-width characters — which is exactly what Emacs’s current-column does and where grapheme-cluster segmentation (UAX #29) eventually comes in.
• textr is a learning project — a from-scratch gedit clone I’m building to learn Rust, with a headless, UI-agnostic core and thin frontends. The 2D↔1D bridge above is one small, load-bearing piece of its View.

References

1. How is code compiled?
2. How is it distributed?
3. Where is encapsulation enforced?

Java accumulated its answers over time. Source files and packages came from the language, JARs became the distribution unit, and JPMS arrived in 2017 to add strong encapsulation and explicit dependency readability.

Rust and Cargo started with a more integrated model, designed together around a standard package and build system.

The two systems answer the same three questions but put the answers in different places. Java spreads them across pieces assembled over two decades — packages and the module system in the language, JARs and Maven outside it — that must stay consistent without anything forcing them to. Rust concentrates compilation and encapsulation in the crate and distribution in the Cargo package, with far less overlap.

The terms

Both toolchains reuse a few of the same words for different things, so it helps to fix the vocabulary before the code.

Rust / Cargo

• Crate — the unit the compiler compiles, and the unit of encapsulation. A crate is either a library crate or a binary crate.
• Package — what Cargo versions, builds, and publishes, described by Cargo.toml. A package holds at most one library crate and any number of binary crates.
• Module (mod) — a namespace inside a crate. Modules form a tree, and that tree is where visibility is enforced.
• Workspace — a set of packages that share one Cargo.lock and one build.

Java / Maven / JPMS

• Package — a namespace declared per source file (package in.systemhalted.gateway.api;).
• Module (JPMS) — a named group of packages with a module-info.java that declares what it reads and what it exports. The unit of strong encapsulation.
• Artifact (Maven) — the versioned, publishable unit, identified by groupId:artifactId:version and shipped as a JAR.
• JAR — the packaging and distribution format.

Shared

• Dependency — code from another package, artifact, crate, or module that this component uses. Maven resolves and fetches a Java dependency, while JPMS separately determines module readability through requires. Cargo resolves a Rust dependency and makes the corresponding crate available to the compiler from the same manifest declaration.

The words already collide. A Rust crate is closest to a JPMS module, a Rust module to a Java package, and a Cargo package to a Maven artifact. None of these is exact — a crate resembles a JPMS module in encapsulation and dependency structure more than in the mechanics of compilation — and “module” means almost opposite-scale things in the two worlds. The rest of this post lines these up against the three questions and shows where the words look alike but aren’t.

The same project, twice

Consider a small API gateway library with a public API, a configuration type, and an internal routing engine.

Here is the Java version, built with Maven and JPMS:

gateway/
├── pom.xml
└── src/main/java/
    ├── module-info.java
    └── in/systemhalted/gateway/
        ├── api/
        │   └── RouteHandler.java
        ├── config/
        │   └── GatewayConfig.java
        └── internal/
            └── Router.java

And the Rust version:

gateway/
├── Cargo.toml
└── src/
    ├── lib.rs
    ├── api.rs
    ├── api/
    │   └── handlers.rs
    ├── config.rs
    ├── internal.rs
    └── internal/
        └── router.rs

Four files reveal most of the architectural difference: pom.xml, module-info.java, Cargo.toml, and lib.rs. They do not map one-to-one. Understanding why is most of understanding the two systems.

The four files

pom.xml: identity and dependencies

The POM belongs to Maven, not to the Java language. It declares the artifact’s coordinates and the dependencies required to build it:

<groupId>in.systemhalted</groupId>
<artifactId>gateway</artifactId>
<version>1.4.0</version>
<dependencies>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.17.1</version>
    </dependency>
</dependencies>

Maven resolves the dependency, downloads the JAR, and constructs the classpath or module path. javac knows nothing about the POM.

module-info.java: readability and exports

The module descriptor lives at the root of the Java source tree, at src/main/java/module-info.java. It is part of the Java language and is compiled by javac.

module in.systemhalted.gateway {
    requires com.fasterxml.jackson.databind;
    exports in.systemhalted.gateway.api;
    exports in.systemhalted.gateway.config;
    // in.systemhalted.gateway.internal is not exported
}

The descriptor declares two boundaries: which modules this module reads, and which packages it exposes.

Jackson therefore appears twice. The POM tells Maven to obtain the JAR; the descriptor tells JPMS that this module may read it. These are separate systems, and nothing forces them to agree: a dependency can sit on the module path without a matching requires, or be declared with requires without being supplied by the build. The two names even differ — the Maven artifactId is jackson-databind, the JPMS module name com.fasterxml.jackson.databind. Coordinate and module identity are separate namespaces.

Cargo.toml: the package manifest

Cargo.toml describes the Cargo package:

[package]
name = "gateway"
version = "1.4.0"

[dependencies]
serde_json = "1"

The gateway package here is a single library crate, src/lib.rs. Add a src/main.rs and it would hold two crates, a library and a binary. The crate is where visibility and module boundaries apply.

Declaring a dependency in Cargo.toml both resolves it and makes it available during compilation; Cargo invokes rustc with the corresponding --extern arguments. There is no separate language-level file holding a second copy of the dependency declaration.

lib.rs: the crate root

lib.rs is not a manifest. It is source code — the root of the library crate and the starting point of the crate’s module tree:

// src/lib.rs
pub mod api;
pub mod config;
mod internal;

pub mod api; loads src/api.rs and makes the module publicly reachable; mod internal; loads src/internal.rs but does not make it reachable outside the crate.

This is where part of the Java module descriptor’s role moves into source. The descriptor’s exports in.systemhalted.gateway.api; and the crate root’s pub mod api; are not exact equivalents, but serve the same purpose: defining the externally reachable surface. In Rust the dependency side lives in Cargo.toml, the visibility side in ordinary source.

Where the module tree comes from

The deeper difference is not the file names. It is who defines the structure.

Java packages are declared independently

Every Java source file declares its package, such as package in.systemhalted.gateway.internal;. Build tools conventionally mirror that name in the source directory, as in/systemhalted/gateway/internal/Router.java. But the package declaration defines membership: no parent package registers the class, and no other source file has to mention it.

A Java package is a flat namespace. The dots suggest hierarchy, but there is none: a.b and a.b.c are separate packages, and a.b.c is not semantically nested inside a.b.

Rust modules form an explicit tree

In Rust, a file does not join the crate merely by existing on disk; a parent module must declare it. In src/lib.rs, mod internal; brings src/internal.rs into the crate. In turn, src/internal.rs’s pub mod router; brings src/internal/router.rs into the tree, and src/api.rs’s pub mod handlers; brings src/api/handlers.rs. Remove the mod router; declaration and the compiler never reads router.rs, even if the file remains on disk. The filesystem follows the module tree; it does not create it.

This explains why the layout carries both internal.rs and an internal/ directory: internal.rs defines the module, internal/ holds its child modules. A leaf module such as config needs only config.rs. The older internal/mod.rs layout still works, but the internal.rs-plus-internal/ style is now common.

Visibility follows the tree

Once the tree exists, visibility is evaluated along paths through it. Consider:

// src/internal/router.rs
pub struct Router { /* ... */ }

pub(crate) fn rebuild_routes() {
}

pub(super) fn debug_dump() {
}

pub(in crate::internal) fn merge() {
}

fn parse_segment() {
}

These declarations represent different visibility scopes.

pub is bounded by the path

This is the first rule that often surprises Java developers. pub struct Router; does not necessarily make Router reachable outside the crate; every module along the path must also be public. The full path is crate::internal::router::Router, but lib.rs declared mod internal;, not pub mod internal;. The private internal module blocks the path: Router is public within the visibility that path allows, but the path itself is not externally reachable.

Visibility belongs to both the item and its route through the module tree.

Visibility can name an ancestor

Rust can express scopes relative to the tree: pub(crate) is visible throughout the crate, pub(super) within the parent module and its descendants, pub(in crate::internal) within the named module and its descendants. The path in pub(in path) must name an ancestor of the current module. A function inside crate::internal::router cannot declare pub(in crate::api), because api is a sibling branch, not an ancestor.

Java has no equivalent, because its packages do not form a semantic tree. For top-level types and cross-package access, Java mainly offers public and package-private; class members also have private and protected, but none express visibility to a named package subtree.

Before JPMS, a package named internal was mostly a warning:

This is internal. Please do not use it.

JPMS made that boundary enforceable across modules by withholding exports. But inside the module, Java remains flat.

Visibility, keyword by keyword

External reachability is path-based in Rust and package-based in JPMS. Each Rust modifier grants a scope defined by the module tree; the nearest Java construct is often not a keyword at all, because Java expresses external reachability through the module descriptor rather than a modifier on the declaration.

Java approximates pub(crate) by placing a public class in a package omitted from exports — not a Java visibility level, but an effect of the module boundary. Rust privacy also differs from Java package privacy: a private Rust item is visible to its defining module and that module’s descendants, while a package-private Java member is visible to all code in the same package.

Imports reveal the architecture

The different namespace models also shape imports.

Java:

import java.util.List;
import in.systemhalted.gateway.api.RouteHandler;

Rust:

// src/internal/router.rs
use std::collections::HashMap;
use crate::api::RouteHandler;

And from a child module, use super::Router;.

A Java import aliases a fully qualified name. That name belongs to a global package namespace and says nothing about where the importing class sits relative to the imported one. There is no Java equivalent of super::, because Java packages have no parent-child relationship.

A Rust use navigates the module tree:

• crate:: starts at the crate root
• self:: starts at the current module
• super:: starts at the parent module
• an external crate is named directly

The module tree is therefore more than an encapsulation mechanism. It is the coordinate system for names throughout the crate.

Re-exports and transitive readability

Rust and JPMS both allow one component’s API to depend on another, but the mechanisms are very different.

JPMS propagates readability

module in.systemhalted.gateway.api {
    requires transitive in.systemhalted.gateway.types;
}

Any module that requires gateway.api also reads gateway.types, without declaring gateway.types itself. The dependency remains a separate module: its classes keep their original package and module identity, and consumers can access only the packages that gateway.types exports. requires transitive changes the readability graph.

Rust republishes names

In src/lib.rs, pub use gateway_types::RouteConfig; gives consumers a public path through the current crate, gateway::RouteConfig. Rust can also re-export an item from a private internal module with pub use crate::internal::router::Router;: the internal module remains private, but Router becomes available at gateway::Router.

This works because Router is public and reachable from lib.rs. A re-export cannot override privacy; it can only republish an item the re-exporting module can already access.

This gives Rust an important form of indirection: the implementation can live at crate::internal::router::Router while the public API stays gateway::Router, and the internal module tree can change without changing the path exposed to consumers.

Java cannot do this directly, because a class’s package is part of its identity.

Scaling up: modules and workspaces

Real projects rarely remain a single component. Suppose the gateway is split into two libraries, gateway-api and gateway-core.

Maven multi-module build

gateway/
├── pom.xml
├── gateway-api/
│   ├── pom.xml
│   └── src/main/java/
│       ├── module-info.java
│       └── in/systemhalted/gateway/api/
└── gateway-core/
    ├── pom.xml
    └── src/main/java/
        ├── module-info.java
        └── in/systemhalted/gateway/core/

The root POM aggregates the modules:

<packaging>pom</packaging>
<modules>
    <module>gateway-api</module>
    <module>gateway-core</module>
</modules>

gateway-core declares the Maven dependency in its POM and the JPMS dependency in its module descriptor.

Cargo workspace

gateway/
├── Cargo.toml
└── crates/
    ├── gateway-api/
    │   ├── Cargo.toml
    │   └── src/
    │       └── lib.rs
    └── gateway-core/
        ├── Cargo.toml
        └── src/
            └── lib.rs

The root manifest defines the workspace:

[workspace]
members = ["crates/*"]

gateway-core declares a path dependency:

[dependencies]
gateway-api = { path = "../gateway-api" }

The crates/ directory is only convention; the members declaration defines the workspace. Each member remains a normal Cargo package with its own manifest, source tree, and crate root. A workspace adds organization above packages without weakening the boundaries between their crates.

Dependency management

The Maven and Cargo mappings are close but not identical. A Cargo workspace shares one resolved dependency graph through Cargo.lock. Centralized dependency declarations belong in [workspace.dependencies], and member packages inherit them with serde = { workspace = true } — closer to Maven’s dependencyManagement than the lock file itself.

At the build level, both construct a dependency graph and build components in order: cargo build from the workspace root resembles a Maven reactor build like mvn package from the parent project.

The architectural consequence is the same: a crate boundary inside a workspace is still a real boundary. gateway-core sees only what gateway-api exposes publicly, just as one JPMS module sees only the packages another exports. Splitting a project into crates or JPMS modules is therefore not directory organization; it changes the enforceable architecture.

Split packages and legacy constraints

JPMS arrived after two decades of classpath-based Java. That history created migration mechanisms and compatibility layers:

• automatic modules
• the unnamed module
• classpath fallback
• command-line overrides
• split-package failures

A split package occurs when two or more named modules define types in the same Java package. JPMS rejects that arrangement: a given package must belong to a single module. Rust rules out the analogous structure by construction — a Rust module belongs to one crate’s module tree and is introduced by declarations within that crate. No module spans two crates. There is no pre-crate Rust ecosystem to accommodate.

The escape hatch survives, too. Drop a JAR onto the legacy --class-path instead of the module path and its packages fall back into the unnamed module, outside JPMS entirely — strong encapsulation holds only as long as everyone stays on the module path. Rust has no such fallback: nothing sits outside the crate system, so its boundaries always hold.

This is not evidence that Rust’s designers were smarter; they had a different starting point. But the consequence matters: crate boundaries apply uniformly across the ecosystem, and tools can assume the model is real.

The runtime model

JPMS also governs deep runtime reflection. Java frameworks often inspect constructors, fields, and methods reflectively; whether that access is permitted depends on ordinary access checks and whether the containing package is open. A module can open one package to a specific framework:

module in.systemhalted.gateway {
    opens in.systemhalted.gateway.config
        to com.fasterxml.jackson.databind;
}

The command line can also modify the boundary with --add-opens.

Rust has no general runtime reflection facility comparable to Java reflection. Its major metaprogramming mechanisms — procedural macros and derive macros — operate during compilation, expanding into ordinary Rust code that remains subject to the language’s visibility rules. Rust therefore needs no equivalent of JPMS opens; the problem does not arise, because the runtime model is different.

The mapping

What each language should envy

Return to the three questions. Rust answers all of them inside the language and its build tool, the same way for every crate. Java answers them across pieces assembled over twenty years. That difference is the source of what each side can envy.

Java developers should envy Rust’s uniformity. Every Rust crate, from the standard library to a small third-party library, participates in the same crate and module model. There is no parallel classpath world, no automatic-module transition state, no split-package compatibility problem, and no reflective framework asking for runtime access to private implementation details.

Rust developers should envy the explicit JPMS boundary document. A module-info.java file states, in one compact artifact:

• which modules are readable
• which packages are exported
• which packages are opened
• which services are used or provided

Cargo can expose the resolved dependency graph through cargo metadata, and Rust tooling can resolve public re-exports, but a crate’s external namespace is assembled through source declarations rather than summarized in one descriptor. For architectural governance at module and package granularity, JPMS provides a cleaner artifact.

Neither system describes the complete public API: an exported Java package still has to be inspected for its public classes and methods, and a Rust crate still has to be analyzed for its public items and re-exports. But JPMS makes the high-level boundary unusually explicit.

Rust could make the crate the language’s central architectural unit, and Cargo built its package and dependency model around it: one place answers compilation and encapsulation, one place answers distribution. JPMS had no such freedom. It had to coexist with packages, JARs, the classpath, reflection-heavy frameworks, and two decades of existing code; its answers to the three questions were already spread apart before it arrived.

That historical constraint explains most of the irregularity in JPMS. It also explains most of the uniformity in Cargo.

The turn-based loop comes first because its artifact is the foundation. Working turn by turn with an agent forces a team to answer a basic question: how do we know a change is good? Writing the answer down as a runnable check produces the skill file. Nothing about this requires autonomy. It only requires noticing what a person does before accepting a result, and recording it.

Second is the goal-based loop, which consumes that check. An exit criterion is only useful if there is a way to evaluate it, and the evaluation is the check the turn-based stage wrote down. Defining the criterion is work of its own. A team has to say what done means in language precise enough for an evaluator to apply, and that precision usually does not exist yet. Producing it is the real output of this stage.

After the criterion is in place, a schedule or an event trigger follows, and this is the rung where teams most often stall. A schedule is the easiest of the four artifacts to create, so it tends to appear early and stay. A prompt that runs every morning feels like automation, and nobody revisits it. The better path is to treat the schedule as provisional from the day it is created, and to replace it with an event trigger when the source system offers one. A calendar entry is a guess about when the world changes. An event is a report that it did.

A proactive routine can only come last, because it consumes everything: the check, the criterion, the trigger, and the operational guardrails around all three. At this stage the question stops being technical. A routine that touches production systems needs an owner, a budget, and a way to be turned off, and those are organisational decisions. A platform team can supply the machinery. Only the owning team can supply the ownership.

Skipping a rung defers work rather than removing it. A proactive routine adopted before the exit criterion exists will need that criterion defined during an incident instead of during design. A goal-based loop adopted before the check is written down will evaluate against an evaluator’s guess. The artifacts get produced either way. The order decides whether they are produced deliberately or under pressure.

So the most capable loop is not the right starting point. The right starting point is the artifact underneath the rung a team is already on, finished and written down before the next step up.

In a turn-based loop, the check lives in your judgement. The agent gathers context, does the work, checks itself, and hands back a result. You decide whether to accept it, redirect, or discard. Your judgement does not scale past a certain volume of work. What helps is writing the check down as something the agent can run before it comes back to you. Start the dev server, exercise the changed control, take screenshots before and after, run the affected tests. A skill file that encodes these steps moves the check from one person’s head to a file the whole team can read and improve. Every loop above this one assumes that file exists.

A goal-based loop moves the check into the exit criterion. You state what done looks like, and an evaluator model tests the condition each time the agent tries to stop. A measurable criterion, such as a Lighthouse score or a latency threshold on a specific endpoint, stops the loop on something you can defend after the fact. A vague criterion stops the loop early, because the agent convinces itself it is done, or lets it run long, because there is no way to be sure. Most teams do not have exit criteria for their own work written in language a person could act on, let alone an agent. The first attempt at a goal-based loop usually exposes that gap.

With a time-based loop, the trigger carries the check. The same prompt runs every N minutes against the current state of the world: summarising a channel, sweeping a queue, checking a pull request for review comments. What usually breaks here is the interval, not the prompt. A short interval against a slow-moving source wastes tokens, and a long interval against a fast-moving source misses changes that need a response. When the source system can emit an event, such as a webhook or a queue depth alert, an event trigger ties each run to a change that actually happened. A time-based loop works better as a stage on the way to an event-driven one than as a destination.

A proactive loop spreads the check across the whole system around the agent. A routine watches a stream of bug reports, alerts, or tickets, and runs a goal-based loop for each item until someone turns it off. This is a queue consumer whose judgement comes from a model, so it needs the same properties as any production consumer: idempotency, rate limits, spend caps, observability, a shutoff, and an owner. The Anthropic post covers token budgeting and routing routine subtasks to smaller models, which matter for cost. The harder question is ownership. A routine without a clear owner will eventually do the wrong thing with nobody responsible for noticing.

Three of these places build on each other directly. The check written down at the turn-based level is what the goal-based loop evaluates, and the criterion defined at the goal-based level is what the proactive routine applies to each item on the stream. The trigger sits beside this chain rather than inside it: a schedule or an event source decides when the runs happen, not whether their results can be trusted. A proactive routine built on unverified goals does not automate anything useful. It repeats whatever was already broken, at a cost that grows with how often it runs.

So the loops are the visible layer, and the durable artifacts sit underneath them: the skill file, the exit criterion, the trigger, and the observability around the routine. A team can locate its missing artifact by looking at what it still does by hand. If UI changes are verified manually at the end of every session, the skill file has not been written. Case-by-case judgement on whether a fix is done usually means there is no measurable exit criterion. A script that someone remembers to start every morning is a schedule that has not been written down, and after that an event trigger. Hand triage of a steady stream of reports is a routine that has not been built yet. These artifacts outlive the specific prompt, the specific model, and the specific team. Writing them down is the work the loops post points to.

That post was about the boundary between what a type system can prove and what only a test can. There is a separate limitation worth looking at, one that has little to do with that particular bug. An example test only runs the input I give it. My save test proves something about the string abcdef written to a temporary file, and nothing about the other inputs the function will see. I chose the input, so the input agrees with me. Writing more example tests does not remove this, because I choose those inputs too.

From examples to a property

Take insert and delete on the document model. The usual way to test them is with an example.

#[test]
fn insert_then_delete_restores_buffer() {
    let mut doc = Document::from_str("hello");
    doc.insert(5, " world");        // "hello world"
    doc.delete(5, 11);              // back to "hello"
    assert_eq!(doc.text(), "hello");
}

This test passes, and it is not a bad test. But look at what it actually asserts: that inserting " world" at index 5 of "hello", and then deleting that range, gives back "hello". That is one point in a large space of possible inputs. The rule I care about is not about "hello" at all. It is this: for any document, inserting a string and then deleting exactly that range returns the original document. I wrote a test about one case of the rule because a single case is the only thing an example test lets me state.

Property-based testing lets me state the rule and let the framework choose the inputs. Instead of picking the input, I describe the range of valid inputs, state what must be true, and the framework generates many cases and tries to find one that fails. In Rust the common tool for this is the proptest crate¹.

use proptest::prelude::*;

proptest! {
    #[test]
    fn insert_then_delete_is_identity(base in ".*", ins in ".*", at in 0usize..=usize::MAX) {
        let mut doc = Document::from_str(&base);
        let at = at % (doc.len() + 1);          // an offset into the buffer
        let before = doc.text().to_string();

        doc.insert(at, &ins);
        doc.delete(at, at + ins.len());

        prop_assert_eq!(doc.text(), before);
    }
}

proptest runs this body a few hundred times with different generated values, and the test passes only if the property holds for all of them. I am no longer asserting a fact about "hello". I am asserting a fact about insert and delete.

Reading the generator line

Most of the work in that test is in the signature, so it is worth going through it slowly.

Each name in strategy clause defines one generated argument. A strategy is the proptest term for something that knows two things: how to produce a random value, and how to shrink that value toward something smaller when a test fails. On each run, proptest draws a value from each strategy and calls the body with them. The in reads a little like the in of a for loop, but it is not walking a fixed list. It is drawing from a described set of inputs.

base in ".*" and ins in ".*" generate strings. The literal ".*" is read as a regular expression, and the strategy produces strings that match it. In that expression . means any character and * means zero or more of them, so ".*" matches almost any string: the empty string, "hello", a single emoji, a two-byte character such as "é", whitespace, or control characters. This wide range is deliberate. ".*" will produce characters that take more than one byte, and those are the inputs I do not think to type by hand, which turns out to matter here. If I wanted a narrower set of inputs I would use a narrower expression, such as "[a-z]{1,10}" for one to ten lowercase letters, or "[0-9]+" for a run of digits. Here I want the widest set I can get.

at in 0usize..=usize::MAX generates an integer. The strategy here is an ordinary Rust range. proptest provides a Strategy implementation for the standard range types, so a range can be handed to it directly as a generator that produces values inside the range. Reading the bounds from left to right: 0usize is the low end, zero, written as a usize, which is Rust’s pointer-sized unsigned integer and the type used for indexing. The ..= operator makes the range inclusive, so the upper value is part of the range. usize::MAX is the largest value a usize can hold. The usize suffix on 0usize is not cosmetic. Both ends of a range must have the same type, and without it the compiler cannot tell whether I meant a u32, an i64, or something else. So at is any index from zero to the maximum.

This raises a fair question. The generated integer can be as large as usize::MAX, about 1.8 billion billion on a 64-bit machine, while a short document has only a handful of positions. A five-byte string like "hello" has just six offsets, zero through five. So almost every raw value the generator produces lands far past the end of the buffer rather than inside it. Why draw from the whole usize range at all? The reason is that the three strategies are evaluated independently. The generator for at does not know how long the base string it is paired with will be, so it cannot produce an index that is valid for that particular document. The first line of the body deals with this.

let at = at % (doc.len() + 1);   // an offset into the buffer

The modulo maps any generated integer into the range 0..=doc.len(), where doc.len() is the length of the document in bytes. The range is inclusive at the top because you can insert at the end of the buffer as well as inside it. Generating a wide value and then mapping it into range is a common pattern with proptest, because it keeps every generated value usable instead of throwing away the ones that fall out of range. There is a more precise tool for producing an index into a generated collection, the prop_flat_map combinator, which lets one strategy depend on the value another produced. For a single index the modulo is simpler.

So the signature reads as follows: for any strings base and ins, and any index at mapped into range, the body must hold. proptest runs the body many times with different values, and the test passes only if the property holds for all of them.

When a property fails

Run this property as written and it does not reach the assertion. It panics, and not on "hello". It panics on an input I would not have typed into a test by hand.

A raw generated input is not much use as a bug report. If the framework told me the test failed on a four thousand character random string, that would be a puzzle to work through, not a defect I can read. What makes property-based testing practical is shrinking. When proptest finds a failing input, it does not report that input directly. It looks for the smallest and simplest input that still fails, using shorter strings, smaller indices, and values closer to zero, and reports that instead. The failure shrinks down to something like this.

minimal failing input:
    base = "é"
    ins  = ""
    at   = 1

panic: byte index 1 is not a char boundary

That is a report I can act on, and the surprise is where the panic comes from. It is not a bug in insert or delete. It is my own line, at % (doc.len() + 1). In Rust a string index is a byte offset, and len() returns a length in bytes, so the modulo gives me some offset between zero and the byte length. But not every byte offset is a place you are allowed to edit. An edit must land on a UTF-8 character boundary, and insert panics when it does not. The character é is two bytes, so the only valid positions are 0 and 2; the offset 1 falls inside the character. The generator produced the smallest string and index that break an assumption I had written into the test without noticing it: that any offset from zero to the length is a valid place to insert.

The fix is to generate only offsets that fall on character boundaries. Rust gives me the boundaries through char_indices, and I add the end of the string as the last position.

use proptest::prelude::*;

fn char_boundary_offsets(s: &str) -> Vec<usize> {
    s.char_indices()
        .map(|(i, _)| i)
        .chain(std::iter::once(s.len()))
        .collect()
}

proptest! {
    #[test]
    fn insert_then_delete_is_identity(base in ".*", ins in ".*", raw_at in 0usize..=usize::MAX) {
        let offsets = char_boundary_offsets(&base);
        let at = offsets[raw_at % offsets.len()];   // a valid edit position

        let mut doc = Document::from_str(&base);
        let before = doc.text().to_string();

        doc.insert(at, &ins);
        doc.delete(at, at + ins.len());

        prop_assert_eq!(doc.text(), before);
    }
}

There is a more general point here, beyond the Unicode detail. A property test does not only run more inputs than an example. It makes me say what a valid input is. Writing the example test, I never had to define the set of legal insertion points, because I only used position 5 of "hello", which happens to be legal. The property could not be written until I stated, in code, that a legal position is a character boundary. So the generator did not simply find a case I had forgotten. It exposed an assumption I had never written down, which is what happens when a rule has to hold for every input rather than for the one I picked.

The type system does not close this gap. String guarantees the buffer is valid UTF-8, but an index into it is a plain usize, and the type says nothing about whether that usize lands on a boundary. That check happens at runtime, and until I wrote the property it was happening nowhere in my tests.

Where property-based testing helps, and where it does not

It would be easy to read this and decide that property tests should replace example tests. They should not, any more than tests replace types. Each answers a different question, and property testing has a cost. A good property is harder to find than a good example, and a weak property is worse than no test at all.

The cost is in finding the rule. “For input "hello", expect "hello world"” takes a second to write. “For all documents and all edits, this relationship holds” takes real thought, and if the rule is slightly wrong the test is either flaky or empty. The common mistake is a vacuous property, one that cannot fail.

prop_assert!(doc.len() >= 0);       // len() is usize; always true

That runs a few hundred times, passes every time, and checks nothing. A property that is merely true is not the same as a property that constrains the code. The useful properties describe a relationship tight enough that a wrong implementation cannot satisfy it: round-trips such as decode(encode(x)) == x, idempotence such as f(f(x)) == f(x), order-independence where applying two edits in either order gives the same result, and conservation of some quantity such as length or character count. If I cannot name a relationship like that, an example test is the honest choice, and a property adds nothing.

Here is the rule I settled on. A property test earns its place when there is a rule that must hold across many inputs, not a single value I am checking. save then load should return the same buffer, and that is a rule. “The About dialog shows version 2.1” is a single value, and writing it as a property gains nothing.

Types, examples and properties

I now think of three tools rather than two, each covering what the previous one cannot.

Types rule out illegal shapes before the program runs. A Result that must be used, an Option that cannot be read without handling the empty case, a closed enum that forces every branch to be handled. The compiler settles these for every input at once, and this is the cheapest of the three.

Example tests check specific behaviour that the types allow but do not pin down. That inserting " world" at index 5 of "hello" gives "hello world" and not "worldhello" is a fact about values, and the type checker has no opinion on it. An example fixes one such fact.

Property tests cover the space in between, the behaviour that must hold across all the inputs the type still allows and that no single example can cover. The UTF-8 boundary case sits there. It is legal according to the type, missed by the examples, and wrong in general. The useful thing is to be clear about how behaviour will be evaluated before writing the code, which is the test-first instinct. A property is one clear way to state that evaluation, because it forces me to say what must always be true rather than what happened to be true the one time I ran the code.

The three fit together in order. Push what you can into types, so the compiler proves it for every input. Use examples to record the specific behaviour you have decided on. Use properties for the rules that must hold everywhere and that you would never cover by hand.

Summary

An example test checks the inputs I thought of. A property-based test states a rule and lets the framework generate inputs that try to break it. That is the difference that matters. On my own I tend to write the tests my code already passes, because I imagine the same cases when I write the code and when I test it. The generator does not share that imagination, and shrinking makes what it finds small enough to read and fix. Writing the property also forces me to say what a valid input is, which is often where the real gap turns out to be. The reason to reach for property testing is not that the code is otherwise untestable. It is that it exercises the inputs I would not have chosen, and in a text editor those are often the inputs a real user will produce.

References and Notes

Looking for alternatives, I came across a Git feature that I had heard about before but never used seriously: worktree. At first glance, worktree looks like a convenience feature. It allows multiple working directories to be attached to the same repository so that different branches can be checked out simultaneously. That is useful on its own, but what caught my attention was the implication behind it.

Most Git users, myself included, tend to think of a repository as a working directory that happens to contain a .git directory. Git does not view things this way. From Git’s perspective, the repository is the object database, and the working directory is one view into that repository. Worktree exists because Git was never fundamentally limited to a single working directory. Most of us simply use it that way.

The Mental Model Most of Us Carry

Consider a typical repository:

my-project/
├── src/
├── docs/
├── pom.xml
└── .git/

Most developers naturally view this as a single unit: a repository, a working directory, and a currently checked-out branch. That mental model is not wrong, but it is incomplete. The branch is not stored in the working directory. The commit history is not stored in the working directory. The object database is not stored in the working directory. Those things live inside .git, and the files visible in the working directory are a projection of a particular commit from the repository.

Once viewed from that perspective, an interesting question emerges: why should a repository be limited to a single projection? Git’s answer is that it isn’t.

Creating a Worktree

Suppose the current repository is checked out on a feature branch:

git switch feature/customer-notifications

A second working directory can be created using:

git worktree add ../customer-notifications-main main

Git creates a new directory alongside the existing one:

projects/
├── customer-notifications/
└── customer-notifications-main/

The first directory remains on the feature branch, and the second is checked out on main. Both can be opened in separate terminals, editors, or IDE windows, and changes in one working tree do not affect the other. From a developer’s perspective, it feels almost like having two clones, except that Git is not creating another repository.

Looking at What Git Actually Creates

After creating a worktree, the repository structure changes slightly. Inside .git, a new directory appears:

.git/
├── objects/
├── refs/
├── logs/
└── worktrees/

Listing the contents of .git/worktrees/ reveals a subdirectory for each additional worktree, containing files such as HEAD, gitdir, commondir, and index. The exact contents may vary across Git versions, but the important observation is that Git is maintaining metadata for an additional working tree while continuing to share the underlying repository. This is why creating a worktree is extremely fast. Git is not duplicating the commit graph, objects, tags, or references. It is creating another working directory and associating it with the existing repository.

Shared and Separate State

The easiest way to understand worktrees is to identify what is shared and what is independent. The object database, branch references, tags, hooks, and configuration are all shared across worktrees. Each worktree has its own working directory contents, its own index, and its own HEAD pointing somewhere into the shared refs/heads/. This separation is what allows two branches to be active simultaneously without two repositories, and it has a few practical consequences worth noting.

A git fetch in any worktree updates the shared refs, so the new commits are immediately visible from every worktree attached to the repository. Hooks installed in .git/hooks/ run regardless of which worktree triggered them, which catches some people out when they expect per-worktree hook behavior. And because branch refs are shared, a branch checked out in one worktree cannot be checked out in another, which is the next thing worth looking at.

Why Git Refuses Certain Operations

Suppose feature/customer-notifications is already checked out in one worktree. Attempting to check it out in another produces an error:

$ git worktree add ../another-copy feature/customer-notifications
fatal: 'feature/customer-notifications' is already checked out at '/path/to/customer-notifications'

This can initially seem restrictive, but the restriction makes sense given the shared-ref design. A branch is a movable reference, and if two working directories could manipulate the same branch independently, Git would have no way to determine the intended state of that reference. The restriction prevents ambiguity and protects branch state from concurrent updates.

Worktree Versus Stash

Worktree and stash solve different problems, even though both come up when someone needs to switch context briefly. A stash temporarily preserves uncommitted changes so another task can be performed in the same working directory, while a worktree creates an entirely separate workspace. If I need to quickly pull main, perform a rebase, and continue working, a stash is reasonable. If I expect to spend hours investigating another branch while keeping the current work untouched, a worktree is the better choice. The distinction is subtle but important: a stash preserves state, while a worktree preserves context.

Worktree Versus Another Clone

Historically, some developers solved this problem by maintaining multiple clones of the same repository. That approach works, but every clone maintains its own object database, references, and repository metadata, which means fetches must be performed in each clone separately and disk usage grows linearly with the number of copies. Worktrees share those resources. For smaller repositories the difference is negligible, but for larger repositories containing years of history it becomes noticeable. More importantly, worktrees communicate intent. They represent multiple views into the same repository rather than multiple copies of the repository itself.

The Bare Repository Approach

One workflow worth mentioning treats worktrees as the primary interface rather than an occasional convenience. The repository is cloned bare, and every branch the developer wants to work on becomes its own worktree:

git clone --bare git@github.com:example/project.git project.git
cd project.git
git worktree add ../project-main main
git worktree add ../project-feature feature/customer-notifications

There is no “main” working directory in this model. Every working directory is a worktree, and the bare repository at the root contains only the shared state. Some developers organize their worktrees inside a .worktrees/ subdirectory of the project to keep things tidy. Whether this layout is worth adopting depends on how often parallel branches are needed, but it makes Git’s underlying model unusually visible.

Managing Worktrees

Git provides a few commands for managing worktrees. git worktree list shows all active worktrees and the branches they point to:

/home/palak/projects/my-project          a1b2c3d [feature/customer-notifications]
/home/palak/projects/my-project-main     e4f5g6h [main]

git worktree remove <path> removes a worktree. If a worktree directory is removed manually outside of Git, the stale metadata can be cleaned up with git worktree prune. Most day-to-day usage rarely requires anything beyond these commands.

Final Thoughts

Before learning about worktrees, I implicitly assumed that a Git repository and a working directory were inseparable concepts. Git itself makes no such assumption. A repository is an object database containing commits, trees, blobs, references, and metadata, while a working directory is one representation of repository state at a particular point in time. Viewed from that perspective, worktrees feel less like an advanced feature and more like a natural consequence of Git’s architecture.

The feature is useful for avoiding branch switching, preserving context, and keeping parallel streams of work isolated. But the more interesting lesson is the one it reveals about Git itself. Many of the limitations we assume are built into our tools are actually limitations of the mental models we carry around. Git worktree is a small reminder that those two things are not always the same.

These are some of the notes.

A green test is not a correct program

The function performed following tasjs - write the buffer to a path, then mark the document as having no unsaved changes.

pub fn save(&mut self, path: &Path) -> io::Result<()> {
    std::fs::write(path, self.text());   // the Result here is silently dropped
    self.modified = false;
    Ok(())
}

and I had also written the following test:

#[test]
fn save_writes_contents_and_clears_modified() {
    let path = std::env::temp_dir().join("save_test.txt");
    let mut doc = Document::from_str("abc");
    doc.insert(3, "def"); // buffer is now "abcdef", with unsaved changes

    doc.save(&path).unwrap();

    assert_eq!(std::fs::read_to_string(&path).unwrap(), "abcdef");
    assert!(!doc.is_modified()); // a successful save clears the dirty flag

    let _ = std::fs::remove_file(&path);
}

It built a document, made a change, saved it to a temporary file, read the file back, checked that the contents matched, and confirmed the document no longer reported unsaved changes. The test passed, which would normally be the end of it, except that the implementation had a bug on the first line.

std::fs::write returns a Result, because writing to disk can fail in any of the ordinary ways: the disk fills up, the directory does not exist, the process lacks permission. The function ignored that return value, so a failed write would still fall through to self.modified = false and then Ok(()). The editor would report the file as saved, clear the unsaved-changes marker, and lose the user’s work, which is the one bug a text editor cannot ship. The test stayed green the whole time because it only ran the happy path, and a passing happy-path test says nothing about the behaviour on the failure paths it never visits.

The compiler reviews shape, not intent

I missed the bug, but the compiler flagged it:

warning: unused `Result` that must be used

Rust marks Result as #[must_use]¹, so the lint fires when a fallible call’s return is dropped on the floor. Most languages would stay silent here; Rust does not, and it was right to complain.

The warning is real help, but it is still only a warning. I could have written this instead and quieted it:

let _ = std::fs::write(path, self.text());   // explicitly discard it
self.modified = false;
Ok(())

That compiles cleanly. The type checker is satisfied: the signature returns a Result, a Result is returned, the borrow rules² hold, the shape of the program is right. What the compiler has no way of checking is what I meant by it – that a failed write must not clear the modified flag, that it must not be reported as success, that in a text editor a dirty buffer is not a boolean but a promise to the user.

This is what the compiler is good at and where it stops. It checks that the types line up, that fallibility is visible in signatures, that a Result is not accidentally ignored, that the borrows are valid; it does not check that the program does the right thing with any of those things once it has them.

A regression test only counts once you have seen it fail

The fix is one character – the ? operator – which turns the dropped result into a propagated one:

std::fs::write(path, self.text())?;
self.modified = false;
Ok(())

If the write fails, the function exits before clearing the modified flag. The interesting part is not the fix but the realisation that I had no test that would have caught the bug. Adding ? without adding a test would leave me trusting the code for no reason at all, so I wrote the test that should have existed from the start.

#[test]
fn failed_save_surfaces_error_and_keeps_modified() {
    let mut doc = Document::from_str("important data");
    doc.insert(0, "!"); // the document now has unsaved changes

    // a directory that does not exist, so the OS write is forced to fail
    let bad_path = std::env::temp_dir().join("no_such_dir").join("file.txt");
    let result = doc.save(&bad_path);

    assert!(result.is_err());     // the failure must reach the caller
    assert!(doc.is_modified());   // and the buffer must still be dirty
}

Then I put the bug back, ran the suite, watched this test go red, restored the fix, and watched it go green. Putting the bug back to confirm the test fails for the right reason sounds like ceremony but is not – a regression test you have never seen fail is one you trust by faith, and the first section already showed how little a passing test is worth on its own.

Types and tests answer different questions

This is the rule I would keep if I could keep only one. A type system is good at making certain kinds of lies impossible: a value that may be absent cannot pretend to be present, a fallible operation cannot pretend to be infallible, a closed set of cases cannot pretend one branch does not exist. That is powerful, but it is structural. Rust can make the failure visible, warn me when I accidentally ignore it, and force the code to admit that saving may fail, but it cannot encode the editor’s rule that if saving fails, the document must remain dirty. That rule lives at the level of behaviour, not shape.

You can push more behaviour into types than that suggests. The typestate pattern³ encodes a rule into a type, so that an illegal operation does not compile instead of failing a test. The save bug can be narrowed this way by making the clearing of the modified flag depend on a value that only a successful write can produce:

struct Saved;   // nothing outside this module can construct one

fn write(&self, path: &Path) -> io::Result<Saved> { /* ... */ }
fn mark_clean(&mut self, _proof: Saved) { self.modified = false; }

Now mark_clean cannot be called without a Saved, and the only thing that hands one back is a write that returned Ok, so the exact bug from the first section is harder to write by accident.⁴ The boundary between what a type can prove and what only a test can prove is not fixed; it moves with how much you choose to encode.

What stays out of reach, at least in Rust, is value-level correctness – the relationship between specific input and output values. That needs dependent types, which Rust does not have. The type checker has no opinion on whether inserting " world" at index 5 of "hello" gives "hello world" or "worldhello"; both are valid Strings, and which one you meant is a question only a test can answer. The instruction that falls out of this is short: do not test what the type system has already made impossible, and do test the behaviour the type system is silent about. A small example came up in the next step. A document needs to remember the file it came from, but a new, untitled buffer has not come from anywhere, so the honest type for that field is Option<PathBuf>, and Rust will not let you read the path without handling the case where there is none. The “I forgot to check for the missing file” bug is not caught by a test here; the program that contains it does not compile, and writing a test for it would only re-prove what the compiler has already proven.

“Test everything that can fail” is too blunt

I used to carry a simple rule: every operation that can fail gets a failure test. It made sense in the Java world I grew up in, and after the save bug I wrote one for open too – a test asserting that opening a missing file returns an error. Then I deleted it.

The save failure test is worth keeping, but not because saving can fail. It is worth keeping because save changes state on the way to succeeding: it clears the modified flag. The assertion in that test that actually matters is not assert!(result.is_err()) but assert!(doc.is_modified()), which guards the state that a failed save could otherwise leave wrong. open is different; it builds a value and returns it, changing nothing and leaving nothing wrong when it fails. The only thing a failure test there could check is whether I propagated the error instead of swallowing it, which is thin, with no state behind it to protect.

So the rule sharpened into something narrower: a failure test earns its place in proportion to the state a failure could leave wrong, and where failure only propagates cleanly there is little for it to guard. That is not an argument for testing less out of laziness, but for putting tests where things can actually break instead of spreading them evenly out of habit.

The point

This is not an argument against tests or against types, but for knowing which of the two you are leaning on at any given moment. Lean only on tests and you will keep writing assertions to re-establish guarantees a good type system would give you for free, at compile time, for every input. Lean only on types and you will ship code that is well-formed and quietly wrong: green in every structural sense, still losing files. The healthier division is to let the type system carry what it can – null-safety, exhaustiveness, fallibility made visible, and as much of your state machine as you are willing to encode – and to spend your tests on the behaviour that is left over.

That line moves. Typestate carries more than people expect, and dependent types would carry more again, but wherever you draw it for a given program the rule is the same: do not ask a type to prove what only a test can, or a test to re-prove what the type already guarantees. How far the line can move also depends on the language. Rust leans on two guarantees that Java’s type system holds you to less strictly. The first is that absence has a single shape: Rust has no null, so a value that might not be there is an Option and the compiler will not let me touch it without handling the empty case, whereas Java’s Optional⁵ sits beside null rather than replacing it and leaves .get() available to learn my mistake at runtime. The second is move semantics: Java can make a proof-of-save exist through a private constructor and a factory, but it cannot make it spent, so nothing would stop me clearing the flag with the same token twice; Rust takes the value away when it is used, and that is the part Java has no equivalent for. In both cases the work does not vanish – it moves from the compiler to my tests and my discipline. The reason to reach for Rust here is not that the editor is otherwise impossible, which it plainly is not, but that a more expressive type system settles more of the program’s correctness before a single test runs.

References and Notes

This behavior causes some interesting bugs and awkward APIs. We will explore some of them in this post.

1. List<String>.class does not exist

We are all familiar with List.class, String.class, Integer.class but List<String>.class is an illegal construct in Java. At runtime, Java does not have a separate class object for List<String> versus List<Integer>.

That is why APIs that often accept this Class<T> type break down for generic types.

For example, this API looks clean:

<T> T read(String json, Class<T> type)

It works well for a normal object:

User user = read(json, User.class);

However, the following will be impossible:

List<User> users = read(json, List<User>.class);

There is no List<User>.class to pass. You may be tempted to make it work by trying:

List<User> users = read(json, List.class);

But now the runtime only knows that the target type is List. It does not know that the list is supposed to contain User objects.

This is where bugs begin. A JSON library, for example, may deserialize the JSON array into a List<LinkedHashMap> instead of a List<User>. The code may compile, but fail later when you try to use one of the elements as a User.

List<User> users = read(json, List.class); 

User user = users.get(0); // may fail at runtime

The problem is not that the list may be empty. That would be a normal collection issue. The type erasure problem is that even if the list contains elements, those elements may not be of the type the source code appears to promise.

To solve this, libraries invented alternate ways to carry generic type information into runtime APIs.

Jackson uses TypeReference<T>:

List<User> users = objectMapper.readValue(json, new TypeReference<List<User>>() {});

Gson uses TypeToken<T>:

List<User> users =
    gson.fromJson(json, new TypeToken<List<User>>() {}.getType());

Spring uses ParameterizedTypeReference<T>:

ResponseEntity<List<User>> response =
    restTemplate.exchange(
        url,
        HttpMethod.GET,
        null,
        new ParameterizedTypeReference<List<User>>() {}
    );

These APIs look awkward because they are working around the same missing runtime concept: a class-like object that represents a fully parameterized generic type. If Java supported class literals for parameterized types such as List<User>.class, many of these APIs could have been simpler.

2. instanceof List<String> does not work

Another place where type erasure shows up is runtime type checking. In Java, we commonly use instanceof to check the type of an object.

if (obj instanceof String) {
...
}

But Java won’t let you write

if (obj instanceof List<String>) {
 ...
}

The reason is that List<String> is not fully available at runtime. After type erasure, the runtime can check whether the obj is a List, but it cannot directly check whether it is specifically a List<String>.

So, Java only allows this:

if (obj instanceof List<?>) {
...
}

This tells us that obj is some kind of List. It does not tell us what kind of elements the list contains and we must inspect each element individually, if we really care:

if (obj instanceof List<?> list &&
    list.stream().allMatch(String.class::isInstance)) { //this check returns true for an empty list as well
     ...
}

This works, but it is much more verbose than a normal runtime type check. It also changes the nature of the check. We are no longer asking the JVM, “Is this a List<String>?” We are asking, “Is this a List, and do all of its current elements happen to be strings?”

3. Method overloads can clash after erasure

Java lets you overload methods when their parameters are different. For example, this is fine:

void process(String name) {...}

void process(Integer id) {...}

At runtime these are still different. However, with generics, method overloading gets a little tricky.

You may think that this will work:

void process(List<String> names) {...}

void process(List<Integer> ids) {...}

But Java rejects this because after type erasure both effectively are the same method:

void process(List names) {...}
void process(List ids) {...}

Both have the same erased signature void process(List). The compiler sees the method collision and throws this error:

name clash: process(List<Integer>) and process(List<String>) have the same erasure

The workaround usually is to give methods different names processNames(List<String> names) and processIds(List<Integer> ids).

Another option is to introduce wrapper types:

record Names(List<String> names) {}
record Ids(List<Integer> ids) {}

void process(Names names) {...}

void process(Ids ids) {...}

This works because Names and Ids are real runtime types. They survive erasure, unlike List<String> and List<Integer>.

4. Generic arrays are painful

Type erasure also makes arrays and generics uncomfortable together.

In Java, arrays know their component type at runtime. For example, a String[] knows that it is an array of String. If you try to put an Integer into it, the JVM can detect the problem and throw an ArrayStoreException.

String[] names = new String[10];
Object[] values = names;

values[0] = 42; // ArrayStoreException at runtime

Generics work differently. A List<String> does not carry String as a full runtime type in the same way. After erasure, it is mostly just a List.

That mismatch is why Java does not allow this:

T[] values = new T[10]; // illegal

The runtime does not know what T really is, so it cannot create an array with the correct component type.

The workaround is to pass the component type explicitly:

static <T> T[] createArray(Class<T> type, int size) {
    return (T[]) java.lang.reflect.Array.newInstance(type, size);
}

Usage:

String[] names = createArray(String.class, 10);
Integer[] ids = createArray(Integer.class, 10);

The generic type T is known to the compiler, but not enough is available at runtime. So the API asks the caller to pass a Class<T> token manually.

This is another cost of type erasure. The type appears obvious in the source code, but runtime code still needs an explicit type descriptor to do the right thing.

Why these workarounds work

Type erasure is not total. Java still records generic type information in the class file, in a metadata section called the Signature attribute. This is kept for declarations: fields, method parameters and return types, and a class’s generic superclass and interfaces. What gets erased is the type of a value at runtime. A List<String> object and a List<Integer> object share the same List.class, so an object cannot tell you its element type. But a declaration can.

That retained information is readable through reflection. Field.getGenericType(), Method.getGenericReturnType(), and Class.getGenericSuperclass() return a java.lang.reflect.Type, which can be a ParameterizedType such as List<User> rather than a plain List.

This is the trick behind TypeReference, TypeToken, and ParameterizedTypeReference. Writing new TypeReference<List<User>>() {} creates an anonymous subclass, and its generic superclass TypeReference<List<User>> is a declaration. So List<User> is preserved in that subclass’s metadata, and the library recovers it with getClass().getGenericSuperclass():

Type type = new TypeReference<List<User>>() {}
    .getClass()
    .getGenericSuperclass(); // ParameterizedType: List<User>

Conclusion

Once your code crosses a runtime boundary, assume the generic type will not be there. Design the API to carry the type explicitly, through a Class<T> token, a TypeReference<T>, a wrapper type, or a type tag you store in the data yourself, rather than trusting that the compiler’s view of the type survives into runtime.

I wanted to do this without losing my existing shell configuration, installed tools, IDE state, and Claude Code session history.

It turns out a complete rename is very doable. The data-loss fear is mostly misplaced. The real risk is not deleted files, but stale hard-coded paths. In this post, I cover exactly how I did it.

A Little Gotcha Before We Begin

usermod refuses to touch an account that is logged in or has running processes:

usermod: user old-user is currently used by process NNNN

Since this was the only admin account on the machine, I couldn’t rename it while logged into it. The fix is a throwaway second admin account that does the surgery while the real account is fully logged out.

With that, here are the steps to change the identity of an existing user.

Step 1 — Create a temporary admin

While still logged in as the old user:

sudo adduser tempadmin
sudo usermod -aG sudo tempadmin

Step 2 — Log out completely and switch

This part matters: Log Out of GNOME — not lock, not “Switch User”. The old session must actually end. Then log in as tempadmin at the greeter and open a terminal.

Step 3 — Confirm the old account is truly idle; if needed terminate the sessions

who                                   # old user should not appear
pgrep -u old-user                     # should print nothing
sudo loginctl terminate-user old-user 2>/dev/null
pgrep -u old-user || echo "clear - safe to proceed"

Proceed to Step 4 if the message says clear - safe to proceed.

Before renaming, record the account’s current numbers so you can confirm nothing shifted afterward — and so you’re not assuming mine:

id old-user        # note the uid=, gid=, and the full groups= list

On my machine both the UID and GID were 1000 (the first account the Ubuntu installer creates). Yours may differ — on a multi-user box, a migrated system, or one with service accounts, the first human user isn’t always 1000. Whatever your values are, they should be identical before and after; keeping them unchanged is exactly why we never pass -u.

Step 4 — The actual rename

Three commands do the core work:

# Rename the login AND move/rename the home dir (last arg is the CURRENT name):
sudo usermod -l systemhalted -d /home/systemhalted -m old-user

# Rename the matching private group; the GID stays unchanged:
sudo groupmod -n systemhalted old-user

# Update the display/full name shown at the login screen:
sudo usermod -c "systemhalted" systemhalted

Why this is safe: the UID stays unchanged, 1000 on my machine — usermod -l changes the login name and -d -m moves the home directory’s contents, but the UID stays put because I’m not passing -u. With the numeric owner unchanged, usermod -m moves the home directory and tries to adapt ownership, permissions, ACLs, and extended attributes under the new path — though the manpage notes some cases may still need manual fixing. In the normal same-UID case, the intent is relocation rather than deletion, but I still treated backup as mandatory. The account’s authentication entry is preserved through the rename, so the same password keeps working.

Step 5 — Fix what usermod doesn’t touch

This is where the real work hides. usermod renames the account, but several things keep pointing at the old path or the old name.

(a) Hard-coded paths. A scan of my home turned up ~196 files under .config, .local, and .claude containing the literal string /home/old-user — almost all of it, in my case, JetBrains and VS Code state. Editing them in bulk is a trap: many are binary or fragile JSON, and a stray sed can corrupt them. The clean, reversible fix is a single compatibility symlink:

sudo ln -s /home/systemhalted /home/old-user

Now every stale /home/old-user/... reference resolves transparently. No file edits, no corruption risk. It’s not entirely free, though — a lingering symlink can confuse backups, scripts, future users, or security scans — so keep it (documented) until you’re confident nothing references the old path, then remove it.

(b) The user crontab spool isn’t auto-renamed (the same goes for at jobs, if you use them):

f=/var/spool/cron/crontabs/old-user
sudo test -f "$f" && sudo mv "$f" /var/spool/cron/crontabs/systemhalted \
  && sudo chown systemhalted:crontab /var/spool/cron/crontabs/systemhalted

(c) GNOME’s AccountsService keeps per-user login-screen prefs keyed by name:

f=/var/lib/AccountsService/users/old-user
sudo test -f "$f" && sudo mv "$f" /var/lib/AccountsService/users/systemhalted

(d) Claude Code session history. Claude Code names its project folders after the working directory. Mine were all -home-old-user-*, so prior session history was keyed to the old path:

cd /home/systemhalted/.claude/projects

for d in -home-old-user*; do
  mv "./$d" "./${d/-home-old-user/-home-systemhalted}"
done

Two gotchas bit me here. First, the directory names start with -, so every tool treats them as options — prefix paths with ./ (or use --) or you’ll get a wall of invalid option -- 'h'. Second, a -home-systemhalted folder already existed from a session run after the home move, so the bare rename would collide. I merged that one instead of moving it:

cp -a --update=none "./-home-old-user/." "./-home-systemhalted/" \
  && rm -rf "./-home-old-user"

--update=none means any file that already exists at the destination is left untouched — it’s never overwritten, regardless of timestamps.

Step 6 — Verify before trusting it

getent passwd systemhalted    # home, shell, GECOS all correct
getent group  systemhalted    # same group, GID unchanged (1000 on mine)
id systemhalted               # the important one
ls -ld /home/systemhalted     # owned systemhalted:systemhalted
ls -ld /home/old-user         # symlink -> /home/systemhalted

The line I cared most about was id, confirming the renamed account was still in both sudo and docker — i.e. admin and container access carried over intact.

Step 7 — Reboot and live in it

sudo reboot

Log in as systemhalted with the old password. Terminal opens in /home/systemhalted, the IDEs launch with their state, docker ps works, my shell tooling works, and Claude Code shows all the prior history.

Step 8 — Remove the temporary admin

Only after the renamed account is confirmed working:

sudo deluser --remove-home tempadmin

Worth knowing: if deluser ever runs without --remove-home (or partially), it leaves two orphans behind — /home/tempadmin and /var/lib/AccountsService/users/tempadmin — which you then clean up by hand:

sudo rm -rf /home/tempadmin
sudo rm /var/lib/AccountsService/users/tempadmin

Rollback, just in case

Everything is reversible from the tempadmin session until you delete it. If anything looked wrong, I had this ready:

sudo rm /home/old-user
sudo usermod -l old-user -d /home/old-user -m systemhalted
sudo groupmod -n old-user systemhalted
sudo usermod -c "Old User" old-user

Important Notes and Observations

• No data was lost in my case — a same-UID rename moves files; it doesn’t delete them by design.
• The danger is stale paths, not lost files — and one symlink neutralizes all of them.
• Never bulk-sed IDE state — it’s binary/JSON and corrupts easily.
• Secrets mostly survive, but check them. GNOME Keyring is unlocked by your login password, which didn’t change, so keyring-backed secrets keep working (maybe a one-time unlock prompt). Saved Wi-Fi depends on setup — NetworkManager profiles may be system-wide, per-user, or keyring-backed — so verify rather than assume.
• The compatibility symlink is practical, not free. It saves you from editing hundreds of files, but document it and remove it once you’re sure nothing uses the old path.

The whole thing took one reboot and zero reinstalls. The account that started life as old-user is now systemhalted as far as normal desktop and shell tooling are concerned. (The kernel itself never cared — to it, credentials are numeric UIDs and GIDs, not names; the login name was only ever a label on top of UID 1000.)

Disclaimer: This worked on my single-admin Ubuntu/GNOME machine, and I’m sharing it as a record of what I did — not as a guaranteed recipe for yours. Renaming a user touches login, ownership, and system state, so treat every command here as something to understand before you run it, not to paste blindly. Setups differ (other desktops, network/LDAP accounts, encrypted or NFS home directories, services running as the user), and any of those can change the outcome. Take a backup first, keep the tempadmin escape hatch until you’ve confirmed everything works, and proceed at your own risk. The username here (old-user) is a placeholder; the real one has been obfuscated.

The structure, if you can call it that, was two systems running side by side and neither of them working. I had read about Zettelkasten somewhere, read a book about it, watched some videos, and set up the standard folders — Inbox, Input, Output, Zettel — without ever using them. Alongside that I had an older hierarchy of Work, Personal, and Writing, where I was actually saving things, more or less at random. Whichever folder I thought of first got the note. I had been telling myself this was fine because the notes were captured, and search would do the rest. Search did not do the rest. Most of the time I did not know what to search for.

Why I finally bothered

I am an engineering manager. In a normal week I need notes from team 1:1s, architecture decisions, books I am reading, an AI/ML study plan I run at weekends, ideas for writing, astrophotography logs, and guitar stuff. None of that is unusual. What forced the cleanup was that several times in a couple of months, I needed a specific note before a meeting and could not find it in time.

The problem was not that I had too many notes. It was that I had never decided what the system was supposed to do.

Simplification

One Saturday I sat down and wrote out what I actually use notes for: work, personal life, things I am learning, writing I am working on, a daily journal, random ideas, and an archive for things I am done with. That came out as eight buckets, which I numbered so they would sort in the order I wanted:

00 Inbox
10 Work
20 Personal
30 Learning
40 Writing
50 Journal
60 Ideas
99 Archive

Each has a few subfolders. Work has Team Management, Meetings, Architecture, and Decisions. Learning has Books, Articles, Videos, Courses, and a Subjects folder where I keep my own synthesis on topics I am studying over time. Writing splits into Ideas, Drafts, Articles, Poetry, Stories, Books, and Published. The old Zettelkasten folders went into Archive. I was not using them, and pretending I would was part of why the system was a mess.

I learned later that what I had built was close to a combination of two methods I had not heard of when I started. PARA (Projects, Areas, Resources, Archives) organises by how actionable something is, not by topic. Johnny Decimal is a numbering scheme that gives every folder a permanent ID. People combine the two, and what I had was a rough version of that combination. It was nice to know but did not change anything. The structure worked because it matched how I think, not because it had a name.

The rule that fixed the overlap

The other thing I had been doing wrong, which the cleanup made obvious, was mixing notes and tasks. Things that belonged in Reminders were living in Apple Notes, and the other way around, and I was sometimes updating both for the same thing without realising it.

The rule I went with is simple. If it has a “done” state, it goes in Reminders. If it does not, it goes in Notes. Tasks, follow-ups, and things to keep an eye on go to Reminders; knowledge, drafts, and references go to Notes. It sounds obvious written down, but I had not been doing it. Once I started, the overlap was gone, and so was most of the duplication that came with it.

What I’d actually tell someone

The cleanup itself was not the hard part. The hard part was admitting that the Zettelkasten folders I had set up years earlier were aspirational, not active. I was impressed by how useful it was for Niklas Luhmann and thought it would be useful for me as well in the same way it was for him. However, these folders had been sitting there making the system look organised while making the actual mess worse.

Most of the work was being honest about what I really do with notes — capture things, look them up later, occasionally write something longer — and building the smallest structure that supported those three things. No folders that existed only because some method I had read about said they should.

If you are sitting on your own version of a thousand unfiled notes, do not go looking for a method to copy. Write down what you actually use notes for, build the folders that match, and leave room to fix it later. The other thing — and I am saying this because I ignored my own advice for years — is that half an hour once or twice a week spent clearing out the Inbox goes a long way. The cleanup itself is a one-time event. Keeping it clean is not.

What stood out to me, reading through it, was less the model architecture than the engineering discipline behind the choice of where to use which kind of model. The authors did not, in fact, reach for an LLM as the answer to every problem, which is increasingly unusual in this space. Instead, they appear to have asked, at each stage of the pipeline, what the cheapest model that does the job correctly actually is, and to have used that. Two of the numbers in the paper are particularly striking:

• An auto-encoder handles intent detection in 0.009 seconds instead of 1.6 seconds.

• A fine-tuned BERT model handles routing in 0.3 seconds instead of ~2 seconds.

The LLM, in their architecture, only enters the picture once this fast triage stage has narrowed the problem space — at which point it is being used for what large language models are actually good at, which is reasoning over a focused problem and generating narrative explanations for a human reader.

They also, sensibly, avoid the naive Text-to-SQL approach that has become almost a cliché in this area. Instead, they describe an augmented querying flow that brings business context, internal APIs, and a plan-and-execute style of decomposition to bear on the question. The system breaks a query into a sequence of steps, fetches the right data at each step, and produces explanations that a human reader can actually act on, rather than a single SQL query that is correct in the abstract but unhelpful in context. In their reported evaluation, the system holds 90th-percentile latency under roughly 13 seconds, with about 89% relevance and correctness as judged by human raters — which, for an agentic pipeline of this kind, is a meaningfully better number than I would have expected.

The most important idea in the paper, however, is, I think, the one that follows from all of this rather than being stated outright. It is the implicit observation that, in the model described, the user is no longer expected to navigate a dashboard at all. They are expected, instead, to ask questions of a system that already understands the schema, the metrics, the seasonality of the business, and the internal vocabulary that the team uses to talk about what is happening — and to receive, in response, an explanation of what happened and, where the data supports it, why it happened.

This is not, to be clear, an argument for replacing BI teams or data engineers; on the contrary, it is an argument for amplifying them. What this kind of system promises, if it works at the scale claimed, is to turn the underlying data infrastructure into something a non-specialist part of the business can actually query in plain language, and to receive a structured explanation back rather than a chart that someone still has to interpret. The future of analytics, on this view, is less likely to be a continued proliferation of dashboards and more likely to be a steady shift toward conversational understanding layered on top of structured data — with dashboards retained, in the end, mostly for the cases in which a human still wants to look at the underlying time series themselves.

If you are working on analytics platforms, on GenAI agents, or on data products of any meaningful scale, I would encourage you to read the paper end to end. It is, by some distance, the most thoughtful treatment of where to put which kind of intelligence in an agentic data system that I have come across in the last year.

References

In Part 7 of this series, we looked at summation error and how Kahan compensation can keep a long reduction from quietly losing low-order bits. This post is about a different and, in some ways, more consequential precision trap: the moment at which a value is rounded down to fewer digits in order to be displayed, stored, or reported.

Rounding looks like a small detail in the code, and it almost never is one. It is the place where many otherwise-correct calculations end up disagreeing with audits, ledgers, and reconciliations, often in ways that are difficult to diagnose because the disagreement only appears at scale. Most teams I have seen pick a rounding mode in roughly the same way that most people pick a film on a streaming service — by going with whichever option happened to be already playing — and in Java, the option that happens to be already playing is very often HALF_UP, on the implicit assumption that this is “normal” rounding.

“We’ll use HALF_UP. That’s normal rounding. Done.”

Then a pricing engine ships, or a statement generator, or a ledger, or a tax calculator, or an amortization schedule — and at some point afterwards it becomes apparent that rounding is policy, not arithmetic. Or, more precisely, it is arithmetic with consequences, and the choice of which rounding mode to apply turns out to be a product decision wearing mathematical clothing. This post is about how to make that choice deliberately, and about why HALF_UP is not the universal solvent it is so often assumed to be.

Quick Reference: Rounding Mode Cheat Sheet

Key rules:

• Avoid rounding double when policy matters: use BigDecimal from strings → The Classic Trap
• Decide when to round, not just how → Rounding Timing
• For money, use scaled integers internally → Money Pattern

The real problem: ties (the 5s)

The drama in rounding, almost without exception, is not really about choosing between 2.341 and 2.34. It is about ties — values that lie exactly halfway between two representable steps at the precision you are rounding to.

At two decimal places, the troublesome values are the ones that look like this:

• 1.005
• 2.675
• 10.125

It is worth being precise about what counts as a tie, because the term is sometimes applied loosely to values that are not, strictly speaking, halfway between anything. A value is a tie at a given scale only when the first discarded digit is exactly 5 and all subsequent discarded digits are exactly 0. So 1.005 is a tie at two decimal places, and so is 1.00500…, but 1.0051 is not a tie — it is, on inspection, closer to 1.01 than to 1.00, and any reasonable rounding mode will produce 1.01 regardless of how it handles ties.

The reason ties matter so much in practice is that, depending on which way the rounding mode resolves them, the cumulative effect across many operations can be quite different. If you consistently push ties in one direction — say, always upward — you introduce a systematic bias into every aggregate that depends on those rounded values. Sometimes that bias is exactly what the domain wants (retail prices, for instance, often round in a particular direction by convention); often it is not, occasionally it is actually prohibited by regulation or contract, and even when it is none of those things, it is the kind of pattern that tends to make reconciliation teams unhappy when the totals begin to drift.

Java’s rounding toolbox

Java’s standard library, in java.math.RoundingMode, exposes a fairly complete set of rounding policies. The interesting question, in the end, is rarely which of them is “available” — they all are — but rather which of them matches the contract the domain actually wants enforced. The available modes are:

• HALF_UP - round to nearest; if exactly halfway, round away from zero (1.5 -> 2, -1.5 -> -2)
• HALF_DOWN - round to nearest; if exactly halfway, round toward zero (1.5 -> 1, -1.5 -> -1)
• HALF_EVEN - round to nearest; if exactly halfway, round to the result whose last kept digit is even (banker’s rounding, also called round-half-to-even)
• UP - always away from zero
• DOWN - always toward zero (truncate)
• CEILING - toward positive infinity (-1.231 -> -1.23, 1.231 -> 1.24 at 2dp)
• FLOOR - toward negative infinity (-1.231 -> -1.24, 1.231 -> 1.23 at 2dp)
• UNNECESSARY - throw if rounding would be required (which I have come to think of as a “make bugs loud” mode)

It is also worth recalling that calling setScale(...) without specifying a rounding mode will, if rounding turns out to be required, throw — which is a sensible default but is occasionally surprising:

new BigDecimal("1.234").setScale(2); // throws ArithmeticException

And the most important practical rule, which I find myself repeating in code review more often than any other rule in this area:

If you care about exact decimal policy, avoid rounding a double. Round a BigDecimal created from a string (or from an exact integer scale).

The classic trap: new BigDecimal(double)

This is the trap that catches almost everyone at least once, and it is worth showing in code rather than just describing in prose:

import java.math.BigDecimal;
import java.math.RoundingMode;

public class RoundingTrap {
  public static void main(String[] args) {
    BigDecimal a = new BigDecimal(1.005);               // from double
    BigDecimal b = new BigDecimal("1.005");             // from string

    System.out.println(a.setScale(2, RoundingMode.HALF_UP)); // 1.00
    System.out.println(b.setScale(2, RoundingMode.HALF_UP)); // 1.01
  }
}

The expectation, of course, is that both values round to 1.01. What actually happens is that the first one rounds to 1.00, because 1.005 as a binary floating-point value is not, in fact, exactly 1.005 — it is a hair less than that, by exactly the small amount that binary floating point cannot quite represent the decimal fraction 0.005. When you then ask BigDecimal to round that approximate value, it dutifully rounds the value it actually has, which falls just below the tie boundary.

If you find yourself, in a finance system, looking at a result in which 1.005 appears to round to 1.00 under HALF_UP, the right interpretation is not that Java is being eccentric. The right interpretation is that you have fed an approximate binary representation into an exact decimal rounding routine, and that the routine has, accurately and unhelpfully, given you the answer that corresponds to the value it was actually given.

What about BigDecimal.valueOf()?

There is a third constructor in this space which sits between the two previous options and which readers trip over often enough that it is worth treating explicitly: BigDecimal.valueOf(double).

BigDecimal a = new BigDecimal(1.005);        // Dangerous: uses exact binary representation
BigDecimal b = BigDecimal.valueOf(1.005);    // Safer: uses Double.toString() internally
BigDecimal c = new BigDecimal("1.005");      // Safest: exact decimal from string

BigDecimal.valueOf(double) will, in most everyday cases, behave as if you had used the string constructor, because it routes the conversion through Double.toString() internally — and Double.toString is specified to return the shortest decimal representation that round-trips back to the same double. So when you start from a literal like 1.005, valueOf gives you the decimal 1.005 rather than the binary-exact value, which is what most callers actually wanted.

A reasonable rule of thumb, based on which entry point one is using:

• Use new BigDecimal("...") for literals and external decimal inputs that arrive as strings
• Use BigDecimal.valueOf(double) only when you already have a double in hand and need the best possible decimal view of it (still risky after computations, but better than new BigDecimal(double))
• Never use new BigDecimal(double) unless you explicitly want the exact binary-to-decimal conversion, which is rarely what callers expect

It is worth one further caution. If the double you are wrapping has come out of arithmetic — rather than out of a literal — then the value you are now “viewing” through valueOf may already be quite far from the decimal you think you started with. valueOf is an honest decimal view of whatever double it receives; it is not a cleansing ritual that retroactively repairs precision lost earlier in the calculation.

Why HALF_UP can be the wrong default

HALF_UP is, in fairness, intuitive. It matches what most of us were taught with a pencil and paper in school, and it has the agreeable property of making the answer easy to predict in any individual case. The trouble is that, when applied repeatedly to many values, it can introduce a systematic drift, because every tie is resolved in the same direction.

It is worth pausing for a moment on a small irony of the IEEE 754 standard here. IEEE 754’s default rounding mode for binary floating-point arithmetic is round to nearest, ties to even — which is, in effect, HALF_EVEN. Each individual double operation in Java is specified to behave as if its result had been rounded that way. The mild irony, then, is that when developers reach for BigDecimal precisely because they are now worried about decimal correctness, many of them then choose HALF_UP, thereby reintroducing the very bias that IEEE 754 had originally been designed to avoid.

Bias demo: HALF_UP vs HALF_EVEN

Suppose, in order to make this concrete, that you have a population of values which all happen to land exactly on ties at two decimal places. (This kind of thing happens more often than people tend to expect, particularly after divisions and intermediate scaling steps.) The behaviour of the two main candidate rounding modes, side by side, looks like this:

• HALF_UP: 1.005 -> 1.01, 1.015 -> 1.02, 1.025 -> 1.03 … always nudging up
• HALF_EVEN: 1.005 -> 1.00, 1.015 -> 1.02, 1.025 -> 1.02 … nudging toward whichever side has an even last digit, and so alternating which side “wins” on ties

The structural difference is that HALF_EVEN alternates which direction it pushes ties, while HALF_UP always pushes them the same way. This is the underlying reason why banking and accounting systems often prefer HALF_EVEN: across a large enough population of values, the alternation cancels out into something close to zero net bias, while HALF_UP accumulates a small but consistent upward drift that becomes visible at scale.

The bias in action

A short program makes the difference clearer than any description does:

import java.math.BigDecimal;
import java.math.RoundingMode;

public class BiasDemonstration {
  public static void main(String[] args) {
    String[] ties = {"1.005", "1.015", "1.025", "1.035", "1.045",
                     "1.055", "1.065", "1.075", "1.085", "1.095"};
    
    BigDecimal sumHalfUp = BigDecimal.ZERO;
    BigDecimal sumHalfEven = BigDecimal.ZERO;
    BigDecimal trueSum = BigDecimal.ZERO;
    
    System.out.println("Value   -> HALF_UP / HALF_EVEN");
    for (String tie : ties) {
      BigDecimal bd = new BigDecimal(tie);
      BigDecimal roundedUp = bd.setScale(2, RoundingMode.HALF_UP);
      BigDecimal roundedEven = bd.setScale(2, RoundingMode.HALF_EVEN);
      
      System.out.println(tie + " -> " + roundedUp + " / " + roundedEven);
      
      sumHalfUp = sumHalfUp.add(roundedUp);
      sumHalfEven = sumHalfEven.add(roundedEven);
      trueSum = trueSum.add(bd);
    }
    
    System.out.println();
    System.out.println("HALF_UP sum:   " + sumHalfUp);
    System.out.println("HALF_EVEN sum: " + sumHalfEven);
    System.out.println("True sum:      " + trueSum.setScale(2));
  }
}

And the resulting output:

Value   -> HALF_UP / HALF_EVEN
1.005 -> 1.01 / 1.00
1.015 -> 1.02 / 1.02
1.025 -> 1.03 / 1.02
1.035 -> 1.04 / 1.04
1.045 -> 1.05 / 1.04
1.055 -> 1.06 / 1.06
1.065 -> 1.07 / 1.06
1.075 -> 1.08 / 1.08
1.085 -> 1.09 / 1.08
1.095 -> 1.10 / 1.10

HALF_UP sum:   10.55
HALF_EVEN sum: 10.50
True sum:      10.50

Across these ten values, the difference between HALF_UP and the true sum is five cents. That is, on its own, an unimpressive amount of money. The point of the demonstration, however, is that the same bias scales linearly: across millions of transactions of this kind, the same five-cents-per-ten pattern can become a meaningful, persistent, and entirely systematic drift in favour of one party.

How HALF_UP can favor one party at scale

When this kind of pattern shows up in a system that processes millions of transactions in which ties are common, what HALF_UP does, in effect, is to consistently favour whichever side of the transaction benefits from rounding upward. If you happen to be that party, you may, over time, find that your books “win” fractions of a cent more often than they “lose”; and that is, generally, fine — until regulators, auditors, or customers do their own arithmetic and notice the same pattern from the other side.

Which is to say, the question worth asking is not really “which rounding mode is mathematically correct?”, because none of them is uniquely correct in any abstract sense. The right question, as it almost always is in this kind of design choice, is:

Which rounding policy matches the domain contract?

Rounding is not only mode. It’s also when.

There is a second axis to this kind of decision, which gets less attention than it deserves but which often matters at least as much as the choice of mode itself. The axis is when in a calculation rounding actually happens. Two broad designs are common in production code:

1. Round at every step (easy, often wrong)
2. Keep high precision internally, round only at boundaries (harder, usually right)

Example: line items and invoices

To make the trade-off concrete, consider a fairly standard invoicing setup:

• Price per item carries four decimal places of precision
• The currency, for display and settlement, is two decimals
• Taxes are computed on totals rather than per-line-item (which is the more common convention)

If rounding happens too early in this kind of calculation, the invoice total can end up disagreeing with what the underlying ledger says it ought to be. The four candidate policies, in this small space, are:

• Round each line item to two decimals, then sum the rounded values
• Sum the unrounded line items at full precision, then round once at the total
• Compute tax per line item and then sum the taxes
• Compute tax once on the total, then round the tax once

All four of these are, in some sense, “reasonable”, and none of them is wrong in the abstract. Only one of them, however, will actually agree with the rules your business has adopted, and the only way to make sure you have picked the right one is to choose explicitly, document the choice somewhere where it can be referred to later, and write tests that pin the chosen behaviour in place.

A practical Java pattern: Money as scaled integer

For currency in particular, the simplest representation that is also genuinely correct in most cases is, in my experience, to store amounts as integer minor units — long-typed cents, in the case of a US-style currency — to do all of the internal arithmetic in integers, and to convert back to a decimal representation only at the boundaries where the value is being displayed or transmitted. This pattern avoids a great deal of BigDecimal overhead, and it eliminates the entire class of bug that arises from rounding the same value twice in slightly different ways during a calculation.

Simple Money type (cents)

A reasonable starting point looks something like this:

import java.math.BigDecimal;
import java.math.RoundingMode;

public final class Money {
  private final long cents;

  private Money(long cents) { this.cents = cents; }

  public static Money ofDollars(String amount) {
    // Parse exact decimal dollars, then scale to cents.
    // This version rounds permissively; see ofDollarsStrict for validation.
    BigDecimal bd = new BigDecimal(amount).setScale(2, RoundingMode.HALF_EVEN);
    return new Money(bd.movePointRight(2).longValueExact());
  }

  /**
   * Strict version: reject inputs that aren't exactly 2 decimals.
   * @throws ArithmeticException if rounding would be required
   */
  public static Money ofDollarsStrict(String amount) {
    BigDecimal bd = new BigDecimal(amount).setScale(2, RoundingMode.UNNECESSARY);
    return new Money(bd.movePointRight(2).longValueExact());
  }

  public Money plus(Money other) {
    return new Money(Math.addExact(this.cents, other.cents));
  }

  /**
   * Multiply by a factor with explicit rounding policy.
   * Notice we round only at the boundary where we return to cents.
   * In production, consider taking a BigDecimal factor to avoid parsing repeatedly.
   */
  public Money times(String factor, RoundingMode mode) {
    BigDecimal bd = BigDecimal.valueOf(cents)
        .movePointLeft(2)
        .multiply(new BigDecimal(factor))
        .setScale(2, mode);
    return new Money(bd.movePointRight(2).longValueExact());
  }

  public BigDecimal toBigDecimal() {
    return BigDecimal.valueOf(cents).movePointLeft(2);
  }

  @Override
  public String toString() { return toBigDecimal().toPlainString(); }
}

The pattern at work in this small class is, I think, worth pulling out explicitly. Inputs are parsed from strings rather than from doubles; the scaling is always explicit; the rounding mode is a parameter at any point that policy has to be applied; and the internal representation is integer cents rather than BigDecimal. The code is a little longer than the obvious naive version would have been, but each of those choices removes a class of bug rather than papering over one.

There is one design choice in ofDollars worth calling out explicitly: it silently rounds inputs like "10.129" to "10.13". That is sometimes exactly what is wanted — for display amounts, for instance, where the upstream code has already validated the value — but for payments and ledgers it is often safer to use ofDollarsStrict instead, which simply rejects any input that is not already at exactly two decimals. The strict version forces validation to happen at the point where the value enters the system, rather than letting an off-scale value drift in and quietly become representable.

For multi-currency systems, the natural extension is to add a Currency field to the type and to ensure that arithmetic across different currencies is rejected outright. The core pattern, however, stays the same: integer minor units internally, with rounding occurring only at clearly identified boundaries.

Quick guide: what each rounding mode is “for”

The following is a field guide rather than a strict ruleset; the right answer in any given system is always whichever mode matches the domain contract, but these are the rough shapes of cases I have run into often enough to recommend by default.

Use HALF_EVEN when…

You are aggregating a large number of rounded values and you want to minimise systematic bias in the aggregate. This is the right default for most accounting ledgers, for interest accrual across many accounts, and for any large-scale reporting in which the total over many ties is what matters more than the result of any individual rounding step.

Use HALF_UP when…

The domain explicitly expects “schoolbook” rounding — the kind of rounding most people learned in primary school, where 0.5 always rounds up. This is appropriate for retail display prices, for some tax jurisdictions in which the rule is set by statute, and for human-facing calculations where the convention is part of the user experience and surprising the user with banker’s rounding would, on balance, do more harm than the bias it would prevent.

Use DOWN when…

Truncation is explicitly what the policy requires — typically in fee calculations or in conservative estimates where the rule is something like “we may not exceed this cap under any circumstances,” and where rounding in the more usual direction could produce an out-of-bounds value.

Use CEILING or FLOOR when…

These differ from UP and DOWN in that the direction is interpreted relative to the number line, rather than relative to zero, which makes them the right choice when sign matters:

• CEILING is “toward positive infinity” (-1.231 -> -1.23, 1.231 -> 1.24 at 2dp)
• FLOOR is “toward negative infinity” (-1.231 -> -1.24, 1.231 -> 1.23 at 2dp)

These tend to be the right choice for constraints, limits, and compliance rules of the form “must be at least X” or “must be at most Y”.

Use UNNECESSARY when…

You want the program to fail loudly the instant it encounters a value that you did not expect to need rounding. This sounds, when written down, like an irritation; in my experience it is, in fact, an enormously useful debugging and validation aid:

BigDecimal subtotal = new BigDecimal("12.34");
BigDecimal rate = new BigDecimal("0.075");
BigDecimal tax = subtotal.multiply(rate);

// Fail fast if tax isn't exactly representable at 2 decimals as required by policy
tax = tax.setScale(2, RoundingMode.UNNECESSARY);

The exception thrown by UNNECESSARY is not, in this kind of usage, a nuisance to be caught and ignored. It is a spotlight that the runtime is shining on an assumption you had forgotten you were making, and being told about that assumption explicitly is almost always more useful than letting it quietly produce a slightly wrong answer further downstream.

The moral of the story

HALF_UP, in summary, is not really wrong; it is simply not universal. Rounding modes, taken together, are not “implementation details” of any kind. They are product decisions, made of policy rather than mathematics, and they need to be picked the way that any other policy is picked — explicitly, with tests, and with a paper trail that lets future readers understand why a particular choice was made and what would have to change for the choice to be revisited.

TL;DR

• Ties (first discarded digit is 5, rest are zeros) are where rounding policy matters most.
• HALF_UP is intuitive but can introduce bias at scale.
• HALF_EVEN often reduces bias for aggregates.
• Avoid rounding double values when policy matters. Use BigDecimal created from strings or scaled integers.
• Prefer BigDecimal.valueOf(double) over new BigDecimal(double) when you must start from a double - but remember it’s not a cleansing ritual.
• Decide when you round, not just how you round.
• Prefer rounding once at boundaries, not repeatedly in the middle.
• Use UNNECESSARY to catch “we assumed this would be exact” bugs early.

In Part 6 of this series, we looked at how to defend against NaN by validating values at boundaries rather than scattering checks throughout the code. This post is about a different and, in its own way, sneakier kind of failure mode in floating-point arithmetic — the case where your sum is computed correctly at every individual step, every individual operation does exactly what the standard says it should, and the final answer is still meaningfully wrong.

There is a quiet bug that hides inside almost every large numerical reduction in production code, and it has, in my experience, almost nothing to do with mistakes in logic and almost everything to do with how floating-point addition is forced to behave. Floating point has fixed precision: every addition rounds back to a fixed number of significant digits, and the bits that do not fit are discarded. Most of the time, the discarded bits are too small to matter. Some of the time — particularly when you add a large number of values together, or when the values you are adding span many orders of magnitude — the cumulative effect of those tiny dropped bits becomes large enough to be visible in the answer.

This post is about the full story of Kahan summation, which is a small algorithmic trick with a surprisingly large impact on this kind of error. Kahan summation does not, and cannot, make floating-point arithmetic exact; what it does is keep your running totals from quietly losing meaningful contributions when you add many numbers, by tracking the bits that rounding has thrown away and folding them back into the next operation.

The innocent-looking loop

The classic numerical-analysis textbook example for this kind of error is, for historical reasons, usually written in Fortran — it is the lingua franca of mid-twentieth-century numerics — and it looks innocuous enough that it is easy to read past it without realising what it is about to do.

S = 0.0
DO 4 I = 1, N
  YI = ...
4 S = S + YI

The whole drama, such as it is, lives in the last line: S = S + YI. People then drop in a line that, at first reading, sounds slightly mystical:

Rounding or truncation in the addition can contribute to a loss of almost $\log_{10}(N)$ significant decimal digits in S.

The $\log_{10}(N)$ in that sentence does not, however, come from any sort of wizardry. It is simply what happens when fixed precision collides with a running total that keeps growing — the floating-point grid on which the running total can land gets coarser as the total gets larger, and so the smallest things you are still trying to add to it get progressively harder to see.

Why you can lose about log10(N) digits

Suppose, for the sake of intuition, that we are working in base-10 floating point with p significant decimal digits. (IEEE 754 is base 2, but the intuition transfers cleanly enough that working in base 10 is worth it for the explanation.) Suppose further that the YI values are roughly the same size and mostly the same sign, so that the partial sums genuinely grow rather than cancelling each other out, and let |Y| denote a typical magnitude.

After N terms, then, the true sum is roughly:

And here is the key floating-point constraint to keep in mind: a p-digit floating number around magnitude |S| cannot represent arbitrarily small changes. Near S, the spacing between representable values is approximately:

It helps to think of $\Delta S$ as the width of the grid lines at the altitude where S happens to live. When you compute S = S + YI, the result is rounded back onto that grid, and any increment that is much smaller than $\Delta S$ at that altitude can simply vanish, because there is no representable value between S and S + YI in the new precision regime.

By the end of the loop, |S| \approx N|Y|, so the grid spacing near the final sum is:

Comparing the grid spacing to the size of a typical addend gives:

That ratio, in the end, is what the textbooks are getting at. Multiplying by N shifts magnitudes by $\log_{10}(N)$ decimal digits, which means that as the running sum grows by a factor of N, the rounding grid spacing grows by exactly the same factor. Relative to the scale of the things you are still trying to add, you have therefore effectively lost about $\log_{10}(N)$ digits of useful precision. Another way to say the same thing is:

A concrete gut punch

Suppose, to make this less abstract, that you are working with about p = 7 significant decimal digits — roughly single-precision territory — and you sum N = 10^6 numbers, each of size about 1.

$\log_{10}(10^6) = 6$

Which leaves you with roughly:

$7 - 6 = 1$ meaningful decimal digit at the scale of 1

By the time S reaches around one million, the grid spacing near S is roughly:

$10^6 \cdot 10^{-7} = 10^{-1} = 0.1$

So adding 0.01 to S at that point can literally do nothing at all — the increment is smaller than the grid spacing in the precision regime where the running total currently lives, and it gets rounded away. The problem, in other words, is not really the single rounding step; it is the cumulative effect of rounding many million times while the running total continues to inflate.

A quick demo: sums that look reasonable and are still wrong

Here is a small example of the same kind of cancellation, in Java this time:

double x = 1e16;
double naive = (x + 1.0) + 1.0 - x;   // commonly prints 0.0
System.out.println(naive);

The two + 1.0 additions in this expression happen while the running value is around 1e16, and at that scale 1.0 is, in fact, smaller than the spacing between representable doubles. The two ones, therefore, fall through the cracks of the grid entirely, and the answer that arrives back is 0.0 rather than the 2.0 that ordinary arithmetic would have produced.

This is, importantly, not a curious corner case that only shows up in exam questions. If your system does any kind of analytics, pricing, telemetry, risk computation, recommendations, or ranking — anything where N becomes large in routine operation — this kind of error is the default failure mode hiding inside what looks like a perfectly innocent reduction.

Kahan summation: track what gets dropped

Kahan summation is, in technical terms, an instance of compensated summation. The idea is to keep two running numbers instead of one:

• sum: the running total, the way you would normally maintain it
• c: a compensation term that records what rounding threw away on the previous addition

The first holds the running total at full machine precision; the second holds an estimate of the low-order bits that were lost on the most recent addition, so that those lost bits can be folded back into the next addition rather than disappearing forever.

The algorithm

Given a new addend x, the algorithm performs three steps in sequence: it adjusts the addend by what was lost on the previous step, it adds the adjusted addend to the running total, and it then estimates how much was lost on this step and stores that estimate as the new compensation.

y = x - c
t = sum + y
c = (t - sum) - y
sum = t

That, taken on its own terms, is the whole trick. It does not change the underlying floating-point rules — additions are still rounded, the precision is still finite — but it changes the way you accumulate so that the rules hurt you less, by keeping a small ledger of what they have already cost you and applying it as a correction on the next operation.

Java implementation: a small accumulator with big consequences

The accumulator type

A reasonable Java implementation of the algorithm above is a small accumulator object that exposes the operations one would expect of any reduction primitive — adding a single value, combining with another instance of itself, and producing a final result.

public final class KahanAccumulator {
  private double sum;
  private double c; // compensation for lost low-order bits

  public void add(double x) {
    double y = x - c;
    double t = sum + y;
    c = (t - sum) - y;
    sum = t;
  }

  public void combine(KahanAccumulator other) {
    // Merge another partial sum into this one.
    // This improves accuracy, but the result is still order-dependent.
    this.add(other.sum);
    this.add(other.c);
  }

  public double value() {
    return sum;
  }
}

One thing worth saying clearly here, before anyone is tempted to oversell what Kahan summation buys: Kahan improves accuracy substantially, but floating-point addition is still not associative, which means that parallel reductions running over the same data can still produce different totals from one run to the next, simply because the order in which the partial sums get combined varies. Kahan makes that wobble considerably smaller; it does not make it impossible.

Using it with Streams

There are two natural ways to integrate this accumulator with the Java Streams API, depending on whether you happen to have a primitive DoubleStream or a boxed Stream<Double>.

Option A: DoubleStream.collect (best for primitive streams)

DoubleStream has its own collect overload that avoids the boxing cost of the generic Collector machinery, which makes it the better choice when you already have a primitive stream:

import java.util.stream.DoubleStream;

public final class Kahan {

  private Kahan() {}

  public static double sum(DoubleStream stream) {
    KahanAccumulator acc = stream.collect(
        KahanAccumulator::new,
        KahanAccumulator::add,
        KahanAccumulator::combine
    );
    return acc.value();
  }
}

Usage looks like the obvious thing:

double s1 = Kahan.sum(DoubleStream.of(0.1, 0.2, 0.3));
double s2 = Kahan.sum(myDoubleStream.parallel()); // allowed, order still not fixed

Option B: a Collector (nice ergonomics for boxed streams)

When you already have a Stream<Double> for some reason — typically because the upstream code is using boxed types — a Collector is more convenient and idiomatic:

import java.util.stream.Collector;

public final class KahanCollectors {

  private KahanCollectors() {}

  public static Collector<Double, KahanAccumulator, Double> kahanSummingDouble() {
    return Collector.of(
        KahanAccumulator::new,
        (acc, x) -> acc.add(x),
        (a, b) -> { a.combine(b); return a; },
        KahanAccumulator::value
    );
  }
}

Usage, again, is the usual thing:

double s = myStreamOfDoubles.collect(KahanCollectors.kahanSummingDouble());

Correctness: what to test, and what not to promise

It is worth being precise, when introducing this kind of utility into a codebase, about what it does and does not promise, because the difference between the two is the source of most production surprises in this space.

What you should promise

• Better accuracy than naive summation for large N and wide dynamic ranges
• Explicit behavior you control and can review in code

What you should not promise

• Bit-for-bit deterministic results in parallel streams
• Exactness for money or decimal accounting

JUnit test idea: compare against a higher-precision reference

BigDecimal is not, strictly speaking, a perfect oracle for binary floating point — it represents numbers in base 10 — but it is a very useful reference when you want to confirm that one summation strategy is drifting more than another, since BigDecimal arithmetic has effectively unlimited precision for the operations involved here.

import static org.junit.jupiter.api.Assertions.*;
import java.math.BigDecimal;
import java.util.Random;
import java.util.stream.DoubleStream;
import org.junit.jupiter.api.Test;

public class KahanAccumulatorTest {

  @Test
  void kahan_is_usually_better_than_naive_on_wide_range_data() {
    Random r = new Random(0);

    double[] xs = DoubleStream.generate(() -> {
      double sign = r.nextBoolean() ? 1.0 : -1.0;
      double mag = Math.pow(10.0, r.nextInt(20) - 10); // 1e-10 .. 1e9
      return sign * mag * r.nextDouble();
    }).limit(200_000).toArray();

    double naive = 0.0;
    KahanAccumulator kahan = new KahanAccumulator();
    BigDecimal ref = BigDecimal.ZERO;

    for (double x : xs) {
      naive += x;
      kahan.add(x);
      ref = ref.add(BigDecimal.valueOf(x));
    }

    // Reference converted back to double so we compare at double resolution.
    double reference = ref.doubleValue();

    double errNaive = Math.abs(naive - reference);
    double errKahan = Math.abs(kahan.value() - reference);

    // In rare adversarial sequences Kahan can be slightly worse, but typically it's much better.
    assertTrue(errKahan <= errNaive * 2.0,
        String.format("Kahan error (%.2e) vs naive (%.2e)", errKahan, errNaive));
  }
}

Performance: what it costs

Naive summation does exactly one addition per element. Kahan summation, by comparison, does a small handful of extra operations per element — three subtractions and one extra addition, on top of the one that would otherwise have been there. That is, in the end, a small constant factor, and in the great majority of analytics pipelines I have worked on, the accuracy gain is worth far more than the marginal CPU cost.

If you do want to measure the overhead carefully, the right tool to reach for is JMH, and the comparison worth running is between four variants:

• the naive loop
• DoubleStream.sum()
• the Kahan loop directly
• Kahan via DoubleStream.collect

The relative numbers, in my experience, depend more on the surrounding pipeline and the JIT’s behaviour than on the algorithm itself, but the overhead of Kahan tends to be in single-digit percent territory for typical workloads.

When NOT to use Kahan

Kahan is genuinely useful, but it is not the right tool in every situation. There are at least three categories in which I would specifically avoid reaching for it:

• Financial calculations. Use BigDecimal for money. Kahan does not give you decimal semantics, and the kinds of rounding rules that finance and accounting actually require live in a different problem entirely.
• Tiny datasets. When N is small, the cumulative error of naive summation is usually well below any threshold that matters, and the overhead of Kahan, however small, is rarely worth carrying for the negligible benefit.
• Hard real-time or tight latency budgets. Profile first; the overhead is usually small but it is not zero, and in the kind of code that has a hard deadline every iteration, even a few nanoseconds per element can add up to meaningful time.

Variants worth mentioning: Neumaier and pairwise summation

Kahan is, although a good default, not the only available compensated-summation technique. It is worth at least naming a couple of the alternatives, both because they sometimes behave better than Kahan on particular kinds of data and because anyone seriously interested in numerical accuracy will encounter them sooner or later in the literature.

Neumaier summation is a small modification of Kahan that handles a specific case more gracefully: the case in which the next addend is, in magnitude, larger than the current running sum, which is a situation Kahan can mishandle in a few corner cases.

// Neumaier variant
public final class NeumaierAccumulator {
  private double sum;
  private double c;

  public void add(double x) {
    double t = sum + x;
    if (Math.abs(sum) >= Math.abs(x)) {
      c += (sum - t) + x;
    } else {
      c += (x - t) + sum;
    }
    sum = t;
  }

  public double value() {
    return sum + c;
  }
}

Pairwise summation, also known as tree reduction, takes a different approach: rather than tracking compensation explicitly, it reduces error growth by combining numbers of similar magnitude together, so that the running sum never gets too far ahead of the values still being added. Some stream implementations may, behind the scenes, do something resembling this internally; the principal advantage of writing your own is that the behaviour is then explicit, reviewable, and not subject to silent change between JDK versions.

When to use what

Pulling all of the above together, the general guidance I would offer is roughly this. Use BigDecimal when you need decimal semantics, and you mean it; use naive summation when N is small, the magnitudes of the addends are similar, and you genuinely do not care about a few low-order bits; and use Kahan or Neumaier when:

• N is large (thousands to millions of values)
• values span many orders of magnitude
• small contributions matter
• you want better accuracy without dragging BigDecimal everywhere

A note on BigDecimal vs Kahan (so we don’t mix the tools)

It is worth being explicit about this distinction, because I have seen it confused often enough that the confusion is, by itself, a recurring source of bugs. BigDecimal is about decimal semantics — money, accounting, “pennies must add up,” explicit rounding rules — and Kahan is about accumulation error in double when double is genuinely the right representation for the values being summed (metrics, measurements, statistics, ML features, telemetry) but naive summation is bleeding low-order bits over a long enough reduction.

The practical rule of thumb is straightforward enough:

• If your requirements mention cents, statements, taxes, interest, regulatory accuracy, or mandated rounding policies → use BigDecimal (and decide scale + rounding mode explicitly).
• If your requirements mention large aggregates, mixed magnitudes, order sensitivity, or “why did parallel give a different total?” → keep double, but upgrade the summation (Kahan / Neumaier / pairwise), and be explicit about ordering if determinism matters.

One small but persistent footgun while we are on the subject: BigDecimal only stays “decimal-correct” if you construct it correctly. Prefer parsing from a decimal string, or BigDecimal.valueOf(double), over new BigDecimal(double), which faithfully captures the binary representation of the double — including, of course, all of its decimal infidelity.

Closing

Floating point, on its own terms, is not really broken; it is simply finite. The problem is not that rounding happens — rounding is what makes finite-precision arithmetic possible in the first place — but that rounding can happen many millions of times across a single reduction, while the running total continues to grow, and the cumulative effect of all those small dropped bits can become large enough to be wrong in ways that matter.

What Kahan summation does, in the end, is to keep a small ledger of what rounding has thrown away and to fold those discarded bits back into the next addition, so that the long-run drift stays bounded rather than accumulating without limit. The code change involved is genuinely small; the conceptual shift, however — from treating summation as a single line of code to treating it as a deliberate numerical algorithm — is the part that, I think, is worth taking seriously the next time you find yourself reaching for .sum() over a few million doubles.

References

• W. Kahan, “Pracniques: Further remarks on reducing truncation errors,” Communications of the ACM, 8(1), Jan. 1965. DOI (ACM Digital Library): https://dl.acm.org/doi/10.1145/363707.363723

Open PDF (hosted copy): https://convexoptimization.com/TOOLS/Kahan.pdf

Metadata entry (Semantic Scholar): https://www.semanticscholar.org/paper/Pracniques%3A-further-remarks-on-reducing-truncation-Kahan/672a99813f52aed720d3508d6be7db461328b064

• Prof Kahan’s Assorted Notes, https://people.eecs.berkeley.edu/~wkahan/

• N. J. Higham, “The Accuracy of Floating Point Summation”, SIAM Journal on Scientific Computing (1993). https://doi.org/10.1137/0914050

• DoubleStream.sum(): https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/stream/DoubleStream.html#sum()

• DoubleSummaryStatistics: https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/DoubleSummaryStatistics.html

This post, in the modern fashion, started its life with a social media thread. I saw a post about “student drivers” which had, by the time I came across it, evolved into a sweeping generalization about an entire people; someone, apparently in response, had then attempted to generalize Indians in one long, unpunctuated breath, and the resulting exchange was almost a small textbook in the failure mode I want to write about here. The irony, in particular, was not subtle: the corrective offered to a sweeping generalization about one group was, somehow, a sweeping generalization about another group, on the apparent assumption that two equal and opposite errors might add up to honesty.

I should say, before going further, that this is not really a post about Indians, or Americans, or any one group in particular. It is a post about a habit of mind that I think we have, collectively, normalized — the habit of reducing millions of people to a neat sentence and then calling it “an observation.”

The world is too large for our brains

It is worth starting with the underlying mechanism, because the instinct that produces stereotypes is, by itself, neither malicious nor unusual. Human brains are, before they are anything else, pattern machines. We find faces in clouds, infer intent from a raised eyebrow, and build small working models of reality more or less continuously, because raw reality, in its full detail, is far too large to actually hold in mind. That model-building instinct is not in itself evil; it is, in a real sense, a survival adaptation, and most of the time it serves us reasonably well.

The trouble begins, however, when we start to mistake the model for the world. A stereotype, viewed from this angle, is what happens when you take a small sample of human behaviour, add a generous amount of emotion, mix in a few viral anecdotes, and then quietly export the resulting impression as if it were a universal law about an entire population. It tends to feel efficient, satisfying, and like a kind of closure on a question that previously was open — and all three of those feelings, while pleasant, are also exactly what makes the move so easy to perform without noticing. It is, in the end, the principal way in which we end up misreading entire peoples.

“Indians are…” is a sentence that breaks under its own weight

India, to take the example I happen to know best, is not a monolith. It is a continent-sized civilization with many languages, many histories, many moral philosophies, many economic realities, and many different ways of being human, none of which fit comfortably under a single noun. When someone confidently begins a sentence with “Indians are…”, and then completes it with any predicate at all, what they are usually saying — whether they realise it or not — is something closer to “I met some Indians once,” or “I saw a few videos,” or “I had one bad experience,” or “I read a thread on the internet that made me feel righteous, and I am now extrapolating from it.”

The mental move that follows is the one I find most worth flagging, because it is so common and so frictionless that it is usually invisible to the person making it. The brain takes an anecdote — a single experience, a small sample, a memorable encounter — and quietly upgrades it into an identity, into a property of an entire group. That is not insight in any meaningful sense; it is laziness wearing the costume of insight, and it is unusually difficult to argue with, because the speaker can always retreat to “but I am only describing what I have actually seen.”

The equal and opposite mistake

There is a pattern that tends to follow almost mechanically from the kind of move described above, and it is worth treating on its own terms because it is, in some ways, more pernicious than the original. The pattern is this: the response to “Indians are X” is almost never “let us not generalize about people in groups,” but is instead “well, Americans are Y.”

This looks, on its surface, like an attempt at fairness — turning the same lens back on the speaker — but in practice it is just the same error doing a second lap. The United States, after all, is the third-largest country in the world by population, and it contains multitudes: regions that feel like different planets, communities built up from different waves of migration, value systems that clash with each other on a daily basis, and identities that do not fit into any clean box that anyone has yet managed to draw. Broad generalizations about Americans are, on close inspection, every bit as unfair as broad generalizations about Indians, and for exactly the same reasons.

What tends to happen in these exchanges, then, is not a productive conversation but a steady collapse into noise, in which one stereotype is met with its opposite, that opposite is met with another, and by the time anyone has stopped to think, the actual question — whatever it might originally have been — has been buried under a small mountain of confidently stated nonsense. The result, in any sense that the word ought to mean, is not justice; it is just more of the same kind of laziness, moving faster.

The “small loud group” problem

A reasonable-sounding defence of the move I have been describing goes something like this: “I am not generalizing about everyone. I am only talking about a certain kind of person.” That defence is sometimes legitimate, and it is worth taking seriously, because there is no question that social media has a way of amplifying the most extreme voices in any group, that outrage tends to function as the algorithmic fuel of these platforms, and that the loudest voices therefore get the microphone far more often than their actual representativeness in the underlying population would warrant.

What this defence misses, however, is that adult conversation about groups requires adult precision about which claim, exactly, is being made — and there is a meaningful difference between two claims that often get confused for each other. The first is “some people are doing this harmful thing”; the second is “this group is like this.” The first is a claim about behaviour, and it has the useful property of being debatable, measurable, challengeable, and refinable in the light of new evidence. The second is a claim about identity, and what it does, often without the speaker quite noticing, is to turn an observed behaviour into an essential property of the group itself — at which point it becomes very difficult to discuss without sounding either dismissive of the original observation or apologetic on behalf of an entire population.

That distinction — between behaviour and identity — is, I think, the line that matters most in these conversations. It is also, in my experience, the line that is most reliably crossed without anyone quite admitting that they have crossed it.

Why stereotypes feel so good

It is worth being honest, at this stage, about why stereotypes are so persistent, despite the fact that almost everyone, when asked directly, will agree that they are unreliable. The honest answer, I think, is that they offer at least three things which the internet, in its current form, particularly rewards: a feeling of certainty about a complicated subject (“I understand what is going on here”), a substantial saving of time and mental effort (“I do not need to do the work”), and a kind of moral permission (“I am now justified in judging”). Each of those payoffs is, on its own, a real human need, and a stereotype tends to provide all three at once and very cheaply.

Real understanding, by comparison, is comparatively slow. It requires the friction of saying “I might be wrong about this,” and of then doing the harder work of going out and gathering better evidence than the small sample one started with. Philosophers have a name for the disposition involved here — they call it epistemic humility¹ — and it amounts, in the end, to the recognition that our knowledge of any given group of people is always incomplete, our samples always limited, and our confidence in our conclusions always more provisional than we tend to act as if it is. It is, almost by definition, the opposite of the kind of reaction that social media tends to reward, and it is, in any case, considerably harder than typing a hot take.

A better way to speak

If the goal is to be able to criticize an observed pattern without flattening millions of people into cardboard cutouts, there are a few defaults that I have, over time, come to find useful. Most of them are not original to me, and most of them sound obvious when written down — but, as with most discipline, the difficulty is not in the principles themselves but in remembering to apply them at the moment one is most tempted not to.

• Speak about specific behaviors, not identities.
• Use “some” and “in my experience” like seatbelts.
• Separate “what I saw” from “what is true.”
• Ask whether your sample is representative or just memorable.
• When you feel righteous, pause. Righteousness is not a fact-checker.

What unifies most of these, I think, is the underlying observation that, in conversations about groups, precision is a form of kindness. If the goal is to be understood, and to leave the other person with a clearer view of the world rather than a more flattering view of themselves, then the additional words spent saying “some” rather than “all,” or “in my experience” rather than “obviously,” are not weakness or hedging — they are the difference between a sentence that can be true and one that essentially cannot.

Immigrants live inside the blur

I should also say, more personally, that immigrants tend to occupy a particular kind of space in these conversations, and it is one that I have come to understand from the inside. You learn, over the years, to love the country you live in, while also carrying — quietly, most of the time — the experience of being misread by it. You become, in effect, bilingual not only in language but in assumptions, and you spend a non-trivial amount of mental energy translating between the two on any given day.

The forms of that misreading vary considerably. Sometimes you are treated, by complete strangers, as an ambassador for a population of more than a billion people, and asked questions that no individual could possibly be qualified to answer. Sometimes you are treated as an exception to whatever is being said about your group at the time — “you’re not like the others” — which is meant generously and is, in its own way, almost as unkind as the original generalization, because it tacitly accepts the generalization while granting you a personal exemption from it. And sometimes you are simply reduced to a meme, which is at least faster.

Through all of that, you still get up in the morning and show up. You contribute, you build, you teach, you try to belong somewhere without having to dissolve into it in order to do so. And it is from inside that experience, more than anything else, that I want to insist that when someone stereotypes a group, what they are doing is not really “just words.” A stereotype, in practice, is a small social verdict, and small social verdicts have a way of sticking.

The point

It would be, of course, somewhat awkward to end an essay against generalizations with a generalization of my own, and so I want to be careful to avoid the easy line “everyone stereotypes,” which is, in its own quiet way, exactly the kind of move I have spent the last several sections complaining about.

What I will say instead is something more local. After the thread I described at the beginning, I also received a number of kind messages from people in my local community, offering support and, in several cases, apologizing “on behalf of” others. The intent in those messages was generous, and I genuinely appreciated each of them. But I have to admit that, even within that generosity, I felt the shadow of the same underlying mistake — the assumption that there is, somewhere, a “them” coherent enough to speak for. The truth, when one looks closely, is that there usually is not, and that the most useful thing one can do in response to bad generalizations from one direction is not to speak more confidently from the other, but to refuse the framing altogether.

We do not, in the end, defeat stereotypes by pretending that differences between people and groups do not exist; that is its own form of dishonesty, and a particularly fragile one at that. We defeat them, to the extent that we defeat them at all, by being honest about complexity — by trading lazy certainty for careful clarity, by resisting the cheap thrill of the sweeping sentence, and by remembering, especially when it is most tempting to forget, that every “they” we are tempted to talk about is, on closer inspection, made of millions of individual “someones,” each of whom would, given the chance, object to whatever we were about to say about them collectively. The world, in other words, is genuinely detailed; and being honest about that detail, even when it makes our sentences longer, is what taking other people seriously actually looks like.

References and Notes

When IEEE 754 arithmetic encounters an operation for which there is no real-number answer — dividing zero by zero, taking the square root of a negative number, and so on — it does not throw an exception. Instead, it produces a special value called NaN, short for “Not a Number,” and the program continues running as if nothing in particular had happened. This is, as a piece of language design, both a strength and a quiet curse: it allows numerical code to keep flowing in the presence of locally invalid operations, but it also means that NaN tends to slip downstream silently and only surface much later, in log files, in metrics, and on dashboards, long after the operation that actually produced it has scrolled out of view.

The temptation, on first encountering this behaviour, is to start sprinkling if (isNaN) checks throughout the codebase as a defence. This post is, in a sense, about doing the opposite of that — about defending against NaN structurally, using a small number of checks placed deliberately at the right boundaries, rather than scattering a defensive layer through every function in the system.

The shape of the beast

NaN, as the name suggests, is what IEEE 754 hands back from operations that have no meaningful real-number answer. The canonical examples are familiar enough — dividing 0.0 by 0.0, taking the square root of a negative number, taking the logarithm of a negative number — but there is a fourth source that is, in practice, more important than any individual mathematical case: any operation that already involves a NaN will itself produce a NaN. NaN is contagious, in both the best and the worst senses of the word, and that contagion is the mechanism by which a single invalid operation upstream can quietly poison everything that flows from it.

Some of the operations that produce NaN are:

1.	0.0 / 0.0  
2.	Math.sqrt(-1.0)  
3.	Math.log(-1.0)  
4.	Any operation that already contains NaN, because NaN is contagious in the best and worst ways  

Java follows IEEE 754 in all of this. The JVM does not, for the most part, throw an exception when a floating-point operation is invalid; it produces NaN or Infinity and lets the program continue, which is another way of saying that what you are looking at is a silent error.

There is one further property of NaN that catches almost everyone the first time they encounter it. NaN is not equal to anything, not even to itself.

double x = Double.NaN;

System.out.println(x == x);              // false
System.out.println(Double.isNaN(x));     // true
System.out.println(x < 0);              // false
System.out.println(x > 0);              // false
System.out.println(x == 0);             // false

NaN, in other words, does not really behave like a value in the ordinary sense; it is more accurately understood as a signal that has, by the design of the floating-point system, been forced to masquerade as a value. The fact that it wears that disguise convincingly is the source of most of the practical trouble that NaN goes on to cause in production systems.

Defensive programming hell, or what I call “Checkpoint Syndrome”

The instinctive reaction to discovering NaN in production is, I think, entirely understandable. You find a NaN in a log, trace it back to a particular code path, add an isNaN check there, deploy the fix, and move on. The trouble is that, almost without exception, the next NaN that shows up will not be in the same place — it will be in some other path that touches the same data — and so another check goes in. Repeat this process for a few months and you end up with a codebase in which essentially every function is doing its own defensive validation of every input it receives, often with slightly different responses to the same underlying condition.

The result tends to look something like this:

double price = computePrice(input);
if (Double.isNaN(price) || Double.isInfinite(price)) {
    // shrug, return 0?
}

This is what I have come to think of as Checkpoint Syndrome, and the problem with it goes well beyond the visual noise. It is something closer to a small architectural disaster: the checks have a way of spreading everywhere, and yet, despite their ubiquity, they almost never point at the real cause of the bug, because by the time NaN reaches the function being checked, the operation that originally produced it is several layers upstream and is no longer visible at the call site. Each defensive site is also forced to make its own decision about what to do when the check fails, and those decisions tend to drift apart over time, so a single class of upstream bug ends up being silently handled in a dozen inconsistent ways throughout the codebase.

Worst of all — and this is where Checkpoint Syndrome most reliably produces actual financial bugs — the easiest “fix” at any individual site is to convert the invalid value to zero. That has the convenient property of making the immediate symptom go away, while quietly turning “we do not know” into “definitely zero” in every downstream calculation that follows.

The antidote to all of this is not, as it might first appear, to add more checks. The antidote is to place fewer checks but to place them where they actually matter — at the boundaries where invalid values either enter the system or are first produced, once, deliberately, and with a documented response.

The core principle: validate at the edges, compute in the middle

Most NaN outbreaks begin at boundaries:

1.	Parsing and deserialization (CSV, JSON, user input, partner payloads)  
2.	Sensor-style data (telemetry, percentages, rates)  
3.	Divide by something that might be zero or missing  
4.	"This should never happen" conversions (and then it happens)  

The most useful thing you can do in response to this, in my experience, is to establish a single simple contract that the rest of the system can rely on: inside the computation core, all doubles are finite unless explicitly documented otherwise. The computation core is the part of the system that should be allowed to be blissfully boring — the pure math, the algorithms, the business logic that operates on already-validated inputs — and it is the place where you most want to be able to reason about correctness without simultaneously doing border control.

The practical consequence of that contract is that NaN handling becomes concentrated in a small number of choke points, rather than spread thinly across the codebase.

Pattern 1: “Finite by default” as a guardrail

The simplest expression of this discipline is a small helper that asserts finiteness explicitly:

static double requireFinite(double x, String name) {
    if (!Double.isFinite(x)) {
        throw new IllegalArgumentException(name + " must be finite, got " + x);
    }
    return x;
}

Double.isFinite() is, for this purpose, the only check you actually need. It returns true only when its argument is neither NaN nor Infinity, which is exactly what “a normal, usable number” means in most contexts. There is rarely any value in writing two separate checks for isNaN and isInfinite when a single call to isFinite captures both conditions.

The pattern, then, is to use requireFinite at public boundaries and at layer transitions, rather than inside every private method:

public double monthlyPayment(double principal, double annualRate, int months) {
    requireFinite(principal, "principal");
    requireFinite(annualRate, "annualRate");
    if (months <= 0) throw new IllegalArgumentException("months must be positive");

    double r = annualRate / 12.0;

    // Standard annuity formula:
    //   P = L * [r(1+r)^n] / [(1+r)^n - 1]
    // Rewritten with a negative exponent:
    //   P = L * r / (1 - (1+r)^(-n))
    // Same math, often friendlier numerically for large n.
    return principal * r / (1.0 - Math.pow(1.0 + r, -months));
}

What you get from this approach is, I think, three things worth having at the same time. The failure becomes loud and early, rather than silent and downstream. The body of the computation stays free of validation noise. And when the exception does fire, it points at a real contract violation — a specific named argument coming in non-finite — rather than at some mysterious symptom many layers later in the call stack.

Pattern 2: Separate “invalid” from “zero” with a result type

There are situations in which you cannot reasonably throw — typically because the caller is processing a stream of inputs and needs to continue regardless of whether any individual one is valid, while still being able to tell which ones were not. In those situations, the right move is usually to represent the distinction explicitly in the return type, rather than overloading a numeric value to mean both “result” and “no result.”

In Java, you can use a sealed interface to construct a true disjunctive type, of which I have written a longer post for the full story:

sealed interface CalcResult permits CalcResult.Valid, CalcResult.Invalid {
    
    record Valid(double value) implements CalcResult {
        public Valid {
            // Enforce finite values in Valid variant
            if (!Double.isFinite(value)) {
                throw new IllegalArgumentException("Valid result must be finite");
            }
        }
    }
    
    record Invalid(String reason) implements CalcResult {}
    
    // Convenience factory methods
    static Valid ok(double value) {
        return new Valid(value);
    }
    
    static Invalid failed(String reason) {
        return new Invalid(reason);
    }
}

With a type of this shape in place, NaN is no longer a stealth signal hiding inside a numeric channel. An invalid result is a first-class outcome of its own, the Valid variant cannot be constructed with a non-finite value, and the compiler can be made to enforce exhaustive handling at every consumption site.

CalcResult safeDivide(double a, double b) {
    if (!Double.isFinite(a) || !Double.isFinite(b)) {
        return CalcResult.failed("non-finite input");
    }
    if (b == 0.0) {
        return CalcResult.failed("division by zero");
    }
    return CalcResult.ok(a / b);
}

// Pattern matching is exhaustive - compiler forces you to handle both cases
CalcResult result = safeDivide(10.0, 2.0);
switch (result) {
    case CalcResult.Valid(double v) -> 
        System.out.println("Result: " + v);
    case CalcResult.Invalid(String reason) -> 
        System.err.println("Failed: " + reason);
    // No default needed - this is exhaustive
}

The exact shape of the type is, in the end, less important than the discipline it expresses. What matters is that “invalid” is treated as a first-class outcome that the type system insists be handled, rather than as a special value that callers can quietly forget to check for.

Pattern 3: Domain types that make NaN impossible

It is also worth observing that many of the doubles in business systems are not really “real numbers in the wild.” They are money, rates, counts, durations, and percentages — values which have far more structure than double is capable of representing, and which therefore tend to make poor candidates for raw double storage in the first place.

A few examples of the better choice in each case:

1.	Money: store cents as long, or use BigDecimal where precision matters  
2.	Counts: long or int  
3.	Percentages: maybe basis points as int (one basis point = 0.01%, so 550 bps = 5.50%)  
4.	Durations: java.time.Duration  

Tightening the types in this way reduces the surface area on which NaN can even appear in the first place. This is, in the end, the most reliable form of NaN defence available, and it has the additional virtue of being entirely structural: it is enforced by the compiler, rather than by the discipline of the next person to touch the code. A long, after all, simply cannot hold NaN — the language will not allow it.

Pattern 4: Decide where Infinity is acceptable

Part 5 of this series covered Infinity and signed zero in some detail. The practical question, at this stage, is what to do about Infinity in everyday code, and the answer turns mostly on whether your domain has any legitimate use for it.

Infinity is sometimes a meaningful signal:

1.	"Unlimited" limit  
2.	"Unbounded" score  
3.	A mathematical asymptote that you intentionally model  

If your domain does not explicitly accept Infinity as a meaningful value, however, the right thing to do is to treat it exactly as you would treat NaN — reject it at the boundary, using Double.isFinite(). This is precisely why the requireFinite helper above checks for finiteness rather than for isNaN alone: it enforces the rule “no NaN, no Infinity, full stop” in a single line. In the rare cases in which you genuinely need to distinguish between NaN and Infinity, you can still do so by checking each separately, but for the great majority of code, the only useful distinction is finite versus not-finite.

The bug pattern to be alert to is the case where Infinity is tolerated accidentally — quietly admitted into the computation core because no one thought to reject it — and is then multiplied or scaled into a very large number that happens to look plausible at a glance. That is, in effect, how nonsense becomes confident-looking nonsense, and it is harder to catch downstream than NaN, because at least NaN propagates obviously.

Pattern 5: Centralize sanitization, but do not lie

There are also situations in which sanitization is genuinely required, particularly when dealing with messy external data over which you have no control. The two principles I would offer, having been bitten by both, are: do the sanitization once, in a centralised place, and be honest about what you actually did.

The dangerous form of sanitization is the one that looks reasonable at first glance:

double safe = Double.isFinite(x) ? x : 0.0;

The problem with this pattern is that it converts “we do not know” into “definitely zero”, and zero is rarely a neutral value in any system that does arithmetic on it. A safer set of options is to either drop the offending datapoint outright (and to count it, so the volume of dropped values is observable), to mark the result as invalid using one of the result types described above, or to fall back to a documented default that has actual meaning in the domain — and, in any case, to log what happened so that the upstream cause can eventually be addressed.

A common example in aggregation is something like this:

double averageFinite(double[] xs) {
    double sum = 0.0;
    int n = 0;

    for (double x : xs) {
        if (Double.isFinite(x)) {
            sum += x;
            n++;
        }
    }

    // Honest answer: no valid data means undefined result.
    if (n == 0) return Double.NaN;

    return sum / n;
}

The honesty of the function consists in the last detail: when there is no valid data to average, the function returns NaN rather than zero. NaN is, in this case, the right answer — it is exactly what “undefined” means — and silently substituting zero would, once again, be a way of pretending to know something the function actually does not.

Third-party libraries: when NaN arrives by mail

Not every NaN is born in your own code. Some of them are delivered.

The most common cause of imported NaN is calling a math function that can legitimately return NaN for part of its domain — Math.sqrt, Math.log, and so on — and then forgetting that the moment you call out to such a function you have, in effect, crossed a boundary again, and the same boundary discipline applies. The fix is to validate the result immediately, at the point of crossing, while you still have full context about what was being computed:

double result = Math.sqrt(userInput);

// JDK math functions and third-party libs can return NaN.
// Validate the output right away, at the point of crossing.
requireFinite(result, "sqrt result");

The pattern generalises beyond JDK math: any time you call out to code you do not control — libraries, services, model inference endpoints, partner data feeds — the return value is best treated as a fresh boundary, with the same kind of validation you would apply to any other external input. The principle being preserved here is simply that the computation core gets to assume validated inputs, and any time something crosses into the core from elsewhere, the responsibility for validation falls on the crossing point.

Finding the first NaN, not the last one

There is a particular tragedy that tends to play out in long-lived systems, in which NaN is only detected at the very end of a long chain of transformations — in a report, on a dashboard, or in a downstream consumer — and the team then has to work backwards through ten transformations to find the operation that originally produced it. By that point, the NaN you are looking at is the smoke; the fire was several layers upstream, and the hard part of the bug is reconstructing how the smoke got to where it is now.

Two practical habits help with this:

Add “tripwire assertions” in debug builds

In places where NaN should never legitimately exist, it is worth asserting that fact explicitly during development and in tests, so that the first appearance of NaN fails loudly rather than silently propagating:

static void assertFinite(double x, String name) {
    if (!Double.isFinite(x)) {
        throw new AssertionError(name + " became non-finite: " + x);
    }
}

This is particularly worthwhile after major computation steps in numerically sensitive algorithms, where the cost of the assertion is negligible and the benefit of catching the first bad value early is considerable.

Log with context once, not everywhere

If the system needs observability into NaN-related rejections — and most production systems eventually do — the right place for that observability is at the boundary where the rejection happens, because that is where the original context still exists. An input adapter is a good example:

double parseRate(String raw) {
    double x = Double.parseDouble(raw);
    if (!Double.isFinite(x)) {
        // log raw payload id, customer id, partner id, etc.
        throw new IllegalArgumentException("rate must be finite");
    }
    return x;
}

This is the point at which you still have access to the raw payload, the source identity, and any correlating identifiers; once the value has flowed through several layers of the application, all of that context tends to be lost. Logging at the boundary, rather than at the point of eventual symptom, is therefore both cheaper and more useful.

A small NaN hygiene checklist

By way of summary, the practices that have served me well in dealing with NaN in long-running systems are roughly these:

1.	Use Double.isFinite at boundaries where values enter your system or cross layers
2.	Keep computation code clean; assume finite inputs inside the core
3.	Do not convert invalid to zero unless the domain definition says it is correct
4.	Prefer domain types over raw doubles when the value is not truly "a real number"
5.	When you must degrade gracefully, return an explicit invalid result, not a silent sentinel
6.	Instrument the boundary where you reject or drop invalids, so you can find the upstream cause

NaN, taken on its own terms, is not really an enemy. It is the floating-point system’s way of telling you that something invalid happened upstream and that the math, in good faith, could not produce a real-number answer. The defensive task is therefore not to suppress that signal everywhere it appears, but to listen to it where it first appears, deal with it deliberately at the boundary, and let the rest of the system rely on the contract that the boundary enforces. Validate at the edges, keep the core clean, and resist, as steadily as you can, the temptation to silently turn unknown into zero.

1) Consolidated assets into one place

The site had grown to keep static files in two different locations — public/ and assets/ — and the inconsistency had begun to cause exactly the sort of small problems that consistency is supposed to prevent: occasional broken paths, references that pointed to the wrong directory, and a small but real amount of cognitive overhead every time I had to remember which folder a particular file lived in.

I have now merged everything into assets/. CSS, JavaScript, and the favicon files have all moved over from public/, every template has been updated to reference /assets/..., and the web manifest along with the webcmd page have been corrected to match. The reader-facing benefit is mostly negative — fewer broken resources where there is no good reason for any to break — but the structural simplification also makes future maintenance easier, which I expect will matter more over time than any single fix in this round.

2) Modernized the favicon setup

While I was tidying up assets/, I also took the opportunity to modernise the favicon. The site now serves an SVG favicon as the primary asset, with the old ICO file still in place as a fallback for older browsers that do not yet handle SVG icons gracefully. The <link rel="icon"> tags in the templates were updated accordingly. SVG is crisp at any size and looks visibly sharper on modern displays, while the ICO fallback preserves compatibility for the small but persistent set of browsers that still need it.

3) Webcmd was modernised

The /webcmd/ page is still the command-line interface to the site that it always was, but it had drifted out of sync with the rest of the site in a few small ways. It is now using the default site layout and theme, which means it inherits the Nord palette and reads consistently with everything else. The help output, which was previously a wall of text, is now properly semantic — lists and headings rather than line-by-line plaintext — which makes it easier to scan and renders in two columns on desktop while collapsing sensibly on mobile.

A couple of smaller fixes also went in. The find command, which had quietly been listed under the wrong section, is now grouped with the other Searches commands where it belongs. The navigation commands (ph, p, pi, pr) now use relative URLs, which means they continue to work as expected when I am testing locally on localhost — previously they would silently rewrite to the production host, which made local testing more annoying than it had any reason to be.

4) Accessibility pass (WCAG-oriented)

I also did a focused accessibility sweep across the main site templates, with WCAG 2.1 AA as the target — the jsgames/ directory is excluded for now, since it is a different problem with a different set of constraints.

The work was less about adding any one big feature and more about closing a number of small gaps. There is now a “Skip to content” link at the top of every page, and a proper <main> landmark wrapping the primary content, both of which make keyboard and screen-reader navigation considerably faster. The sidebar toggle correctly updates aria-expanded and is now operable from the keyboard rather than only by mouse. The search overlay now traps focus while it is open and restores focus to the original element when it is closed, which is what a modal dialog should do but which my old implementation was not doing. The “Annotate me” control, which had been a clickable <span> for historical reasons, is now a real <button>. And the focus styles across the site have been tightened up so that keyboard users can actually see where they are at any given time.

The cumulative effect, for readers using assistive technology, should be noticeably more predictable behaviour throughout the site, and a meaningful reduction in the number of places where keyboard navigation hits an unexpected dead end.

5) Reduced Disqus noise on non-post pages

Disqus comment-count scripts were, until this round, being loaded on every page of the site, including pages that do not and never will have comments. They are now only loaded on posts that explicitly allow comments. This means there is less third-party JavaScript running on the home page, archive pages, and other non-post views, which both improves page-load performance modestly and reduces the slightly disorienting experience of seeing comment-related widgets briefly flicker on pages that have no comments to count.

6) New documentation

Finally, in the interest of keeping all of the above sustainable rather than letting it bit-rot, I wrote a couple of internal documents to capture the conventions I have adopted. docs/webcmd.md explains how the command engine works and how to add new commands without re-learning the system every time. docs/accessibility.md captures the accessibility conventions I am now trying to maintain, along with a small checklist of things to verify when adding new templates. The README has been updated to point to both, so that next-me, or anyone else who eventually pokes at the codebase, has a fighting chance of doing it correctly.

In short

Taken individually, none of these changes is a headline. Taken together, however, they leave the site noticeably more consistent, more accessible, and easier to extend — which is, I think, what most maintenance work ought to look like when it is going well.

If you notice anything that looks broken or behaves oddly after this round of changes, please do let me know. Otherwise, I will keep working through the small stuff in the background, on the assumption that the cumulative result of doing so over time tends to be worth more than any single big feature would have been.

The wound it pointed at had been described well in an Economic Times editorial titled “The PM gets it right” (Feb 15, 2011). The diagnosis there was simple and, I think, still essentially correct: we had modernised inputs but not marketing, and what existed between the farmer and the consumer was an inefficient chain in which retail price spikes rarely sent any of that upside back to the farm. The editorial was making a policy argument, but underneath it was an economics one — that the marketing layer was, on balance, doing more harm to the producer than the production layer was doing good.

Fifteen years later, I want to revisit that post with newer facts and a clearer view of what has changed and what has not. A fair amount of policy has happened since then, and a fair amount of physical infrastructure has been built; whether either of these has reached the smallest farmer in the chain is the harder question, and the one I want to think about here.

Where the value leaks

The way I have come to think about India’s post-harvest problem is as a collision between two things that do not get along: produce that decays quickly, and a chain that coordinates poorly. Most of the country’s marketed agricultural output passes through a long, multi-step path — harvesting, sorting, grading, transport, wholesale, retail — and each step adds time, friction, and another opportunity for the farmer’s bargaining position to deteriorate. Because the produce itself is decaying with every hour spent in the chain, the side with the least leverage, which is almost always the farmer, has the strongest incentive to settle quickly, on whatever terms are available.

Four chokepoints, in particular, decide how much value reaches the producer. The first is time, in the most physical sense: perishable produce punishes delay, and any link in the chain that adds hours adds losses. The second is scale, because a single farmer with a single field has very little leverage when negotiating with aggregators, traders, or retail buyers, while a collective with many fields and a shared facility has meaningfully more. The third is credit, since being able to wait — to not sell on the day of harvest at whatever price the market is offering — is fundamentally a financial capability, and a farmer without access to short-term finance is, in practice, a price-taker. The fourth is information, because opaque price signals get manipulated, and even where transparency exists it is uneven across mandis, states, and crops.

Back in 2011, I quoted a line from that editorial that I think is still worth repeating, because it captures all four of these in a single sentence: the linkage between the farmer and the consumer is inefficient, wasteful, and subject to manipulation, so shortages trigger hoarding and price spikes at the consumer end without sending those higher prices back to the farmer. The economic point underneath that sentence is unkind but accurate. When the price signal does not reach the farm, the incentive to invest in better acreage, better inputs, or better husbandry erodes, and the long-run consequence is lower productivity than we should otherwise be capable of.

So what has actually changed between 2011 and 2026? There is enough to take seriously, and not enough yet to be complacent about. The next sections walk through what I think are the five most consequential changes — what each is, what it has and has not done, and where it sits relative to the chokepoints described above.

If you grant the four chokepoints described above, then it follows that perishables are not really being bought and sold in the way other goods are; they are being raced against spoilage, and most of what we call the “post-harvest system” is really an attempt to manage that race. Cold chain, warehousing, credit, and transparent markets are, on close inspection, all doing the same underlying work — they are buying time, either physically by slowing decay or financially by reducing the pressure to sell at any price today.

That, viewed in this way, is the main story of India’s post-harvest sector since 2011. We have begun, at significant scale though still unevenly, to build and to finance time. The five sections that follow are the five places where I think this is most visible.

Change 1: We started paying for the boring parts

The Agriculture Infrastructure Fund (AIF) is a government financing facility launched in 2020-21 to support post-harvest and farm-gate infrastructure through interest subvention and credit guarantee support. The instrument is worth dwelling on for a moment, because the choice to attack the problem with a financing facility — rather than with another round of input subsidy — is itself a quiet acknowledgement that the bottleneck has shifted from the farm to everything that happens after it.

As of June 30, 2025, the Press Information Bureau (PIB) reported ₹66,310 crore sanctioned under AIF for 1,13,419 projects, mobilising ₹1,07,502 crore of investment, including 2,454 cold storage projects.

This is not a philosophical shift so much as a cash-flow one, and supply chains, for all their physical hardware, ultimately run on cash flow. If a sorting yard, a small cold room, or a pack-house cannot be built because the cooperative or entrepreneur behind it cannot get five-year money on reasonable terms, the fact that we know cold chain is good for the country is of no use to anyone.

Change 2: Cold chain scaled, and policy got more specific

The Ministry of Food Processing Industries (MoFPI) runs the Integrated Cold Chain and Value Addition Infrastructure (ICCVAI) scheme under the Pradhan Mantri Kisan Sampada Yojana (PMKSY).

As of June 2025, 395 integrated cold chain projects had been approved since 2008, with 291 operational, creating preservation capacity of 25.52 lakh metric tonnes (LMT) per year and processing capacity of 114.66 LMT per year. MoFPI also notes a key policy shift in June 2022: support for fruit and vegetable cold chain projects under that scheme component was discontinued, and the sector was shifted to Operation Greens.

In plain terms, the government has stopped pretending that every crop has the same bottlenecks. Grains, oilseeds, fruits, and vegetables have very different storage curves, very different price cycles, and very different value chains, and a single instrument was never going to serve all of them well. Splitting fruits and vegetables out into a more specialised programme is overdue, and is the sort of unglamorous, mid-level policy adjustment that does not make headlines but probably matters more than most things that do.

For a farmer growing perishables, cold chain is what turns the choice from “sell today or lose everything” into “sell when the price is reasonable” — a small change in framing that completely changes the bargaining position of the producer, and is, in fact, the most direct attack on the perishability chokepoint that any policy lever can deliver.

Change 3: Markets became more digital, and more visible

The electronic National Agriculture Market (e-NAM) integrates regulated wholesale markets (mandis) into an online trading platform.

PIB reported that 1,522 mandis were integrated as of June 30, 2025, with 1,79,41,613 farmers and 4,518 Farmer Producer Organisations (FPOs) registered, and total traded value of ₹4,39,941 crore recorded on the platform.

If 2011 was about price signals being weak, then e-NAM is the most credible attempt to date to strengthen them. A digital common market, in a country with the geographical and linguistic spread of India, matters more than its physical equivalent ever could, because it lets a buyer in one state discover a price in another without either of them needing to travel, and because it exposes the spread between mandis to the producer in a way that previously sat with traders alone.

It is worth being honest about what this can and cannot do, however. A stronger price signal does not automatically fix the physical chain underneath it. A dashboard, however well-designed, cannot refrigerate produce, and digital trading does not, by itself, move a truck across a state line. What e-NAM is good at is reducing one specific chokepoint — information — and that is enough to make it worthwhile, but it cannot reasonably be expected to do work that other parts of the system have to do.

Change 4: Farmer collectivisation moved from slogan to infrastructure

A Farmer Producer Organisation (FPO) is a farmer collective that can aggregate produce, negotiate, and invest in shared capabilities.

The “Formation and Promotion of 10,000 FPOs” scheme hit the 10,000 milestone by Feb 2025, according to PIB. The same release describes equity grants and credit guarantee cover supporting thousands of FPOs.

This matters more than the headline number suggests, because it is the most direct assault on the scale chokepoint described earlier. A single farmer is, by virtue of being one farmer, forced to accept the chain as it is — they cannot afford a sorting line, they cannot finance a cold room, and they cannot threaten to walk away from a buyer because they have nowhere else for their crop to go. A collective changes all three of those conditions simultaneously: it can build shared infrastructure that no individual member could afford, it can absorb buyer-side delays because it is not financially fragile in the same way, and it can negotiate as a credible counterparty rather than as a price-taker.

Produce handled by an FPO can therefore be graded, packed, stored, and sold with leverage that no individual farmer would have on their own — same crop, same season, but a meaningfully different position in the chain.

Change 5: Storage plus credit got sharper

If, in 2011, the farmer’s biggest enemy was forced timing, then by 2026 the problem has slowly mutated into a different one: whether waiting can be financed at all. Storage, in physical terms, has been built; what was missing for a long time was a way to convert that storage into liquidity for the farmer who owned the produce sitting in it.

The Warehousing Development and Regulatory Authority (WDRA) oversees warehousing regulation and electronic warehouse receipt systems, and a Parliamentary Standing Committee report notes the steady growth of pledge finance against electronic Negotiable Warehouse Receipts (eNWRs). The same report notes the launch on 04-03-2024 of e-Kisan Upaj Nidhi, a digital gateway developed by WDRA in association with NABARD (National Bank for Agriculture and Rural Development) and a task force in SBI (State Bank of India), to connect eNWR with onboarded banks.

Of all the changes since 2011, this is the most direct antidote to the original problem, because if a farmer can store and borrow against the stored produce, they are no longer forced to sell at the worst possible moment. The eNWR-plus-credit stack is, in effect, the financial glue that makes the physical investment in cold chain and warehousing actually reach the producer rather than the trader.

The uncomfortable part: loss is still huge

Even with better financing, more cold chain, more digital markets, and stronger farmer collectives, the scale of loss in the system remains uncomfortably large.

A 2024 policy brief by the Indian Council for Research on International Economic Relations (ICRIER) cites a NABARD Consultancy Services (NABCONS) 2020-2022 study estimating food loss in India at about ₹1.53 trillion (USD 18.5 billion) annually due to post-harvest losses (PHL). The brief also makes the broader argument that, on the margin, reducing PHL is often more cost-effective than producing more food only to lose more of it.

So we have, in fairness, built much of the scaffolding that the 2011 critique implicitly demanded. Yet we are still losing value at industrial scale, and the gap between scaffolding and outcomes is where the next decade of work has to happen — not in announcing new schemes, but in making the schemes that already exist actually reach the smallest producer in the chain.

A 2026 reading

The 2011 editorial argued that policy must take marketing as seriously as production. I agreed with that then, and I still agree with it now, but with a slight update to the framing. India’s post-harvest problem is best understood as a systems problem, in which incentives, physics, and finance combine — usually against the weakest player in the chain. It is rarely a single villain that captures the value; it is the cumulative friction of a chain in which every link is a little tilted against the producer.

Each of the four chokepoints I named earlier maps, more or less, onto one of the policy levers that has emerged in the years since 2011. Perishability — the time problem — is being addressed by cold chain investment under ICCVAI/PMKSY and, increasingly, by Operation Greens. The scale problem is being addressed by the FPO programme and the slow consolidation of farmer collectives. The credit problem is being addressed by the AIF on the infrastructure side and by the WDRA’s eNWR-plus-e-Kisan Upaj Nidhi stack on the working-capital side. And the information problem is being addressed, imperfectly but steadily, by e-NAM. None of these levers is a complete solution on its own, but together they describe a more coherent post-harvest stack than India had at any point before 2011.

The honest test, however, is not whether the schemes exist; it is whether the smallest farmer in the chain still has to make the panic-sale choice on the day of harvest. As long as the answer for too many of them is yes, the work the 2011 editorial called for is unfinished, regardless of how good the dashboards look. If, fifteen years from now, the farmer finally has the same thing the consumer has had all along — a real choice about when and to whom to sell — then the work begun in 2020-21 will have done what it set out to do. We are not there yet, but for the first time in a long while, we are at least pointed in the right direction.

References (links)

1. Original 2011 post on SystemHalted:
   https://systemhalted.in/2011/04/26/whats-wrong-with-our-post-harvest-agricultural-supply-chain/

2. Quoted editorial (2011, original link from my post):
   https://economictimes.indiatimes.com/opinion/et-editorial/the-pm-gets-it-right/articleshow/7499198.cms

3. PIB note on Agriculture Infrastructure Fund (AIF) status (as of 30 June 2025):
   https://www.pib.gov.in/PressNoteDetails.aspx?ModuleId=3&NoteId=154999

4. PIB release on e-NAM registrations and traded value (as of 30 June 2025):
   https://www.pib.gov.in/PressReleasePage.aspx?PRID=2151361

5. PIB release on the 10,000 Farmer Producer Organisations (FPOs) milestone (Feb 28, 2025):
   https://pib.gov.in/PressReleasePage.aspx?PRID=2106913

6. Integrated Cold Chain and Value Addition Infrastructure (ICCVAI) status note (PDF; includes June 2025 stats):
   https://static.pib.gov.in/WriteReadData/specificdocs/documents/2025/oct/doc20251029679501.pdf

7. Parliamentary Standing Committee report (PDF; mentions eNWR pledge finance and e-Kisan Upaj Nidhi launch):
   https://sansad.in/getFile/lsscommittee/Consumer%20Affairs%2C%20Food%20and%20Public%20Distribution/18_Consumer_Affairs_Food_and_Public_Distribution_2.pdf?source=loksabhadocs

8. ICRIER Policy Brief 20 (PDF) summarising the NABCONS 2020-2022 loss estimate:
   https://icrier.org/pdf/Policy_Brief_20.pdf

9. WDRA page describing e-Kisan Upaj Nidhi:
   https://wdra.gov.in/web/wdra/e-kisan-upaj-nidhi

Notes

Abbreviations introduced in this post

• AIF: Agriculture Infrastructure Fund
• PIB: Press Information Bureau
• MoFPI: Ministry of Food Processing Industries
• PMKSY: Pradhan Mantri Kisan Sampada Yojana
• ICCVAI: Integrated Cold Chain and Value Addition Infrastructure
• LMT: lakh metric tonnes
• e-NAM: electronic National Agriculture Market
• Mandi: regulated wholesale agricultural market
• FPO: Farmer Producer Organisation
• WDRA: Warehousing Development and Regulatory Authority
• eNWR: electronic Negotiable Warehouse Receipt
• NABARD: National Bank for Agriculture and Rural Development
• SBI: State Bank of India
• NABCONS: NABARD Consultancy Services
• ICRIER: Indian Council for Research on International Economic Relations
• PHL: post-harvest losses

What actually ran

A year, in the end, does not care very much about intent. What it does pay attention to is what actually ran. You can mean well for months on end and still ship nothing of real value, and you can equally well ship one small thing each week and find, by the time December comes around, that you have quietly built up a pile of evidence you did not realise you were accumulating.

My goal for 2026, then, is not reinvention. I am old enough now to mistrust grand reinventions, which I have rarely seen survive contact with February. What I want instead is evidence — a steady, unspectacular sequence of small, honest commits, in code and in everything else.

What I am optimizing for

What I do not want, on the other hand, is a year that looks impressive from a distance and feels hollow up close. I have had years like that before, and they leave very little behind that is worth keeping. What I want instead is a year in which the work is sturdy, the learning is deliberate, the writing is honest, and my attention — increasingly the scarcest resource of all — is mine to direct, rather than something rented out to whatever happens to be loud at the moment.

I would rather end the year with fewer open loops and more finished sentences than the other way around.

Building, without worshipping the build

Engineers, I have noticed, tend to confuse motion for progress. Activity feels like virtue, busy easily passes for important, and the small thrill of being needed in several places at once does a remarkably good job of disguising itself as actual work. I have been guilty of this often enough to recognise it the moment it begins to happen.

This year I want more stillness around the work — less frantic context-switching, and more of the kind of deep, uninterrupted time in which an idea actually becomes real. Real, after all, is the only feature that users can actually use, and it is a feature that almost never emerges from a calendar full of fifteen-minute fragments.

A small promise to myself

A few specific commitments, in plain English. I want to read on purpose this year, rather than leaving it to the algorithm to decide what I think about. I want my learning to have a map, however rough — a sense of where I am going and why I am going there — and I want to keep writing even on the days when the mood for it has gone elsewhere, because craft, in my experience, is what is left over after motivation has left the room.

I also want to treat health as a prerequisite for all of the above, rather than as something I will get to once the work calms down. Work, I have learned the hard way, never calms down on its own; it has to be either kept in proportion deliberately, or accepted on its own terms.

I want to be ambitious without being brittle, which is to say I want to aim for the kind of progress that survives a bad day, a bad week, and the occasional bad month, rather than the kind of progress that depends on everything going well to even register.

And, on a more concrete note, I want to finally ship Consumption Backlog this year — and an iOS version of it, too, if I can find the time.

Closing thoughts

If you are reading this, my hope for you is that 2026 brings you, in some combination, three things: something to learn that changes how you see the world, someone to love who makes the days a little lighter, and something to build that makes you quietly proud when no one in particular is watching. Those are the three I have come to think matter most, in roughly that order, though the order has a way of shuffling depending on the year.

Happy New Year. To whatever extent one of those three is within reach today, that is probably the right place to start, and small is probably the right size to start at.

My core claim is simple: engineers who set up a task with clear instructions, a thin prototype, and hard guardrails tend to have a better experience with AI-assisted coding, and they tend to keep their good faith in the tools. Engineers who don’t, often walk away with confusion, rewrites, and a lingering sense that the agent is “untrustworthy.” And that’s why the same tool creates opposite stories, depending on who’s holding it.

This post is my framework, Discipline First: a trust pipeline for AI-assisted coding. It’s a small kit you can use immediately: an Agent Brief that makes intent hard to misread, guardrails that make failures visible early, and a one-week experiment that turns belief into evidence.

The axis that matters

The agent isn’t the methodology. Your engineering habits are.

I’m going to be deliberately boring about definitions, because the labels are less important than the axis. Whether you call it vibe coding, agentic coding, or AI-assisted coding, the same split shows up: disciplined delivery versus undisciplined delivery. The tools can draft code, refactor code, even propose architectures, but they can’t rescue a vague task from its own vagueness.

Discipline is what turns “the agent wrote something” into “the system changed, and we can explain why, prove it works, and undo it safely if it doesn’t.”

Four engineers walk into the same tool

And once you see it that way, four kinds of engineers show up around AI-assisted coding.

The Skeptic is driven by quality, security, and maintainability. They are not anti tool. They are pro standards. They will try these tools in a sandbox, or allow them under strict review, and they trust what they can validate through time-tested discipline: tests, contracts, architecture checks, and clean interfaces. Their posture is “prove it,” and their lingering doubt is usually about authorship. They suspect only human engineers can reliably meet that bar.

The Dismisser opts out early. Sometimes that’s reflexive, sometimes it’s reasoned: they’ve seen bad suggestions, security risks, legal uncertainty, vendor lock-in, or unreviewable diffs and they decided the trade is not worth it. Their posture is still “already decided,” but the why matters. You don’t convert a Dismisser by arguing about models or demos. You convert them by giving them control of the bar: let them define the quality gates, then run a small, measured experiment in their own codebase that either meets the bar or fails honestly.

The Viber loves speed. They ship fast, accept large diffs, and skip the guardrails that make code testable and observable. Their posture is “speed is truth.” To be fair, that posture has a place: spikes, prototypes, throwaway demos, learning a new stack. The problem is when the same vibe crosses the border into production. That’s where “it seems to work” quietly becomes regressions, mystery failures, and eroded trust. The point of Discipline First is not to shame exploration. It’s to prevent avoidable damage when the stakes are real.

The Disciplined Builder loves AI-assisted coding and still engineers hard. Small tasks, small diffs, acceptance criteria, tests, verification loops, rules files, and a security mindset. They do not care whether the code was written by a human or an agent. They care that it is explainable, reviewable, testable, and observable. Their posture is simple: trust is built.

The only difference between the Skeptic and the Builder is what they believe about that last mile. The Skeptic thinks only humans can deliver it consistently. The Builder has learned how to make the agent earn it.

Discipline First is XP with a faster pair

Discipline First is not a new religion. It is Extreme Programming adapted to a world where your pair can write code at absurd speed. Extreme Programming starts with Values, because Values guide Principles, and Principles are what make Practices hold up under pressure.

The Values are Communication, Simplicity, Feedback, Courage, and Respect.

Communication becomes explicit intent: the Agent Brief and rules files are how you communicate without mind reading. Simplicity becomes a principle you enforce: assume simplicity, slice work into small tasks, keep diffs small, keep releases small. Feedback becomes rapid and objective: failing tests and continuous integration tell the truth early. Courage becomes disciplined restraint: stop the agent when it starts guessing, delete generated code when it bloats, and ship in increments so reality can correct you fast. Respect becomes engineering for humans: keep standards non negotiable, make changes transparent, and leave behind code that future people can understand, test, observe, and safely change.

From those Values and Principles, the Practices follow naturally: pair programming with the agent as the pair and the human as the driver, test driven development as the spec the agent must satisfy, continuous integration as the always on referee, and small releases as the safest way to turn speed into reliability.

The Discipline First kit

1) The Agent Brief

Think of the Agent Brief as a PRD that’s small enough to fit in your head, but sharp enough that the agent can’t “creatively interpret” it.

1. Goal
2. Non-goals
3. Constraints
4. Interfaces
5. Acceptance criteria
6. Risks
7. Test plan
8. Observability
9. Dependencies
10. Recovery and blast radius

Here’s a concrete example (for one service):

Goal: Add rate limiting to POST /payments to reduce abuse and protect downstream dependencies.
Non-goals: No UI changes. No new auth scheme. No changes to other endpoints.
Constraints: Must not change the public API contract. Must keep p95 latency impact under 5%.
Interfaces: POST /payments only; configuration via env var PAYMENTS_RATE_LIMIT_RPS.
Acceptance criteria: Requests above limit return 429 with standard error body; limits are per-customer; logs include rate-limit decision.
Risks: False positives blocking legit traffic; misconfigured limits; uneven behavior across instances.
Test plan: Add functional tests for 200, 429, and boundary conditions; include concurrency test; all tests deterministic in CI.
Observability: Emit metric payments.rate_limited.count; structured log rate_limit_decision with customer id hash; dashboard alert on spikes.
Dependencies: Redis (or in-memory) limiter library already approved; no new infrastructure.
Recovery and blast radius: Feature flag the limiter; default off; rollback is flag flip; document emergency disable procedure.

Then three hard rules.

1. Do not change public APIs unless explicitly permitted.
2. Prefer the smallest diff that can satisfy the brief.
3. Stop and ask when uncertain.

2) Guardrails

Not vibes. Guardrails.

Unit tests and functional tests. Contract checks. Architecture checks. Code review. Small tasks. CI as referee.

If an agent can ship code faster than a human, your only sane response is to make the truth show up faster than the code. We have already seen public “trust cliff” moments when autonomy meets weak guardrails.¹²

Small diffs are not about being precious. They are about control. A small change is easier to review, easier to reason about, easier to test, and easier to roll back. Agents tend to expand scope unless you constrain them, so “small diffs” is both a safety boundary and a way to keep context from exploding.

3) Durable instructions

Modern agentic tooling is quietly reinventing the same old idea: durable project instructions.

Rules files in agentic IDEs make behavior persistent across prompts.³
Repo-level instruction files like AGENTS.md make “how to behave here” predictable.⁴⁵
For conventions that should survive editors and IDEs, EditorConfig gives you portable, version controlled style rules.⁶⁷⁸

Practical rule of thumb:

Use EditorConfig for formatting and style that must survive editors and IDEs.
Use Rules files for tool-specific behavior. What to include in context, how to respond, what not to touch.
Use AGENTS.md for repo-wide agent behavior. Setup commands, test commands, conventions, safety boundaries.

If you only do one thing, do AGENTS.md plus your Agent Brief template. That creates a stable baseline even when prompts change.

A one-week experiment (tests only)

Pick one service. Pick one real user workflow in that service. Then write missing functional tests until the behavior is pinned down.

You will know it’s working when:

The tests fail for the wrong behavior and pass for the right behavior.
They run repeatedly without flakes.
They do not depend on environment quirks or real external systems.
CI stays green and the suite stays fast enough to run often.
Any production code change made only for testability is small and justified.
A human reviewer can read the tests as a spec and agree they capture real behavior.

Measure the boring things: how many rewrites the agent needed, how many times you had to restate intent, how large the diffs got, how much review effort it took, and how quickly you got to “tests green.”

When Discipline First is overkill (and when it breaks)

Discipline First is the safest on ramp for production work. It is not mandatory ceremony for everything.

It is overkill for throwaway prototypes, exploratory spikes, and one-off scripts where the cost of failure is low and the code has no future. In those contexts, “vibes first” can be a valid way to learn quickly.

Discipline First breaks down in predictable ways:

The Brief rots. Intent changes but the kit does not.
Tests become performative. They pass but don’t pin behavior.
Guardrails turn into friction without signal. Slow, flaky, or mis-scoped checks.
The agent is allowed to widen scope. Diffs balloon, review becomes theater.

The fix is the same as always: tighten the slice, keep the checks honest, and scale the process to the risk.

Why measure at all

Because productivity gains are not guaranteed.

One of the clearest public data points so far is a randomized controlled trial from METR (published July 2025) studying experienced open source developers working in codebases they already knew. When AI tools were allowed, developers expected big speedups and later felt faster, but measured completion time was slower on average in that setting.⁹¹⁰¹¹

That result does not mean “AI slows everyone down.” Tools and workflows change fast, task types vary wildly, and many teams report real gains. It means your intuition is not an instrument. Discipline First is not “trust the agent.” It’s “instrument the work.”

Close the loop for each persona

For the Skeptic: run one Discipline First experiment on tests only in a sandboxed branch.

For the Dismisser: pick one internal task, define your own quality gates, and let the experiment decide.

For the Viber: no big diffs, and every change must come with a failing test first.¹²

For the Builder: make discipline the default - publish the Agent Brief template, standardize rules files, and optimize the workflow so the safest path is the easiest path.

Sources and Footnotes

Floating point math is fast, useful, and occasionally haunted. Not philosophically, literally haunted, as in values that aren’t equal to themselves.

NaN != NaN evaluates to true

IEEE 754 formalizes that haunting by defining special values that let computations continue while still signaling trouble. They look like broken math until you realize they’re doing damage control.

Why Special Values Exist in Floating Point

In Part 1 of this series¹, we saw how floating-point numbers work: they use a fixed number of bits to represent the sign, exponent, and significand (mantissa). This representation has limits:

• Largest representable number: Around 1.8 × 10^308 for doubles (Double.MAX_VALUE)
• Smallest positive normalized number: Around 2.2 × 10^-308 (Double.MIN_NORMAL)
• Smallest positive nonzero number: Around 4.9 × 10^-324 (Double.MIN_VALUE)
• Precision: Limited by machine epsilon (about 2.22 × 10^-16 near 1.0, i.e., Math.ulp(1.0) = the gap between 1.0 and the next larger representable double)

But what happens when you compute something that exceeds these limits?

double huge = 1e308;
double overflow = huge * 10;            // Exceeds max value
double gradual = Double.MIN_NORMAL / 2; // Becomes subnormal (gradual underflow)
double underflow = Double.MIN_VALUE / 2; // Falls below min value → +0.0
double undefined = 0.0 / 0.0;           // Mathematically meaningless

IEEE 754 could have made these operations:

1. Throw exceptions (slow, interrupts computation)
2. Wrap around to negative values (confusing, hides errors)
3. Return arbitrary garbage (dangerous)

Instead, it reserves special bit patterns in the exponent field to represent infinity and NaN. These aren’t normal numbers; they are sentinel values that signal “something unusual happened, but computation can continue.”

The Bit Pattern Trick

A double uses 11 bits for the exponent. IEEE 754 reserves special patterns for edge cases:

This means you can check for special values with simple bit operations - no exceptions, no branching overhead in critical inner loops. In Java you normally just use Double.isNaN(x) and Double.isInfinite(x), which are implemented efficiently under the hood.

The Problem: What Should Math Return When It Breaks?

Now that we understand why special values exist (to handle edge cases without crashing), let’s see when they appear.

Consider calculating the average price change across a portfolio:

double totalChange = 0.0;
int validStocks = 0;

for (Stock stock : portfolio) {
    double change = stock.getCurrentPrice() - stock.getPreviousPrice();
    if (isValidChange(change)) {
        totalChange += change;
        validStocks++;
    }
}

double avgChange = totalChange / validStocks; // What if validStocks is 0?

If no stocks had valid data, you’re dividing 0.0 / 0. Should your program:

• Crash immediately?
• Return 0.0 and pretend the average is zero (which is a lie)?
• Return something that screams “this value is meaningless”?

IEEE 754 chose option three. It invented special values so errors can propagate visibly instead of silently corrupting results downstream.

IEEE 754 Special Values

IEEE 754 defines a few “not-a-normal-number” values so computations can keep going in a principled way instead of crashing or silently inventing garbage.

1. Signed Zero: ±0.0

Before we dive into infinity, there’s a subtle detail: IEEE 754 has both +0.0 and -0.0. They both print as 0.0 and compare as equal, but they behave differently in division:

double posZero = 0.0;
double negZero = -0.0;

System.out.println(posZero == negZero);  // true
System.out.println(1.0 / posZero);       // Infinity
System.out.println(1.0 / negZero);       // -Infinity

Signed zero exists so that 1.0 / (tiny positive number → 0) gives +∞ while 1.0 / (tiny negative number → 0) gives −∞. It preserves the direction you approached zero from, which matters for continuity and limit-style reasoning (aka Calculus).

2. Infinity: +∞ and −∞

Infinity appears when a finite result cannot be represented, or when you divide a nonzero number by zero.

double posInf = 1.0 / 0.0;   // +Infinity
double negInf = -1.0 / 0.0;  // -Infinity

System.out.println(posInf);                  // Infinity
System.out.println(negInf);                  // -Infinity
System.out.println(Double.isInfinite(posInf)); // true

Infinity participates in ordering as you’d expect:

• +∞ is greater than every finite number
• −∞ is smaller than every finite number

Arithmetic is mostly “limit-like”:

• finite + (+∞) = +∞
• positive × (+∞) = +∞
• negative × (+∞) = −∞

But some combinations are undefined and produce NaN:

• (+∞) + (−∞) = NaN
• (+∞) × 0.0 = NaN
• (+∞) / (+∞) = NaN

3. NaN: Not a Number

NaN means “this result is undefined,” like 0.0 / 0.0 or √(−1). Once NaN enters a computation, it spreads into any operation involving NaN to produce a NaN:

double nan = 0.0 / 0.0;

System.out.println(nan);           // NaN
System.out.println(nan + 5);       // NaN
System.out.println(nan * 2);       // NaN
System.out.println(Math.sqrt(nan)); // NaN

This “contagious” behavior is intentional. If a value is undefined, any result built on it should also be undefined.

The Weird Rule: NaN ≠ NaN

The key rule that feels like a logic prank but is actually a safety feature:

NaN is not equal to anything, including itself.

double nan = Double.NaN;

System.out.println(nan == nan);        // false (!)
System.out.println(nan != nan);        // true
System.out.println(nan < 5.0);         // false
System.out.println(nan >= 5.0);        // false
System.out.println(Double.isNaN(nan)); // true (correct way)

Why? Because NaN means “undefined result,” and you can’t meaningfully compare undefined values. NaN is unordered by design. There can be many NaN bit patterns (IEEE 754 supports “signaling” and “quiet” NaNs with different payloads), but Java generally treats them as “some NaN” unless you inspect raw bits with Double.doubleToRawLongBits().

Think of it like asking whether two error messages are “the same error.” Even if both say “Error,” you don’t know if they represent the same underlying problem. The comparison itself is meaningless.

Never check for NaN with ==. Use Double.isNaN(x) instead.

NaN and Sorting: Two Different Worlds

This is where things get interesting. Java has two ways to compare doubles, and they behave differently with NaN:

IEEE 754 comparisons (==, <, <=, etc.):

• Any comparison with NaN returns false
• These are what you use in if statements

Java’s total order (Double.compare(), Double.compareTo())²:

• NaN is considered greater than all other values, including +∞
• All NaNs are considered equal
• This is what Arrays.sort() and Arrays.binarySearch() use

double[] values = {3.0, Double.NaN, 1.0, 2.0};
Arrays.sort(values);
System.out.println(Arrays.toString(values)); 
// [1.0, 2.0, 3.0, NaN] -- guaranteed by Java spec

// IEEE comparison says "not sorted":
System.out.println(values[2] <= values[3]); // false (3.0 <= NaN is false)

// Total order says "sorted":
System.out.println(Double.compare(values[2], values[3]) <= 0); // true

The gotcha: If you validate sortedness using <=, you’ll get false negatives when NaN is present. If you need to check ordering, use Double.compare(a, b) <= 0 instead.

IEEE comparisons answer “is this mathematically ordered?” while Java’s total order answers “can we put these in a consistent sequence for sorting?” The Arrays Javadoc basically says exactly that: < is not a total order for doubles, so sorting uses the total order from Double.compareTo.

The good news: Arrays.sort() and Arrays.binarySearch() work correctly with NaN because they use the total order internally. The NaN will consistently end up at the end of the array.

Quick Reference: Comparison Table

Practical Takeaways

1. Never check NaN with ==. Use Double.isNaN(x).
2. Log early for special values when debugging “impossible” totals:

   if (Double.isNaN(result)) {
       log.error("NaN detected at step X");
   }
   if (Double.isInfinite(result)) {
       log.error("Infinity detected at step X");
   }
   

1. Use the right comparison for the job: == for value equality, Double.compare() for ordering.
2. Understand the contagion: Once NaN enters your calculations, it spreads. Trace backward to find the division by zero or invalid operation that spawned it.

The Philosophical Bit

NaN isn’t a bug. It is math raising its hand and saying, politely but firmly:

“I can’t promise anything from here.”

When your balance sheet shows NaN, don’t curse floating point. Ask what division by zero or invalid square root you missed three steps ago. The special values aren’t betraying you; they’re the only honest answer to questions that have no answer.

In the next post, we’ll look at how to actually handle these special cases in production code without littering your logic with endless isNaN() checks.

References and Notes

In my post on associativity and reduce¹, we saw something that feels like a prank. In Example 5, adding 1.0 to 1e16 did not change the value at all.

double x = 1e16;
System.out.println(x + 1.0 == x); // true

That is not Java being cheeky. That is IEEE-754 being literal. A double does not live on a continuous number line. It lives on a grid.

This post answers one question:

How fine is the grid?

Machine epsilon and the first rung above 1.0

One common definition of machine epsilon (ε) is:

The smallest ε > 0 such that 1.0 + ε ≠ 1.0 for double.

This is “the gap from 1.0 to the next representable double above it”.

Note: Some references use ε to mean 2⁻⁵³, which is half this gap and represents the maximum relative rounding error for a correctly rounded operation. In this post, ε means the “next representable number above 1.0” definition, which is 2⁻⁵².

Finding ε in Java

Here is a loop that finds that smallest nudge.

public class MachineEpsilon {
  public static void main(String[] args) {
    double eps = 1.0;

    while (1.0 + (eps / 2.0) != 1.0) {
      eps /= 2.0;
    }

    System.out.println("epsilon = " + eps);
  }
}

Typical output on an IEEE-754 JVM:

epsilon = 2.220446049250313E-16

That value is exactly 2⁻⁵². Since powers of two are exactly representable in binary floating point, there is no approximation error in storing this value. The decimal string 2.220446049250313E-16 is just how Java renders that exact binary value for display, rounded to about 15 decimal digits.²

One quick trap: Double.MIN_VALUE is not machine epsilon. Double.MIN_VALUE is the smallest positive double near zero (about 5×10⁻³²⁴). Machine epsilon is about spacing near 1.0.

The ladder model

Picture a ladder laid across the number line.

Near 1.0, the rungs are extremely close together. As the numbers get bigger, the rungs spread out.

Machine epsilon tells you the rung spacing near 1.0. But what you usually want is the spacing near whatever value you are actually using.

That spacing is called ULP, short for Unit in the Last Place.

Java gives it to you with Math.ulp(x).

How spacing grows with magnitude

Let’s sample the grid at a few scales.

public class UlpSpacing {
  public static void main(String[] args) {
    double[] xs = {1.0, 10.0, 1e8, 1e16};

    for (double x : xs) {
      System.out.printf("x=%-8s  ulp(x)=%s%n", x, Math.ulp(x));
    }
  }
}

Typical output:

x=1.0      ulp(x)=2.220446049250313E-16
x=10.0     ulp(x)=1.7763568394002505E-15
x=1.0E8    ulp(x)=1.4901161193847656E-8
x=1.0E16   ulp(x)=2.0

That last line is the whole “Example 5” mystery solved:

Around 1e16, the grid spacing is 2.0.

So 1e16 + 1.0 lands between rungs and rounds back to 1e16. But 1e16 + 2.0 is exactly one rung up.

double big = 1e16;
System.out.println(big + 1.0 == big); // true
System.out.println(big + 2.0 == big); // false

Powers of two: where spacing jumps

ULP does not grow smoothly. It jumps at powers of two.

Right below 2ᵏ, spacing is one value. At 2ᵏ, spacing doubles.

That is why comparisons that seem symmetric can behave oddly if two values straddle a power-of-two boundary. If you’re comparing values near different powers of two, their ULPs can differ by a factor of 2.

Near zero: subnormals exist and they are weird

For very small magnitudes below Double.MIN_NORMAL (approximately 2.225×10⁻³⁰⁸), double switches to subnormal (also called denormal) representation.

What changes in subnormal land:

Normal doubles have an implicit leading 1. in the mantissa:

• value = (1.fraction) × 2^exponent
• This gives you full precision

Subnormals drop that leading 1.:

• value = (0.fraction) × 2^(minExponent)
• You lose precision gradually as you approach zero

Why they exist:

Without subnormals, there would be a hard cliff from tiny normal numbers straight to 0.0. Subnormals provide gradual underflow - a ramp instead of a cliff.

Key differences:

• Spacing becomes constant at approximately 5×10⁻³²⁴ (the value of Double.MIN_VALUE) rather than scaling with magnitude
• Arithmetic can be slower on some CPUs
• Relative precision is much worse (you may have only a few significant bits left)

Everything in the range (0, Double.MIN_NORMAL) is subnormal. That’s the range from about 4.9×10⁻³²⁴ up to about 2.225×10⁻³⁰⁸.

Most business code never goes near subnormals. Numerical code sometimes does. It is worth knowing that floating-point has an emergency mode near zero that trades precision for continuity.

Why tiny increments vanish when numbers get big

When your running total grows large enough, the local rung spacing can become bigger than the increments you are adding.

So “add a million tiny things to a huge sum” eventually turns into “add nothing, repeatedly,” because the tiny things fall between rungs and get rounded away.

That is not philosophical. It is mechanical.

Why equality checks on doubles are dicey

Sometimes two values that “should be different” land on the same rung. Sometimes two values that “should be equal” get rounded at different times and land on adjacent rungs.

So == is only safe when you mean exact equality:

• Comparing to exact literals: 0.0, 1.0, -1.0
• Checking for special values: infinities, or Double.isNaN(x)
• Comparing sentinel values or results from identical deterministic operations
• Loop counters stored as doubles (though you should use integers instead)

When you need “close enough,” you need a rule that matches your domain.

A common practical pattern is absolute tolerance near zero plus relative tolerance for scale.

public final class DoubleCompare {
  private DoubleCompare() {} // prevent instantiation

  /**
   * Check if two doubles are nearly equal using absolute and relative tolerance.
   * 
   * @param a first value
   * @param b second value
   * @param absTol absolute tolerance (try 1e-9 for many applications)
   * @param relTol relative tolerance (try 1e-9 for many applications)
   * @return true if values are within tolerance
   */
  public static boolean nearlyEqual(double a, double b, double absTol, double relTol) {
    if (Double.isNaN(a) || Double.isNaN(b)) return false;
    if (a == b) return true; // handles infinities and exact matches

    double diff = Math.abs(a - b);
    if (diff <= absTol) return true;

    double maxAbs = Math.max(Math.abs(a), Math.abs(b));
    return diff <= relTol * maxAbs;
  }
}

Choosing tolerance values:

• absTol should match the minimum meaningful difference in your domain. For scientific data measured to 3 decimal places, maybe 1e-3. For pixel coordinates, maybe 0.5.
• relTol is typically something like 1e-9 (about 9 decimal digits of agreement) for general use, or 1e-6 if you’re being more lenient.
• These comparisons are slower than ==. If you’re comparing millions of values in performance-critical code, measure the cost.

This is not the only strategy, but it is harder to misuse than an “ULPs everywhere” helper.

Why parallel reductions can drift

Parallel reductions regroup operations. Floating-point addition is not associative, so regrouping changes when rounding happens.

Here is the smallest “this is why” example.

double a = 1e16;

double left  = (a + 1.0) + 1.0;   // first +1 vanishes, then second +1 vanishes
double right = a + (1.0 + 1.0);   // (1.0 + 1.0) becomes 2.0, which moves one ULP

System.out.println(left == right); // false
System.out.println(left);          // 1.0E16
System.out.println(right);         // 1.0000000000000002E16

Same values, different grouping, different result. That is the reason parallel sums can drift when the data has large magnitudes or mixed scales.

The takeaway

A double gives you roughly the same number of significant bits everywhere (about 15-16 decimal digits), not the same absolute resolution everywhere.³

Machine epsilon tells you the first rung above 1.0.

Math.ulp(x) tells you the rung spacing where you are standing.

And that is why, at 1e16, adding 1.0 is like whispering into a hurricane.

References

In my earlier post on IEEE 754 doubles I showed how a tiny Java example could break your intuition about numbers. The JVM was not being sloppy. It was faithfully following the floating point rules. The surprise came from my mental model, not from the hardware.

BigDecimal is Java’s answer to a different problem: what if I actually need decimal correctness, not fast binary approximation? It is the type you reach for when cents matter, reconciliation matters, or auditors matter.

It is less magical than it looks.

TL;DR: Use new BigDecimal("0.1") for decimal values in your code. Only use BigDecimal.valueOf(double) when you’re already stuck with a double from external sources.

Quick Reference

Before we dive in, here’s what you need to remember:

  
// ✓ Correct ways to create BigDecimal for money  
new BigDecimal("0.1");                   // String literal  
new BigDecimal("123.45");                // String literal  
BigDecimal.valueOf(10).movePointLeft(1); // 10 × 10^-1 = 1.0  

// ✗ Wrong for hardcoded decimal values  
new BigDecimal(0.1);        // Exposes the binary approximation as decimal

Doubles speak binary, your domain speaks decimal

double is brilliant for physics, graphics, simulations, and anything where small error is acceptable. It is terrible at representing human money. The root cause is simple. Doubles are binary fractions. Money is decimal.

0.1 rupee or dollar has no exact representation in binary floating point. When you write:

double x = 0.1;
System.out.println(x);              // prints 0.1 (canonical string)
System.out.printf("%.20f%n", x);    // 0.10000000000000000555...

you are already off by a tiny amount, even though the default printout shows 0.1. Most of the time you are happy to ignore that tiny tail (technically binary approximation). But then you sum millions of rows, or reorder operations, or start comparing for equality, and the tail starts wagging the dog.

BigDecimal cuts across this by working in base 10.

BigDecimal’s mental model

Conceptually, a BigDecimal is two things glued together:

1. An integer representing all the digits, without any decimal point.
2. A scale that says where the decimal point lives.

Formally:

value = unscaledValue × 10^(-scale)

So:

BigDecimal amount = new BigDecimal("123.45");

internally becomes:

• unscaledValue = 12345
• scale = 2
• logical value = 12345 × 10^-2 = 123.45

Because the unscaled integer is exact, decimal values like 0.1, 0.01, 1234567890.12 are also exact. There is no “closest representable value” the way there is with double. You only lose information when you explicitly ask BigDecimal to round (via MathContext or setScale).

How the JDK actually stores it

That is the spec view. Under the hood in OpenJDK, the class looks roughly like this:

public class BigDecimal extends Number
        implements Comparable<BigDecimal> {

    // Compact form when it fits in a long
    private transient long intCompact;

    // Full form when it doesn't
    private BigInteger intVal;

    // Digits after the decimal point
    private int scale;

    // Cached number of significant digits
    private transient int precision;

    // Marker for "no compact long, use intVal instead"
    static final long INFLATED = Long.MIN_VALUE;
}

So BigDecimal actually has two representations for the unscaled value:

• Compact mode: if the unscaled integer fits in a 64-bit long, it lives in intCompact and intVal is null. This is the fast path for “small enough” numbers.
• Inflated mode: if it does not fit, intCompact is set to INFLATED and the digits live in intVal as a BigInteger.

This optimization means small monetary amounts stay fast, while supporting arbitrarily large values when needed.

Your 123.45 example fits happily in compact form:

• intCompact = 12345L
• intVal = null
• scale = 2

Never construct BigDecimal from a double

A classic foot-gun looks like this:

BigDecimal a = new BigDecimal(0.1);
BigDecimal b = new BigDecimal("0.1");

System.out.println(a); // 0.1000000000000000055511151231257827021181583404541015625
System.out.println(b); // 0.1

The first line takes the binary double for 0.1 and converts it directly into an exact decimal. The double is already an approximation, so you get the full fraction printed out.

The second line parses the string "0.1" as a decimal value. There is no binary detour, so you get exactly one tenth.

You have not “fixed” the double by wrapping it in a BigDecimal. You have just made its approximation painfully visible.

What about BigDecimal.valueOf() and canonical strings?

This is where people get confused:

BigDecimal a = new BigDecimal(0.1);
BigDecimal b = BigDecimal.valueOf(0.1);

System.out.println(a);
// 0.1000000000000000055511151231257827021181583404541015625

System.out.println(b);
// 0.1

Same literal 0.1, two different worlds.

The difference is that valueOf goes through the canonical decimal string of the double.

Route 1: new BigDecimal(0.1)

This constructor works directly from the binary bits of the double:

• The double for 0.1 is not exactly one tenth.
• It is some messy binary fraction very close to 0.1.
• new BigDecimal(double) asks: “What is the exact decimal value of this binary fraction?”

So you see the full binary approximation:

0.1000000000000000055511151231257827021181583404541015625

Ugly, but honest.

Route 2: BigDecimal.valueOf(0.1) and canonical decimal strings

valueOf takes a different path:

public static BigDecimal valueOf(double val) {
    return new BigDecimal(Double.toString(val));
}

The key piece here is Double.toString(val). That method does not dump all the internal bits. Instead, it produces the canonical decimal string for that double: