Joist | Blog

Joist 2.3 Released with Rails-style Scopes!

Sat, 08 Aug 2026 00:00:00 GMT

Joist 2.3 is out! (…as of a month ago, we’re behind on the blog post! 😅)

The release highlights are:

Rails-style scopes for naming and composing reusable em.find filters
Scaffolding for GraphQL query resolvers
Paginated em.find calls are now auto-batched
New EnumCollections for many-to-many join tables that point at an enum
Native Temporal support on Node 26, with no polyfill required
Smarter batched queries via constant inlining and boolean pruning
A large round of hot-path performance work, with 20-50%+ speedups across flush, hydration, and em.find micro-benchmarks

Rails-Style Scopes

The headline 2.3 feature is #1892: Rails-style scopes 🎉, which let you name and compose common em.find filters directly on your entity classes.

You declare scopes as static fields, using the scope function:

src/entities/Author.ts

export class Author extends AuthorCodegen {
  static adult = scope({ age: { gte: 18 } });
  static active = scope({ deletedAt: null });
  static popular = scope((a) => a.isPopular.eq(true));
  static hasBooks = scope({ books: true });
  static booksReviewedBy = scope.fn((reviewer: Author) => ({ books: { reviewer } }));
}

And then reuse and chain them throughout your codebase:

await Author.adult.find(em);
await Author.adult.popular.find(em);
await Author.active.where({ firstName: "a1" }).find(em);
await Author.booksReviewedBy(reviewer).find(em);

Scopes are syntax-sugar for Joist’s regular em.find, so they share the same filter syntax and semantics, including relation filters across one-to-many, many-to-one, and many-to-many paths.

A few things scopes can do:

Chain with AND semantics, i.e. Author.adult.popular finds authors that match both the adult and popular scopes
Take parameters via scope.fn, i.e. Author.named("a") or Author.taggedWith("fiction").
Add additional criteria at each call site, i.e. Author.adult.orderBy({ createdAt: "DESC" }).limit(10) or .softDeletes("include").
Drop into em.find anywhere a filter is expected, i.e. em.find(Author, Author.adult) or em.find(Book, { author: Author.adult }).
Composition You can defined new scopes by combining others: static popularAdult = Author.popular.adult.

Similar to regular em.find queries, scopes also support alias for complex predicates (i.e. ANDs and ORs) that a plain inline filter can’t express, like a top-level OR that spans two different joined tables:

export class Author extends AuthorCodegen {
  // Authors with a book that is either titled "b1" or has a 3-star review.
  static titleOrRated = scope((a) => {
    const [b, r] = aliases(Book, BookReview);
    return {
      where: { books: { as: b, reviews: { as: r } } },
      conditions: { or: [b.title.eq("b1"), r.rating.eq(3)] },
    };
  });
}

Joist’s scopes are heavily inspired by Rails scopes 🙏, but are strongly-typed and adapted to fit Joist’s conventions–most notably, scopes always require a trailing .find(em) (or findOne, findOneOrFail, findCount, findIds) so Joist knows which EntityManager to use for loading and identity caching.

See the new Scope Queries docs for the full feature set.

GraphQL Query Scaffolding

Joist has had scaffolding for GraphQL object resolvers & save mutations for awhile now, but we finally got around to adding query scaffolding in #1865.

This means that after adding your authors database table, Joist will (if you’ve enabled the GraphQL scaffolding) codegen not only an Author.ts and AuthorCodegen.ts file, but also will auto-create a schema/Author.graphql file with a basic Author type and a Query extension for fetching authors:

schema/Author.graphql

extend type Query {
  " Newly added in this 2.3 release "
  author(id: ID!): Author!
  " Newly added in this 2.3 release "
  authors(filter: AuthorFilter, limit: Int, offset: Int): AuthorsPage!
}

extend type Mutation {
  " Already scaffolded in earlier releases. "
  saveAuthor(input: SaveAuthorInput!): Author!
}

As well as default implementations authorQuery.ts and authorsQuery.ts:

src/resolvers/author/authorQuery.ts

export const author: Pick<QueryResolvers, "author"> = {
  async author(_, args, ctx) {
    return ctx.em.load(Author, args.id);
  },
};

src/resolvers/author/authorsQuery.ts

export const authors: Pick<QueryResolvers, "authors"> = {
  async authors(_, args, ctx) {
    return paginate(ctx, Author, args);
  },
};

As always, if you need to change any of the scaffolded output, just update it!

Joist will never overwrite your changes.

Batched Paginated Finds

Joist’s em.find has always auto-batched its SELECT queries (part of our N+1 prevention), but previously em.find did not support batching paginated queries, i.e. queries wanting to use OFFSET or LIMIT.

This limitation was not an issue for top-level endpoints doing pagination, like GET /authors?offset=100&limit=50 or query / authors(offset: 100), but it does become an issue if you want to return “only the first N grandchildren” in a nested child, like this GraphQL query:

query {
  authors(filter: { status: "active" }) {
    books {
      " The `reviews` field will invoked once per author, per book "
      reviews(first: 5) {
        rating
      }
    }
  }
}

This reviews(first: 5) is basically “a paginated SELECT in a loop”, which now in #1836 will also be batched into a single SQL query for all children invoking the query.

Because em.find now supports limit and offset, we don’t have a need for legacy em.findPaginated method, which was the previous way to do (unbatched) paginated finds, anymore, so it’s been removed.

Now any em.find can accept limit/offset and still get Joist’s auto-batching:

const [aReviews, bReviews] = await Promise.all([
  em.find(BookReview, { book: b1 }, { limit: 5, orderBy: { id: "ASC" } }),
  em.find(BookReview, { book: b2 }, { limit: 5, orderBy: { id: "ASC" } }),
]);

We covered this in detail in its own Batched Paginated Finds post.

Enum Collections

Joist has historically supported modeling “lists of enums” via int[] enum-array columns, but #1908 adds a second option: a true many-to-many join table between an entity and an enum, modeled as an EnumCollection.

For example, to give Publisher a list of “logo colors”, where Color is an enum in your domain model, create a join table between publishers and the color enum table:

src/migrations/2026-06-29-add-publisher-logo-colors.ts

export function up(b: MigrationBuilder): void {
  createManyToManyTable(b, "publisher_logo_colors", "publishers", {table: "color", column: "logo_color_id"});
}

Joist recognizes that one side points at an enum table and generates an EnumCollection on the entity side (only the entity side; the enum does not get a reverse relation):

src/entities/PublisherCodegen.ts

// Code generated
public class PublisherCodegen extends BaseEntity {
  readonly logoColors: EnumCollection<Publisher, Color> = hasEnumCollection();
}

To the caller it looks like “an array of Colors”, but unlike an enum-array column it’s lazy (because it lives in its own table, it’s not immediately fetched as part of the SELECT * FROM publishers call, so it needs a load hint), and it’s fully integrated with the rest of Joist:

const publisher = await em.load(Publisher, "p:1", "logoColors");
publisher.logoColors.get; // [Color.Red, Color.Blue]
publisher.logoColors.add(Color.Green);
publisher.logoColors.set([Color.Green, Color.Blue]);
await publisher.logoColors.includes(Color.Blue); // probe without loading

It supports filtering (em.find(Publisher, { logoColors: Color.Red })), changes tracking, and reactivity, just like any other collection. See the Enum Collections docs for when to reach for this versus a plain enum-array column.

Native Temporal on Node 26

Joist supports the Temporal API for date/time columns, and historically required the temporal-polyfill package to provide it.

Node 26 ships Temporal natively, so #1889 updates Joist’s native-first / polyfill-fallback detection to transparently use the global Temporal when it’s available, falling back to the polyfill on older Node versions. There’s also a new import { Temporal } from "joist-orm" that resolves to whichever implementation is in use, so codegen and app code don’t need a direct import to either.

Optimized Batched Queries

Two changes make Joist’s batched em.find queries leaner for the Postgres planner.

#1909 adds constant inlining. Previously, every filter value in a batch flowed through the _find CTE, even columns whose value was identical across all the batched calls. Now Joist detects when all batched queries share the same value for a column and inlines it as a plain bound predicate, leaving only the genuinely-varying columns in the CTE:

WITH _find (tag, arg0) AS (
  SELECT unnest($1::int[]), unnest($2::character varying[])
)
SELECT array_agg(_find.tag) as _tags, a.*
FROM authors AS a
CROSS JOIN _find AS _find
WHERE a.deleted_at IS NULL AND a.first_name = $3 AND a.last_name = _find.arg0
GROUP BY a.id

Here first_name = $3 is a plain scalar comparison the planner can satisfy with an index, while only last_name–the column that actually differs between the batched queries–flows through the CTE.

#1879 adds boolean expression pruning, so a common conditional-filter idiom just works:

em.find(Author, {
  name: { ilike: name && `${name}%` },
});

When name is falsy, the ilike: false term is pruned from the query entirely (rather than generating a spurious predicate), while equality operators like eq: false are preserved.

Many-to-Many Tables Without `id` Columns

#1904 lets Joist work with many-to-many join tables that don’t have a surrogate id primary key, i.e. tables whose primary key is just the (foo_id, bar_id) pair. This is a common shape for join tables, and Joist now handles loading, preloading, and mutating their rows just like the id-backed variant.

Performance Work

2.3 includes a large round of hot-path performance work, benchmarked at 100k-entity scale across several independent process runs. The headline numbers (vs. the prior commit, all confirmed):

Faster em.flush dirty checking (#1857): with a mostly-clean unit of work, scanning for dirty entities went from ~4ms to ~0.1ms when nothing dirty (~97%), and from ~9.5ms to ~4.8ms with 100 dirty entities (~49%).
Faster reaction walking (#1857): insert-heavy flushes dropped ~28-47% depending on column count.
Lazier find indexes (#1861): in-memory indexes are now maintained lazily per-field, and only triggered when fields are actually queried, which took a first-build scenario from ~1517ms to ~24ms (~98%), with 30-50% steady-state wins.
Faster auto-batching key calc (#1859): when deduping queries for N+1 prevention, we avoid re-JSON.stringify-ing keys, which dropped high-cost deduping by ~52-58%.
Reactive queue lookups (#1857): ~22-43% faster.
Entity hydration and identity-map loadAll paths: ~17-34% faster.
Already-loaded populate paths: ~22-33% faster.

There’s also a new BENCHMARK-PLAN.md with the methodology and a regression-benchmark suite to keep these from regressing.

Smaller But Notable Changes

A few other changes worth calling out:

#1900 aligns em.find handling of many-to-one and polymorphic behavior when passed empty arrays, em.find({ publisher: [] }) now always means in: [] which means “no rows”.
#1883 excludes em.deleted entities from findCount, and #1871 ensures find does not return pending-delete entities.
#1881 always defines transientFields, which is still only instantiated lazily, but useful for metaprogramming to rely on it.
#1874 caches dataloader lookups/promises during em.populate, and #1868 avoids recursive-property populate deadlocks.
#1873 handles cycles in recursive Properties.
#1891 skips preloading GraphQL fields with arguments (deferring to the resolver instead of unintentionally overloading data).
#1863 restores em.flush return order (broken in a previous perf refactoring), and #1862 keeps loadLens synchronous for already-loaded paths.

There are also misc correctness fixes around STI subtypes, recursive relations, reaction handling, and dependency bumps, plus codegen conflict-avoidance fixes (#1901, #1902, #1903) to keep generated scope/enum/type names from colliding with user symbols.

Upgrading

The main migration notes are:

Replace em.findPaginated calls with em.find, passing the same limit/offset options.
After adding any scope declarations, re-run joist-codegen to refresh the generated <Entity>Scopes types.

Thanks to everyone filing issues, testing next releases, and pushing on Joist’s edge cases.

Batched Paginated Finds

Sun, 21 Jun 2026 00:00:00 GMT

Joist has always been aggressive about preventing N+1s–baking dataloader-style batching “into the data layer” was the original reason we started the Joist project back in ~2020.

This auto-batching means that if you write code like:

// Loop over 100 authors
await Promise.all(authors.map(async (author) => {
  await author.books.load();
}));

Joist sees the repeated books.load() calls (before they go on the wire), batches them together, and issues one SELECT for all the authors’ books.

Similarly, Joist’s em.find is batch-friendly, so if you have several semantically-similar em.finds in flight at the same time:

await Promise.all([
  em.find(Book, { author: a1 }),
  em.find(Book, { author: a2 }),
  em.find(Book, { author: a3 }),
]);

Joist turns that into one batched SQL query instead of three.

One wrinkle was that, historically, this auto-batching did not work paginated finds–i.e. em.find(..., { limit, offset })–until now! 🎉

The GraphQL Use Case

Joist is not a “GraphQL-only” ORM, but we’re definitely “GraphQL-informed”, so let’s look at a GraphQL schema where “lack of paginated finds” is problematic.

Given a typical Author/Book/Review schema:

type Author {
  books: [Book!]!
}

type Book {
  " Allow fetching a subset of the reviews. "
  reviews(first: Int): [BookReview!]!
}

The first argument on reviews allows a client to “limit the book reviews” they retrieve:

query {
  author(id: 1) {
    books {
      reviews(first: 5) {
        rating
      }
    }
  }
}

Normally Joist’s entityResolver can “put the entity on the wire” without any boilerplate, including relations like reviews, but because reviews has this custom first argument, we need implement it with em.find:

export const bookResolvers: BookResolvers = {
  ...entityResolver(getMetadata(Book)),

  async reviews(book, args, ctx) {
    // Normal/boring em.find query
    return ctx.em.find(
      BookReview,
      { book },
      { limit: args.first ?? undefined, orderBy: { id: "ASC" } },
    );
  },
};

However, b/c the GraphQL runtime ends up invoking reviews function “in a loop”, an unbatched implementation of this em.find would cause an N+1, as the query evaluation becomes:

Load author a:1 (good, 1 SQL call)
Load all the author’s books (good, 1 SQL call, returns 100 books)
For each book, load the first five reviews (eh, 100 SQL calls?)

That last line is the scary one. 😬

If the author has 100 books, our initial resolver implementation will issue 100 SELECT ... FROM book_reviews WHERE book_id = ? LIMIT 5 queries–we’ve created an N+1.

Why Pagination is Different

If Joist is so great at auto-batching, why were these paginated em.find calls not batched already?

For Joist’s non-paginated one-to-many loading, batching is really straightforward: we use pretty much the same SELECT * FROM book_reviews that a non-batched SELECT would do, but with a parameter of “an array for all requested book ids” instead of a single book id:

SELECT br.*
FROM book_reviews br
WHERE br.book_id = ANY($1)
ORDER BY br.id ASC

The database returns all reviews for all books, and Joist groups them back by book_id, so we can populate each books reviews collection.

But reviews(first: 5) does not mean “load any five reviews across all books”. It means “load five reviews per book”, i.e. this SQL would be wrong:

-- Wrong SQL
SELECT br.*
FROM book_reviews br
WHERE br.book_id = ANY($1)
ORDER BY br.id ASC
LIMIT 5

The LIMIT 5 applies to the whole result set, so if the first book has five reviews, books two through twenty get nothing.

Given this wrinkle, i.e. it had not been immediately obvious “how to auto-batch paginated queries”, we’d so far pragmatically decided to just not even try. 😅

So, historically, Joist didn’t allow em.find to accept pagination parameters (limit or offset), and instead we had a dedicated em.findPaginated that was specifically not batched, but did accept limit & offset parameters, which callers just had to know to “not call it in a loop”.

The New Pagination SQL

Until now! The latest Joist next release now batches paginated em.find calls.

We achieve this by moving the LIMIT parameter from “a top-level LIMIT” to a “per book LIMIT” using a CROSS JOIN LATERAL statement, which looks somewhat cryptically like this:

WITH _find (tag, arg0) AS (
  SELECT unnest($1::int[]), unnest($2::int[])
)
SELECT _find.tag as tag, _data.*
FROM _find AS _find
CROSS JOIN LATERAL (
  SELECT br.*
  FROM book_reviews AS br
  WHERE br.book_id = _find.arg0
  ORDER BY br.id ASC
  LIMIT $3
) AS _data

The important pieces are:

The _find CTE lets pass us a small in-memory “lookup table” into Postgres, with one row for each of the batched calls
- tag is just a counter of “batch 0”, “batch 1”, “batch 2”
- arg0 is the per-call book_id, i.e. book_id=10, book_id=20, book_id=30 if we’re loading review for books [10, 20, 30]
CROSS JOIN LATERAL runs the inner query once per _find row.
- This means that “each book” gets its own SELECT ... FROM book_reviews ... LIMIT
The LIMIT $3 lives inside the lateral subquery, so it applies per book, not globally.

Tying this back to our GraphQL use case, if a GraphQL query asks for reviews(first: 5) across 20 books, Postgres still receives one review query, and each book gets its own ORDER BY ... LIMIT 5 semantics.

Removing `findPaginated`

Because we’ve now solved the restriction of “we cannot batch paginated finds”, the reason for separate em.find vs. em.findPaginated methods has gone away, so we’ve also just removed findPaginated. 🔪

Now any em.find can use limit/offset parameters, and still get Joist’s robust auto-batching support:

const [aReviews, bReviews] = await Promise.all([
  em.find(BookReview, { book: b1 }, { limit: 5, orderBy: { id: "ASC" } }),
  em.find(BookReview, { book: b2 }, { limit: 5, orderBy: { id: "ASC" } }),
]);

As long as the paginated finds have compatible shapes–same entity, same options like limit / offset / orderBy, and the same logical filter shape–Joist will batch them.

This first-class em.find support also means any service-layer / business-logic code that loops over entities and asks for “the first N children” per entity will get an auto-batched query, even if no GraphQL is involved. 🎉

Why This Matters

This feature finally unlocks a common GraphQL pattern: fetching “the first N children” for a list of parents, without having to write a custom dataloader/resolver or accept an N+1.

Previously, applications using Joist had to choose between:

Avoiding nested first fields all together (honestly what we’d done 😅),
Writing custom dataloader resolvers by hand (👎️), or
Accepting N+1s for “just a small first-five list” fields (also 👎️)

Now the feature “just works”, with a straightforward, boring implementation that leverages em.find, as we continue to deliver on Joist’s “never N+1” mission. 🚀

Simplest Possible Closure Tables

Sun, 07 Jun 2026 00:00:00 GMT

Joist is a TypeScript ORM that lets you declaratively implement your business logic, which we explore in this post by examining this snippet of code:

class Employee {
  /** Create an employee -> all transitive managers closure/m2m table. */
  readonly managersClosure: ReactiveManyToMany<Employee, Employee> =
    hasReactiveManyToMany(
      "managersRecursive",
      (e) => [e, ...e.managersRecursive.get],
    );
}

Which we assert is the simplest possible implementation of closure tables. 💥

It achieves this by combining two of Joist’s primitives:

Recursive collections (the managersRecursive relation), and
Reactive infrastructure like hasReactiveManyToMany

We’ll look at each of these, and also explain closure tables while we’re at it.

Closure Tables

Closure tables are a niche but well known way to store hierarchical data (trees, or arbitrarily nested parent/child data) in a relational database that makes it easy to query “up or down multiple levels of the tree” with a single regular/fast join.

For example, if we have an employees table with a manager_id column, some employees might have “just one manager” (they report directly to the CEO), while other employees might have “their manager, their manager’s manager, their manager’s manager’s manager”, etc. up to the CEO.

Historically, given the “static” nature of SQL queries (i.e. no for loops), queries for “all the transitive managers for this employee” were difficult/impossible to query using just the manager_id column by itself.

Closure tables are a pattern of storing data that make these “find all transitive managers” queries easy.

They do this by, at write-time when manager_id is changed, duplicating each hop from an employee to every single transitive manager as derived data–they don’t replace the manager_id column, they’re calculated from it.

For an example like Fred reports to Bob reports to Jill, the closure table approach stores:

fred.managersClosure=[fred, bob, jill]
bob.managersClosure=[bob, jill]
jill.managersClosure=[jill]

I’m purposefully representing these as “each employee has a collection of ‘all their managers’” because I think that’s the easiest to mental model to reason about.

In contrast, most closure table articles jump to “what does the managers_closure table look like in the database”, which is something like:

Which is correct, but for me was hard to initially reason about.

Instead, I think it’s most intuitive to articulate the “extra table” (the “closure table”) we’re adding as “each employee has a m2m between themselves and all their managers”–because that’s really what it is. 😅

Now with this managers_closure table we can “find all managers” with a regular/boring SQL query with a single INNER JOIN:

SELECT e.*
FROM managers_closure mc
JOIN employees e ON e.id = mc.ancestor_id
WHERE mc.descendant_id = :employee_id
AND mc.ancestor_id <> mc.descendant_id;

When we execute this query with :employee_id = fred, it returns two rows:

id=2 descendant_id=fred ancestor_id=bob
id=3 descendant_id=fred ancestor_id=jill

And successfully returns “all of Fred’s managers” in a simple, fast query, regardless of “how many levels up” we had to go.

…but Recursive CTEs!

Most readers will probably have pointed out that “PostgreSQL solved this” by adding WITH RECURSIVE CTEs.

Which is true!

In modern PostgreSQL, we can often skip the managers_closure table with the following updated query, where the WITH RECURSIVE syntax basically acts as “the missing for loop” for us to walk up a variable number of levels, using only the manager_id column:

WITH RECURSIVE managers AS (
    -- anchor: the employee's direct manager
    SELECT manager_id
    FROM employees
    WHERE id = :employee_id

    UNION ALL

    -- recursive: step up to each manager's manager
    SELECT e.manager_id
    FROM employees e
    JOIN managers m ON e.id = m.manager_id
)
SELECT manager_id
FROM managers
WHERE manager_id IS NOT NULL;

I’ll defer to other blog posts to explain the syntax, but this is really great!

Joist already has first-class support for RECURSIVE CTEs, in that anytime joist-codegen sees a “self-referential foreign-key” (i.e. the manager_id column that points back at its same employees table), we automatically create a ...Recursive relation that is powered by a RECURSIVE CTE under the hood, i.e.:

const fred = await em.load(Employee, "e:1");
// the regular m2o relation returns bob
console.log(await fred.manager.load());
// the recursive relation returns [bob, jill]
console.log(await fred.managersRecursive.load());

Where the managersRecursive.load() method will issue a SQL statement exactly like the WITH RECURSIVE CTE above, and get us all their transitive managers.

And in stereotypical “never N+1” fashion, Joist will auto-batch the SQL so that we could load 100 or 1000 employees’ worth of managesRecursive and it would still be a single SQL query. 🎉

Joist’s built-in support puts the power of RECURSIVE CTEs at your fingertips–just a load / populate call away. 😀

Why Still Use Closure Tables?

So, with this great PostgreSQL feature, are closure tables still applicable?

For us, the answer is usually no–unless there is a very high-performance query to optimize. 🚀

For example, internally we’re prototyping an RBAC-based auth system where permission grants “inherit” down a stack of “permission buckets”–like if Jill is the CEO, she can read the salary data of Bob and Fred, Bob can read the salary data of Fred, etc.

In this setup, we’ll issue “what are your transitive auth permissions?” queries basically all the time, and need these queries to be extremely fast, i.e. easy for the Postgres query planner to optimize & execute, which it is still better at doing for the “dumb/single join” of closure table queries, over recursive CTE queries.

So, for this use case, we think it’s worth the write-time cost of generating an old-school permission_bucket_closures table, particularly if Joist can help us implement it as easily as possible.

Closure Table Maintenance

We skipped over one of the biggest cons of closure tables–keeping them up to date.

If we have our same chain of Fred is managed by Bob is managed Jill (that resulted in the 6 managers_closure rows in the image above), but now we add Jan as a layer of management between Bob and Jill, we have a lot of bookkeeping to update.

Specifically we need to add four rows:

Which can be intuitively explained as:

All managers above Jan need Jan as a new report
All employees below Jan need Jan as a new boss,

This doesn’t seem too bad–but what if the hierarchy had more than just a single bob.manager = jan mutation? What if several relationships changed at once?

It starts to get complicated to make it “just work”–until we lean into Joist’s reactivity.

Joist Reactivity for the Win

Let’s look at the two updates we need for “Jan is a new layer of management”:

All managers above Jan need Jan as a new report
All employees below Jan need Jan as a new boss,

The first important insight is to lean into our mental model of “closure tables are just fancy m2m collections”, such that:

All managers above Jan need Jan as a new report –>
- This is really saying set jan.managersClosure=[jan, jill]
- I.e. calcing the m2m for Jan herself will “update the managers above her” because it is “Jan’s m2m collection” that fundamentally controls “who are all of her transitive managers”?
All employees below Jan need Jan as a new boss –>
- This is really saying recalc [fred, bob].managersClosure=[..., jan]
- I.e. re-calcing the m2m for “Jan’s transitive reports” to let them see “now Jan is also a manager of me”

And the second insight is that “recalculating the m2m all of Jan’s downstream reports” is exactly what Joist’s reactivity does when it sees a employee.manager relationship change–it knows bob.manager changed, hence bob.managersRecursive changed, hence anyone “watching bob.managersRecursive” which requires “looking down instead of up” needs to be told about the change.

All of this is achieved by our managersClosure declaration:

readonly managersClosure: ReactiveManyToMany<Employee, Employee> =
  hasReactiveManyToMany(
    "managersRecursive",
    (e) => [e, ...e.managersRecursive.get],
  );

When the jan.manager = jill and bob.manager = jan graph mutation happens, Joist does roughly:

Setting jan.manager = jill triggers reactivity on “who is watching this write?”
- We see managersClosure is watching jan.managersRecursive, and “reverse the hint” to find employee’s below Jan that need to know about this change
- This queues [jan, bob, fred] to have their managersClosure recalc-d
Setting bob.manager = jan also triggers “who is watching this write?”
- We again see managersClosure is watching bob.managersRecursive, and “reverse the hint” to find employee’s below Bob
- This queues [bob, fred] to have their managersClosure recalc-d
  - …technically this is a noop b/c they’re already queued
We recalc the [jan, bob, fred].managersClosure relations all at once
- Joist resolves the managersRecursive reactive hint for all three entities
  - If any relations are already loaded, we use the in-memory results
  - If any relations are not loaded, we issue a single batched CTE query for their data
- When the batched SQL resolves, the managersClosure lambda is invoked for each of [jan, bob, fred]
- This sets jan.managersClosure = [jan, jill]
- And sets bob.managersClosure = [bob, jan, jill]
- And sets fred.managersClosure = [fred, bob, jan, jill]
We call em.flush()
- The three mutated managerClosures m2m collections are diffed
- We issue a single INSERT INTO managers_closure with the new rows

This is admittedly a lot 😵, but thankfully it’s all driven by Joist’s internal infra keeping the hasReactiveManyToMany m2m up-to-date, by respecting the managersRecursive reactive hint we declared in the code.

And the key outcome is that the closure table was completely updated for all affected employees, and we only issued 3 SQL calls, 2 reads and 1 write:

A single “down the tree” SELECT to find the employees to recalc
A single “up the tree” SELECT to load the data for each lambda
A single INSERT the new m2m rows.

And because each of these operations is fundamentally batched, it will be the same 3 SQL calls if we’re changing 1 employee/manager relation or 100 employee/manager relationships. 🤯

A pretty amazing result for a 1-liner–can you imagine trying to achieve this with an adhoc, hand-coded implementation? It’s not something I would readily sign up for. 😬

As a disclaimer, our approach does issue two reads / SELECTs before knowing the INSERT to update.

Most of the traditional, SQL-oriented posts like the one we linked above have examples of “jumping to the INSERTs / DELETEs writes”, without any initial SELECTs, by sussing out the algorithm (i.e. non-trivial WHERE clauses) for keeping the m2m/closure table rows in sync across mutations.

This “jump to writes” approach avoids our two up-front SELECTs, so we must concede that Joist’s approach is not “the absolute minimum number of queries”, and we do pay the cost of pulling all employees in the path into memory.

If you have extremely deep (1000+ level?) trees, this next level of optimization could be worth it, albeit it would come with the increased complexity.

Our assertion is that Joist’s performance of “3 fixed queries” will pragmatically be more performant than all but the most hand-optimized implementations, and its low-cost/simplicity means the ROI often wins over a custom approach.

Closing Thoughts

That is our deep-dive of how two Joist primitives, recursive CTE support & reactive m2m relations, can combine to drive a really powerful result–closure tables implemented as a 1-liner.

The neatest aspect, to us, is that when we started our internal RBAC auth system, we did not need to “build closure tables as a new feature of Joist”–instead the solution emerged elegantly as just a combination of our existing features.

To be clear, we’re not expecting readers to rush out and implement closure tables in their application–it is still a niche pattern, and, as we’ve seen, Postgres’s recursive CTEs provide a great modern alternative (which Joist’s recursive relations also put at your fingertips).

Instead our purpose is illustrating the power of Joist’s “declarative business logic” and “reactive calculations”, and encouraging readers to think about how these features could make their lives easier in their own work.

Joist 2.2 Released!

Mon, 18 May 2026 00:00:00 GMT

Joist 2.2 is out, and we didn’t get around to a 2.1 release announcement 😬, so we’re going to cover both the 2.1 and 2.2 releases.

As usual, Joist releases are driven by the problems we’re solving in our day-to-day application work at Homebound, so this release has a mix of headline features, API cleanup, performance work, and correctness fixes.

The biggest changes are:

Collection filters now use EXISTS clauses, avoiding join fanout em.find queries
Joist’s Postgres driver now uses node-pg directly instead of Knex
Soft-deleted rows can now be resurrected by upsert and findOrCreate
Recursive relations now support self-referential many-to-many tables
Derived property APIs were renamed, with new async-capable properties
collection.set(...) now works without preloading the collection

Faster Collection Queries with `EXISTS`

The biggest 2.2 change is #1743, which changes how Joist renders many collection filters.

Previously, a query like this:

// Find authors
await em.find(Author, {
  // Where they have a "child book" with this title
  books: { title: "Domain Modeling" },
  // And also a "child comment" with this text
  comments: { text: "great post" },
});

Would result in multiple LEFT JOINs into the books and comments tables, followed by DISTINCT to collapse duplicate Author rows.

That worked, but can create a lot of intermediate rows: if an author has 10 books and 20 comments, the database can end up considering 200 joined rows for that one author before deduping. Usually this fan out didn’t matter–until the combined rows got really big, at which point it could adversely affect performance on the database server.

Joist now renders collection filters as correlated EXISTS subqueries where possible:

WHERE EXISTS (
  SELECT 1
  FROM books b
  WHERE b.author_id = a.id AND b.title = 'Domain Modeling'
)
AND EXISTS (
  SELECT 1
  FROM comments c
  WHERE c.parent_author_id = a.id AND c.text = 'great post'
)

The original PR measured this from 1.4x to 10.9x faster than join-plus-distinct in benchmarked cases, with one match-all case reducing planner rows processed by up to 542x.

There are a few behavior changes worth calling out:

Queries that still require multiple collection LEFT JOINs (due to having complex WHERE conditions that OR across both child collections in a single clause) may now throw unless allowMultipleLeftJoins: true is set.
beforeFind plugins still see the “pre-EXISTS” AST – this lets the plugin stay focused on the LEFT JOIN as the “logical child collection” and let Joist do any EXISTS optimizations after the fact

Goodbye Knex, Hello `node-pg`

The largest 2.1 breaking change is #1754: Joist’s PostgresDriver now uses node-pg directly instead of taking a Knex instance.

New setup looks like:

import pg from "pg";
import { PostgresDriver } from "joist-orm";

const pool = new pg.Pool(config);
const driver = new PostgresDriver(pool, {
  onQuery(sql) {
    console.log(sql);
  },
});

If you still want Knex for custom query building (i.e. starting with an em.find to build 90% of your query, and then customize the last 10%–a common pattern we use internally), it is still available via joist-orm/knex:

import { createKnex } from "joist-orm/knex";
const knex = createKnex(pool);

Soft-Delete Resurrection

Soft deletes are great until a unique constraint says “you already have a row for that” and your operation blows up.

Joist 2.2 adds “resurrection” support in #1802, with follow-up fixes in #1821 and #1822.

Now, when upsert or findOrCreate would create a new row, Joist first checks for a “matching” soft-deleted row. If one exists, Joist clears it’s deleted_at columns, and returns/updates that row instead:

const author = await em.upsert(Author, {
  ssn: "123-45-6789",
  firstName: "Jane",
  lastName: "Austen",
});

In this example, if a soft-deleted Author with ssn = "123-45-6789" already exists, Joist resurrects that existing row instead of trying to insert a new one.

For upsert, the resurrection identity comes from uniqueBy in the joist-config.json:

{
  "entities": {
    "Author": {
      "uniqueBy": "ssn"
    },
    "Book": {
      "uniqueBy": ["author", "title"]
    }
  }
}

Resurrection is intentionally treated as an update:

The existing id is preserved
beforeUpdate runs
beforeCreate does not run
Ordinary duplicate rows still fail as expected

Recursive Many-to-Many Relations

Joist has supported recursive relations for self-referential foreign keys, i.e. an employee.manager.get will also have a employee.managersRecursive.get backed by Postgres recursive CTE queries.

Joist 2.1 extends that support to self-referential many-to-many tables in #1739.

For example, if users can have many parent users through a join table, Joist can now generate recursive collections for the full ancestry/descendant graph:

// Given [u5] has parents [u4, u3]
// And parent [u4] has parents [u2, u1]

// When accesing u5's parentsRecursive
const u5 = await em.load(User, "u:5", "parentsRecursive");

// Then we see u4, u3, u2, 1
u5.parentsRecursive.get;
// [u4, u4, u2, u1]

Recursive m2m collections support:

Batched loading for multiple seed entities.
Respect in-memory, unflushed changes.
Cycle detection.
Reactive hints, added in #1744.

Cycle handling was also cleaned up and documented in #1745, #1746, and #1747:

config.addCycleRule(
  "parentsRecursive",
  (u) => `User ${u.name} has a cycle in their parent hierarchy`,
);

Derived Property API Cleanup

2.2 also cleans up Joist’s derived-property naming in #1789 and #1819, while adding new SQL-backed derived properties in #1781.

The old hasAsyncProperty API is now hasProperty:

class Author extends AuthorCodegen {
  readonly numberOfBooks = hasProperty("books", (a) => a.books.get.length);
}

The rationale is that properties like numberOfBooks are “just properties” because their lambda is synchronous, and the hasAsyncProperty term (their previous name) is better reserved for properties that use the async keyword for their lambda.

These new “actually async” hasAsyncProperty’s, with their lambda being async, means they can use async database queries to their calculate values:

class Publisher extends PublisherCodegen {
  readonly numberOfBookReviews = hasAsyncProperty((p) =>
    p.em.findCount(BookReview, { book: { author: { publisher: p.id } } }),
  );
}

Similarly, ReactiveQueryField is now named AsyncReactiveField, for reactive fields whose recalculation is backed by a database query:

class Publisher extends PublisherCodegen {
  readonly numberOfBookReviews = hasAsyncReactiveField(
    "id",
    { authors: { books: "reviews" } },
    (p) => p.em.findCount(BookReview, { book: { author: { publisher: p.id } } }),
  );
}

`collection.set` Works Unloaded

In 2.1, set no longer requires a preloaded collection for either many-to-many (#1722) or one-to-many (#1725) relations.

This means you can replace a collection directly:

const book = await em.load(Book, "b:2");
const [t4, t5] = await em.loadAll(Tag, ["t:4", "t:5"]);

book.tags.set([t4, t5]);
await em.flush();

Without first doing:

await book.tags.load();

Joist tracks the intended final set, flushes the diff, composes later .add / .remove calls correctly, and still triggers reactive validation rules from the unloaded set.

For one-to-many relations, Joist also percolates the many-to-one side immediately for loaded or new children:

const publisher = await em.load(Publisher, "p:1");
const [a2, a3] = await em.loadAll(Author, ["a:2", "a:3"]);

publisher.authors.set([a2, a3]);
await em.flush();

This is a small API change but big DX win: most set calls were already saying “make the final collection exactly this”, so requiring a pre-load was unnecessary noise.

`EntityManager.pendingChanges`

2.1 adds a new debugging/tooling API in #1748: em.pendingChanges.

It exposes a high-level snapshot of the unflushed work in the current unit of work:

const a = newAuthor(em);

em.pendingChanges;
// [{ kind: "create", entity: a }]

It also includes relation changes:

a.tags.add(t);

em.pendingChanges;
// [{ kind: "m2m", op: "add", joinTableName: "authors_to_tags", entities: [a, t] }]

This is especially useful for tests, debugging tools, audit-style logic, or any code that wants to inspect what flush() is about to do without poking at internal state.

Markdown-Backed Entity Docs

#1749 adds codegen support for syncing entity documentation between Markdown files and generated TypeScript docs.

The new config options are:

{
  "docs": true,
  "outputDocs": true
}

With this enabled, entity docs can live in Markdown:

## Overview

The Author entity represents a writer who can publish books.

## Fields

### numberOfBooks

The number of books this author has written.

And codegen can sync them into JSDoc and emit runtime metadata:

export const docs = {
  Author: {
    comment: "The Author entity represents a writer who can publish books.",
    fields: {
      numberOfBooks: "The number of books this author has written.",
    },
  },
} as const;

This should make it easier to keep entity docs readable in-repo while still making them available to GraphQL, admin UIs, or other runtime tooling.

Smaller But Notable Changes

A few other changes are worth calling out:

#1762 adds is aliases in test factories, so tests can assert stable factory ids and reference in-flight entities by factory id.
#1718 fixes findOrCreate support for null values.
#1758 removes the implicit entityLimit and DISTINCT from findIds.
#1784 and #1786 make m2m ordering deterministic across loading paths and processes.
#1798 avoids stack overflows in BatchLoader.loadAll with very large key arrays.
#1820 adds plugin hooks around validation: beforeValidate and afterValidate.
#1825 runs default values in Joist’s trusted context, so plugins/field logging don’t treat default initialization as user writes.

There are also many correctness fixes in reaction handling, load hints, subtype matching, stale preload caches, soft-delete joins, docs formatting, and dependency bumps across both release trains.

Upgrading

The main migration notes are:

Update PostgresDriver setup to pass a pg.Pool instead of Knex.
Rename old hasAsyncProperty usages to hasProperty (codemod).
Rename ReactiveQueryField usages to AsyncReactiveField (codemod).
Review queries that intentionally rely on multiple collection LEFT JOINs; most should be faster automatically, but unusual cases may need allowMultipleLeftJoins: true or optimizeJoinsToExists: false.

Thanks to everyone filing issues, testing next releases, and pushing on Joist’s edge cases.

Dual Release Channels

Fri, 23 Jan 2026 00:00:00 GMT

For awhile now, we’ve struggled with Joist release announcements–as in we just don’t do them. 😅

The primary reason is that, unlike a VC-funded or even bootstrapped product-focused company, Joist’s development has historically been 100% driven by what we need at Homebound, delivering business value in our day-to-day feature work.

So, when we need a new Joist feature to make our feature delivery easier/suck less, we typically need it “right now”–and so we build it and release it, without much concern for “should this warrant a major release?” or “Could we bundle a few of these major features up into a marketing-oriented release announcement?”

We mostly just need to “get back to work”, so we merge our Joist PRs and move on.

While doing so, we’ve admittedly been somewhat loose with semantic versioning. We do use semantic release to drive our releases in CI, but we’ve shied away from the ! of denote “breaking changes”, and have just stayed 1.x release world for…well, years at this point. We’ve just kept incrementing 1.1, 1.2, all the way up to 1.470-something. 😬

But with Joist 2.0 freshly released, we’re going to try a different approach: dual release channels.

Stable: This channel will follow semantic versioning (major.minor.patch) and be the default artifact in npmjs.
- I.e. this channel what you’ll get with a yarn add joist-orm
- In the periodic releases, we’ll bundle up a month-or-two of work into an “official release”, with appropriate documentation and release notes.
- We’ll trigger these releases by periodically merging main into our release branch
Next: This channel will deploy on every merge to main and use the next tag in npmjs.
- I.e. this channel what you’ll get with a yarn add joist-orm@next
- next version numbers will be “the last stable release” plus “an incrementing number”, so if 2.1.0 was the last stable release, next releases would be 2.1.0-next.1, 2.1.0-next.2, etc.
- Next releases will not follow semantic versioning–they could be a mix of bug fixes, new non-breaking features, and breaking changes.
- This is basically our release process today, albeit with the new next tagging

Our hope is that these two channels will let us both:

Keep releasing new features immediately to next, for pulling into our internal repositories
- Without blocking our internal work on “the next major release in 3 months”, and
- Without worrying about artifically burning/increasing major release numbers,
Periodically batch up changes into a stable release, and
- Announce the new features/wins as an official release, with release notes, etc.

We’re optimistic about the approach, and mostly regret that we didn’t start doing this sooner! 😅

(Of course, even though this approach is “new to us”, the dual channel approach is definitely not new–React releases are done this way, and I assume for the same rationale: given Facebook has an internal mono repo, they likely want “the latest release” asap (similar to our internal builds), while using the periodic releases for the general public, who don’t necessarily want wake up every morning to a possibly breaking change.)

Bringing Back Unnest

Sun, 11 Jan 2026 00:00:00 GMT

The latest Joist release brings back leveraging the Postgres unnest function (see the PR), for a nice 9% bump on our alpha latency-oriented benchmarks (joist_v2 is the merged PR):

What is `unnest`?

unnest is a Postgres function that takes an array and returns a set of rows, one for each element in the array.

The simplest example is converting one array and turning it into a set of rows:

-- Pass in 1 array, get back 3 rows
select unnest(array[1, 2, 3]) as i;
 i
---
 1
 2
 3
(3 rows)

But you can also use multiple unnest statements to get multiple columns on those rows:

select
  unnest(array[1, 2, 3]) as id,
  unnest(array['foo', 'bar', 'zaz']) as first_name;
 id | first_name
----+------------
  1 | foo
  2 | bar
  3 | zaz
(3 rows)

Why `unnest` is useful

unnest is great for bulk SQL statements, such as inserting 10 authors in one INSERT; without INSERT you might have 4 columns * 10 authors = 40 query parameters:

INSERT INTO authors (first_name, last_name, publisher_id, favorite_color)
  VALUES (?, ?, ?, ?). -- #a1
    (?, ?, ?, ?), -- #a2
    (?, ?, ?, ?), -- #a3
    (?, ?, ?, ?), -- #a4
    (?, ?, ?, ?), -- #a5
    (?, ?, ?, ?), -- #a6
    (?, ?, ?, ?), -- #a7
    (?, ?, ?, ?), -- #a8
    (?, ?, ?, ?), -- #a9
    (?, ?, ?, ?) -- #a10

Where we have 10 first_name parameters, 10 last_name parameters, etc., but with unnest we can instead send up “just 1 array per column”:

INSERT INTO authors (first_name, last_name, publisher_id, favorite_color)
  SELECT * FROM unnest(
    $1::varchar[], -- first_names
    $2::varchar[], -- last_names
    $3::int[],     -- publisher_ids
    $3::varchar[] -- colors
  )

The benefits of fewer query parameters are:

Smaller SQL statements going over the wire (one of our benchmarks saw 41kb of SQL without unnest, and 350 bytes with unnest)
“Stable” SQL statement that don’t change for each “number of authors being updated”, so will have better prepared statement cache hit rates
Observability tools like Datadog will also better group “stable” SQL statements with a fixed number of parameters

This is generally a well-known approach, i.e. TimeScale had a blog post highlighting a 2x performance increase, albeit you have to get to fairly large update sizes to have this much impact.

Bringing it back?

Joist had previously used unnest in our INSERTs and UPDATEs, but we’d removed it because it turns out unnest is finicky with array columns–it “overflattens” and requires reactangular arrays.

(I.e. array columns like varchar[] for storing multiple values favorite_colors = ['red', 'blue'] in a single column.)

The 1st unfortunate thing with unnest is that it “overflattens”, i.e. if we want to update two author’s favorite_colors columns using unnest, we’d intuitively think “let’s just pass an array of arrays”, one array for each author:

-- Pass two arrays, with two elements each
-- We expect to get back two rows of {red,blue} and {green,purple}
select * from unnest(array[array['red','blue'],array['green','purple']]);
 unnest
--------
 red
 blue
 green
 purple
(4 rows)

…wait, we got 4 rows instead.

Unfortunately this is just how unnest works–when given 2-dimensional arrays (like a matrix), it creates a row per each value/cell in matrix.

Another unfortunate wrinkle with unnest is that our intuitive “array of arrays” creates fundamentally invalid arrays if the authors have a different number of favorite colors:

-- Try to create 1 array of {red,blue} and 1 array of {purple}
select * from unnest(array[array['red','blue'],array['purple']]);
-- ERROR:  multidimensional arrays must have array expressions with matching dimensions

Our error is treating the varchar[][] as “an array of arrays”, when fundamentally Postgres treats it as “a single array, of two dimensions”, like mathematical n-dimensional arrays or matrices: they must be “rectangular” i.e. every row of our m x n matrix must be the same length (we’ve been trying to create “jagged” multidimensional arrays, which is not supported).

One final wrinkle is, not only must all rows be the same length, but think about nullable columns–how could we set a1 favorite_colors='red', 'blue'] but then set a2 favorite_colors=null? With unnests strict array limitations we cannot.

The combination of these issues is why we’d previously removed unnest usage, but now have introducing our own unnest_arrays custom function that solves each of these problems.

`unnest_arrays` Custom Function

Our custom unnest_arrays function works around unnests limitations by coordinating with the Joist runtime to create 2-dimensional arrays that satisfy Postgres’s requirements, but still produce the desired values:

When updating favorite_colors for multiple authors with different number of colors, we pad trailing nulls to the end of each author’s colors array, until the array is rectangular
When updating favorite_colors to null, we also pad a single leading null to indicate the desired nullness (and pad a “not-null marker” for other rows).

This is simpler to see with an example, of updating three authors:

Author 1 should update favorite_colors=red,green,blue
Author 2 should update favorite_colors=green
Author 3 should update favorite_colors=null

We are able to issue a SQL UPDATE like:

WITH data AS (
  SELECT
    unnest($1) as id
    unnest_arrays($2) as favorite_colors,
)
UPDATE authors SET favorite_colors = data.favorite_colors
FROM data WHERE authors.id = data.id

And our favorite_colors array looks like:

-- Created by the Joist runtime by reading the Author's favoriteColors
-- property and then adding padding as needed to a rectangular 2D array
array[
  array['', 'red', 'green', 'blue'], -- a:1
  array['', 'green', null, null], -- a:2
  array[null, null, null, null]] -- a:3
]

This array is passed our unnest_arrays custom function that knows about each of these conventions:

CREATE OR REPLACE FUNCTION unnest_arrays(arr ANYARRAY, nullable BOOLEAN = false, OUT a ANYARRAY)
  RETURNS SETOF ANYARRAY
  LANGUAGE plpgsql IMMUTABLE STRICT AS
$func$
BEGIN
  FOREACH a SLICE 1 IN ARRAY arr LOOP
    -- When invoked for nullable columns, watch for the is-null/is-not-null marker
    IF nullable THEN
      -- If we should be null, drop all values and  return null
      IF a[1] IS NULL THEN a := NULL;
      -- Otherwise drop the is-not-null  marker
      ELSE a := a[2:array_length(a, 1)];
      END IF;
    END IF;
    -- Drop all remaining/trailing nulls
    a := array_remove(a, NULL);
    RETURN NEXT;
  END LOOP;
END

And that’s it; we get out the other side our desired rows:

select unnest_arrays(array[
  array['', 'red', 'green', 'blue'], -- a:1
  array['', 'green', null, null], -- a:2
  array[null, null, null, null]] -- a:3
, true);
  unnest_arrays
------------------
  {red,green,blue}
  {green}

(3 rows)

Pros/Cons

Our solution has a few pros/cons:

Pro: We’ve restored our ability to use unnest for all of our batched SELECTs, UPDATEs, and INSERTs 🎉
Con: Joist users with array columns in their schemas will need to create the unnest_arrays function
- This is a one-time migration so seems reasonable
Con: With all the null padding tricks, we’re giving up the ability to have null values within our array values
- I.e. we cannot have favorite_colors=[red, null, blue]
- For our domain modeling purposes, this is a fine/acceptable tradeoff, b/c we’ve always modeled varchar[] columns as string[] and not Array<string | null> – we actively don’t want nulls in our varchar[] columns anyway

So far these pros/cons are worth it 🚀; but, as always, we’ll continue adjusting our approach as we learn more from real-world use cases & usage.

Using CTEs and Query Rewriting for Versioning

Fri, 05 Dec 2025 00:00:00 GMT

Joist is an ORM primarily developed for Homebound’s GraphQL majestic monolith, and we recently shipped a long-awaited Joist feature, SQL query rewriting via an ORM plugin API, to deliver a key component of our domain model: aggregate level versioning.

We’ll get into the nuanced details below, but “aggregate level versioning” is a fancy name for providing this minor “it’s just a dropdown, right? 😰” feature of a version selector across several major subcomponents of our application:

Where the user can:

“Time travel” back to a previous version of what they’re working on ⌛,
Draft new changes (collaboratively with other users) that are not seen until they click “Publish” to make those changes active ✅

And have the whole UI “just work” while they flip between the two.

As a teaser, after some fits & painful spurts, we achieved having our entire UI (and background processes) load historical “as of” values with just a few lines of setup/config per endpoint–and literally no other code changes. 🎉

Read on to learn about our approach!

Aggregate What Now?

Besides just “versioning”, I called this “aggregate versioning”–what is that?

It’s different from traditional database-wide, system time-based versioning, that auditing solutions like cyanaudit or temporal FOR SYSTEM_TIME AS OF queries provide (although we do use cyanaudit for our audit trail & really like it!).

Let’s back up and start with the term “aggregate”. An aggregate is a cluster of ~2-10+ “related entities” in your domain model (or “related tables” in your database schema). The cluster of course depends on your specific domain–examples might be “an author & their books”, or “a customer & their bank accounts & profile information”.

Typically there is “an aggregate parent” (called the “aggregate root”, since it sits at the root of the aggregate’s subgraph) that naturally “owns” the related children within the aggregate; i.e. the Author aggregate root owns the Book and BookReview children; the Customer aggregate root owns the CustomerBankAccount and CustomerProfile entities.

Historically, Aggregate Roots are a pattern from Domain Driven Design, and mostly theoretically useful–they serve as a logical grouping, which is nice, but don’t always manifest as specific outcomes/details in the implementation (at least from what I’ve seen).

Why Version An Aggregate?

Normally Joist blog posts don’t focus on specific domains or verticals, but for the purposes of this post, it helps to know the problem we are solving.

At Homebound, we’re a construction company that builds residential homes; our primary domain model supports the planning and execution of our procurement & construction ops field teams.

The domain model is large (currently 500+ tables), but two key components are:

The architectural plans for a specific model of home (called a PlanPackage), and
The design scheme for products that go into a group of similar plans (called a DesignPackage)–i.e. a “Modern Farmhouse” design package might be shared across ~4-5 separate, but similarly-sized/laid out, architectural plans

Both of these PlanPackages and DesignPackages are aggregate roots that encompass many child PlanPackage... or DesignPackage... entities within them:

What rooms are in the plan? PlanPackageRooms
What materials & labor are required (bricks, lumber, quantities)? PlanPackageScopeLines
What structural/floor plan options do we offer to customers? PlanPackageOptions
- How do these options change the plan’s materials & labor? A m2m between PlanPackageScopeLines and PlanPackageOptions
What are the appliances in the kitchen? DesignPackageProducts
What spec levels, of Essential/Premium/etc, do we offer? DesignPackageOptions
- How do these spec levels change the home’s products? A m2m between DesignPackageProducts and DesignPackageOptions

This web of interconnected data can all be modeled successfully (albeit somewhat tediously)–but we also want it versioned! 😬

Change management is extremely important in construction–what was v10 of the PlanPackage last week? What is v15 of the PlanPackage this week? What changed in each version between v10 and v15? Are there new options available to homebuyers? What scope changed? Do we need new bids? Etc.

And, pertintent to this blog post, we want each of the PlanPackages and DesignPackages respective “web of interconnected data” (the aggregate root & its children) versioned together as a group with application-level versioning: multiple users collaborate on the data (making simultaneous edits across the PlanPackage or DesignPackage), co-creating the next draft, and then we “Publish” the next active version with a changelog of changes.

After users draft & publish the plans, and potentially start working on the next draft, our application needs to be able to load a complete “aggregate-wide snapshot” for “The Cabin Plan” PlanPackage v10 (that was maybe published yesterday) and a complete “aggregate-wide snapshot” of “Modern Design Scheme” DesignPackage v8 (that was maybe published a few days earlier), and glue them together in a complete home.

Hopefully this gives you an idea of the problem–it’s basically like users are all collaborating on “the next version of the plan document”, but instead of it being a single Google Doc-type artifact that gets copy/pasted to create “v1” then “v2” then “v3”, the collaborative/versioned artifact is our deep, rich domain model (relational database tables) of construction data.

Schema Approach

After a few cough “prototypes” cough of database schemas for versioning in the earlier days of our app, we settled on a database schema we like: two tables per entity, a main “identity” table, and a “versions” table to store snapshots, i.e. something like:

authors table: stores the “current” author data (first_name, last_name, etc), with 1 row per author (we call this the “identity” table)
author_versions table: stores snapshots of the author over time (author_id, version_id, first_name, last_name), with 1 row per version of each author (we call this the “version” table)

This is an extremely common approach for versioning schemas, i.e. it’s effectively the same schema suggested by PostgreQL’s SQL2011Temporal docs, albeit technically they’re tracking system time, like an audit trail, and not our user-driven versioning.

This leads to a few differences: the SQL2011Temporal history tables use a _system_time daterange column to present “when each row was applicable in time” (tracking system time), while we use two FKs that also form “an effective range”, but the range is not time-based, it’s version-based: “this row was applicable starting at first=v5 until final=v10”.

So if we had three versions of an Author in our author_versions table, it would look like:

id=10 author_id=1 first_id=v10 final_id=v15 first_name=bob last_name=smith
- From versions v10 to v15, the author:1 had a firstName=bob
id=11 author_id=1 first_id=v15 final_id=v20 first_name=fred last_name=smith
- From versions v15 to v20, the author:1 had a firstName=fred
id=11 author_id=1 first_id=v20 final_id=null first_name=fred last_name=brown
- From versions v20 to now, the author:1 had a lastName=brown

We found this _versions table strikes a good balance of tradeoffs:

It only stores changed rows–if v20 of the aggregate root (i.e. our PlanPackage) only changed the Authors and two of its Books, there will only be 1 author_versions and two book_versions, even if the Author has 100s of books (i.e. we avoid making full copies of the aggregate root on every version)
When a row does change, we snapshot the entire row, instead of tracking only specific columns changes (storing the whole row takes more space, but makes historical/versioned queries much easier)
- Technically we only include mutable columns in the _versions table, i.e. our entities often have immutable columns (like type flags or parent references) and we don’t bother copying these into the _versions tables.
We only store 1 version row “per version”–i.e. if, while making PlanPackage v20, an Authors first name changes multiple times, we only keep the latest value in the draft author_versionss row.
- This is different from auditing systems like SQL2011Temporal, where _history rows are immutable, and every change must create a new row–we did not need/want this level of granularity for our application-level versioning.

With this approach, we can reconstruct historical versions by finding the singular “effective version” for each entity, with queries like:

select * from author_versions av
join authors a on av.author_id = a.id
where a.id in ($1)
  and av.first_id <= $2
  and (av.final_id is null or av.final_id < $3)

Where:

$1 is whatever authors we’re looking for (here a simple “id matches”, but it could be an arbitrarily complex WHERE condition)
$2 is the version of the aggregate root we’re “as of”, i.e. v10
$3 is also the same version “as of”, i.e. v10

The condition of first_id <= v10 < final_id finds the singular author_versions row that is “effective” or “active” in v10, even if the version itself was created in v5 (and either never replaced, or not replaced until “some version after v10” like v15).

…but what is “Current”?

The authors and author_versions schema is hopefully fairly obvious/intuitive, but I left out a wrinkle: what data is stored in the authors table itself?

Obviously it should be “the data for the author”…but which version of the author? The latest published data? Or latest/WIP draft data?

Auditing solutions always put “latest draft” in authors, but that’s because the application itself is always reading/writing data from authors, and often doesn’t even know that the auditing author_versions tables exist–but in our app, we need workflows & UIs to regularly read “the published data” & ignore the WIP draft changes.

So we considered the two options:

The authors table stores the latest draft version (as in SQL2011Temporal), or
The authors table stores the latest published version,

We sussed out pros/cons in a design doc:

Option 1. Store “latest draft” data in authors
- Pro: Writes with business logic “just read the Author and other entities” for validation
- Con: All readers, even those that just want the published data, must use the _versions table to “reconstruct the published/active author”
Option 2. Store the “latest published” data in authors
- Pro: The safest b/c readers that “just read from authors” will not accidentally read draft/unpublished data (a big concern for us)
- Pro: Any reader that wants to “read latest” doesn’t have to know about versioning at all, and can just read the authors table as-is (also a big win)
- Con: Writes that have business logic must first “reconstruct the draft author” (and its draft books and its draft book reviews if apply cross-entity business rules/invariants) from the _versions tables

By “reconstruct the [published or draft] author”, what we mean is that instead of “boring CRUD code” that just SELECTs from the authors, books, etc. tables, version-aware code must:

a) actually read from the author_verisons, book_versions tables,
b) know “which version” it should use to find those authors/books
- …and we might be looking for “PlanPackage v10” but “DesignPackage v8” so track multiple, contextual versions within a single request, not just a single “as of” timestamp
c) do version-aware graph traversal, i.e. “boring walk the graph” code like for (const book in author.books) that is ubiquitous in our codebase needs the author.books relation to use the version-aware book_versions.author_id instead of the books.author_id, b/c the Book might have changed its Author over time.
- So nearly every graph navigation needs checked for “is this relation actually versioned?”, and if so, opt-into the more complex, version-aware codepath.

This “reconstruction” problem seemed very intricate/complicated, and we did not want to update our legacy callers (which were mostly reads) to “do all this crazy version resolution”, so after the usual design doc review, group comments, etc., we decide to store latest published in authors.

The key rationale being:

We really did not want to instrument our legacy readers to know about _versions tables to avoid seeing draft data (we were worried about draft data leaking into existing UIs/reporting)
We thought “teaching the writes to ‘reconstruct’ the draft subgraph” when applying validation rules would be the lesser evil.

(You can tell I’m foreshadowing this was maybe not the best choice.)

Initial Approach: Not so great

With our versioning scheme in place, we started a large project for our first versioning-based feature: allowing home buyers to deeply personalize the products (sinks, cabinet hardware, wall paint colors, etc) in the home their buying (i.e. in v10 of Plan 1, we offered sinks 1/2/3, but in v11 of Plan 1, we now offer sinks 3/4/5).

And it was a disaster.

This new feature set was legitimately complicated, but “modeling complicated domain models” is supposed to be Joist’s sweet spot–what was happening?

We kept having bugs, regressions, and accidental complexity–that all boiled down to the write path (mutations) having to constantly “reconstruct the drafts” that it was reading & writing.

Normally in Joist, a saveAuthor mutation just “loads the author, sets some fields, and saves”. Easy!

But with this versioning scheme:

the saveAuthor mutation has to first “reconstruct the latest-draft Author from the author_versions (if it exists)
any writes cannot be “just update the Author row” (that would immediately publish the change), they need to be staged into the draft AuthorVersion
any Joist hooks or validation rules also need to do the same thing:
- validation rules have to “reconstruct the draft” view of the author + books + book reviews subgraph of data they’re validating,
- hooks that want to make reactive changes must also “stage the change” into a draft BookVersion or BookReviewVersion instead of “just setting some Book fields”.

We had helper methods for most of these incantations–but it was still terrible.

After a few months of this, we were commiserating about “why does this suck so much?” and finally realized–well, duh, we were wrong:

We had chosen to “optimize reads” (they could “just read from authors” table).

But, in doing so, we threw writes under the bus–they needed to “read & write from drafts”–and it was actually our write paths that are the most complicated part of our application–validation rules, side effects, business process, all happen on the write path.

We needed the write path to be easy.

New Approach: Write to Identities

We were fairly ecstatic about reversing direction, and storing drafts/writes directly in authors, books, etc.

This would drastically simplify all of our write paths (GraphQL mutations) back to “boring CRUD” code:

// Look, no versioning code!
const author = await em.load(Author, "a:1");
if (author.shouldBeAllowedToUpdateFoo) {
  author.foo = 1;
}
await em.flush();

…but what about those reads? Wouldn’t moving the super-complicated “reconstruct the draft” code out of the writes (yay!), over into “reconstruct the published” reads (oh wait), be just as bad, or worse (which was the rationale for our original decision)?

We wanted to avoid making the same mistake twice, and just “hot potato-ing” the write path disaster over to the read path.

Making Reads not Suck

We spent awhile brainstorming “how to make our reads not suck”, specifically avoiding manually updating all our endpoints’ SQL queries/business logic to do tedious/error-prone “remember to (maybe) do version-aware reads”.

If you remember back to that screenshot from the beginning, we need our whole app/UI, at the flip of a dropdown, to automatically change every SELECT query from:

-- simple CRUD query
SELECT * FROM authors WHERE id IN (?) -- or whatever WHERE clause

To a “version-aware replacement”:

-- find the right versions row
SELECT * FROM author_versions av
WHERE av.author_id in (?) -- same WHERE clause
  AND av.first <= v10 -- and versioning
  AND (av.final IS NULL or av.final > v10)

And not only for top-level SELECTs but anytime authors is used in a query, i.e. in JOINs:

-- simple CRUD query that joins into `authors`
SELECT b.* FROM books
  JOIN authors a on b.author_id = a.id
  WHERE a.first_name = 'bob';

-- version-aware replacement, uses books_versions *and* author_versions
-- for the join & where clause
SELECT bv.* FROM book_versions bv
  -- get the right author version
  JOIN author_versions av ON
    bv.author_id = av.author_id AND (av.first <= v10 AND (av.final IS NULL or a.final > v10))
  -- get the right book version
  WHERE bv.first <= v10 AND (bv.final IS NULL or bv.final > v10)
  -- predicate should use the version table
  AND av.first_name = 'bob';

1. Initial Idea: Using CTEs

When it’s articulated like “we want every table access to routed to the versions table”, a potential solution starts to emerge…

Ideally we want to magically “swap out” authors with “a virtual authors table” that automatically has the right version-aware values. How could we do this, as easily as possible?

It turns out a CTE is a great way of structuring this:

-- create the fake/version-aware authors table
WITH _authors (
  SELECT
    -- make the columns look exactly like the regular table
    av.author_id as id,
    av.first_name as first_name
    -- but read the data from author_versions behind the scenes
  FROM author_versions
  WHERE (...version matches...)
)
-- now the rest of the application's SQL query as normal, but we swap out
-- the `authors` table with our `_authors` CTE
SELECT * FROM _authors a
WHERE a.first_name = 'bob'

And that’s (almost) it!

If anytime our app wants to “read authors”, regardless of the SQL query it’s making, we swapped authors-the-table to _authors-the-CTE, the SQL query would “for free” be using/returning the right version-aware values.

So far this is just a prototype; we have three things left to do:

Add the request-specific versioning config to the query
Inject this rewriting as seamlessly as possible
Evaluate the performance impact

2. Adding Request Config via CTEs

We have a good start, in terms of a hard-coded prototype SQL query, but now we need to get the first <= v10 AND ... pseudo code in the previous SQL snippets actually working.

Instead of a hard-coded v10, we need queries to use:

Dynamic request-specific versioning (i.e. the user is currently looking at, or “pinning to”, PlanPackage v10), and
Support pinning multiple different aggregate roots in the same request (i.e. the user is looking at PlanPackage v10 but DesignPackage v15)

CTEs are our new hammer 🔨–let’s add another for this, calling it _versions and using the VALUES syntax to synthesize a table:

WITH _versions (
  SELECT plan_id, version_id FROM
   -- parameters added to the query, one "row" per pinned aggregate
   -- i.e. `[1, 10, 2, 15]` means this request wants `plan1=v10,plan2=v15`
  VALUES (?, ?), (?, ?)
), _authors (
  -- the CTE from before but joins into _versions for versioning config
  SELECT
    av.author_id as id,
    av.first_name as first_name
  FROM authors a
  JOIN _versions v ON a.plan_id = v.plan_id
  -- now we know:
  -- a) what plan the author belongs to (a.plan_id, i.e. its aggregate root)
  -- b) what version of the plan we're pinned to (v.version_id)
  -- so we can use them in our JOIN clause
  JOIN author_versions av ON (
    av.author_id = a.id AND
    av.first_id >= v.version_id AND (
      av.final_id IS NULL OR av.final_id < v.version_id
    )
  )
)
SELECT * FROM _authors a WHERE a.first_name = 'bob'

Now our application can “pass in the config” (i.e. for this request, plan:1 uses v10, plan:2 uses v15) as extra query parameters into the query, they’ll be added as rows to the _versions CTE table, and then the rest of the query will resolve versions using that data.

Getting closer!

One issue with the query so far is that we must ahead-of-time “pin” every plan we want to read (by adding it to our _versions config table), b/c if a plan doesn’t have a row in the _versions CTE, then the INNER JOINs will not find any p.version_id available, and none of its data will match.

Ideally, any plan that is not explicitly pinned should have all its child entities read their active/published data (which will actually be in their author_versions tables, and not the primary authors tables); which we can do with (…wait for it…) another CTE:

WITH _versions (
  -- the injected config _versions stays the same as before
  SELECT plan_id, version_id FROM VALUES (?, ?), (?, ?)
), _plan_versions (
   -- we add an additional CTE that defaults all plans to active unless pinned in _versions
  SELECT
    plans.id as plan_id,
    -- prefer `v.version_id` but fallback on `active_version_id`
    COALESCE(v.version_id, plans.active_version_id) as version_id,
  FROM plans
  LEFT OUTER JOIN _versions v ON v.plan_id = plans.id
)
_authors (
  -- this now joins on _plan_versions instead _versions directly
  SELECT
    av.author_id as id,
    av.first_name as first_name
  FROM authors a
  JOIN _plan_versions pv ON a.plan_id = pv.id
  JOIN author_versions av ON (
    av.author_id = a.id AND
    av.first_id >= pv.version_id AND (
      av.final_id IS NULL OR av.final_id < pv.version_id
    )
  )
)
-- automatically returns & filters against the versioned data
SELECT * FROM _authors a WHERE a.first_name = 'bob'

We’ve basically got it, in terms of a working prototype–now we just need to drop it into our application code, ideally as easily as possible.

3. Injecting the Rewrite Automatically

We’ve discovered a scheme to make reads automatically version-aware–now we want our application to use it, basically all the time, without us messing up or forgetting the rewrite incantation.

Given that a) we completely messed this up the 1st time around 😕, and b) this seems like a very mechanical translation 🤖, what if we could automate it? For every read?

Our application already does all reads through Joist (of course 😅), as EntityManager calls:

// Becomes SELECT * FROM authors WHERE id = 1
const a = em.load(Author, "a:1");
// Becomes SELECT * FROM authors WHERE first_name = ?
const as = em.find(Author, { firstName: "bob" });
// Also becomes SELECT * FROM authors WHERE id = ?
const a = await book.author.load();

It would be really great if these em.load and em.find SQL queries were all magically rewritten–so that’s what we did. 🪄

We built a Joist plugin that intercepts all em.load or em.find queries (in a new plugin hook called beforeFind), and rewrites the query’s ASTs to be version-aware, before they are turned into SQL and sent to the database.

So now what our endpoint/GraphQL query code does is:

function planPackageScopeLinesQuery(args) {
  // At the start of any version-aware endpoint, setup our VersioningPlugin...
  const plugin = new VersioningPlugin();
  // Read the REST/GQL params to know which versions to pin
  const { packageId, versionId } = args;
  plugin.pin(packageId, versionId);
  // Install the plugin into the EM, for all future em.load/find calls
  em.addPlugin(plugin);

  // Now just use the em as normal and all operations will automatically
  // be tranlated into the `...from _versions...` SQL
  const { filter } = args;
  return em.find(PlanPackageScopeLine, {
    // apply filter logic as normal, pseudo-code...
    quantity: filter.quantity,
  });
}

How does this work?

Internally, Joist parses the arguments of em.find into an AST, ParsedFindQuery, that is a very simplified AST/ADT version of a SQL query, i.e. the shape looks like:

// An AST/ADT for an `em.find` call that will become a SQL `SELECT`
type ParsedFindQuery = {
  // I.e. arrays of `a.*` or `a.first_name`
  selects: ParsedSelect[];
  // I.e. the `authors` table, with its alias, & any inner/outer joins
  tables: ParsedTable[];
  // I.e. `WHERE` clauses
  condition?: ParsedExpressionFilter;
  groupBys?: ParsedGroupBy[];
  orderBys: ParsedOrderBy[];
  ctes?: ParsedCteClause[];
};

After Joist takes the user’s “fluent DSL” input to em.find and parses it into this ParsedFindQuery AST, plugins can now inspect & modify the query:

class VersioningPlugin {
  // List of pinned plan=version tuples, populated per request
  #versions = []

  // Called for any em.load or em.find call
  beforeFind(meta: EntityMetadata, query: ParsedFindQuery): void {
    let didRewrite = false;
    for (const table of [...query.tables]) {
      // Only rewrite tables that are versioned
      if (hasVersionsTable(table)) {
        this.#rewriteTable(query, table);
        didRewrite = true;
      }
    }
    // Only need to add these once/query
    if (didRewrite) {
      this.#addCTEs(query);
    }
  }

  #rewriteTable(query, table) {
    // this `table` will initially be the `FROM authors AS a` from a query;
    // leave the alias as-is, but swap the table from "just `authors`" to
    //  the `_authors` CTE we'll add later
    table.table = `_${table.table}`;

    // If `table.table=author`, get the AuthorMetadata that knows the columns
    const meta = getMetadatFromTableName(table.table);

    // Now inject the `_authors` CTE that to be our virtual table
    query.ctes.push({
      alias: `_${table.table}`,
      columns: [...],
      query: {
        kind: "raw",
        // put our "read the right author_versions row" query here; in this blog
        // post it is hard-coded to `authors` but in the real code would get
        // dynamically created based on the `meta` metadta
        sql: `
          SELECT
            av.author_id as id,
            av.first_name as first_name,
            -- all other author columns...
          FROM authors a
          JOIN _plan_versions pv ON a.plan_id = pv.id
          JOIN author_versions av ON (
            av.author_id = a.id AND
            av.first_id >= pv.version_id AND (
              av.final_id IS NULL OR av.final_id < pv.version_id
            )
          )
        `
      }
    })
  }

  // Inject the _versions and _plan_versions CTEs
  #addCTEs(query) {
    query.ctes.push( {
      alias: "_versions",
      columns: [
        { columnName: "plan_id", dbType: "int" },
        { columnName: "version_id", dbType: "int" },
      ],
      query: {
        kind: "raw",
        bindings: this.#versions.flatMap(([pId, versionId]) => unsafeDeTagIds([pId, versionId])),
        sql: `SELECT * FROM (VALUES ${this.#versions.map(() => "(?::int,?::int)").join(",")}) AS t (plan_id, version_id)`,
      },
    },
    // Do the same thing for _plan_versions
    query.ctes.push(...);
  }
}

This is very high-level pseudo-code but the gist is:

We’ve mutated the query to use our _authors CTE instead of the authors table
We’ve injected the _authors CTE, creating it dynamically based on the AuthorMetadata
We’ve injected our _versions and _plan_versions config tables

After the plugin’s beforeFind is finished, Joist takes the updated query and turns it into SQL, just like it would any em.find query, but now the SQL it generates automatically reads the right versioned values.

4. Performance Evaluation

Now that we have everything working, how was the performance?

It was surprisingly good, but not perfect–we unfortunately saw a regression for reads “going through the CTE”, particularly when doing filtering, like:

WITH _versions (...),
 _plan_versions (...),
 _authors (...),
)
-- this first_name is evaluating against the CTE results
SELECT * FROM _authors a WHERE a.first_name = 'bob'

We were disappointed b/c we thought since our _authors CTE is “only used once” in the SQL query, that ideally the PG query planner would essentially “inline” the CTE and pretend it was not even there, for planning & indexing purposes.

Contrast this with a CTE that is “used twice” in a SQL query, which our understanding is that then Postgres executes it once, and materializes it in memory (basically caches it, instead of executing it twice). This materialization would be fine for smaller CTEs like _versions or _plan_versions, but on a potentially huge table like authors or plan_package_scope_lines, we definitely don’t want those entire tables sequentially scanned and creating “versioned copies” materialized in-memory before any WHERE clauses were applied.

So we thought our “only used once” _authors CTE rewrite would be performance neutral, but it was not–we assume because many of the CTE’s columns are not straight mappings, but due to some nuances with handling drafts, ended up being non-trivial CASE statements that look like:

-- example of the rewritten select clauses in the `_authors` CTE
SELECT
  a.id as id,
  a.created_at as created_at,
  -- immutable columns
  a.type_id as type_id,
  -- versioned columns
  (CASE WHEN _a_version.id IS NULL THEN a.first_name) ELSE _a_version.first_name END) as first_name,
  (CASE WHEN _a_version.id IS NULL THEN a.last_name) ELSE _a_version.last_name END) as last_name,

And we suspected these CASE statements were not easy/possible for the query planner to “see through” and push filtering & indexing statistics through the top-level WHERE clause.

So, while so far our approach has been “add yet another CTE”, for this last stretch, we had to remove the _authors CTE and start “hard mode” rewriting the query by adding JOINs directly to the query itself, i.e. we’d go from a non-versioned query like:

-- previously we'd "just swap" the books & authors tables to
-- versioned _books & _authors CTEs
SELECT b.* FROM books
  JOIN authors a ON b.author_id = a.id
  WHERE a.first_name = 'bob';

To:

SELECT
  -- rewrite the top-level select
  b.id as id,
  -- any versioned columns need `CASE` statements
  (CASE WHEN bv.id IS NULL THEN b.title ELSE bv.title) AS title,
  (CASE WHEN bv.id IS NULL THEN b.notes ELSE bv.notes) AS notes,
  -- ...repeat for each versioned column...
  -- ...also special for updated_at...
FROM books
  -- keep the `books` table & add a `book_versions` JOIN directly to the query
  JOIN book_versions bv ON (bv.book_id = b.id AND bv.first_id >= pv.version_id AND ...)
  -- rewrite the `ON` to use `bv.author_id` (b/c books can change authors)
  JOIN authors a ON bv.author_id = a.id
  -- add a author_version join directly to the query
  JOIN author_versions av ON (av.author_id = a.id AND av.first_id >= pv.version_id AND ...)
  -- rewrite the condition from `a.first_name` to `av.first_name`
  WHERE av.first_name = 'bob';

This is a lot more rewriting!

While the CTE approach let us just “swap the table”, and leave the rest of the query “reading from the alias a” & generally being none-the-wiser, now we have to find every b alias usage or a alias usage, and evaluate if the SELECT or JOIN ON or WHERE clause is touching a versioned column, and if so rewrite that usage to the bv or av respective versioned column.

There are pros/cons to this approach:

Con: Obviously the query is much trickier to rewrite
Pro: But since our rewriting algorithm is isolated to the VersioningPlugin file, we were able to “refactor our versioned query logic” just once and have it apply everywhere which was amazing 🎉
Pro: The “more complicated to us” CTE-less query is actually “simpler to the Postgres query planner” b/c there isn’t a CTE “sitting in the way” and so all the usual indexing/filtering performance optimizations kicked in, and got us back to baseline performance

Removing the _authors / _books CTEs and doing “inline rewriting” (basically what we’d hoped Postgres would do for us with the “used once” CTEs, but now we’re doing by hand) gave us a ~10x performance increase, and returned us to baseline performance, actually beating the performance of our original “write to drafts” approach. 🏃

Skipping Some Details

It would make the post even longer, so I’m skipping some of the nitty-gritty details like:

Soft deletions–should entities “disappear” if they were added in v10 and the user pins to v8? Or “disappear” if they were deleted in v15 and the user is looking at v20?
- Initially we had our VersionPlugin plugin auto-filter these rows, but in practice this was too strict for some of our legacy code paths, so in both “not yet added” and “previously deleted” scenarios, we return rows anyway & then defer to application-level filtering
Versioning m2m collections, both in the database (store full copies or incremental diffs?), and teaching the plugin to rewrite m2m joins/filters accordingly.
Reading updated_at from the right identity table vs. the versions table to avoid oplock errors when drafts issue UPDATEs using plugin-loaded data
Ensuring endpoints make the pin and addPlugin calls without accidentally loading “not versioned” copies of the data they want to read into the EntityManager, which would cache the non-versioned data, & prevent future “should be versioned” reads for working as expected.
Migrating our codebase from the previous “by hand” / “write to drafts” initial versioning approach, to the new plugin + “write to identities” approach, which honestly was a lot of fun–lots of red code that was deleted & simplified by the new approach. 🔪

Thankfully we were able to solve each of these, and none turned into dealbreakers that compromised the overall approach. 😅

Wrapping Up

This was definitely a long-form post, as we explored the Homebound problem space that drove our solution, rather than just a shorter announcement post of “btw Joist now has a plugin API”.

Which, yes, Joist does now have a plugin API for query rewriting 🎉, but we think it’s important to show how/why it was useful to us, and potentially inspire ideas for how it might be useful to others as well (i.e. an auth plugin that does ORM/data layer auth 🔑 is also on our todo list).

That said, we anticipate readers wondering “wow this solution seems too complex” (and, yes, our production VersionPlugin code is much more complicated than the pseudocode we’ve used in this post), “why didn’t you hand-write the queries”, etc 😰.

We can only report that we tried “just hand-write your versioning queries”, in the spirit of KISS & moving quickly while building our initial set of version-aware features, for about 6-9 months, and it was terrible. 😢

Today, we have versioning implemented as “a cross-cutting concern” (anyone remember Aspect Oriented Programming? 👴), primarily isolated to a single file/plugin, and the rest of our code went back to “boring CRUD” with “boring reads” and “boring writes”.

Our velocity has increased, bugs have decreased, and overall DX/developer happiness is back to our usual “this is a pleasant codebase” levels. 🎉

If you have any questions, feel free to drop by our Discord to chat.

Thanks

Thanks to the Homebound engineers who worked on this project: Arvin, for bearing the brunt of the tears & suffering, fixing bugs during our pre-plugin “write to drafts” approach (mea cupla! 😅), ZachG for owning the rewriting plugin, both Joist’s new plugin API & our internal VersioningPlugin implementation 🚀, and Roberth, Allan, and ZachO for all pitching in to get our refactoring landed in the limited, time-boxed window we had for the initiative ⏰🎉.

Lazy Fields for 30x speedup without Decorators or Transforms

Mon, 06 Oct 2025 00:00:00 GMT

Joist is an ActiveRecord-style TypeScript ORM, such that we purposefully mimic how Rails declares relations, i.e. as class fields:

// simplified example
class Author {
  books = hasMany("books");
}

These class fields are very ergonomic, i.e. no decorators (which we purposefully avoid) or other boilerplate, so much so that at Homebound, we’ve written (or codegen-d) literally thousands of these class fields in our domain models.

Fields are Expensive at Scale

This approach has worked really well, with the only downside being that JavaScript’s class fields are eagerly initialized.

This means when new Author is called, all the fields in class Author, like books = hasMany(...) above, and any others like publisher = hasOne(...), etc., are immediately created & allocated in memory.

These class fields are created & assigned every time, for every author instance, even if the code that called new Author doesn’t end up using the books relation (maybe it only uses publisher).

This eager-ness is fine when Author has a handful of relations, but in sufficiently-complicated domain models, some of our core entities have 30 relations, and the cost of creating 30 relations x 1,000 of rows can start to add up.

So our goal is to keep the books = hasMany("books") syntax, but somehow make the fields lazy, so we only pay for what we use.

First Approach: Codegen

Not surprisingly, our first solution was to leverage code generation.

For an application’s default, foreign-key-based relations like one-to-many, many-to-one, etc., Joist already generates a AuthorCodegen class that has the boilerplate relations defined for free.

I.e. in Joist, you get getters & setters like author.firstName, and relations like author.books and publisher.author all for free, from the generated AuthorCodegen base class.

Because these ...Codegen classes are not handwritten, it was easy to just swap the field-based code over to getters:

// in AuthorCodegen.ts, generated by joist-codegen
class AuthorCodegen {
  // lazy cache of relations
  #relations: Record<string, any> = {};

  // Not a field anymore, but looks like one to callers
  get books(): OneToMany<Book> {
    return (this.#relations["books"] ??= hasMany("books"));
  }
}

// in Author.ts, written by the user
class Author extends AuthorCodegen {
  // we get the lazy relations for free
}

Because get books() is a method in a JavaScript class, the books function is installed just once on the Author.prototype, not each individual author instance, and so when we create new Author the only field to initialize is a single shared #relations cache, and there “nothing to do” for books–it doesn’t need an allocation, it’s just waiting to be called, on the prototype–it’s lazy.

This works great! We get lazy initialization, hasMany("books") is only invoked when books is accessed, and no one cares that the generated output is a little longer/more verbose.

However, Joist also has a robust set of user-defined relations, for rich domain modeling, i.e. things like ReactiveField:

// in Author.ts, written by the user
class Author extends AuthorCodegen {
  numberOfBooks = hasReactiveField(...);
}

And because this line is fundamentally a user-/engineer-written relation, we couldn’t apply our usual codegen hammer.

Second Approach: AST Transformers

Given we want “this field…but rewritten into a getter”, that sounded like a good job for AST rewriting!

So our next approach was using ts-patch to run a custom transform during the tsc compilation process that scanned the AST for class fields following our has... convention, and rewrote them to getters.

So the source Author.ts file would have the books = hasBooks field syntax, but the tsc emitted Author.js file would have a get books() { ... } getter, very similar to the AuthorCodegen getters from the previous approach.

This was pretty good!

The pros/cons were:

Pro: We achieved lazy initialization :tada:
Pro: Engineers kept writing the ergonomic relation = hasSome(...) field syntax
Pro: No runtime overhead
Con: Potentially different test vs. production behavior, if tests did not use ts-jest & invoke the transformer
Con: We’re tightly coupled to ts-patch and tsc-driven output

This last con has been becoming more painful, as more alterative performance-improving solutions to “transpile/load TypeScript” have been popping up.

We’re fans of tsx, Bun, and Node’s built-in TypeScript support–all of which are amazingly fast, but all have spotty or missing transform support (likely on purpose, b/c transforms themselves would only slow things down), and so we’ve not been able to leverage these newer tools.

This had been fine, but as our majestic monolith codebase grows (and grows), we’re more & more motivated to switch to a faster build/load process–without giving up the lazy fields performance optimization.

Final Approach: JavaScript Prototypes

We spent quite awhile brainstorming how to “keep fields…but not have them initialized”–literally months of “here & there” thinking about it, since this was not our top priority, but always an itch, lurking, tingling, just waiting to be scratched.

We were willing to use proxies, or subtypes, or really any magic–but could not think of a way to avoid the hasMany function in books = hasMany from running when the constructor is called.

Stepping back, we established that, tactically, we need to achieve two things:

Create an instance without the fields
Move the relations onto the prototype

When articulated this way, a native-JavaScript solution starts to emerge, and even become obvious in retrospect.

Skipping field initializers with Object.create

The insight to avoid field initializers is they happen “when the constructor is called”–what if we just don’t call the constructor? I.e. don’t call new Author?

Ideally we’d like to create an empty author instance (i.e. with no fields assigned, so no hasMany functions invoked), but hooked up to the Author class’s prototype, so it would still “quack like an Author”.

Turns out in JavaScript this is a one-liner:

const empty = Object.create(Author.prototype);

And it has all the behaviors we’re looking for:

console.log(empty instanceof Author); // true!
console.log(empty.someGetterOnPrototype); // works
console.log(empty.someClassField); // undefined but expected

Granted, “don’t use new Author” sounds like a weird approach, but serendipitously Joist already requires all entity creation to go through the EntityManager, i.e.:

// Creating new authors must use em.create
const newAuthor = em.create(Author);
// Loading existing authors must use em.load/em.find
const oldAuthor = await em.load(Author, "a:1");

So our codebases were already decoupled from the new operator, and using em.create / em.load instead, as Joist uses these em.create & em.load methods to precisely control (and optimize) the entity creation/lifecycle.

This was lucky! Our em.create / em.load API meant we already had a single choke point to swap out an optimized Object.create-based instantiation flow (instead of using new), and have the entire codebase benefit, with very few/ideally zero changes.

So far this is almost too easy–we’ve got an empty author, but of course author.books.load() does not work yet (it was a field that we skipped), so how do we get that books relation, and all the other relations, back?

Moving Relations to the Prototype

We need a caller accessing author.books to still work, and ideally “find books” on the Author’s prototype, which means something like:

The JavaScript runtime looks for a "books" key defined directly on the author instance, but does not find one (because we skipped the new Author constructor)
The JavaScript runtime looks up the prototype chain, for a "books" key, and finds Author.prototype["books"], and invokes that, with author bound to this

This flow would allow us to define books “just once” on the Author.prototype and have all author instances “get the books key” basically for free: they would inherit the "books" key from their prototype, without any extra per-author instantiation costs.

Granted, different authors can have different books, so we don’t want Author.prototype.books to be a literal Book[] array of books–but we could define it as a function that, when called with this=author1 or this=author2, returns the books for that specific author.

…which is essentially a getter (i.e. we’ve just described how getter methods in JavaScript classes work: they are just syntax sugar for “put this function on the prototype”).

So, how do we get this Author.prototype.books getter defined?

Because everything in JavaScript is “just an object”, including prototypes, we can dynamically define keys on the prototype with just an Object.defineProperty:

Object.defineProperty(Author.prototype, "books", {
  configurable: true,
  get(this: any) {
    return (this.relations["books"] ??= hasBooks());
  },
});

So easy!

We just need to call this defineProperty for author.books, author.publisher, and all other relations on Author–but how do we know what those relations are?

Probing for Relation Metadata

We want to figure out “what are the fields on Author that we should move to the prototype”? Ideally just at runtime, i.e. without any AST parsing.

Who can tell us what the fields are? …the constructor!

This is ironic, because we’ve been trying so hard to avoid the constructor (avoid calling new)–but if we just call it once, during boot, we can use it as a one-time “probe” to discover the fields, and then never call it again.

We end up with a process like:

When Joist boots, create a single “fake” new Author to let the constructor assign fields like books = hasMany(...) during the traditional object instantiation process
Teach hasBooks(...) to return a LazyField marker/wrapper that isn’t the true relation (it cannot actually hold/fetch Books), but instead a lazy version that lets us:
- a) identify “this author.books value is a lazy field”, and
- b) later ask it to create the live relation, only when needed/accessed.
For every Object.entries(author) where value intanceof LazyField, do an Object.defineProperties to “move” that field to the prototype, and call lazyField.create() when lazily accessed:

Object.defineProperty(Author.prototype, lazyField.fieldName, {
  configurable: true,
  get(this: any) {
    return (this.__fields[lazyField.fieldName] ??= lazyField.create(this));
  },
});

And that’s it–we’ve done a one-time/on-boot “probe” of new Author, to find its fields/relations, moved them to the prototype, and now going forward we can do our constructor-avoiding Object.create(Author.prototype) call.

When callers access the relations like author.books, author.publisher they will still be there, and still “look like fields” to the type system, but they’ll be lazily created.

Non-Relation Transient Fields

The approach so far works great for “class fields that are relations”, like hasMany or hasReactiveField, but what about “just regular fields”, like if you want to have some extra/random class field on an Author?

Joist already had an established convention for these “not actually persisted but sometimes useful fields”, which was wrapping them in a transientFields object:

class Author extends AuthorCodegen {
  transientFields = {
    someSpecialFlag: false,
  };

  // use the transient field in some business logic
  get someLogic(): boolean {
    return this.transientFields.someSpecialFlag ? 1 : 2;
  }
}

We created this convention solely for developer ergonomics: we want it to be abundantly clear that someSpecialFlag is not a database column, and wrapping these one-off fields in a transientFields object has been a great way to communicate that.

Here again we got lucky–because transientFields is a known convention/hardcoded name, even though we’re skipping the new operator (which normally defines the transientFields POJO on each instance), we can apply the same “move to the prototype” trick.

We defineProperty a Author.prototype.transientFields getter to lazily create each instances’ transientFields POJO on first access:

Object.defineProperty(cstr.prototype, "transientFields", {
  get(this: any) {
    // Give each instance its own lazily-created copy of transientFields
    const copy = structuredClone(value);
    // Once defined on the instance, this prototype getter won't be hit
    // again, b/c author.transientFields will immediately find its instance
    // level copy
    Object.defineProperty(this, "transientFields", { value: copy });
    return copy;
  },
});

So everything still works!

Performance Achieved

So, why do all this?

We recorded benchmarks from our internal codebase, creating one of our core entities, which has 30+ relations, across these 3 approaches:

The vanilla new cstr() with no field optimizations at all
Our previous joist-transform-properties ts-patch approach
The new Object.create + prototype approach

benchmark                            avg (min … max) p75 / p99    (min … top 1%)
---------------------------------------------------- -------------------------------

new cstr() vanilla js          4.98 µs/iter   5.00 µs   █▅
                      (4.11 µs … 176.31 µs)   8.92 µs   ██▃
                    ( 32.00  b …   1.16 mb)  29.87 kb ▁████▅▃▂▂▁▁▁▁▁▁▁▁▁▁▁▁

joist-transform-properties   258.26 ns/iter 258.62 ns      █
                    (248.18 ns … 331.49 ns) 279.27 ns      ██
                    (296.04  b … 661.47  b) 496.61  b ▂▁▂▂████▃▄▃▂▂▂▂▁▁▁▁▁▁

Object.create()              157.89 ns/iter 156.26 ns ██
                    (152.97 ns … 305.13 ns) 222.49 ns ██
                    (351.71  b … 708.99  b) 441.30  b ██▃▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Note that the 1st results 4.98µs is microseconds, while the 2nd and 3rd are nanoseconds, so after normalizing & comparing them:

Our new Object.create is 30x faster that vanilla new Author
The new Object.create is also 1.6x faster than our previous AST-transform approach (likely b/c transientFields is lazy now too)

Perhaps more importantly, for memory usage:

The new Object.create uses 70x less memory than vanilla new Author
The new Object.create uses basically the same memory as the AST approach (expected)

We’re really happy with these results!

Onward

Looking at what we’ve achieved, we’ve made all Joist relations/class fields lazy, with no decorators like @lazy or @ManyToOne, or our prior AST transforms; instead, just by taking the time to study the problem, and then leaning into one of JavaScript’s core native features–prototypes!

This work unblocks our current priority: improving the DX of our large, 500-table majestic monolith codebase, to feel as light & nimble as smaller codebases–and doing so without regressing real-world application performance.

Avoiding ORM Decorators

Sun, 27 Jul 2025 00:00:00 GMT

Joist is an entity-based ORM, i.e. an authors table gets an Author class, to hold business logic (both simple and complex validation rules, reactive fields, lifecycle hooks, etc.):

Author.ts

class Author extends AuthorCodegen {
  // Example of trivial business logic...
  get fullName(): string {
    return this.firstName + ' ' + this.lastName;
  }
}

It’s common for other entity-based ORMs to use decorators (in JavaScript/TypeScript, also called annotations in Java) to define the domain model itself, i.e. use a code-first approach.

For example, in MikroORM:

Author.ts

@Entity()
export class User {
  @PrimaryKey()
  id!: number;

  @Property({ unique: true })
  email!: string;

  @Property()
  firstName!: string;

  @Property()
  lastName!: string;

  @OneToMany(() => Order, order => order.user)
  orders = new Collection<Order>(this);
}

Or Java’s Hibernate:

Author.java

@Entity
@Table(name = "users")
public class User {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(unique = true, nullable = false)
    private String email;

    @Column(name = "first_name", nullable = false)
    private String firstName;

    @Column(name = "last_name", nullable = false)
    private String lastName;

    @OneToMany(mappedBy = "user", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
    private List<Order> orders;
}

Joist avoids decorators, and instead uses a schema-first approach, of reading, and code generating, the domain model from the database schema.

This results in pleasantly succinct entity files that are reminiscent of Rails ActiveRecord:

class User extends UserCodegen {
}

But there are a few more reasons than just succinctness for this approach.

TypeScript Decorators are Painful

Because TypeScript decorators shipped years before the official JavaScript decorators, they are clunky to use, as they require build-time infra to bake their metadata into production builds/artifacts.

When using tsc, this is as simple as turning on emitDecoratorMetadata, but then projects are limited to tsc-derived tooling like ts-node, instead of more modern tooling like tsx.

Admittedly, this should get better with official JavaScript decorators now shipped, but ORMs will have to migrate over to the new standard, which has slightly different (and less powerful, afaiu) semantics.

For Joist, instead of reading/encoding metadata from @Column decorators, Joist’s joist-codegen looks at the database directly, and then generates a metadata.ts file that is imported on boot, and “just works” (this is similar to Mikro’s EntitySchema approach, except that Joist always uses this approach, and makes it dead simple, instead of being something users have to fiddle with.)

Decorators are not DRY

For simple primitive fields, like:

  @PrimaryKey()
  id!: number;

The @PrimaryKey line is just fine, but once decorators get more complex, they can become repetitive, i.e. for something like Hibernate’s OneToMany:

    @OneToMany(mappedBy = "user", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
    private List<Order> orders;

The decorator arguments can get more & more complex, which is bad because it means:

The engineer writing orders has to repeat “we want our m2o relations to behave like (…the typical arguments we’ve decided to use…)” across the codebase, or
The engineer using orders has to wonder “how was this m2o relation configured? is it like all of our others, or different?”

Instead, Joist uses a declarative/automated approach (code generation): the “rules” or “output” for varchar columns is always the same, the output for timestamptz columns is always the same, for FK columns, etc.

Joist’s approach drives consistency across the codebase: if we have 5 foreign key columns, or 100 foreign key columns, or 500 timestamptz columns (across our entire schema), they will all act “the same way”.

This is especially important in large codebases, where consistency is key to maintainability and readability (avoiding violating the Principle of Least Surprise).

Customization via Rules

If applications need different output than Joist’s default output (i.e. handling timestamptz columns differently), Joist’s preference is to encode these as config-file rules/flags that are applied across the codebase to a pattern of columns, instead of repeatively configured/decided on a column-by-column basis.

A great example of this is our temporal config flag, which flips all date columns from being mapped as the built-in JavaScript Date type to Temporal-based types, which are much more ergonomic & correct to work with.

Other than temporal, we don’t have many other customizations available, primarily because we’ve not needed them yet–if you do, feel free to open an issue on GitHub!

Schema Should be Source of Truth

Joist’s view is that, once your application is in production, writing “diff-based” migrations (i.e. node-pg-migrate) is the better than “code-first” schema management.

This is because the diff-based migrations are heavily grounded in “what is the production data now”, vs. code-first migrations (from decorators or Prisma’s domain model file), which make it almost “too easy” to make large, sweeping domain model changes, without engineer’s really realizing (or at least deferring) how the production data will be brought along to the new world.

Using `joist-config.json`

Joist is able to get ~90-95% of metadata it needs directly from the database, but there is always that last 5% of config that is not available in the database itself–things like renaming “other side” relations, STI inheritance behavior, and a few other things.

For these, Joist uses a joist-config.json file (see its documentation).

This has been fine, but is still kind of a “least terrible” approach–eventually Joist might push all config into the database schema (via COMMENTS fields, which we use already for renaming FKs columns, or other tricks), or trying a “schema in a DSL” approach, similar to entgo’s schemadef (which is admittedly a “code first” approach, but doesn’t intermix the “schema definition” with “entity definition” like decorators do).

Pipelining for 3-6x Faster Commits

Sun, 20 Apr 2025 00:00:00 GMT

I’ve known about Postgres’s pipeline mode for a while, and finally have some prototyping of pipelining in general, and alpha builds of Joist running with pipeline mode (coming soon!).

This post is an intro to pipelining, using postgres.js and mitata to benchmark some examples.

What is Pipelining?

Pipelining, as a term in networking, allows clients to send multiple requests, immediately one after each other, without first waiting for the server to respond.

Without Pipelining

Using NodeJS talking to Postgres for illustration, the default flow of SQL statements, without pipelining, involves a full round-trip network request for each SQL statement:

Send an INSERT authors
…wait several millis for work & response…
Send an INSERT books
…wait several millis for work & response…
Send an INSERT reviews
…wait several millis for work & response…

Note that we have to wait for both:

The server to “complete the work” (maybe 1ms), and
The network to deliver the responses back to us (maybe 2ms)

Before we can continue sending the next request.

This results in a lot of “wait time”, for both the client & server, while each is waiting for the network call of the other to transfer over the wire.

With Pipelining

Pipelining allows us to remove this “extra wait time” by sending all the requests at once, and then waiting for all responses:

Send INSERT authors
Send INSERT books
Send INSERT reviews
…wait several millis for all 3 requests to complete…

The upshot is that we’re not waiting on the network before sending the server more work to do.

Not only does this let our client “send work” sooner, but it lets the server have “work to do” sooner as well–i.e. as soon as the server finishes INSERT authors, it can immediately start working on INSERT books.

Transactions Required

One wrinkle with pipelining is that if 1 SQL statement fails (i.e. the INSERT authors statement), all requests that follow it in the pipeline are also aborted.

This is because Postgres assumes the later statements in the pipeline relied on the earlier statements succeeding, so once earlier statements fail, the later statements are considered no longer valid.

This generally means pipelining is only useful when executing multi-statement database transactions, where you’re executing a BEGIN + some number of INSERT, UPDATE, and DELETE statements + COMMIT, and we already expect them to all atomically commit.

Serendipitously, this model of “this group of statements all need to work or abort” is exactly what we want anyway for a single backend request that is committing its work, by atomically saving its work to the database in a transaction–and is exactly what Joist’s em.flush does. :-)

Benchmarking Wire Latency

Per above, network latency between your machine & the database is the biggest factor in pipelining’s performance impact.

This can make benchmarking difficult and potentially misleading, because benchmarks often have the “web backend” and “the database” on the same physical machine, which means there is effectively zero network latency.

Thankfully, we can use solutions like Shopify’s toxiproxy to introduce an artificial, deterministic amount of latency to the network requests between our Node process and the Postgres database.

toxiproxy is particularly neat in that it’s easy to run as a docker container, and control the latency via POST commands to a minimal REST API it exposes:

docker-compose.yml

services:
  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.12.0
    depends_on:
      db:
        condition: service_healthy
    ports:
      - "5432:5432"
      - "8474:8474"
    volumes:
      - ./toxiproxy.json:/config/toxiproxy.json
    command: "-host=0.0.0.0 -config=/config/toxiproxy.json"

toxiproxy.json

[
  {
    "name": "postgres",
    "listen": "0.0.0.0:5432",
    "upstream": "db:5432",
    "enabled": true
  }
]

toxi-init.sh

curl -X POST http://localhost:8474/reset
curl -X POST http://localhost:8474/proxies/postgres/toxics -d '{
  "name": "latency_downstream",
  "type": "latency",
  "stream": "downstream",
  "attributes": { "latency": 2 }
}'

Is all we need to control exactly how much latency toxiproxy injects between every Node.js database call & our docker-hosted postgres instance.

Leveraging postgres.js

We’ll look at Joist’s pipeline performance in a future post, but for now we’ll stay closer to the metal and use postgres.js to directly execute SQL statements in a few benchmarks.

We’re using postgres.js instead of the venerable node-pg solely because postgres.js implements pipelining, but node-pg does not yet.

postgres.js also has an extremely seamless way to use pipelining–any statements issued in parallel (i.e. a Promise.all) within a sql.begin are automatically pipelined for us.

Very neat!

The Postgres pipelining docs make a valid point that pipelining requires async behavior, which in traditional blocking languages like Java & C, is a significant complexity increase, such that pipelining may not be worth the trade-off.

However, JavaScript is already async & non-blocking, so submitting several requests in parallel, and waiting for them to return, is extremely natural; it’s just Promise.all:

// Example of how easy/natural submitting parallel requests is in JS
const [response1, response2, response3] = await Promise.all([
  sendRequest1(),
  sendRequest2(),
  sendRequest3(),
]);

Benchmarks

0. Setup

We’ll use mitata for timing info–it is technically focused on CPU micro-benchmarks, but its warmup & other infra make it suitable to our async, I/O oriented benchmark as well.

For SQL statements, we’ll test inserting tag rows into a single-column table–for these tests, the complexity/cost of the statement itself is not that important, and a simple insert will do.

We have a few configuration parameters, that can be tweaked across runs:

numStatements the number of tags to insert
toxiLatencyInMillis the latency in millis that toxiproxy should delay each statement

As we’ll see, both of these affect the results–the higher each becomes (the more statements, or the more latency), the more performance benefits we get from pipelining.

1. Sequential Inserts

As a baseline benchmark, we execute numStatements inserts sequentially, with individual awaits on each INSERT:

bench("sequential", async () => {
  await sql.begin(async (sql) => {
    for (let i = 0; i < numStatements; i++) {
      await sql`INSERT INTO tag (name) VALUES (${`value-${nextTag++}`})`;
    }
  });
});

We expect this to be the slowest, because it is purposefully defeating pipelining by waiting for each INSERT to finish before executing the next one.

2. Pipelining with return value

This is postgres.js’s canonical way of invoking pipelining, returning a string[] of SQL statements from the sql.begin lambda:

bench("pipeline string[]", async () => {
  await sql.begin((sql) => {
    const statements = [];
    for (let i = 0; i < numStatements; i++) {
      statements.push(sql`INSERT INTO tag (name) VALUES (${`value-${nextTag++}`})`);
    }
    return statements;
  });
});

We expect this to be fast, because of pipelining.

3. Pipelining with Promise.all

This last example also uses postgres.js’s pipelining, but by invoking the statements from within a Promise.all:

bench("pipeline Promise.all", async () => {
  await sql.begin(async (sql) => {
    const statements = [];
    for (let i = 0; i < numStatements; i++) {
      statements.push(sql`INSERT INTO tag (name) VALUES (${`value-${nextTag++}`})`);
    }
    await Promise.all(statements);
  });
});

This is particularly important for Joist, because even within a single em.flush() call, we’ll execute a single BEGIN/COMMIT database transaction, but potentially might have to make several “waves” of SQL updates (technically only when AsyncReactiveFields are involved), and so can’t always return a single string[] of SQL statements to execute.

We expect this to be fast as well.

Performance Results

I’ve run the benchmark with a series of latencies & statements.

1ms latency, 10 statements:

toxiproxy configured with 1ms latency
numStatements 10
clk: ~4.37 GHz
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
runtime: node 23.10.0 (x64-linux)

benchmark                   avg (min … max)
-------------------------------------------
sequential                    15.80 ms/iter
pipeline string[]              4.16 ms/iter
pipeline Promise.all           4.21 ms/iter

summary
  pipeline string[]
   1.01x faster than pipeline Promise.all
   3.8x faster than sequential

1ms latency, 20 statements:

toxiproxy configured with 1ms latency
numStatements 20
clk: ~4.52 GHz
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
runtime: node 23.10.0 (x64-linux)

benchmark                   avg (min … max)
-------------------------------------------
sequential                    30.43 ms/iter
pipeline string[]              4.55 ms/iter
pipeline Promise.all           4.51 ms/iter

summary
  pipeline Promise.all
   1.01x faster than pipeline string[]
   6.74x faster than sequential

2ms latency, 10 statements:

toxiproxy configured with 2ms latency
numStatements 10
clk: ~4.53 GHz
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
runtime: node 23.10.0 (x64-linux)

benchmark                   avg (min … max)
-------------------------------------------
sequential                    28.85 ms/iter
pipeline string[]              7.27 ms/iter
pipeline Promise.all           7.54 ms/iter

summary
  pipeline string[]
   1.04x faster than pipeline Promise.all
   3.97x faster than sequential

2ms latency, 20 statements:

toxiproxy configured with 2ms latency
numStatements 20
clk: ~4.48 GHz
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
runtime: node 23.10.0 (x64-linux)

benchmark                   avg (min … max)
-------------------------------------------
sequential                    55.05 ms/iter
pipeline string[]              9.17 ms/iter
pipeline Promise.all          10.13 ms/iter

summary
  pipeline string[]
   1.1x faster than pipeline Promise.all
   6x faster than sequential

So, in these benchmarks, pipelining makes our inserts (and ideally future Joist em.flush calls!) 3x to 6x faster.

A few notes on these numbers:

1-2ms latency I think is a generally correct/generous latency, based on what our production app sees between an Amazon ECS container and RDS Aurora instance.

(Although if you’re using edge-based compute this can be as high as 200ms :-O)
10 statements per em.flush seems like a lot, but if you think about “each table that is touched”, whether due to an INSERT or UPDATE or DELETE, and include many-to-many tables, I think it’s reasonable for 10-tables to be a not-uncommon number.

Note that we assume your SQL statements are already batched-per-table, i.e. if you have 10 author rows to UPDATE, you should be issuing a single UPDATE authors that batch-updates all 10 rows. If you’re using Joist, it already does this for you.

Pipelining FTW

I created this raw SQL benchmark to better understand pipelining’s l-wlevel performance impact, and I think it’s an obvious win: 3-6x speedups in multi-statement transactions.

As a reminder/summary, to leverage pipelining you need three things:

A postgresql driver that supports it,
Be executing multi-statement transactions, and
Structure your code such that all the transaction statements are submitted in parallel

This last one is where Joist is the most helpful–it’s em.flush() method automatically generates the INSERTs, UPDATEs, and DELETEs for your changes, and so it can automatically submit them using a Promise.all, and not require any restructuring in your code.

In a future/next post, we’ll swap these raw SQL benchmarks out for higher-level ORM benchmarks, to see pipelining’s impact in more realistic scenarios.

Evolution of Defaults

Thu, 27 Feb 2025 00:00:00 GMT

Joist’s mission is to model your application’s business logic, with first-class support for domain modeling features & concepts.

A great example of this is Joist’s support for something as simple as default values: for example, the Author.status field should default to Active.

Joist’s default values support grew from “the simplest thing possible” (requiring adhoc patterns that engineers would copy/paste around) to a robust, first-class feature (an explicit setDefault API that “just works”).

This is a microcosm of Joist’s goal to identify the repeated patterns and pain points involved in “building a domain model”, and provide elegant features with a great DX.

Version 1. Schema Defaults

Joist’s initial defaults support was purposefully “as simple as possible”, and limited to DEFAULTs declared in the database schema, i.e. an is_archived field that defaults to FALSE, or a status_id that defaults to DRAFT:

CREATE TABLE example_table (
    id SERIAL PRIMARY KEY,
    is_archived BOOL DEFAULT false,
    status_id INTEGER DEFAULT 1,
);

Joist’s codegen would recognize these, and “apply them immediately” when creating an entity:

const a = em.create(Author, {});
expect(a.status).toBe(AuthorStatus.Draft); // already Draft
expect(a.isArchived).toBe(false); // already false

This was super-simple, and had a few pros:

Pro: The status is immediately within the em.create
- I.e. you don’t have to wait for an em.flush to “see the database default”
- Any business logic can immediately start using the default
Pro: No duplication of “draft is the default” between the database schema & TypeScript code
Con: Only supports static, hard-coded values
- Ideally we’d like to write lambdas to calculate defaults, based on business logic

Version 2. beforeCreate hooks

Being limited to static DEFAULT values is not great, so the first way of implementing more complicated “dynamic defaults” was using Joist’s beforeCreate hooks:

/** Any author created w/non-zero amount of books defaults to Published. */
authorConfig.beforeCreate("books", a => {
  if (a.status === undefined) {
    a.status = a.books.get.length > 0 ? AuthorStatus.Published : AuthorStatus.Draft;
  }
})

This was a quick-win b/c Joist already supported beforeCreate hooks, but had a few cons:

Pro: Supports arbitrary business logic
- The load hint easily enables cross-entity calculations
Con: The default logic isn’t ran until em.flush
- Harder for business logic to rely on
- Creates inconsistency between “hard-coded defaults” (applied immediately in em.create) and “dynamic defaults” (applied during flush)
Con: Susceptible to hook ordering issues
- If our default’s value depends on other defaults, it is hard to ensure the other “runs first”
Con: Boilerplate/imperative (not really a first-class feature)
- The code has to 1st check if a.status is already set (not a huge deal, but boilerplate)
- There is nothing in the code/API that identifies “this is a default”, instead we just have an adhoc pattern of “this is how our app sets defaults”
Con: Caused duplication with test factories
- Our test factories often wanted “the same defaults” applied, but Joist’s factories are synchronous, which meant any logic that was “set in beforeCreate” wouldn’t be seen right away.
- To work around this, we often “wrote twice” default logic across our entities & test factories–not great!

Version 3: Adding setDefault

We lived with the Version 1 & 2 options for several years, because they were “good enough”, but for the 3rd version, we wanted to start “setting defaults” on the road to being “more than just good enough”.

Specifically, we wanted a first-class, idiomatic way to “declaratively specify a field’s default value” instead of the previous “manually check the field in a beforeCreate hook”.

So we added config.setDefault, which accepts the field name, it’s dependencies (if any), and a lambda that would calculate the default value:

/** Calculate the Author.status default, based on number of books. */
authorConfig.setDefault("status", "books", (a) => {
  // a.books.get is available, but a.firstName is not, b/c it's not listed as a dependency
  return a.books.get.length > 0 ? AuthorStatus.Published : AuthorStatus.Draft;
})

This was a great start, but we pushed it out knowingly half-baked:

Pro: Provided scaffolding of a better future
- Gave an idiomatic way to “declare defaults”
Pro: The type system enforces that the lambda only calls fields explicitly listed in the dependency param
- This reused our ReactiveField infra and is great for ensuring dependencies aren’t missed
Con: The dependencies weren’t actually used yet
- “…ship early!”
Con: setDefault lambdas were still not invoked until em.flush
- So we still had the “write defaults twice” problem with test factories

Version 4: Dependency Aware

After having the setDefault API in production for a few months, the next improvement was to capitalize on “knowing our dependencies” and allow defaults to depend on other defaults.

For example, maybe our Author.status default needs to know whether any of the books are published (which itself is a default):

// In `Author.ts`
authorConfig.setDefault("status", { books: "status" }, a => {
  const anyBookPublished = a.books.get.some(b => b.status === BookStatus.Published);
  return anyBookPublished ? AuthorStatus.Published : AuthorStatus.Draft;
})

// In `Book.ts`
bookConfig.setDefault("status", {}, b => {
  // Some business logic that dynamically determines the status
  return BookStatus.Published;
});

Now, if both a Book and an Author are created at the same time, em.flush will ensure that the Book.status is calculated before invoking the Author.status default–we’ve solved our ordering issue!

This was a major accomplishment–cross-entity defaults had been a thorn in our side for years.

(Fwiw we readily admit this is a rare/obscure need–in our domain model of 100s of entities, we have only ~2-3 of these “cross-entity defaults”, so we want to be clear this is not necessarily a “must have” feature–but, when you need it, it’s extremely nice to have!)

Pro: Finally unlocked cross-entity defaults
Con: Still have the “write defaults twice” problem with factories

Version 5: Teaching Factories!

The next DX iteration was solving the duplication of “factories want the defaults too!”.

Looking more closely at this issue, Joist’s test factories are synchronous, which means we can create test data easily without any awaits:

// Given an author
const a = newAuthor(em);
// And a book
const b = newBook(em, { author: a });
// And setup something else using b.title
// ...if there is "default title logic", it will not have ran yet, which
// can be confusing for tests/other logic expecting that behavior
console.log(b.title);

The lack of awaits is very nice! But it does mean, if we really wanted b.title to immediately reflect its production default, we had recode the default logic into the newBook factory:

export function newBook(em: EntityManager): DeepNew<Book> {
  return newTestInstance(em, Book, {
    title: "recode the Book default logic here",
  });
}

As before, for a while this was “good enough”–but finally in this iteration, we taught the factories to leverage their “each test’s data is already in memory” advantage and just invoke the defaults immediately during the newTestInstance calls.

This works even for setDefaults that use load hints, like “author status depends on its books”:

// In `Author.ts`
authorConfig.setDefault("status", { books: "status" }, a => {
  const anyBookPublished = a.books.get.some(b => b.status === BookStatus.Published);
  return anyBookPublished ? AuthorStatus.Published : AuthorStatus.Draft;
})

In production, Joist can’t assume “the author’s books are already in-memory”, so em.flush would first load / await for the a.books to be loaded, and then invoke the lambda.

However, because our tests know that a.books is already in memory, they can skip this await, and immediately invoke the lambda.

Pro: We finally can remove the factory’s “write it twice” defaults

Version Next: findOrCreates

Always looking ahead, the next itch we have is that, currently, default lambdas that call async methods like em.find or em.findOrCreate are still skipped during newTestInstance and only run during em.flush.

Which means, for these defaults, we still have remnants of the “write it twice” defaults anti-pattern–albeit very few of them!

We should be able to lift this restriction as well, with a little bit of work (…maybe :thinking:, the newBook call is fundamentally synchronous, so maybe not).

Slow Grind to Perfection

Wrapping up, besides a “walk down memory lane”, the larger point of this post is highlighting Joist’s journey of continually grinding on DX polish–we’re about five years into Joel’s Good Software Takes 10 Years, so only another 5 to go! :smile:

Of course, it’d be great for this evolution to happen more quickly–i.e. if we had a dependency-aware, factory-aware, amazing setDefault API from day one.

But, often times jumping to an abstraction can be premature, and result in a rushed design–so sometimes it doesn’t hurt to “sit with the itch” for a little while, evolve it through multiple iterations of “good enough”, until finally a pleasant/robust solution emerges.

And, perhaps most pragmatically, small iterations helps spread the implementation out over enough hack days that it can actually get shipped. :ship:

Recursive Relations

Sat, 20 Jul 2024 00:00:00 GMT

Joist’s development is currently very incremental, and doesn’t have “big release” milestones & release notes, but we recently released a notable new feature: recursive relations. Check them out! :tada:

Is Joist the Best ORM, Ever?

Sat, 11 May 2024 00:00:00 GMT

I’ve been working on the Joist docs lately, specifically a Why Joist? page, which ended up focusing more on “why Domain Models?” than a feature-by-feature description of Joist.

Which is fine, but a good friend (and early Joist user) proofread it, and afterward challenged me that I was being too humble, and I should be more assertive about Joist being “THE BEST ORM FOR TYPESCRIPT AND POSTGRES” (his words), as he listed off his own personal highlights:

If it compiles, it works. “If you love TypeScript, you’ll love Joist.”
It’s “really effing fast” (no N+1s, ever).
We solve many common problems for you (auto-batching updates, handling the insertion order of related entities, and have many patterns for enums, polymorphic relations, etc.)
Factories make testing amazing.

All of these are true.

But in thinking about his challenge, of pitching Joist specifically as “the best ORM for TypeScript & Postgres”, I actually think I can be even more bullish and assert Joist is, currently, the best ORM, in any language, ever, TypeScript or otherwise.

Which is crazy, right? How could I possibly assert this?

I have three reasons; admittedly the first two are not technically unique to Joist, but both foundational to its design and implementation, and the third that is one of Joist’s “special sauces”:

JavaScript’s ability to solve N+1s via the event loop, and
TypeScript’s ability to model loaded-ness in its type system.
Joist’s “backend reactivity”

No N+1s: JavaScript’s Event Loop

I’ve used many ORMs over the years, going back to Java’s Hibernate, Ruby’s ActiveRecord, and a few bespoke ones in between.

Invariably, they all suffer from N+1s.

I don’t want to repeat Joist’s existing Avoiding N+1s docs, but basically “entities are objects with fields/methods that incrementally lazy-load their relations from the database” is almost “too ergonomic”, and tempts programmers into using the abstraction when they shouldn’t (i.e. in a loop), at which point N+1s are inevitable.

Again as described in “Avoiding N+1s”, JavaScript’s event loop forcing all I/O calls to “wait just a sec”, until the end of the event loop tick, gives Joist an amazing opportunity, of course via dataloader, to de-dupe all the N+1s into a single SQL call.

For everything.

This works so well, that personally I don’t know that I ever want to work in a programming language/tech stack that cannot use this trick (at least to build backend/line-of-business applications).

Granted, JavaScript is not the only language with an event loop–async Rust is a thing, Python has asyncio, and even Vert.x on the JVM provides it (I prototyped “dataloader ported to Vert.x” several years ago), and either Rust or the JVM (Scala!) would be pretty tempting just in terms of “faster than JavaScript” performance.

But the event loop is only part of the story–another critical part is TypeScript’s type system.

Other TypeScript ORMs like Prisma & Drizzle “solve” N+1s by just not modeling your domain as entities (with lazy-loaded relations), and instead force/assume a single/large up-front query that returns an immutable tree of POJOs.

This does remove the most obvious N+1 footgun (lazy-loaded relations), but it also fundamentally restricts your ability to decompose business logic into smaller/reusable methods, because now any logic that touches the database must be done “in bulk” directly by your code, and often crafted in SQL specifically to how each individual endpoint is accessing the data.

(Concretely, if you had a saveAuthor endpoint with logic/queries to validate “this author is valid”, and now write a batch saveAuthors endpoint, you could not reuse the “written for one entity” logic without rewriting it to work at the new endpoint’s grouped/batch level of granularity. Or similar for saveBook logic that you want to use within a saveAuthor that also upserts multiple children books.)

Instead, Joist’s auto-batching lets you ergonomically write code at the individual entity abstraction level (whether in a loop, or in per-entity validation rules or lifecycle hooks), but still get performant-by-default batched queries.

Loaded Subgraphs: TypeScript’s Type System

After solving N+1s with the event loop, the next biggest ergonomic problem in traditional, entity-based ORMs is tracking (or basically not tracking) loaded-ness in the type system.

Because you can’t have your entire relational database in memory, domain models must incrementally load their data from the database, as your business logic’s codepaths decide which parts they need to read.

This was another downfall of the Hibernate/ActiveRecord ORMs: there was no notion of “is this relation loaded yet?”, and so any random relation access could trigger the surprise of an expensive database I/O call, as that relation was lazy-loaded from the database.

Joist solves this by statically typing all relations as “unloaded” by default, i.e. accessing an Author’s books requires calling a1.books.load(), which returns a Promise (which is also key to the N+1 prevention above).

Which is great, I/O calls are now obvious, but “do an await for every relation access” would really suck (we tried that), so Joist goes further and uses TypeScript’s type system to not only track individual relation loaded-ness (like author1.books or book2.authors), but mark entire subgraphs of entities as populated/loaded relations and hence synchronously accessible:

// Load the Author plus the specific books + reviews subgrpah
const a1 = await em.load(Author, "a:1", {
  populate: { books: { reviews: "comments" } },
});

// a1 is typed as Loaded<Author, { books: { reviews: "comments" } }>
// Tada, no more await Promise.all
a1.books.get.forEach((book) => {
  book.reviews.get.forEach((review) => {
    console.log(review.comments.get.length);
  });
})

This combination of:

Explicit .load() / await calls for any I/O, but leveraging
Mapped types to allow compiler-checked synchronous access

For me, is also something that I never want to work without again. It’s just so nice.

Unlike JavaScript not having a monopoly on the event loop, for these mapped types I believe TypeScript effectively does have a lock on this capability, from a programming language/type system perspective.

Creating “new types” in other programming languages is generally handled by macros (Scala and Rust), or I suppose Haskell’s higher-kinded-types. But, as far as I know, none of them can combine TypeScript “mapped type + conditional type” features in a way that would allow this “take my user-defined type (Author)” and “this user-defined populate hint type” and fuse them together into a new type, that is “the author with this specific subgraph of fields marked as loaded”.

I’m happy to be corrected on this, but I think TypeScript is the only mainstream programming language that can really power Joist’s Loaded<Author, { books: "reviews" }>-style adhoc typing of subgraphs, or at least this easily.

Other TypeScript ORMs (Prisma, Drizzle, Kysley, etc.) also leverage TypeScript’s mapped types to create dynamic shapes of data, which is legitimately great.

However, they all have the fundamental approach of issuing “one-shot” queries that return immutable trees of POJOs, directly mapped from your SQL tables, and not subgraphs of entities that can have non-SQL abstractions & be further incrementally loaded as/if needed (see Why Joist for more on this).

You can generally see, for both issues covered so far (N+1s and statically-typed loaded-ness), most TypeScript ORMs have “solved” these issues by just removing the features all together, and restricting themselves to be “sophisticated query builders”.

Joist’s innovation is keeping the entity-based, incremental-loading mental model that is historically very popular/idiomatic for ORMs (particularly Ruby’s ActiveRecord), and just fundamentally fixing it to not suck.

Joist’s Backend Reactivity

This 3rd section is the first feature that is unique to Joist itself: Joist’s “backend reactivity”.

Many ORMs have lifecycle hooks (this entity was created, updated, or deleted–which Joist does as well), to organize side effects/business logic of “when X changes, do Y”.

But just lifecycle hooks by themselves can become tangled, complicated, and a well-known morass of complexity and “spooky action at a distance”.

This is because they’re basically “Web 1.0” imperative spaghetti code, where you have to manually instrument each mutation that might trigger a side effect.

(Concretely, lets say you have a rule that needs to look at both an author and its books. With raw lifecycle hooks, you must separately instrument both the “author update” and “book update” hooks to call your “make sure this author + books combination is still valid” logic. This can become tedious and error-prone, to get all the right hooks instrumented.)

Instead, Joist’s reactive fields and reactive validation rules take the lessons of “declarative reactivity” from the Mobx/Solid/reactivity-aware frontend world, and bring it to the backend: reactive rules & fields declare in one place what their “upstream dependencies” are, and Joist just handles wiring up the necessary cross-entity reactivity.

This brings a level of ease, specificity, and rigor to what are still effectively lifecycle hooks under the hood, that really makes them pleasant to work with.

The declarative nature of Joist’s domain model-wide reactivity graph is also very amenable to DX tooling & documentation generation, but we’ve not yet deeply explored/delivered any functionality that leverages it.

Conclusion: Best ORM Ever?

So, these three features are what back up my exaggerated “best ORM ever” assertion.

If tomorrow, I suddenly could not use Joist, and had to find another ORM to use (or, in general, build any sort of application backend on top of a relational database), in any current/mainstream programming language, without a doubt I would want:

Bullet-proof N+1 prevention,
Tracking loaded relation/subgraph state in the type system, and
Backend reactivity, for declarative cross-entity validation rules and reactive fields.

And Joist is the only ORM that does all three of these: two of which are uniquely enabled by the JavaScript/TypeScript stack, and the third just part of Joist’s own innovation.

Disclaimer 1: Uncomfortably Bold Claims

I usually don’t like making bold/absolutist claims, like “this or that framework is ‘the best’” or “technology x/y/z is terrible” or what not.

I did enough of that early in my career, and at this point I’m more interested in “what are the trade-offs?” and “what’s the best tool for this specific use case?”

So, I hold two somewhat incongruent thoughts in my head, as I am both:

Very confident that Joist is “the best” way to build application backends on top of a relational database, for a large majority of use cases/teams/codebases, but I also
Recognize it’s “framework” / entity approach (see Why Joist) might be either too opinionated or too much abstraction for some people’s tastes, and just in general choices & alternatives are always great to have.

My guess is if you tried Joist, you would quickly come to like it, but it’s also perfectly fine if not!

Disclaimer 2: Still a Lot To Do

Similar to the two incongruent thoughts above, another two semi-contradictory thoughts is the disclaimer that:

Joist’s core is very solid and vetted by 4+ years of production usage & continual iteration at Homebound, but also
There’s still a lot of work to do, obviously supporting other databases, but also the myriad fun, incremental improvement ideas we’re tracking in the issue tracker, and of course even more that we’ve not thought of yet.

Feedback

If you have thoughts, questions, or feedback, please let us know! Feel free to join the Joist discord, or file issues on the GitHub repo if you try Joist and run into any issues.

Despite all the hubris in this post, we are still a very small project & community, and so have a lot of growth and improvement ahead of us.

Thanks for the read!

New NextJS Sample App

Sun, 21 Apr 2024 00:00:00 GMT

We’ve added a new NextJS + Joist sample app that shows how Joist can be used in a NextJS application, with several benefits:

Automatic N+1 Prevention
JSON Payload/Props Creation
Optional Join-based Preloading

This post gives a short overview; if you’d like to watch a video, we also have a YouTube video that walks through the sample app.

Two Render Tree Approaches

While building the sample app, we found two fundamental ways of structuring a NextJS app’s render tree:

Fewer RSCs (left side), that prop drill data to the Client Components
- table.tsx is a server component that loads all data for the tree
- author-rcc-card.tsx and book-rcc-preview.tsx are client components that accept prop-drilled data
Mostly RSCs (right side), with Client Components only at the bottom
- table.tsx is a server component but only loads what it needs
- author-rsc-card.tsx and book-rsc-preview.tsx are RSC and do their own data loading

The top-level Table / table.tsx component renders each of these side-by-side, so we can see the differences, and observe some pros/cons of each approach.

With mostly RSC components, it’s easy to decompose data loading away from the top-level component.

For example, the AuthorRscCard can make its own data loading calls, and even if it’s render many pages on the page, Joist will de-dupe across the N sibling AuthorRscCards, and batch into a single SQL call.

type AuthorCardProps = {
  /** RSCs can accept the domain model enities as a prop. */
  author: Author;
  addBook: (id: string) => Promise<void>;
};

/** The RSC version of AuthorCard can load it's own data. */
export async function AuthorRscCard({ author, addBook }: AuthorCardProps) {
  // This will be auto-batched if many cards render at once
  const books = await author.books.load();
  // Or if you wanted a tree of data, this will also be auto-batched
  const loaded = await author.populate({ books: { reviews: "ratings" } });
  return <div>...jsx</div>;
}

This is nice because it allows the AuthorRscCard to be more self-sufficient, and allow the parent table component to be unaware of its children loading details.

With mostly Client components, the opposite happens, and only the parent can make database / EntityManager calls, and so is responsible for loading all the data for its children, and passing it as JSON via props:
```
type AuthorCardProps = {
  /** RCCs must accept a POJO of `Author` + all nested data. */
  author: AuthorPayload;
  addBook: (id: string) => Promise<void>;
};

/** The RCC version of AuthorCard accepts the `AuthorPayload`. */
export function AuthorRccCard({ author, addBook }: AuthorCardProps) {
  // can only use data already available on `author`
}
```
Even though the up-front data load can become awkward, it does give more opportunities for optimizations; for example Joist can use join-based preloading to load a single tree of Author + Book + Review entities in a single SQL call, which is even better optimization than the “one query per layer” N+1 prevention of the RSC-based approach.

Automatic N+1 Prevention

In either approach, Joist’s N+1 prevention auto-batches database calls, even if they are made across separate component renders.

I.e. in the RSC components:

The top-level Table component makes 1 SQL call for all Author entities.
All 2nd-level AuthorRscCard cards each make their own author.books.load() (or author.populate(...)) call, but because they’re all rendered in the same event loop, Joist batches all the load calls into 1 SQL call
Any 3rd-level components would have their load calls batched as well.

In the React Client Component approach, this auto-batching is admittedly not as necessary, assuming a singular top-level component, like Table, loads all the data at once anyway (although, as mentioned later, Joist can optimize that as well).

See the Avoiding N+1s section of our docs for more information.

JSON Payload/Props Creation

Since the client components cannot make their own async data calls, the top-level Table components is responsible for loading all the data into a JSON payload, and passing it down to the children as props.

Joist entities have an easy way of doing this is, via a toJSON method that takes the shape of data to create:

// Define the shape of data to create
export const authorHint = {
  id: true,
  firstName: true,
  books: {
    id: true,
    title: true,
    reviews: ["id", "rating"],
  },
  customField: (a) => a.id + a.title,
} satisfies JsonHint<Author>;

// This typedef can be used in the client-side props, or to match any
// endpoint-based respones types like for REST/OpenAPI.
export type AuthorPayload = JsonPayload<Author, typeof authorHint>;

const payload = await a.toJSON(authorHint);

The toJSON implementation will:

Load any relations that are not yet loaded from the database
Output only the keys that are requested in the authorHint
Call any lambdas like customField to generate custom values

As with previous examples, all data loading is N+1 safe, and also potentially join-based preloaded.

See the toJSON docs for more information.

This recursive toJSON payload generation is a relatively new feature of Joist, so if you have feature ideas that would make it more useful, please let us know!

Join-Based Preloading

The last optimization that Joist can do is join-based preloading, which can be used in either the RSC or RCC approach.

This is also a newer feature that requires opt-ing in to, but in em.ts you can add a preloadPlugin:

/** Returns this request's `EntityManager` instance. */
export const getEm = cache(() => {
  // Opt-in to preloading
  const preloadPlugin = new JsonAggregatePreloader();
  return new EntityManager({}, { driver, preloadPlugin });
});

This will allow Joist to load a deep tree/subgraph of entities in a single SQL call.

For example, normally a Joist em.find a call like:

const a = await em.find(
  Author,
  { id: 1 },
  {populate: { books: "reviews" } },
);
// Now access all the data in memory
console.log(a.books.get[0].reviews.get[0].rating)

Will issue three SQL calls:

SELECT * FROM authors WHERE id = 1;
SELECT * FROM books WHERE author_id = 1;
SELECT * FROM reviews WHERE book_id IN (1, 2, 3, ...);

But with the preloadPlugin enabled, it will use a single SQL call that uses CROSS JOIN LATERAL and json_agg to return the author’s books, and the book’s reviews (omitted for brevity) in a single row:

select a.id, _b._ as _b from authors as a
  cross join lateral
    -- create a tuple for each book, and aggregate then into an array of books
    select json_agg(json_build_array(_b.id, _b.title, _b.foreword, _b.author_id) order by _b.id) as _
    from books _b
    where _b.author_id = a.id
  ) _b
  where a.id = ? limit ?

Joist’s join-based preloading is still a beta feature, so if you run into any issues, please let us know!

What about Complex Queries?

So far, our queries have focused on loading “just entities”, and then putting those on the wire (or rendering them to HTML).

This is because Joist’s focus is on building robust domain models, and specifically helping solve the “write-side” of your application’s business logic (running the correct validation rules, lifecycle hooks, reactive updates), and less so on the “read-side” of complex queries (i.e. that using aggregates using GROUP BY, multiple nested subqueries/projections/etc.).

As such, Joist does not yet have a sophisticated query builder that can create arbitrary SQL queries, like Kysley or Drizzle.

Instead, Joist encourages an approach that uses its robust write-side features to create materialized columns in the database, such that the majority of your pages/responses really can be served by “super simple SELECT statements”, instead of using complicated queries to calculate aggregates on-the-fly.

Although you can of course use both approaches, and just use a lower-level query builder where needed.

Sample App Feedback

Joist’s roots come from the GraphQL world, so this sample app was our first foray into using it for a NextJS application. If we’ve missed any key features that would make it easier to use Joist in a NextJS app, please let us know!

Joist | Blog

Joist 2.3 Released with Rails-style Scopes!

Rails-Style Scopes

GraphQL Query Scaffolding

Batched Paginated Finds

Enum Collections

Native Temporal on Node 26

Optimized Batched Queries

Many-to-Many Tables Without id Columns

Performance Work

Smaller But Notable Changes

Upgrading

Batched Paginated Finds

The GraphQL Use Case

Why Pagination is Different

The New Pagination SQL

Removing findPaginated

Why This Matters

Simplest Possible Closure Tables

Closure Tables

…but Recursive CTEs!

Why Still Use Closure Tables?

Closure Table Maintenance

Joist Reactivity for the Win

Closing Thoughts

Joist 2.2 Released!

Faster Collection Queries with EXISTS

Goodbye Knex, Hello node-pg

Soft-Delete Resurrection

Recursive Many-to-Many Relations

Derived Property API Cleanup

collection.set Works Unloaded

EntityManager.pendingChanges

Markdown-Backed Entity Docs

Smaller But Notable Changes

Upgrading

Dual Release Channels

Bringing Back Unnest

What is unnest?

Why unnest is useful

Bringing it back?

unnest_arrays Custom Function

Pros/Cons

Using CTEs and Query Rewriting for Versioning

Aggregate What Now?

Why Version An Aggregate?

Schema Approach

…but what is “Current”?

Initial Approach: Not so great

New Approach: Write to Identities

Making Reads not Suck

1. Initial Idea: Using CTEs

2. Adding Request Config via CTEs

3. Injecting the Rewrite Automatically

4. Performance Evaluation

Skipping Some Details

Wrapping Up

Thanks

Lazy Fields for 30x speedup without Decorators or Transforms

Fields are Expensive at Scale

First Approach: Codegen

Second Approach: AST Transformers

Final Approach: JavaScript Prototypes

Skipping field initializers with Object.create

Moving Relations to the Prototype

Probing for Relation Metadata

Non-Relation Transient Fields

Performance Achieved

Onward

Avoiding ORM Decorators

TypeScript Decorators are Painful

Decorators are not DRY

Customization via Rules

Schema Should be Source of Truth

Using joist-config.json

Pipelining for 3-6x Faster Commits

What is Pipelining?

Without Pipelining

With Pipelining

Transactions Required

Many-to-Many Tables Without `id` Columns

Removing `findPaginated`

Faster Collection Queries with `EXISTS`

Goodbye Knex, Hello `node-pg`

`collection.set` Works Unloaded

`EntityManager.pendingChanges`

What is `unnest`?

Why `unnest` is useful

`unnest_arrays` Custom Function

Using `joist-config.json`