Joist 2.5: State Machines, PostgreJS, Read Locks, Metrics
Introduction
Section titled “Introduction”Joist 2.5 is out, with quite a few new features–software is never done. 😅
State Machines 🤯
Section titled “State Machines 🤯”Joist 2.5 finally adds support for first-class state machines, built specifically for managing the business logic for the ubiquitous status fields/enums in domain models. For example, a book’s publishing workflow could look like:
import { bookConfig as config } from "./entities";
// Declare allowed transitionsconfig.setTransitions("status", { Draft: ["Review"], Review: ["Draft", "Published"], Published: [], // Once published, the book cannot change state.});
// Load the reviews and validate each attempt to publish.config.addTransitionRule("status", { to: "Published" }, "reviews", (book) => { if (book.reviews.get.length === 0) { return "A book needs at least one review before publishing"; }});
// Enqueue the job in the same transaction as the status change.config.addTransitionReaction("status", { to: "Published", phase: "commit" }, (book, ctx) => { return enqueuePublicationJob(ctx, book);});The new APIs are just three new methods on each entity’s config API:
setTransitionsdeclares allowed transitions, i.e. aBook.statusenum can go fromDrafttoReviewbut not fromPublishedback toDraftaddTransitionRuleadds rules that will be checked/invoked for each specific status transitionaddTransitionReactionsimilarly allows reacting to each specific status transition.
Note that we purposefully did not adopt an off-the-shelf framework, like XState, because our feature is very tailored towards a) driving status/state fields in a domain model, and b) integrating into Joist’s existing rules & reactions concepts as idiomatically as possible.
To a certain extent, you could already write ad hoc state machines with Joist’s existing APIs, but it was hard to capture each explicit state change, i.e. if a single request causes a Book status to “jump ahead” through a Pending → Signed → Paid flow.
The biggest win of these new transition rules/reactions APIs is that they correctly recognize these multi-hop flows and will always trigger the transitions & rules at each individual step.
See the docs for more.
PostgreJS Driver 🚀
Section titled “PostgreJS Driver 🚀”Joist 2.5 ships support for a new database driver, postgrejs, which is an alternative to the venerable node-pg.
The biggest upshot is that postgrejs has first-class support for Joist’s “lazy rows” optimization, where when 1000s of Author rows are read off the socket from postgres, we don’t immediately put all of their columns onto the heap as JS values, and instead wait until their individual getters are accessed.
This can make reading subsets of entity data ~1.5-2.5x faster. 🚀
This optimization has already been available for Joist node-pg users, via a lazyRows opt-in on our stock PostgresDriver, but internally we have to monkey patch node-pg’s internal handling of the socket buffers, which is admittedly a little suss. 🙈
So having a database driver with 1st-class support for our optimization is amazing–kudos to the postgrejs developer, Eray Hanoglu, for being super-responsive in adding support for this optimization feature directly to his driver.
See the PR for benchmarks–the tldr is that both Joist’s existing lazyRows hack & PostgreJS beat stock node-pg by ~1.5-2.5x (on benchmarks for reading a subset of fields on an entity), by using exactly the same optimization.
But now with PostgreJS we can do it without the monkey patch hackery. 🎉
See the docs for setup details.
Read-Drift Protection
Section titled “Read-Drift Protection”Joist has always leaned into optimistic locks for scaling, because it minimizes the time of each database transaction as much as possible.
For example, updating an Author entity looks like:
- Load Author
a:1by doing aSELECT * FROM authors WHERE id = 1outside a transaction- Potentially do many other SQL calls/business logic for the request
- Mutate the author in-memory
- Open a transaction with
BEGIN - Issue
UPDATE authors SET name = 'updated' WHERE id = 1 AND updated_at = 2 - Commit the transaction with
COMMIT
The “op lock” is the UPDATE ... AND updated_at = 2, which catches “what if someone else wrote to the authors row in between step 1 and step 3”–i.e. did it “drift” out from under us?
Which works great–but what if you need to read state but then not update the row?
For example, you might have business logic like “if the book.status = Draft, then …” but between you reading the books row with status=Draft, and committing your change (to a different entity/table like book_reviews), someone else updated the book status out from under you?
Enter Postgres’s FOR SHARE read locking feature. This allows the above flow to become:
- Load Book
b:1by doing aSELECT * FROM books WHERE id = 1outside a transaction- Potentially do many other SQL calls/business logic for the request
- [*] Mutate another entity
br:1in-memory - Open a transaction with
BEGIN - [*] Issue
SELECT 1 FROM books WHERE id = 1 AND updated_at = 2 FOR SHARE– the read lock- Issue
UPDATE book_reviews SET rating = 5 WHERE id = 1 AND updated_at = 2– the normal write that depends on thebooksrow having not changed
- Issue
- Commit the transaction with
COMMIT
The new query in step 4 now “makes sure no one wrote to b:1”, but does so without actually writing/updating the row itself, which is historically what Joist’s op locks have required.
Joist provides three APIs to use this new feature:
-
em.assertUnchanged(entity)manually says “fail the transaction if any other transaction touched this entity”// Allow a rating only for draft booksconst review = await em.load(BookReview, reviewId, "book");const book = review.book.get;if (!book.isDraft) {throw new Error("Ratings only allowed on draft books");}// Tell the txn to fail if book changed since we loaded item.assertUnchanged(book);review.rating = 5;await em.flush();This is the most explicit approach to having Joist use
FOR SHARElocks. -
Reactive rules & reactions can use
assertUnchanged: trueto opt all entities they read into read lock protectionimport { bookReviewConfig as config } from "./entities";// Runs anytime rating changesconfig.addRule({ assertUnchanged: true }, { rating: {}, book_ro: "status_ro" }, (review) => {if (review.rating && !review.book.get.isDraft) {return "Ratings only allowed on draft books";}});This declaratively adds read-locks to the entities in the reactive hint.
-
Specific fields can be opted-in to read lock protection across the app by adding
assertUnchanged: truetojoist-config.json.{"entities": {"Book": {"tag": "b","fields": {"status": { "assertUnchanged": true }}}}}Now any business logic reading
book.statusorbook.isDraftto make “state-driven” decisions is automatically protected withFOR SHARElocks.
A final callout is that using FOR SHARE locks incorrectly risks deadlocks, i.e. if one connection wants to lock “book 1 first, then review 2” but another connection is locking “review 2 first, then book 1”.
However, when Joist issues the FOR SHARE SQL within em.flush, we always acquire locks in the same table-alpha order, reducing deadlock risk through consistent lock ordering–does your “just a query builder” ORM do that? Didn’t think so. 😅 (Apologies, apologies.)
See the docs for more details.
Request-Level Entity Metrics
Section titled “Request-Level Entity Metrics”Another “late to the party” addition in Joist 2.5 is the ability to report per-validation rule / per-reaction metrics to your observability platform like Datadog.
This allows building dashboards/alerts on “which validation rules are running more than expected”, “when a request takes a long time, which rules/reactions was it running, and how many times?”, etc.
You can opt in to metrics tracking with setMetrics during request setup:
// Creating the em for a given request...const em = newEntityManagerForRequest();const metrics = new EntityMetrics();em.setMetrics(metrics);Then after the request is finished, you can use metrics.snapshot() to get back stats:
{ evaluations: [ { name: "Author.numberOfBooks", count: 12, totalMillis: 3.42 }, { name: "Author.addRule(Author.ts:86)", count: 8, totalMillis: 1.27 }, ],}See the docs for more details & code to copy/paste & report these to a Datadog-ish platform.
pgvector Support
Section titled “pgvector Support”Joist can do AI too! 😅 We now support pgvector & the vector(N) column type, for maintaining vector values and doing filtering/ordering by cosine distance, etc.
Note that, since vectors are typically large, this feature pairs well with Joist’s existing lazy fields feature, i.e. you can mark vector columns as lazy: true in joist-config.json, and they won’t be returned by default for the majority of callers that most likely don’t need the vector pulled into memory.
See the docs.
Easier Entity Cloning
Section titled “Easier Entity Cloning”Joist has provided an em.clone API for “copy/pasting a subgraph of entities”, and in this release it provides a new overrides prop that allows more immediately setting fields of cloned entities to known values:
// Copy/paste books from author1 to author2const a1 = await em.load(Author, "a:1", "books");const a2 = await em.load(Author, "a:2");const copiedBooks = await em.clone(a1.books.get, { overrides: { Book: () => ({ author: a2 }) },});This “copy & re-parent” pattern was already doable with em.clone, using the postClone callback to explicitly set the books’ author after the fact, but that would lead to the new books “temporarily being owned by the 1st author”, and then “quickly being moved to the 2nd author”, which could trigger unnecessary hooks & reactions.
Now we can directly say “they always belong to the 2nd author”.
Also in Joist 2.5, the overall em.clone API is more type-safe because callers can pass per-entity type callbacks for customizing the cloned entities:
const a1 = await em.load(Author, "a:1");const copiedBooks = new Map<Book, Book>();const copiedAuthor = await em.clone(a1, { // Declare the subgraph to copy/paste deep: { books: "reviews" }, // Add per-entity hooks to guide the copy/paste skipIf: { Book: (book) => book.title === "Excluded" }, overrides: { Book: (book) => ({ title: `${book.title} COPY` }), BookReview: () => ({ rating: 0 }), }, postClone: { Book: (original, clone) => { copiedBooks.set(original, clone); }, },});Each callback knows its entity type, so book.title and the returned creation options are type-checked without casts or instanceof checks.
See the docs.
Upgrading
Section titled “Upgrading”Upgrade your Joist packages together and rerun joist-codegen to refresh generated metadata/types.
All new features, i.e. the new PostgreJS driver, read locking, and metrics, are opt-in; adopting 2.5 does not require switching drivers or enabling them.
Laundry List
Section titled “Laundry List”As always, the long tail of smaller features, bug fixes, and performance fixes:
- Paginated
em.finds now use the lazy row optimization. #2133. - No-op field assignments no longer dirty Joist’s internal value caches. #2139.
em.upsertandem.findOrCreatecan now resurrect soft-deleted entities that use array columns as part of their identity/unique constraint. #2083.- EntityManager transaction-hook registrations now accept
{ forceTransaction: true }for when you have side-effects like enqueuing background jobs that need to happen via transaction hooks, even if there are not any entities to write duringem.flush(), see docs and #2123. toMatchEntitysupports lazy fields in assertions. #2092.runnow correctly mirrors lazy fields during tests. #2093.runnow correctly invalidates Joist’s internal value caches during tests. #2140.- Joist’s internal in-memory indexes skip already-deleted entities, fixing failed/stale
findWithNewOrChangedlookups. #2082. - Null/empty-array markers go through the column’s codec, fixing some Temporal and custom-serde array jankiness. #2132.
- Tell agents to stop misusing
getEntitiesin assertions jfc. #2106 · #2113 · #2115.

