Partial Update APIs

Joist has built-in support for building “partial update”-style APIs on top of your domain model.

Partial update APIs, whether they are implemented over REST or GraphQL or GRPC, typically follow the conventions of:

• A create or update can include only a subset of fields, and any field not included in the subset is left as-is
• An update can use null as a marker to mean “unset this field”
• Updating a parent’s collection of children can be done incrementally, i.e. w/o knowing the full set of children

Joist has dedicated EntityManager.createOrUpdatePartial and Entity.setPartial APIs to help implement APIs that follow these conventions with as little boilerplate as possible.

Info

These methods are particularly useful for implementing GraphQL APIs, where Joist’s normal EntityManager.create and Entity.set methods have TypeScript types that are too strict for GraphQL’s looser/less strict type system.

For example, a SaveAuthorInput firstName that is generated by graphql-code-generator might be typed as firstName: string | null | undefined which does not match Joist’s more idiomatic Author.firstName: string | undefined typing.

Unsetting Fields with null Marker

A common pattern for partial-update style APIs is to treat null and undefined differently, i.e. { firstName: null } specifically means “unset the firstName property”, while firstName being not present (i.e. either { firstName: undefined } or an empty {}) means “do not change firstName”.

As mentioned, in a GraphQL input type this might be typed as SaveAuthorInput having a firstName: string | null | undefined field.

Normally the null-ness of input’s firstName will cause issues with Joist’s “preferred undefined” convention:

// This is typed as string | null | undefined
const { firstName } = saveAuthorInput;
const author = em.load(Author, "1");
// The normal `.set` will not compile
author.set({ firstName });
}

Instead, Joist provides a setPartial that allows code to opt-in to the partial-semantic behavior and an RPC layer’s potentially-less-strict typing:

// This is typed as string | null | undefined
const { firstName } = saveAuthorInput;
const author = em.load(Author, "1");
// Compiles successfully
author.setPartial({ firstName });
}

Specifically, the semantics of Entity.setPartial is that:

• For a required field firstName:
  • { firstName: "foo" } will update firstName
  • { firstName: undefined } will do nothing
  • { firstName: null } will cause a validation error b/c firstName is required and cannot be null
• For an optional field lastName:
  • { lastName: "bar" } will update lastName
  • { lastName: undefined } will do nothing
  • { lastName: null } will unset lastName (i.e. set it as undefined)
• For collections like books:
  • { books: [b1] } will set the collection to just b1
  • { books: null } will set the collection to []
  • { books: undefined } will do nothing

The EntityManager.createPartial and EntityManager.createOrUpdatePartial methods both have these semantics as well.

Info

Arguably, the ideal partial-update type for Author in this scenario would be:

interface SaveAuthorInput {
  firstName: string | undefined;
  lastName: string | null | undefined;
}

But sometimes it’s a challenge to get the RPC framework, e.g. GraphQL in this instance, to generate a type that exactly matches this, which is why Joist offers these more flexible setPartial methods.

Saving Parents with Children

To save both a parent and multiple potentially-new-or-existing children, Joist provides EntityManager.createOrUpdatePartial.

An example usage is:

// Given some RPC types (in this instance GraphQL)
interface SaveAuthorInput {
  id?: string | null;
  firstName?: string | null;
  books?: SaveBookInput[];
}
interface SaveBookInput {
  id?: string | null;
  title?: string | null;

}

// When the client's request comes in...
const input: SaveAuthorInput = {
  // Updating author 1
  id: "a:1",
  // To have 3 books
  books: [
    // And the 1st book is new
    { title: "new book" },
    // And the 2nd book already exists but has no changes
    { id: "b:1" },
    // And the 3rd book already exists but has a new title
    { id: "b:2", title: "updated" },
  ]
}

// Then we can apply all of those changes via
// a single call
await em.createOrUpdatePartial(Author, input);

Admittedly, your RPC/GraphQL API convention for parent/children inputs has to fairly closely follow what Joist’s own partial update / createOrUpdatePartial convention, but assuming you do so, Joist can reduce a very large amount of CRUD boilerplate in an RPC/GraphQL API.

Note

Unlike EntityManager.create, which is synchronous, EntityManager.createOrUpdatePartial is async and needs to be await-d because it may require SQL calls to look up existing entities, e.g. the b:1 and b:2 IDs in the above example.

Info

Joist’s createOrUpdatePartial behavior, while developed independently, is effectively similar Objection.js’s upsertGraph operation.

Incremental Collection Updates

Joist’s default behavior for any collection set (e.g. Entity.set, Entity.setPartial, EntityManger.createOrUpdatePartial, etc.) is for the collection to be exhaustively set to the new value, for example:

const author = newAuthor(em);
// Start out with two books
author.set({ books: [b1, b2] });
// Later set with a third book
author.set({ books: [b3] });
// Passes b/c `set` has implicitly removed the 1st two books
expect(author.books.get.length).toEqual(1);

However, when partially updating entities via an RPC call, it’s often convenient to change only a single child of the collection, especially for APIs where the child itself doesn’t have a dedicated operation (i.e. saving an invoice line item can only be done via the invoice API).

To support these APIs, setPartial and createOrUpdatePartial will both opt-in to incremental collection semantics if they detect an extra op hint key on the children. For example:

const author = newAuthor(em);
// Start out with two books
author.set({ books: [b1, b2] });
// Later add a third book
author.setPartial({ books: [
  // And include the `op` hint
  { op: "include", title: "b3" }
]});
// Passes b/c `setPartial` saw `op` and worked incrementally
expect(author.books.get.length).toEqual(3);

The valid values of an op keyword are:

• { op: "delete", id: ... } will remove the child and EntityManager.delete it for hard deletion
• { op: "remove", id: ... } will remove the child but not call EntityManager.delete
• { op: "include", id: ... } will include (i.e. add if needed, or just update if it already an existing child) the child

If any child has an op key, then all children must have an op key, to be the most clear/consistent to the caller that incremental semantics are being used.

Note

The op key is not an actual column stored in your database or domain model, i.e. the Book entity should not have an op field.

Instead, op can be just an extra key on the RPC/GraphQL input types to specifically driven Joist’s incremental collection semantics.

The key name op was chosen for both succinctness and also low probably of overlapping with real fields in the domain model.

Caution

Because incremental semantics are enabled by the presence of an op key, if the collection is empty it will fundamentally look like an exhaustive set that clears the collection.

// An empty collection will always clear the books, and can't be
// treated as a incremental operation
author.setPartial({ books: [] });

This means that if a client is trying to optimize its call by sending “only changed children”, when no children have been changed then it should not include the child key in the payload at all, to avoid unintentionally clearing the collection.

Technically, Joist currently supports a 4th { op: "incremental" } keyword that can act as a “fake child” and will be ignored (i.e. not treated as an entity to add/remove from the collection), but will still enable incremental semantics and so avoid the “empty list clears the collection” gotcha.

Info

If you’re working in React, the form-state library has built-in knowledge of Joist’s op keyword/incremental collection semantics and makes it easy to craft optimized/only-what-changed payloads with a single form.changedValue call.

Legacy Incremental Collection Updates

Joist’s initial version of incremental updates used separate delete and remove keys instead of op:

const author = newAuthor(em);
// Start out with three books
author.set({ books: [b1, b2, b3] });
// Then delete b1, remove b2, leave b3 alone, and add b4
author.setPartial({ books: [
  { id: "b:1", delete: true },
  { id: "b:2", remove: true },
  { id: "b:4" },
]});

These are still supported, but are soft-deprecated and the op key is preferred going forward.

Info

We moved away from delete and remove keywords for two reasons:

1. They are more likely to overlap with existing fields in the domain model, and
2. It is ergonomically easier for our frontend UI to bind to an always-present op key, and just flip its value from include/delete when the user toggles adding/removing rows, instead of adding/removing entire delete / remove keys.