Data Integrity

This guide covers unique constraints, versioning with optimistic concurrency, and idempotency — mechanisms for ensuring data correctness in concurrent environments.

Unique Constraints

Unique constraints ensure that a field (or combination of fields) has a unique value across all items of an entity type. In DynamoDB, there are no built-in unique indexes — effect-dynamodb enforces uniqueness using sentinel items and transactional writes.

Declaration

const Users = Entity.make({
  model: User,
  entityType: "User",
  primaryKey: {
    pk: { field: "pk", composite: ["userId"] },
    sk: { field: "sk", composite: [] },
  },
  timestamps: true,
  unique: {
    email: ["email"], // single-field uniqueness
    tenantEmail: ["tenantId", "email"], // compound uniqueness
    username: ["username"], // another single-field
  },
})

How It Works

For each unique constraint, the system creates a sentinel item — a separate DynamoDB item whose existence proves that a value is taken.

Sentinel key format:

PK: $myapp#v1#user.email#alice@example.com
SK: $myapp#v1#user.email

The sentinel contains a back-reference to the entity’s primary key (_entity_pk / _entity_sk), naming the item that owns it.

Transactional Enforcement

All mutations involving unique constraints are automatically wrapped in DynamoDB transactions. Sentinel operations are sparse — they are emitted only for constraints whose composing fields are all present on the record (see Sparse Constraints below).

Operation	Transaction Items
Put (new item)	Entity item + sentinel per unique field whose composites are all set (condition: attribute_not_exists(pk))
Put (over an existing item)	Entity item + for each changed unique value, release old sentinel + put new sentinel
Update (unique field unchanged)	Entity item only (no sentinel operations)
Update (undefined → defined)	Entity item + put new sentinel
Update (defined → undefined)	Entity item + release old sentinel
Update (defined → defined, changed)	Entity item + release old sentinel + put new sentinel
Delete	Entity item + release sentinel per unique field whose composites were set

A write releases only a sentinel the item owns. An item can hold a unique value without owning its sentinel: the constraint was added after the item was written, the item was written outside the library, or a ttl’d reservation expired and another item then claimed the value. That other item owns the sentinel now, and deleting it would let the value be taken twice. So before releasing a sentinel, a put, update, upsert, delete, soft delete, transaction put or purge reads it, releases it only if its back-reference names this item, and conditions the delete on that still being true. Each sentinel a write would release costs one consistent GetItem. A release whose reservation changed hands in between fails an update, or a delete with a .condition(), with ConcurrentModification on the unique fields; a put, an upsert, a transaction put and a delete with no condition plan the write again from a fresh read (up to three attempts). purge releases the sentinels of the live item and of every tombstone, one at a time.

A put of an entity with unique constraints reads the item first. Over an existing item it rotates the sentinels of changed values and leaves unchanged ones alone, so it never orphans a sentinel, and it keeps the item’s createdAt unless the input supplies one. A put replaces the whole item, so if another writer changes, creates or deletes the item between the read and the write, the put reads it again and retries: the last writer wins, as with a plain PutItem. Only a race lost on every attempt fails, with OptimisticLockError on a versioned entity and ConcurrentModification on an unversioned one. create doesn’t read the item: it must be missing anyway.

upsert reads the item once on these entities, and validates the whole input as a create would. A missing item is created with its sentinels. An existing item is updated from that read, with sentinels rotated for changed unique values only, under the same guards as an update; fields the input omits keep their stored values. A concurrent create or delete in between is retried the other way, and a sentinel release that raced is planned again from a fresh read, as a put’s is; a concurrent change of the item itself fails the upsert, as it fails an update. A race lost on every attempt fails with OptimisticLockError or ConcurrentModification, and a value another item holds fails with UniqueConstraintViolation. On entities without unique constraints or retain, upsert stays a single UpdateItem, unless its input omits a defaulted index composite (see Defaults and Unique Constraints).

Error Handling

When a unique value is already taken, the write fails with UniqueConstraintViolation:

  // #region unique-error-handling
  yield* db.entities.Users.put({
    userId: "u-1",
    email: "alice@example.com",
    username: "alice",
    tenantId: "t-acme",
    displayName: "Alice",
  })

  // Same email → UniqueConstraintViolation
  const duplicateResult = yield* db.entities.Users.put({
    userId: "u-2",
    email: "alice@example.com",
    username: "bob",
    tenantId: "t-acme",
    displayName: "Bob",
  })
    .asEffect()
    .pipe(
      Effect.catchTag("UniqueConstraintViolation", (e) => {
        // e.entityType === "User"
        // e.constraint === "email" or "username"
        // e.fields === { email: "alice@example.com" }
        return Effect.succeed(
          `Blocked: constraint="${e.constraint}", fields=${JSON.stringify(e.fields)}`,
        )
      }),
    )
  // Blocked: constraint="email", fields={"email":"alice@example.com"}

Compound Constraints

Compound constraints combine multiple fields. The sentinel key includes all field values:

unique: {
  tenantEmail: ["tenantId", "email"],
}
// Sentinel PK: $myapp#v1#user.tenantemail#t-acme#alice@example.com

This allows alice@example.com to exist in different tenants but not twice within the same tenant.

Sparse Constraints

A unique constraint is sparse with respect to its composing fields: a sentinel is only written when every field is present on the record. If any composite is undefined or null, the constraint is silently skipped — the record is excluded from that constraint, just as a sparse GSI excludes records with missing key composites.

This matters for any unique constraint that references an optional model field:

class Vehicle extends Schema.Class<Vehicle>("Vehicle")({
  vehicleId: Schema.String,
  accountId: Schema.String,
  name: Schema.NonEmptyString,
  deviceBinding: Schema.optional(Schema.String),  // optional
  transponderId: Schema.optional(Schema.String),  // optional
}) {}

const Vehicles = Entity.make({
  model: Vehicle,
  entityType: "Vehicle",
  primaryKey: { pk: { field: "pk", composite: ["vehicleId"] }, sk: { field: "sk", composite: [] } },
  unique: {
    nameInAccount: ["accountId", "name"],   // required composites — always enforced
    deviceBinding: ["deviceBinding"],        // sparse — only enforced when set
    transponderId: ["transponderId"],        // sparse — only enforced when set
  },
})

Two records can coexist with the same constraint left unset:

// Both succeed — no sentinel is written for deviceBinding/transponderId on either record
yield* db.entities.Vehicles.create({ vehicleId: "v-1", accountId: "acct-1", name: "Truck A" })
yield* db.entities.Vehicles.create({ vehicleId: "v-2", accountId: "acct-1", name: "Truck B" })

Update transitions claim and release the sentinel as the field becomes set or unset:

// undefined → defined: the new value is now reserved
yield* db.entities.Vehicles.update({ vehicleId: "v-1" }).set({ deviceBinding: "device-xyz" })

// Another record trying to claim the same value is blocked
const conflict = yield* db.entities.Vehicles.update({ vehicleId: "v-2" })
  .set({ deviceBinding: "device-xyz" })
  .asEffect()
  .pipe(Effect.catchTag("UniqueConstraintViolation", (e) => Effect.succeed(e.constraint)))
// "deviceBinding"

// defined → undefined: the value is released and can be claimed elsewhere
yield* db.entities.Vehicles.update({ vehicleId: "v-1" }).pipe(Entity.remove(["deviceBinding"]))
yield* db.entities.Vehicles.update({ vehicleId: "v-2" }).set({ deviceBinding: "device-xyz" })
// Succeeds — v-1 released the value

Sparse semantics mean you can declare a unique constraint on any optional field without worrying that records leaving that field unset will collide on a synthesized “missing value” key.

Defaults and Unique Constraints

A decoding default never creates a sentinel. When a write omits a unique field that has a Schema.withDecodingDefault, the field holds only its default, and no value is reserved for it. If that field is also an index composite, the default is stored so the item stays indexed, and the field is listed in a hidden string-set attribute, __edd_d__. Decoded models never include it. The field gets its sentinel once a write supplies its value. A .remove() of the field stores the default again, keeps the item indexed under it, and releases the old value’s sentinel without creating one for the default.

An upsert stores an omitted defaulted index composite only when it creates the item. Over an existing item it keeps the stored value, whether or not the entity has unique constraints. Such an upsert reads the item first to tell the two apart.

Versioning

When versioned is enabled on an Entity, every item carries a version field that auto-increments on each write.

Declaration

The simplest form just adds a version field with no retention:

const SimpleEntity = Entity.make({
  // ...
  versioned: true,  // adds version field, no retention
})

With version retention and optional TTL:

const VersionedUsers = Entity.make({
  model: User,
  entityType: "VersionedUser",
  primaryKey: {
    pk: { field: "pk", composite: ["userId"] },
    sk: { field: "sk", composite: [] },
  },
  timestamps: true,
  versioned: {
    retain: true, // keep version history
    ttl: Duration.days(90), // auto-expire old versions
  },
  unique: {
    email: ["email"],
    username: ["username"],
  },
})

You can also customize the version field name:

const CustomEntity = Entity.make({
  // ...
  versioned: { field: "revision", retain: true },
})

How It Works

• Put of a new item sets version = 1
• Put over an existing item continues it: the next version, the same incarnation token and createdAt, and a retain snapshot of the item it replaces (see below)
• Update uses SET version = version + 1 (DynamoDB atomic counter)
• The version field appears in Entity.Record<E> (readable) but not in Entity.Input<E> or Entity.Update<E> (system-managed)

Version Retention

With retain: true, every mutation stores a snapshot of the previous state as a separate DynamoDB item:

PK: $myapp#v1#user#u-1  SK: $myapp#v1#user           ← current (version 4)
PK: $myapp#v1#user#u-1  SK: $myapp#v1#user#v#0000001  ← snapshot v1
PK: $myapp#v1#user#u-1  SK: $myapp#v1#user#v#0000002  ← snapshot v2
PK: $myapp#v1#user#u-1  SK: $myapp#v1#user#v#0000003  ← snapshot v3

All versions are co-located (same partition key) for efficient access. Version numbers are zero-padded for correct lexicographic sort order.

Version history is never overwritten. An update, delete, restore or put that would write a v#N snapshot holding a different state from the row already there fails with a ValidationError and writes nothing. A write from outside the library that changes an item without bumping its version can still end up in the next v#N snapshot.

Puts on Versioned Entities

A put of a versioned entity reads the item first, so a put over an existing item continues it instead of resetting it to version 1. It takes the next version, keeps the item’s incarnation token and createdAt (unless the input supplies one), and with retain writes a snapshot of the item it replaces. A race with another writer is retried, and the last writer wins; a race lost on every attempt fails with OptimisticLockError. A soft-deleted item counts as missing. create doesn’t read the item; it fails with ConditionalCheckFailed on an existing one. An upsert of a retain entity reads the item once and snapshots the item it replaces.

With retain, a deleted item’s version history outlives it, and the key can be used again without purge. A delete, hard or soft, snapshots the item’s final state at its own version in the same transaction. A put, create, upsert or transaction put of the missing item reads the highest version retained for its key and continues after it (a retain create makes no other read: one Limit 1 query of its history, plus, when the primary sort key has composites, a keys-only Limit 1 query for history an earlier release wrote, which it reads in full only if that finds some): an item deleted at version 3 comes back at version 4, with a new incarnation token and its own v#0000004 snapshot. The earlier history is never overwritten, and a writer still holding version 3 fails with OptimisticLockError. A restore of the old tombstone while the new item is live fails with ItemNotDeleted.

Transaction.transactWrite and EventStore additionalItems write a put of a versioned or unique-constrained entity exactly as its own put does: they read the item, continue or create it, snapshot it and rotate its sentinels, all in the one transaction. A race between that read and the transaction is retried. A taken unique value fails with UniqueConstraintViolation; only an op’s own .condition() fails as a condition (TransactionCancelled, or AdditionalItemConditionFailed from append). DynamoDB allows one operation per item in a transaction, so it rejects a transaction that touches one item twice, and the write fails with a ValidationError naming both operations and the item; that includes two puts that swap unique values, which touch the same sentinels. Nothing is written, and the rejection is never mistaken for a lost race and retried, even when DynamoDB reports it as a failed condition. One whose items pass DynamoDB’s 4 MB limit, where a retain put counts twice (its item and its snapshot), or that holds an item over DynamoDB’s 400 KB per-item limit, is rejected by DynamoDB, so nothing is written; it fails with a ValidationError naming the item and its key (for 4 MB, the largest items), with DynamoDB’s error as its cause. A transaction still can’t delete an entity with unique, retain or softDelete (EDD-9048). Batch.write still sends puts of a versioned entity as create-only transactions (see Batch Operations).

Querying Versions

// Get a specific version snapshot
const v1 = yield* db.entities.VersionedUsers.getVersion({ userId: "v-1" }, 1)

// Query version history (most recent first)
const history = yield* db.entities.VersionedUsers.collect(
  VersionedUsers.versions({ userId: "v-1" }),
)

TTL on Snapshots

With ttl: Duration.days(90), version snapshots get a DynamoDB TTL attribute. DynamoDB automatically deletes expired items. The current item is never TTL’d — only historical snapshots.

Optimistic Concurrency

Versioning enables optimistic locking. Pass expectedVersion on update to ensure no one else has modified the item since you read it.

  // #region optimistic-concurrency
  // Create a versioned user
  const user = yield* db.entities.VersionedUsers.put({
    userId: "v-1",
    email: "versioned@example.com",
    username: "versioned-alice",
    tenantId: "t-acme",
    displayName: "Alice",
  })
  // user.version === 1

  // Update twice to build history
  yield* db.entities.VersionedUsers.update({ userId: "v-1" }).set({ displayName: "Alice V2" })
  // version is now 2

  const current = yield* db.entities.VersionedUsers.update({ userId: "v-1" }).set({
    displayName: "Alice V3",
  })
  // current.version === 3

  // Update with optimistic lock — must match current version
  yield* db.entities.VersionedUsers.update({ userId: "v-1" })
    .set({ displayName: "Alice V4" })
    .expectedVersion(3)
  // Succeeds: version was 3, now 4

If another process updated the item between your read and write, the version will have changed and the update fails:

  // #region optimistic-lock-error
  // Stale version → OptimisticLockError
  const lockResult = yield* db.entities.VersionedUsers.update({ userId: "v-1" })
    .set({ displayName: "Stale Update" })
    .expectedVersion(2)
    .asEffect()
    .pipe(
      Effect.catchTag("OptimisticLockError", (e) =>
        Effect.succeed(`Blocked: expected version ${e.expectedVersion}, actual ${e.actualVersion}`),
      ),
    )
  // Blocked: expected version 2, actual 4

Under the Hood

The entity adds a ConditionExpression to the DynamoDB update:

ConditionExpression: #version = :expected
ExpressionAttributeNames: { "#version": "version" }
ExpressionAttributeValues: { ":expected": { N: "5" } }

If the condition fails, DynamoDB returns ConditionalCheckFailedException together with the stored item, and the entity tells the two causes apart by the stored version:

• A lost version race is an OptimisticLockError carrying the real actualVersion.
• A failed .condition() is a ConditionalCheckFailed.
• An update of a missing item is an ItemNotFound.

That holds on every update path, including retain entities. Earlier versions mixed these up in three cases:

• A versioned update with expectedVersion and a .condition() reported a failed condition as OptimisticLockError(-1).
• A retain update with a .condition() did the same.
• A retain path update with a .condition() reported a lost version race as ConditionalCheckFailed.

A versioned update of a missing item with expectedVersion also used to give OptimisticLockError(-1).

Read-then-write updates

Some updates read the item before writing: a unique-field change, a computed change (.add(), .append(), …) to an index composite, and any update on a retain entity. They write a guarded update of only what changed, so a concurrent change to an unrelated attribute is preserved. If another writer changed something the update read, it fails without writing: with OptimisticLockError on a versioned entity, and with ConcurrentModification on an unversioned one. Retrying re-reads the item.

Versioned entities carry a hidden __edd_i__ attribute, a token set when the item is created and added to existing items on their next guarded write. It never appears in decoded models, only in asNative output and raw items. Version-checked writes require it to match, so an item deleted and recreated at the same version is never mistaken for the original.

An update too wide for one DynamoDB expression (4 KB, 300 operators) writes the whole item instead. A versioned entity writes it under its version condition. An unversioned one writes it under the fallback guard soft delete uses (see Lifecycle). A concurrent write from outside the library to an attribute that guard doesn’t cover is lost, which 1.22.0 also did for every such update. A .condition() too large to fit beside the guard is sent as built: DynamoDB rejects it, nothing is written, and the write fails with a ValidationError stating the expression’s size and how much of it is your .condition(). Operators are counted as DynamoDB counts them: BETWEEN counts once, and in an update expression each +, - and function counts, but a SET clause’s = doesn’t.

Items written before an entity was versioned read as version 0, and expectedVersion(0) addresses them; their first versioned write produces version 1. An item that has the __edd_i__ token but no version is different: its version was removed outside the library. Reads and the writes that check the version refuse it with a ValidationError instead of reading it as version 0, and a query over a partition holding one fails as a whole. Restore the version attribute to read it again, or remove the item: purge removes it on any entity, and on an entity without unique constraints or retain a plain hard delete does too, since it reads nothing and writes no history.

Known limitations, both inherent:

• On unversioned entities, nothing can prove an unguarded attribute unchanged. So the item a unique-field update returns may show stale values for attributes it neither reads nor writes, wide items use the fallback guard, and the whole-item write of a wide update can overwrite writers outside the library. Use versioned where that matters.
• A plain .expectedVersion(n) can’t detect a delete-and-recreate that has climbed back to version n on an entity without retain. Its versions restart at 1, so it takes n − 1 updates after the recreate. A retain item continues after its retained history instead.

Idempotency

Idempotency keys prevent duplicate processing of the same request. In effect-dynamodb, this is implemented as a unique constraint with TTL — no new concept needed.

Declaration

const Payments = Entity.make({
  model: Payment,
  entityType: "Payment",
  primaryKey: {
    pk: { field: "pk", composite: ["paymentId"] },
    sk: { field: "sk", composite: [] },
  },
  timestamps: true,
  unique: {
    idempotencyKey: { fields: ["idempotencyKey"], ttl: Duration.hours(1) },
  },
})

The Payment model is a standard Schema.Class:

class Payment extends Schema.Class<Payment>("Payment")({
  paymentId: Schema.String,
  amount: Schema.Number,
  currency: Schema.String,
  idempotencyKey: Schema.String,
}) {}

How It Works

1. Client sends a request with a unique idempotencyKey (e.g., a UUID).
2. The entity creates a sentinel item for the key.
3. If the same idempotencyKey is sent again within the TTL window, the sentinel already exists and the write fails with UniqueConstraintViolation.
4. After the TTL expires, DynamoDB deletes the sentinel, allowing the key to be reused.

An update that changes the key gives its new sentinel the same TTL. Once a reservation has expired, another item can claim the value, and the item that first held it no longer owns the sentinel, so its later writes leave the new reservation alone (see Transactional Enforcement).

  // #region idempotency-usage
  const requestId = "idem-abc-123"

  // First request — succeeds
  const payment = yield* db.entities.Payments.put({
    paymentId: "pay-001",
    amount: 99.99,
    currency: "USD",
    idempotencyKey: requestId,
  })

  // Retry with same idempotency key → blocked
  const retry = yield* db.entities.Payments.put({
    paymentId: "pay-002",
    amount: 99.99,
    currency: "USD",
    idempotencyKey: requestId,
  })
    .asEffect()
    .pipe(
      Effect.catchTag("UniqueConstraintViolation", (e) =>
        e.constraint === "idempotencyKey"
          ? Effect.succeed("Duplicate request prevented")
          : Effect.fail(e),
      ),
    )
  // The sentinel has a TTL of 1 hour
  // After expiry, the same key can be reused

Transaction Item Budget

DynamoDB limits transactions to 100 items. When multiple features are enabled, each mutation consumes more of this budget:

Mutation	Items Used
Simple put (no features)	1
Put + 1 unique constraint	2
Put + 2 unique constraints + version snapshot	4
Update (unique field changed) + version snapshot	4
Put over an existing item (unique field changed) + version snapshot	4
Soft delete + 2 sentinels + version snapshot	5

Plan your entity design accordingly. For entities with many unique constraints, ensure the total transaction items stay well under 100.

The budget is not checked before sending. A write whose transaction passes 100 items is sent, rejected whole by DynamoDB, so nothing is written, and fails with TransactionOverflow (entityType, operation, itemCount, limit), with DynamoDB’s error as its cause. The same applies to DynamoDB’s other write limits (an item over 400 KB, a transaction over 4 MB, a partition key over 2048 bytes, a sort key over 1024 bytes), which fail with a ValidationError naming the item and its key — for a key, the attribute too, e.g. the partition key "pk" of the User put, at pk=… is 2100 bytes, longer than DynamoDB's 2048-byte limit for a partition key. Key sizes count UTF-8 bytes, and apply to the table’s key and to every index key the item carries; the library composes keys from your composite attributes, so a long composite value (a free-text field, a URL) is what usually crosses them. See Guarded puts expand in the API reference.

How DynamoDB Sizes Items

The library never checks a DynamoDB limit before sending. It measures a request only after DynamoDB has rejected it, to name the item at fault, and to choose where Batch.write splits its transactions of versioned puts. Both use one measure: DynamoDB’s own size accounting, measured against DynamoDB Local and confirmed at the boundaries. An item of 409,600 bytes is accepted and 409,601 is rejected. A transaction of 4,194,304 bytes is accepted and 4,194,305 is rejected. “400 KB” and “4 MB” are binary units.

An item is the sum, over its attributes, of the attribute name’s UTF-8 bytes plus the value’s size. There is no per-item or per-attribute overhead.

Type	Size
S (string)	its UTF-8 bytes
B (binary)	its byte length
BOOL, NULL	1
N (number)	0 is 1. Otherwise 1, plus one byte per base-100 digit pair of the magnitude, plus 1 if negative (see below)
SS, NS, BS (sets)	the sum of the elements’ sizes, with no overhead
L (list)	3, plus each element’s size + 1
M (map)	3, plus each entry’s name (UTF-8) + value size + 1

Numbers are counted in pairs of digits. The pairs are aligned to the decimal point, and pairs that are all zeros are dropped from both ends. So 1, 10, 99, 100, 1000000, 0.5 and 1E-10 are 2 bytes. 101, 120, 1.5 and 12.34 are 3. 12345 and 1.234 are 4. A negative number adds 1 (-1 is 3, -123 is 4), except at the maximum precision of 38 digits, so no number takes more than 21 bytes.

A transaction counts each action and adds them up:

• A Put counts its item. An Update, Delete or ConditionCheck counts only its key, never the stored item or the item an update produces.
• Each action’s ConditionExpression and UpdateExpression count as well, at their written length, plus every placeholder occurrence they contain: a #name adds the attribute name’s UTF-8 bytes, and a :value adds the value’s size. A placeholder used twice counts twice.

Expressions count towards the 4 MB transaction limit but not towards an item’s 400 KB. A 409,600-byte item with a condition is accepted, alone or in a transaction.

Batch.write splits its create-only versioned-put transactions at exactly 100 actions and 4,194,304 bytes by this count. Each put includes its attribute_not_exists condition and the key attribute name it substitutes. The application’s own transactions (Transaction.transactWrite, EventStore.append and commandHandler, entity writes) are never split: they are sent whole, and a size rejection comes back diagnosed.

AWS documents only “the aggregate size of the items”, so production DynamoDB might count a request differently. When DynamoDB rejects a transaction for size that the library measured as fitting, the ValidationError gives both numbers: the library’s measure, and the Payload Size DynamoDB reported. On DynamoDB Local that figure is a running total, taken at the action where it stopped counting. The library does not re-split or retry that request.

What’s Next?

• Lifecycle — Soft delete, restore, purge, and version retention
• Advanced — Rich updates, batch operations, conditional writes