You don't pay for a crawl.
You pay for a change.

Not every tracked field needs the same treatment. Where the monitored fields are flat scalars — a vessel's sanctions programs, a trial's status string — a direct hash over those fields is already deterministic, since the field set and its order are fixed by the code, not by whatever order the source happened to return them in. Where a monitored field is a genuinely nested object — FDA Orange Book's per-product patent and exclusivity data, for example — a plain hash isn't safe on its own: sorting only the top-level keys leaves a nested object's own key order unsorted.

That gap is not hypothetical. An earlier version of this fleet used exactly that top-level-only JSON.stringify(obj, Object.keys(obj).sort()) on a nested field, and two semantically identical records could still hash differently depending on what order the source API returned a nested object's keys in — producing false STATUS_CHANGE classifications, and a STATUS_CHANGE is a billable event. The fix, used specifically where a tracked field is genuinely nested, is the fully recursive canonicalizer below — it sorts object keys at every nesting level, not just the top one. Flat-field actors hash directly instead; there is no nesting for a recursive step to protect against there.

function canonicalize(value) {
  if (Array.isArray(value)) return value.map(canonicalize)
  if (value !== null && typeof value === 'object') {
    return Object.keys(value)
      .sort()
      .reduce((acc, key) => {
        acc[key] = canonicalize(value[key])
        return acc
      }, {})
  }
  return value
}

// The bug this replaced: JSON.stringify(obj, Object.keys(obj).sort())
// only sorts the replacer's TOP-LEVEL key list. Nested objects keep
// whatever key order the source happened to send them in, so two
// semantically identical records could hash differently — and a
// STATUS_CHANGE is a billable event. canonicalize() recurses into
// every array and object first, so key order can never affect the
// fingerprint.
const fingerprint = createHash('sha256') // sha1 on a few older actors
  .update(JSON.stringify(canonicalize(monitoredFields)))
  .digest('hex')

A fingerprint is only useful if it survives past the run that computed it. Apify's run-scoped Actor.getValue() is discarded when the run ends, so the delta engine instead opens a named key-value store — one that persists across every scheduled run of that actor, not just the current one — and reads and writes the last-seen fingerprint for each record key there.

// Run-scoped — discarded when the run ends, useless for diffing
// across scheduled runs:
//   await Actor.getValue('LAST_SEEN')

// Named, cross-run store — persists between scheduled runs:
const stateStore = await Actor.openKeyValueStore('delta-state-v1')
const lastSeen = await stateStore.getValue(recordKey)

const classification = !lastSeen
  ? 'NEW'
  : lastSeen.fingerprint === fingerprint
    ? 'SNAPSHOT_NO_DIFF' // unchanged — never re-delivered, never billed
    : 'STATUS_CHANGE'    // or an actor-specific event: UPDATED, AMENDED...

await stateStore.setValue(recordKey, { fingerprint, seenAt: Date.now() })

Every record's current fingerprint is compared against the last-seen fingerprint in the named store and classified into one of four outcomes. Three of them are real event names drawn live from this fleet's own deltaEvents lists — grouped below, with a count of how many of the 18 actors emit each one. The fourth, the no-diff case, is deliberately not a delta event at all.

A global unhandled-exception handler on process catches both uncaughtException and unhandledRejection before the process exits, and records the stack trace along with memory and run-duration telemetry at the moment of failure — so a crash leaves a diagnosable trail instead of a silent run failure in the Apify Console.

Individual request retries use full-jitter exponential backoff — a random delay bounded by a doubling base and a cap — rather than a fixed retry interval, so a batch of failures doesn't all retry against the source site at the same instant. A request that exhausts its retry budget is written to a dead-letter queue — a named key-value store recording the failure — instead of being silently dropped from the dataset.

process.on('uncaughtException', (err) => captureAndExit(err))
process.on('unhandledRejection', (reason) => captureAndExit(reason))

async function captureAndExit(err) {
  await deadLetterStore.setValue('failure-' + Date.now(), {
    stack: err instanceof Error ? err.stack : String(err),
    memoryMb: Math.round(process.memoryUsage().rss / 1024 / 1024),
    durationMs: Date.now() - runStartedAt,
  })
  process.exit(1)
}

// Full-jitter exponential backoff between retries:
function backoffMs(attempt, base, cap) {
  return Math.random() * Math.min(cap, base * Math.pow(2, attempt))
}

// A request that exhausts its retry budget is written here instead
// of being silently dropped from the dataset:
await deadLetterStore.setValue('exhausted-' + recordKey, {
  recordKey: recordKey,
  attempts: MAX_RETRIES,
  lastError: lastError && lastError.message,
})