# Connectors and Trust Boundaries

Search Engineering · Diagnostic Runbook · Chapter 11

Interactive edition: https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries

Classify the external boundary, then test provenance, freshness, failure semantics, retries, and effects.

## Start with the common causes

1. [Authorization data is stale](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#acl_sync_drift)
   The index and source disagree after a policy-relevant change.
2. [A required source fails open](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#unsafe_connector_fallback)
   The agent completes without evidence the task declared necessary.
3. [An unknown effect is repeated](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#ambiguous_write_replay)
   Duplicate writes appear after timeouts or reconnects.

## Browse all symptoms

### Source integrity

- [Indexed authorization data is stale](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#acl_sync_drift): Indexed authorization attributes lag or disagree with their source.
- [Transport succeeds after meaning changes](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#connector_schema_drift): A connector response still parses but no longer means what the consumer expects.
- [An assertion has no attributable source](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#connector_provenance_missing): A policy-relevant assertion cannot be traced to a source version and observation time.

### Identity

- [Identity context expired before use](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#identity_claim_expired): A connector supplies an expired or revoked identity assertion.

### Availability and retries

- [A required source fails open](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#unsafe_connector_fallback): A missing required source silently becomes a broader or unsupported completion.
- [Throttling creates more traffic](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#rate_limit_retry_amplification): Parallel branches and retries multiply load after throttling.

### Effects

- [An unknown effect is repeated](https://mutapa.dev/search-engineering/reference/connectors-and-trust-boundaries.md#ambiguous_write_replay): A timed-out action is repeated before its original commit status is known.

## Deeper inspection and interventions

## Boundary definition

A connector translates between the trajectory and an external source of records, identity, live facts, or effects. Its role determines which failures are retryable and whether the trajectory may continue, narrow, reauthorize, reconcile, or stop.

## First response

Record connector role, operation, target, schema version, source and record version, observation time, authority reference, attempt count, latency, typed error, retry budget, idempotency reference, and reconciliation state. Do not retry an action until its prior outcome is known or safely deduplicated.

## Symptom catalog

<a id="acl_sync_drift"></a>

### `acl_sync_drift`: Indexed authorization data is stale

**Signal.** Recently changed group, tenant, classification, or deletion state disagrees between source and index.

**Confirm.** Compare source record version and observation time with the indexed authorization attributes.

**Correct.** Reconcile changed and deleted records, bound maximum staleness, and stop using stale policy inputs when the bound is exceeded.

<a id="identity_claim_expired"></a>

### `identity_claim_expired`: Identity context expired before use

**Signal.** A previously valid session or delegated call begins failing at one receiver.

**Confirm.** Validate issuer, audience, validity, revocation, principal, actor, and clock assumptions.

**Correct.** Reauthorize through the identity boundary and reevaluate policy. Never extend validity by copying old claims into state.

<a id="connector_schema_drift"></a>

### `connector_schema_drift`: Transport succeeds after meaning changes

**Signal.** Fields are missing, renamed, retyped, or semantically changed while transport still succeeds.

**Confirm.** Validate a recorded response against the expected schema and semantic fixtures.

**Correct.** Reject or quarantine mismatched payloads, version the adapter, and replay from the source after compatibility is restored.

<a id="unsafe_connector_fallback"></a>

### `unsafe_connector_fallback`: A required source fails open

**Signal.** An unavailable required source produces a fluent answer, widened query, or cached response with no partial-status label.

**Confirm.** Disable the connector and compare the output with the declared completion condition.

**Correct.** Mark sources required or optional by task. Narrow optional failures explicitly; stop when required evidence is unavailable.

<a id="rate_limit_retry_amplification"></a>

### `rate_limit_retry_amplification`: Throttling creates more traffic

**Signal.** Throttling increases total request volume, concurrent branches, or tail latency.

**Confirm.** Join attempts by trajectory and connector target. Inspect `Retry-After`, backoff, concurrency, and remaining budget.

**Correct.** Coordinate retry and concurrency budgets across branches, add jitter, and stop when the remaining trajectory budget cannot accommodate another attempt.

<a id="ambiguous_write_replay"></a>

### `ambiguous_write_replay`: An unknown effect is repeated

**Signal.** Duplicate effects appear after timeouts or reconnects.

**Confirm.** Locate all attempts for the same operation and idempotency key. Determine whether the first request committed.

**Correct.** Reconcile the original operation, reuse the same key only with the same payload, and reject key reuse with different arguments.

<a id="connector_provenance_missing"></a>

### `connector_provenance_missing`: An assertion has no attributable source

**Signal.** A classification, identity claim, or live fact cannot be tied to its source and observation time.

**Confirm.** Follow the assertion from response through transformation, storage, and decision event.

**Correct.** Persist source id, record and schema version, observed time, transformation version, and decision reference beside the governed derivative.

## Diagnostic questions

1. Is this connector supplying indexed data, verified identity, a live read, or an effect?
2. Which source, schema, provenance, and freshness assertions must hold before its payload is usable?
3. Is the source required for the completion condition, or can the result be explicitly narrowed?
4. Are retries bounded by time, attempts, and fan-out?
5. Can an ambiguous write be reconciled through the original operation or idempotency key?

## Exit criteria

- Each connector has one declared role and a role-specific contract.
- Policy-relevant assertions carry provenance and bounded freshness.
- Required-source failures stop; optional failures narrow visibly.
- Retries are budgeted, and ambiguous effects reconcile before replay.