Data Flow¶

This document describes the end-to-end data pipeline that transforms raw access data from source systems into decay scores, review decisions, and automated remediations. All inter-service communication flows through Apache Kafka (KRaft mode).

End-to-End Sequence¶

The following sequence diagram traces a single access event from ingestion through remediation.

sequenceDiagram
    autonumber
    participant SRC as Source System<br/>(e.g. Entra ID)
    participant CON as Connector
    participant K1 as Kafka<br/>verity.events.raw.{platform}
    participant IW as Ingest Worker
    participant K2 as Kafka<br/>verity.events.normalised
    participant NE as Normalise Engine
    participant PG as PostgreSQL<br/>(TimescaleDB)
    participant DE as Decay Engine
    participant K3 as Kafka<br/>verity.scores.updated
    participant RG as Review Generator
    participant K4 as Kafka<br/>verity.reviews.created
    participant WE as Workflow Engine<br/>(Temporal)
    participant K5 as Kafka<br/>verity.reviews.decided
    participant RE as Remediation Executor
    participant K6 as Kafka<br/>verity.remediations.completed
    participant AW as Audit Writer
    participant CH as ClickHouse

    SRC->>CON: Poll / webhook
    CON->>K1: Produce raw event
    K1->>IW: Consume raw event
    IW->>IW: Validate & deduplicate
    IW->>K2: Produce validated event
    K2->>NE: Consume validated event
    NE->>NE: Map to canonical model
    NE->>PG: Upsert Principal, Asset, AccessGrant, AccessEvent
    NE->>K2: Produce normalised event
    K2->>DE: Consume normalised event
    DE->>PG: Read grants & peer groups
    DE->>DE: Compute 6-factor decay score
    DE->>PG: Write access_scores
    DE->>K3: Produce score update
    K3->>RG: Consume score update
    RG->>RG: Evaluate policy thresholds
    RG->>PG: Create review_packet
    RG->>K4: Produce review created
    K4->>WE: Consume review created
    WE->>WE: Start Temporal workflow
    WE->>WE: Assign → Notify → Await decision
    WE->>K5: Produce review decided
    K5->>RE: Consume decided review
    RE->>SRC: Execute remediation action
    RE->>PG: Update grant status
    RE->>K6: Produce remediation completed

    Note over AW,CH: All services also emit to verity.audit.all
    K6->>AW: Consume audit events
    AW->>CH: Append to verity_audit

Kafka Topic Catalogue¶

All topics use the verity. prefix and follow the naming convention verity.<domain>.<event-type>[.<qualifier>].

Raw Event Topics¶

Pipeline Topics¶

Infrastructure Topics¶

Topic Configuration Defaults¶

Data Lifecycle¶

Access Events¶

Access Scores¶

Reviews¶

Access Review Workflow¶

The review lifecycle is implemented as a Temporal workflow in the workflow-engine service. The following state machine describes all possible transitions.

stateDiagram-v2
    [*] --> CREATED: Review packet generated
    CREATED --> ASSIGNED: Reviewer matched by policy
    ASSIGNED --> NOTIFIED: Email / Slack notification sent

    NOTIFIED --> DECIDED: Reviewer submits decision
    NOTIFIED --> ESCALATED: SLA breached (no response)

    ESCALATED --> DECIDED: Escalated reviewer decides

    DECIDED --> REMEDIATED: Decision = Revoke or Downgrade
    DECIDED --> CLOSED: Decision = Maintain

    REMEDIATED --> CLOSED: Remediation confirmed

    CLOSED --> [*]

    state DECIDED {
        [*] --> Revoke
        [*] --> Downgrade
        [*] --> Maintain
    }

Workflow States¶

Temporal Workflow Details¶

Error Handling & Dead-Letter Queues¶

Every consumer implements a retry-then-DLQ pattern:

flowchart LR
    A[Kafka Topic] --> B{Consumer}
    B -->|Success| C[Process & Commit]
    B -->|Transient Error| D[Retry — exponential backoff<br/>max 3 attempts]
    D -->|Recovered| C
    D -->|Exhausted| E[Dead-Letter Topic<br/>verity.dlq.{original-topic}]
    E --> F[Manual Investigation]

Dead-letter messages retain the original headers (x-original-topic, x-retry-count, x-error-message) for investigation and replay.