System Overview¶

Verity is an Access Decay Intelligence Platform that continuously scores how "decayed" every access grant has become, then orchestrates reviews and automated remediation. The platform is implemented as a Python 3.12 monorepo containing 19 microservices organised across five processing planes.

High-Level Architecture¶

The following C4 Context diagram illustrates the major system boundaries, external integrations, and data stores.

C4Context
    title Verity — System Context Diagram

    Person(reviewer, "Access Reviewer", "Reviews flagged access grants and approves or revokes them.")
    Person(admin, "Platform Admin", "Configures connectors, policies, and platform settings.")

    System_Boundary(verity, "Verity Platform") {
        System(api, "API Gateway", "FastAPI — REST API for all client interactions.")
        System(ingest, "Ingest Plane", "Connectors + Ingest Worker — pulls access data from source systems.")
        System(normalise, "Normalise Plane", "Normalise Engine — maps raw events to the canonical model.")
        System(score, "Score Plane", "Decay Engine — computes 6-factor access-decay scores.")
        System(review, "Review Plane", "Review Generator + Workflow Engine — creates and orchestrates reviews.")
        System(remediate, "Remediate Plane", "Remediation Executor — revokes or modifies access in source systems.")
        SystemDb(pg, "PostgreSQL + TimescaleDB", "Operational store — principals, assets, grants, scores, reviews.")
        SystemDb(ch, "ClickHouse", "Audit & analytics — immutable event log, compliance reports.")
        SystemQueue(kafka, "Kafka (KRaft)", "Event streaming backbone between all planes.")
        System(redis, "Redis", "Caching layer — score look-ups, session data, rate limiting.")
        System(temporal, "Temporal", "Durable workflow orchestration for review lifecycle.")
    }

    System_Ext(azuread, "Microsoft Entra ID", "Identity provider & access-data source.")
    System_Ext(snowflake, "Snowflake", "Data warehouse access-data source.")
    System_Ext(databricks, "Databricks", "Analytics platform access-data source.")
    System_Ext(fabric, "Microsoft Fabric", "Data platform access-data source.")
    System_Ext(dashui, "Dashboard UI", "Next.js SPA for reviewers and admins.")

    Rel(reviewer, dashui, "Uses")
    Rel(admin, dashui, "Manages")
    Rel(dashui, api, "HTTPS / JSON")
    Rel(api, pg, "SQL")
    Rel(api, redis, "Cache reads")
    Rel(ingest, azuread, "Microsoft Graph API")
    Rel(ingest, snowflake, "Snowflake SQL API")
    Rel(ingest, databricks, "Databricks REST API")
    Rel(ingest, fabric, "Fabric REST API")
    Rel(ingest, kafka, "Produces raw events")
    Rel(kafka, normalise, "Consumes raw events")
    Rel(normalise, pg, "Writes normalised data")
    Rel(normalise, kafka, "Produces normalised events")
    Rel(kafka, score, "Consumes normalised events")
    Rel(score, pg, "Reads grants / Writes scores")
    Rel(score, kafka, "Produces score events")
    Rel(kafka, review, "Consumes score events")
    Rel(review, temporal, "Starts workflows")
    Rel(review, kafka, "Produces review events")
    Rel(kafka, remediate, "Consumes decided reviews")
    Rel(remediate, azuread, "Revokes access")

Service Inventory¶

Verity comprises 12 application services and 5 infrastructure components.

Application Services¶

Infrastructure Components¶

Technology Stack¶

The Five Planes¶

Verity's processing pipeline is organised into five sequential planes. Each plane is independently scalable and communicates exclusively through Kafka topics.

flowchart LR
    subgraph INGEST["① Ingest Plane"]
        direction TB
        CAD[connector-azure-ad]
        CSF[connector-snowflake]
        CDB[connector-databricks]
        CFB[connector-fabric]
        IW[ingest-worker]
        CAD --> IW
        CSF --> IW
        CDB --> IW
        CFB --> IW
    end

    subgraph NORMALISE["② Normalise Plane"]
        NE[normalise-engine]
    end

    subgraph SCORE["③ Score Plane"]
        DE["decay-engine<br/><small>scoring.py — 6-factor formula</small><br/><small>batch.py — Polars batch</small>"]
    end

    subgraph REVIEW["④ Review Plane"]
        RG[review-generator]
        WE["workflow-engine<br/><small>(Temporal)</small>"]
        RG --> WE
    end

    subgraph REMEDIATE["⑤ Remediate Plane"]
        RE[remediation-executor]
    end

    INGEST -- "verity.events.raw.*" --> NORMALISE
    NORMALISE -- "verity.events.normalised" --> SCORE
    SCORE -- "verity.scores.updated" --> REVIEW
    REVIEW -- "verity.reviews.decided" --> REMEDIATE

    subgraph CROSSCUT["Cross-Cutting Services"]
        AG[api-gateway]
        AW[audit-writer]
        CR[compliance-reporter]
        DU[dashboard-ui]
    end

    AG -. "REST API" .-> INGEST
    AG -. "REST API" .-> SCORE
    AG -. "REST API" .-> REVIEW
    AW -. "verity.audit.all" .-> CROSSCUT

    subgraph STORAGE["Data Stores"]
        PG[(PostgreSQL + TimescaleDB)]
        CH[(ClickHouse)]
        KF[[Kafka — KRaft]]
        RD[(Redis)]
        TM[Temporal Server]
    end

Plane Details¶

① Ingest Plane¶

② Normalise Plane¶

③ Score Plane¶

④ Review Plane¶

⑤ Remediate Plane¶

Shared Libraries¶

All services depend on a set of shared libraries in the libs/ directory.

Deployment Topology¶

flowchart TB
    subgraph K8S["Kubernetes Cluster"]
        subgraph ns_verity["namespace: verity"]
            api[api-gateway ×2]
            cad[connector-azure-ad ×1]
            csf[connector-snowflake ×1]
            cdb[connector-databricks ×1]
            cfb[connector-fabric ×1]
            iw[ingest-worker ×3]
            ne[normalise-engine ×2]
            de[decay-engine ×2]
            rg[review-generator ×1]
            we[workflow-engine ×1]
            re[remediation-executor ×1]
            aw[audit-writer ×2]
            cr[compliance-reporter ×1]
            du[dashboard-ui ×2]
        end
        subgraph ns_infra["namespace: verity-infra"]
            pg[(PostgreSQL + TimescaleDB)]
            ch[(ClickHouse)]
            kf[[Kafka Brokers ×3]]
            rd[(Redis Sentinel)]
            tm[Temporal Server]
        end
    end
    api --> pg
    api --> rd
    iw --> kf
    ne --> kf
    ne --> pg
    de --> pg
    de --> kf
    rg --> kf
    we --> tm
    re --> kf
    aw --> ch
    aw --> kf

Each service is deployed as a Kubernetes Deployment with horizontal pod autoscaling (HPA) based on CPU and Kafka consumer lag. Infrastructure components run in a dedicated verity-infra namespace with StatefulSet controllers.