Audit Writer¶

Path: services/audit/audit-writer/ · Type: Consumer

The Audit Writer maintains the immutable audit trail for the entire Verity platform. It consumes all audit events from Kafka and writes them to ClickHouse, providing a durable, append-only record of every action taken by the system and its users.

Architecture¶

graph LR
    K1[verity.audit.all] --> AW[Audit Writer]
    AW --> CH[(ClickHouse\nverity_audit)]

Data Flow¶

sequenceDiagram
    participant Kafka as verity.audit.all
    participant AW as Audit Writer
    participant CH as ClickHouse

    Kafka->>AW: Audit event batch
    AW->>AW: Validate & transform
    AW->>CH: INSERT INTO verity_audit (batch)
    CH-->>AW: Acknowledge
    AW->>Kafka: Commit offsets

ClickHouse Schema¶

The verity_audit table uses a MergeTree engine ordered by timestamp for efficient time-range queries:

Column	Type	Description
`event_id`	`UUID`	Unique event identifier
`event_type`	`LowCardinality(String)`	Event category (e.g., `REVIEW_STARTED`, `DECISION_SUBMITTED`)
`actor_id`	`UUID`	User or service that triggered the event
`actor_type`	`LowCardinality(String)`	`USER`, `SYSTEM`, `SERVICE`
`resource_type`	`LowCardinality(String)`	Entity type affected
`resource_id`	`UUID`	Entity ID affected
`action`	`LowCardinality(String)`	`CREATE`, `UPDATE`, `DELETE`, `REVOKE`, `APPROVE`
`details`	`String`	JSON-encoded event-specific context
`timestamp`	`DateTime64(3, 'UTC')`	Event timestamp
`ingested_at`	`DateTime64(3, 'UTC')`	Write timestamp

Immutability Guarantees¶

Append-only: The writer only performs INSERT operations — no UPDATE or DELETE.
No mutation: The ClickHouse table is configured without ALTER or DELETE permissions for the writer service account.
Checksumming: Each batch is written with a SHA-256 checksum stored as metadata for tamper detection.
Replication: ClickHouse is deployed with ReplicatedMergeTree for durability.

Retention Policy¶

Policy	Value	Description
Retention period	7 years	Required for SOX, HIPAA, and GDPR compliance
TTL	`MODIFY TTL timestamp + INTERVAL 2555 DAY`	Automatic expiration after 7 years
Compression	`ZSTD(3)`	Column-level compression for storage efficiency
Partitioning	Monthly (`toYYYYMM(timestamp)`)	Partition pruning for fast time-range queries

Configuration¶

Variable	Required	Default	Description
`AUDIT_KAFKA_BOOTSTRAP`	Yes	—	Kafka bootstrap servers
`AUDIT_CONSUMER_GROUP`	No	`verity-audit-writer`	Kafka consumer group ID
`AUDIT_CLICKHOUSE_URL`	Yes	—	ClickHouse HTTP URL
`AUDIT_BATCH_SIZE`	No	`500`	Events per ClickHouse insert batch
`AUDIT_FLUSH_INTERVAL_SECONDS`	No	`5`	Max seconds before flushing a partial batch
`AUDIT_LOG_LEVEL`	No	`INFO`	Python log level

Observability¶

Metric	Type	Description
`audit_events_written_total`	Counter	Events written to ClickHouse (by event type)
`audit_batch_size`	Histogram	Events per batch insert
`audit_write_duration_seconds`	Histogram	ClickHouse insert latency
`audit_consumer_lag`	Gauge	Kafka consumer group lag
`audit_errors_total`	Counter	Write failures