Kubernetes & Helm Deployment¶
Verity ships a comprehensive Helm chart at infra/helm/verity/ for deploying to Kubernetes. The chart is designed for Azure Kubernetes Service (AKS) but works with any conformant Kubernetes cluster.
Chart Structure¶
infra/helm/verity/
├── Chart.yaml # Chart metadata and dependencies
├── values.yaml # Default values
├── values-staging.yaml # Staging overrides
├── values-prod.yaml # Production overrides
├── charts/ # Sub-charts
│ ├── analytics/
│ ├── api/
│ ├── audit/
│ ├── connectors/
│ ├── datastores/
│ ├── decision/
│ ├── frontend/
│ ├── ingestion/
│ └── remediation/
└── templates/ # Shared templates
├── configmap.yaml
├── secrets.yaml
├── networkpolicies.yaml # 21 network policies
├── prometheus-rules.yaml
├── rbac.yaml
└── namespace.yaml
Sub-Charts¶
Each service group has its own sub-chart under charts/:
| Sub-chart | Services | Description |
|---|---|---|
analytics |
decay-engine, peer-analyser, anomaly-detector | Access decay scoring and analysis |
api |
api-gateway | FastAPI REST API |
audit |
audit-writer, compliance-reporter | Audit trail and compliance reporting |
connectors |
All platform connectors | Source platform data ingestion |
datastores |
PostgreSQL, ClickHouse, Redis, Kafka | Data layer (typically disabled in production) |
decision |
review-generator, workflow-engine | Review packet generation and Temporal workflows |
frontend |
dashboard-ui | React web application |
ingestion |
identity-resolver, asset-classifier, event-enricher | Event enrichment pipeline |
remediation |
remediation-executor | Access revocation execution |
External Dependencies (Production)¶
In production, infrastructure is managed externally — the datastores sub-chart is disabled:
| Dependency | Azure Service | Configuration |
|---|---|---|
| PostgreSQL | Azure Database for PostgreSQL (Flexible Server) | global.postgresql.host |
| ClickHouse | Self-managed or ClickHouse Cloud | global.clickhouse.host |
| Kafka | Azure Event Hubs (Kafka protocol) | global.kafka.bootstrapServers |
| Redis | Azure Cache for Redis | global.redis.url |
| Temporal | Self-managed on AKS | global.temporal.host |
# values-prod.yaml (excerpt)
postgresql:
enabled: false # Use Azure Database for PostgreSQL
redis:
enabled: false # Use Azure Cache for Redis
kafka:
enabled: false # Use Azure Event Hubs
global:
postgresql:
host: verity-prod-pg.postgres.database.azure.com
kafka:
bootstrapServers: verity-prod-eh.servicebus.windows.net:9093
securityProtocol: SASL_SSL
redis:
url: rediss://verity-prod-redis.redis.cache.windows.net:6380
Shared Templates¶
ConfigMap¶
The verity-config ConfigMap centralises non-secret configuration shared by all services:
data:
DB_HOST: "verity-prod-pg.postgres.database.azure.com"
DB_PORT: "5432"
KAFKA_BOOTSTRAP_SERVERS: "verity-prod-eh.servicebus.windows.net:9093"
REDIS_URL: "rediss://verity-prod-redis.redis.cache.windows.net:6380"
TEMPORAL_HOST: "verity-temporal:7233"
LOG_LEVEL: "INFO"
LOG_FORMAT: "json"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317"
Secrets (Azure Key Vault Integration)¶
Secrets are managed via the Azure Key Vault CSI driver using SecretProviderClass:
| Secret | Key Vault Object | Kubernetes Secret |
|---|---|---|
| Database password | db-password |
verity-db-credentials |
| ClickHouse password | clickhouse-password |
verity-clickhouse-credentials |
| Kafka connection string | kafka-connection-string |
verity-kafka-credentials |
| Redis password | redis-password |
verity-redis-credentials |
| Azure AD client secret | azure-ad-client-secret |
verity-azure-ad |
| JWT signing key | jwt-signing-key |
verity-jwt |
Network Policies¶
When networkPolicy.enabled: true (default), 21 NetworkPolicy resources enforce least-privilege network access:
graph LR
subgraph Allowed Traffic
CONN["Connectors"] -->|"9092"| KAFKA["Kafka"]
CONN -->|"5432"| PG["PostgreSQL"]
ING["Ingestion"] -->|"9092"| KAFKA
ING -->|"5432"| PG
ING -->|"6379"| REDIS["Redis"]
DE["Decay Engine"] -->|"5432"| PG
DE -->|"9092"| KAFKA
DE -->|"6379"| REDIS
DEC["Decision"] -->|"5432"| PG
DEC -->|"9092"| KAFKA
DEC -->|"7233"| TEMPORAL["Temporal"]
REM["Remediation"] -->|"7233"| TEMPORAL
REM -->|"5432"| PG
AUD["Audit"] -->|"9092"| KAFKA
AUD -->|"5432"| PG
API["API Gateway"] -->|"5432"| PG
API -->|"6379"| REDIS
AN["Analytics"] -->|"8123"| CH["ClickHouse"]
AN -->|"5432"| PG
end
NGINX["Ingress"] -->|"8000"| API
NGINX -->|"3000"| FE["Frontend"]
A default deny-all policy blocks all traffic not explicitly allowed. DNS (UDP 53) egress is allowed for all services that need it.
RBAC¶
Two roles are created in the verity namespace:
verity-role— Read access to ConfigMaps, Secrets, Pods, and Deploymentsverity-temporal-worker-role— Read access to ConfigMaps and Secrets for Temporal workers
Both are bound to the verity-service-account ServiceAccount, which is annotated for Azure Workload Identity.
Prometheus Rules¶
Alert rules are defined in prometheus-rules.yaml (see Monitoring):
| Alert | Condition | Severity |
|---|---|---|
AuditWriteLagHigh |
verity_audit_write_lag_seconds > 30 for 2m |
Critical |
RemediationFailed |
Any failures in 5m window | Critical |
ConnectorStopped |
No events ingested for 10m | Warning |
ReviewSLABreach |
CRITICAL review open > 4h | Critical |
ScoreComputationSlow |
p99 latency > 5s for 5m | Warning |
APIHighErrorRate |
5xx rate > 5% for 5m | Critical |
Resource Configuration¶
Default Values (values.yaml)¶
| Service | Replicas | CPU Request | Memory Request | CPU Limit | Memory Limit |
|---|---|---|---|---|---|
| connectors | 2 | 250m | 512Mi | 1 | 1Gi |
| ingestion | 2 | 500m | 1Gi | 2 | 2Gi |
| decayEngine | 2 | 2 | 4Gi | 2 | 4Gi |
| decision | 2 | 500m | 1Gi | 1 | 1Gi |
| remediation | 2 | 250m | 512Mi | 1 | 1Gi |
| apiGateway | 2 | 500m | 1Gi | 1 | 1Gi |
| analytics | 1 | 500m | 1Gi | 2 | 2Gi |
| audit | 2 | 250m | 512Mi | 1 | 1Gi |
| frontend | 2 | 200m | 256Mi | 500m | 256Mi |
Production Values (values-prod.yaml)¶
Production scales up significantly with higher replicas and resource limits:
| Service | Replicas | CPU Limit | Memory Limit |
|---|---|---|---|
| connectors | 4 | 2 | 2Gi |
| ingestion | 4 | 4 | 4Gi |
| decayEngine | 4 | 4 | 8Gi |
| decision | 3 | 2 | 2Gi |
| remediation | 3 | 2 | 2Gi |
| apiGateway | 4 | 2 | 2Gi |
| analytics | 2 | 4 | 4Gi |
| audit | 3 | 2 | 2Gi |
| frontend | 3 | 1 | 512Mi |
Staging Values (values-staging.yaml)¶
Staging uses minimal resources (1 replica per service, reduced CPU/memory).
Horizontal Pod Autoscaler¶
For production, configure HPA for latency-sensitive services:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: verity-api-gateway
namespace: verity
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: verity-api-gateway
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Installation Commands¶
Prerequisites¶
# Azure CLI login
az login
az aks get-credentials --resource-group rg-verity --name aks-verity
# Install Helm
brew install helm # macOS
# or: curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Install / Upgrade¶
Verify¶
# Check all pods are running
kubectl get pods -n verity
# Check services
kubectl get svc -n verity
# Check ingress
kubectl get ingress -n verity
# View logs
kubectl logs -n verity -l app.kubernetes.io/component=api-gateway --tail=100 -f
Rollback¶
# List release history
helm history verity -n verity
# Rollback to previous revision
helm rollback verity -n verity
# Rollback to a specific revision
helm rollback verity 3 -n verity