Skip to content

Kubernetes & Helm Deployment

Verity ships a comprehensive Helm chart at infra/helm/verity/ for deploying to Kubernetes. The chart is designed for Azure Kubernetes Service (AKS) but works with any conformant Kubernetes cluster.

Chart Structure

infra/helm/verity/
├── Chart.yaml                  # Chart metadata and dependencies
├── values.yaml                 # Default values
├── values-staging.yaml         # Staging overrides
├── values-prod.yaml            # Production overrides
├── charts/                     # Sub-charts
│   ├── analytics/
│   ├── api/
│   ├── audit/
│   ├── connectors/
│   ├── datastores/
│   ├── decision/
│   ├── frontend/
│   ├── ingestion/
│   └── remediation/
└── templates/                  # Shared templates
    ├── configmap.yaml
    ├── secrets.yaml
    ├── networkpolicies.yaml    # 21 network policies
    ├── prometheus-rules.yaml
    ├── rbac.yaml
    └── namespace.yaml

Sub-Charts

Each service group has its own sub-chart under charts/:

Sub-chart Services Description
analytics decay-engine, peer-analyser, anomaly-detector Access decay scoring and analysis
api api-gateway FastAPI REST API
audit audit-writer, compliance-reporter Audit trail and compliance reporting
connectors All platform connectors Source platform data ingestion
datastores PostgreSQL, ClickHouse, Redis, Kafka Data layer (typically disabled in production)
decision review-generator, workflow-engine Review packet generation and Temporal workflows
frontend dashboard-ui React web application
ingestion identity-resolver, asset-classifier, event-enricher Event enrichment pipeline
remediation remediation-executor Access revocation execution

External Dependencies (Production)

In production, infrastructure is managed externally — the datastores sub-chart is disabled:

Dependency Azure Service Configuration
PostgreSQL Azure Database for PostgreSQL (Flexible Server) global.postgresql.host
ClickHouse Self-managed or ClickHouse Cloud global.clickhouse.host
Kafka Azure Event Hubs (Kafka protocol) global.kafka.bootstrapServers
Redis Azure Cache for Redis global.redis.url
Temporal Self-managed on AKS global.temporal.host
# values-prod.yaml (excerpt)
postgresql:
  enabled: false     # Use Azure Database for PostgreSQL
redis:
  enabled: false     # Use Azure Cache for Redis
kafka:
  enabled: false     # Use Azure Event Hubs

global:
  postgresql:
    host: verity-prod-pg.postgres.database.azure.com
  kafka:
    bootstrapServers: verity-prod-eh.servicebus.windows.net:9093
    securityProtocol: SASL_SSL
  redis:
    url: rediss://verity-prod-redis.redis.cache.windows.net:6380

Shared Templates

ConfigMap

The verity-config ConfigMap centralises non-secret configuration shared by all services:

data:
  DB_HOST: "verity-prod-pg.postgres.database.azure.com"
  DB_PORT: "5432"
  KAFKA_BOOTSTRAP_SERVERS: "verity-prod-eh.servicebus.windows.net:9093"
  REDIS_URL: "rediss://verity-prod-redis.redis.cache.windows.net:6380"
  TEMPORAL_HOST: "verity-temporal:7233"
  LOG_LEVEL: "INFO"
  LOG_FORMAT: "json"
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317"

Secrets (Azure Key Vault Integration)

Secrets are managed via the Azure Key Vault CSI driver using SecretProviderClass:

Secret Key Vault Object Kubernetes Secret
Database password db-password verity-db-credentials
ClickHouse password clickhouse-password verity-clickhouse-credentials
Kafka connection string kafka-connection-string verity-kafka-credentials
Redis password redis-password verity-redis-credentials
Azure AD client secret azure-ad-client-secret verity-azure-ad
JWT signing key jwt-signing-key verity-jwt

Network Policies

When networkPolicy.enabled: true (default), 21 NetworkPolicy resources enforce least-privilege network access:

graph LR
    subgraph Allowed Traffic
        CONN["Connectors"] -->|"9092"| KAFKA["Kafka"]
        CONN -->|"5432"| PG["PostgreSQL"]
        ING["Ingestion"] -->|"9092"| KAFKA
        ING -->|"5432"| PG
        ING -->|"6379"| REDIS["Redis"]
        DE["Decay Engine"] -->|"5432"| PG
        DE -->|"9092"| KAFKA
        DE -->|"6379"| REDIS
        DEC["Decision"] -->|"5432"| PG
        DEC -->|"9092"| KAFKA
        DEC -->|"7233"| TEMPORAL["Temporal"]
        REM["Remediation"] -->|"7233"| TEMPORAL
        REM -->|"5432"| PG
        AUD["Audit"] -->|"9092"| KAFKA
        AUD -->|"5432"| PG
        API["API Gateway"] -->|"5432"| PG
        API -->|"6379"| REDIS
        AN["Analytics"] -->|"8123"| CH["ClickHouse"]
        AN -->|"5432"| PG
    end

    NGINX["Ingress"] -->|"8000"| API
    NGINX -->|"3000"| FE["Frontend"]

A default deny-all policy blocks all traffic not explicitly allowed. DNS (UDP 53) egress is allowed for all services that need it.

RBAC

Two roles are created in the verity namespace:

  • verity-role — Read access to ConfigMaps, Secrets, Pods, and Deployments
  • verity-temporal-worker-role — Read access to ConfigMaps and Secrets for Temporal workers

Both are bound to the verity-service-account ServiceAccount, which is annotated for Azure Workload Identity.

Prometheus Rules

Alert rules are defined in prometheus-rules.yaml (see Monitoring):

Alert Condition Severity
AuditWriteLagHigh verity_audit_write_lag_seconds > 30 for 2m Critical
RemediationFailed Any failures in 5m window Critical
ConnectorStopped No events ingested for 10m Warning
ReviewSLABreach CRITICAL review open > 4h Critical
ScoreComputationSlow p99 latency > 5s for 5m Warning
APIHighErrorRate 5xx rate > 5% for 5m Critical

Resource Configuration

Default Values (values.yaml)

Service Replicas CPU Request Memory Request CPU Limit Memory Limit
connectors 2 250m 512Mi 1 1Gi
ingestion 2 500m 1Gi 2 2Gi
decayEngine 2 2 4Gi 2 4Gi
decision 2 500m 1Gi 1 1Gi
remediation 2 250m 512Mi 1 1Gi
apiGateway 2 500m 1Gi 1 1Gi
analytics 1 500m 1Gi 2 2Gi
audit 2 250m 512Mi 1 1Gi
frontend 2 200m 256Mi 500m 256Mi

Production Values (values-prod.yaml)

Production scales up significantly with higher replicas and resource limits:

Service Replicas CPU Limit Memory Limit
connectors 4 2 2Gi
ingestion 4 4 4Gi
decayEngine 4 4 8Gi
decision 3 2 2Gi
remediation 3 2 2Gi
apiGateway 4 2 2Gi
analytics 2 4 4Gi
audit 3 2 2Gi
frontend 3 1 512Mi

Staging Values (values-staging.yaml)

Staging uses minimal resources (1 replica per service, reduced CPU/memory).

Horizontal Pod Autoscaler

For production, configure HPA for latency-sensitive services:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: verity-api-gateway
  namespace: verity
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: verity-api-gateway
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Installation Commands

Prerequisites

# Azure CLI login
az login
az aks get-credentials --resource-group rg-verity --name aks-verity

# Install Helm
brew install helm   # macOS
# or: curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Install / Upgrade

helm upgrade --install verity infra/helm/verity/ \
  --namespace verity \
  --create-namespace \
  --values infra/helm/verity/values.yaml \
  --values infra/helm/verity/values-staging.yaml \
  --set global.image.tag=$(git rev-parse HEAD) \
  --wait \
  --timeout 10m
helm upgrade --install verity infra/helm/verity/ \
  --namespace verity \
  --create-namespace \
  --values infra/helm/verity/values.yaml \
  --values infra/helm/verity/values-prod.yaml \
  --set global.image.tag=v1.0.0 \
  --wait \
  --timeout 15m

Verify

# Check all pods are running
kubectl get pods -n verity

# Check services
kubectl get svc -n verity

# Check ingress
kubectl get ingress -n verity

# View logs
kubectl logs -n verity -l app.kubernetes.io/component=api-gateway --tail=100 -f

Rollback

# List release history
helm history verity -n verity

# Rollback to previous revision
helm rollback verity -n verity

# Rollback to a specific revision
helm rollback verity 3 -n verity

Uninstall

helm uninstall verity -n verity