Service Discovery Stack

Overview

The Service Discovery Stack provides service mesh capabilities, secrets management, and service registry for the HealthFlow NDP platform using HashiCorp Consul and Vault.

Stack Architecture

Components

1. HashiCorp Consul

Purpose: Service discovery, health checking, and service mesh

Version: Consul 1.18.x

Features:

• Service Discovery: Automatic service registration and DNS-based discovery
• Health Checking: Built-in health checks for services
• KV Store: Distributed key-value store for configuration
• Service Mesh: Secure service-to-service communication with Consul Connect
• Multi-datacenter: Support for federation across data centers

Use Cases:

• Service registration and discovery
• Configuration management
• Distributed locks
• Service segmentation
• Health monitoring

2. HashiCorp Vault

Purpose: Secrets management and data protection

Version: Vault 1.16.x

Features:

• Dynamic Secrets: Generate secrets on-demand with automatic rotation
• Data Encryption: Encrypt data in transit and at rest
• Leasing & Renewal: All secrets have a lease with automatic renewal
• Revocation: Secrets can be revoked immediately
• Audit Logging: Detailed audit trail of all operations
• High Availability: Support for HA deployment with Consul backend

Secrets Managed:

• Database credentials (PostgreSQL, MySQL, MongoDB, Redis)
• API keys and tokens
• TLS certificates
• Encryption keys
• Cloud provider credentials
• Third-party service credentials (EDA, MOH, Insurance)

3. Consul Agents

Purpose: Local service mesh proxy and health checking

Deployment: DaemonSet (runs on all nodes)

Features:

• Local DNS interface for service discovery
• Health check execution
• Service registration
• Sidecar proxy for service mesh

Kubernetes Manifests

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: discovery-stack
  labels:
    name: discovery-stack
    stack: infrastructure

Consul StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: discovery-stack
spec:
  serviceName: consul
  replicas: 3
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - consul
            topologyKey: kubernetes.io/hostname
      containers:
      - name: consul
        image: hashicorp/consul:1.18
        args:
          - "agent"
          - "-server"
          - "-bootstrap-expect=3"
          - "-ui"
          - "-data-dir=/consul/data"
          - "-bind=0.0.0.0"
          - "-client=0.0.0.0"
          - "-advertise=$(POD_IP)"
          - "-retry-join=consul-0.consul.discovery-stack.svc.cluster.local"
          - "-retry-join=consul-1.consul.discovery-stack.svc.cluster.local"
          - "-retry-join=consul-2.consul.discovery-stack.svc.cluster.local"
          - "-domain=cluster.local"
          - "-datacenter=healthflow-dc1"
          - "-disable-host-node-id"
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        ports:
        - containerPort: 8500
          name: http
        - containerPort: 8600
          name: dns
        - containerPort: 8300
          name: server
        - containerPort: 8301
          name: serf-lan
        - containerPort: 8302
          name: serf-wan
        volumeMounts:
        - name: consul-data
          mountPath: /consul/data
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /v1/status/leader
            port: 8500
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /v1/status/peers
            port: 8500
          initialDelaySeconds: 10
          periodSeconds: 5
  volumeClaimTemplates:
  - metadata:
      name: consul-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 20Gi

Consul Service

apiVersion: v1
kind: Service
metadata:
  name: consul
  namespace: discovery-stack
  labels:
    app: consul
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: consul
  ports:
  - name: http
    port: 8500
    targetPort: 8500
  - name: dns
    port: 8600
    targetPort: 8600
  - name: server
    port: 8300
    targetPort: 8300
  - name: serf-lan-tcp
    port: 8301
    targetPort: 8301
    protocol: TCP
  - name: serf-lan-udp
    port: 8301
    targetPort: 8301
    protocol: UDP

Consul UI Service

apiVersion: v1
kind: Service
metadata:
  name: consul-ui
  namespace: discovery-stack
spec:
  type: ClusterIP
  selector:
    app: consul
  ports:
  - name: http
    port: 8500
    targetPort: 8500

Vault StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  namespace: discovery-stack
spec:
  serviceName: vault
  replicas: 3
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - vault
            topologyKey: kubernetes.io/hostname
      containers:
      - name: vault
        image: hashicorp/vault:1.16
        args:
          - "server"
        env:
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: VAULT_API_ADDR
          value: "http://$(POD_IP):8200"
        - name: VAULT_CLUSTER_ADDR
          value: "https://$(POD_IP):8201"
        ports:
        - containerPort: 8200
          name: http
        - containerPort: 8201
          name: internal
        volumeMounts:
        - name: config
          mountPath: /vault/config
        - name: vault-data
          mountPath: /vault/data
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 512Mi
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
        livenessProbe:
          httpGet:
            path: /v1/sys/health?standbyok=true
            port: 8200
          initialDelaySeconds: 60
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
            port: 8200
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: vault-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 10Gi

Vault ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: discovery-stack
data:
  vault.hcl: |
    ui = true
    
    storage "consul" {
      address = "consul:8500"
      path    = "vault/"
    }
    
    listener "tcp" {
      address     = "0.0.0.0:8200"
      tls_disable = 1
    }
    
    api_addr = "http://POD_IP:8200"
    cluster_addr = "https://POD_IP:8201"
    
    disable_mlock = false

Vault Service

apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: discovery-stack
spec:
  type: ClusterIP
  selector:
    app: vault
  ports:
  - name: http
    port: 8200
    targetPort: 8200
  - name: internal
    port: 8201
    targetPort: 8201

Service Dependencies

Deployment Instructions

1. Create Namespace

kubectl create namespace discovery-stack

2. Deploy Consul

# Deploy Consul StatefulSet
kubectl apply -f consul/statefulset.yaml

# Deploy Consul Services
kubectl apply -f consul/service.yaml
kubectl apply -f consul/service-ui.yaml

# Wait for Consul to be ready
kubectl wait --for=condition=ready pod -l app=consul -n discovery-stack --timeout=300s

3. Verify Consul

# Check Consul pods
kubectl get pods -n discovery-stack -l app=consul

# Check Consul cluster members
kubectl exec -n discovery-stack consul-0 -- consul members

# Access Consul UI (port-forward)
kubectl port-forward -n discovery-stack svc/consul-ui 8500:8500
# Visit http://localhost:8500

4. Deploy Vault

# Create Vault config
kubectl apply -f vault/configmap.yaml

# Deploy Vault StatefulSet
kubectl apply -f vault/statefulset.yaml

# Deploy Vault Service
kubectl apply -f vault/service.yaml

# Wait for Vault pods
kubectl wait --for=condition=ready pod -l app=vault -n discovery-stack --timeout=300s

5. Initialize Vault

# Initialize Vault (run once)
kubectl exec -n discovery-stack vault-0 -- vault operator init -format=json > vault-keys.json

# IMPORTANT: Save vault-keys.json securely! It contains unseal keys and root token

# Extract unseal keys
UNSEAL_KEY_1=$(cat vault-keys.json | jq -r '.unseal_keys_b64[0]')
UNSEAL_KEY_2=$(cat vault-keys.json | jq -r '.unseal_keys_b64[1]')
UNSEAL_KEY_3=$(cat vault-keys.json | jq -r '.unseal_keys_b64[2]')
ROOT_TOKEN=$(cat vault-keys.json | jq -r '.root_token')

# Unseal Vault on all pods
for pod in vault-0 vault-1 vault-2; do
  kubectl exec -n discovery-stack $pod -- vault operator unseal $UNSEAL_KEY_1
  kubectl exec -n discovery-stack $pod -- vault operator unseal $UNSEAL_KEY_2
  kubectl exec -n discovery-stack $pod -- vault operator unseal $UNSEAL_KEY_3
done

# Login to Vault
kubectl exec -n discovery-stack vault-0 -- vault login $ROOT_TOKEN

6. Configure Vault

# Enable secrets engines
kubectl exec -n discovery-stack vault-0 -- vault secrets enable -path=secret kv-v2
kubectl exec -n discovery-stack vault-0 -- vault secrets enable database

# Enable authentication methods
kubectl exec -n discovery-stack vault-0 -- vault auth enable kubernetes

# Configure Kubernetes auth
kubectl exec -n discovery-stack vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc.cluster.local:443"

Vault Secrets Configuration

Database Secrets Engine

# Configure PostgreSQL dynamic secrets
kubectl exec -n discovery-stack vault-0 -- vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="ndp-app" \
  connection_url="postgresql://{{username}}:{{password}}@postgresql.data-stack:5432/ndp?sslmode=disable" \
  username="vault" \
  password="vault-password"

# Create role for dynamic credentials
kubectl exec -n discovery-stack vault-0 -- vault write database/roles/ndp-app \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Static Secrets

# Store static secrets
kubectl exec -n discovery-stack vault-0 -- vault kv put secret/ndp/api \
  eda_api_key="eda-secret-key" \
  moh_api_key="moh-secret-key"

# Store Redis password
kubectl exec -n discovery-stack vault-0 -- vault kv put secret/ndp/redis \
  password="redis-password"

Application Access Policy

# Create policy for NDP applications
kubectl exec -n discovery-stack vault-0 -- vault policy write ndp-app - <<EOF
# Read application secrets
path "secret/data/ndp/*" {
  capabilities = ["read", "list"]
}

# Generate database credentials
path "database/creds/ndp-app" {
  capabilities = ["read"]
}
EOF

# Bind policy to Kubernetes service account
kubectl exec -n discovery-stack vault-0 -- vault write auth/kubernetes/role/ndp-app \
  bound_service_account_names=prescription-service,dispense-service \
  bound_service_account_namespaces=applications \
  policies=ndp-app \
  ttl=1h

Consul Service Registration

Manual Service Registration

# Register a service in Consul
kubectl exec -n discovery-stack consul-0 -- consul services register - <<EOF
{
  "Name": "postgresql",
  "Address": "postgresql.data-stack.svc.cluster.local",
  "Port": 5432,
  "Tags": ["database", "postgres"],
  "Check": {
    "TCP": "postgresql.data-stack.svc.cluster.local:5432",
    "Interval": "10s"
  }
}
EOF

Automatic Service Registration

Applications can register themselves using Consul's Kubernetes sync:

apiVersion: v1
kind: Service
metadata:
  name: prescription-service
  namespace: applications
  annotations:
    "consul.hashicorp.com/service-name": "prescription-service"
    "consul.hashicorp.com/service-tags": "ndp,api,v1"
    "consul.hashicorp.com/service-port": "8080"
spec:
  selector:
    app: prescription-service
  ports:
  - port: 8080
    targetPort: 8080

Resource Requirements

Estimates

These are rough estimates for development/staging. Production should be scaled based on load.

Service	CPU	Memory	Storage	Replicas	Notes
Consul	0.5 core	1 GB	20 GB	3	HA cluster
Vault	0.5 core	512 MB	10 GB	3	HA cluster
Consul Agent	0.1 core	128 MB	-	Per node	DaemonSet

Best Practices

Vault Security

1. Never commit unseal keys to Git
2. Use auto-unseal in production (Hardware HSM, Transit secrets engine, or local key management)
3. Enable audit logging
4. Rotate root token regularly
5. Use namespaces for multi-tenancy
6. Enable MFA for sensitive operations

Consul Configuration

1. Enable ACLs in production
2. Use TLS for all communication
3. Implement intentions for service mesh security
4. Monitor cluster health
5. Backup Consul data regularly

Secret Management

1. Use dynamic secrets whenever possible
2. Set appropriate TTL for leases
3. Implement secret rotation
4. Use Vault agent for secret injection
5. Audit secret access

Troubleshooting

Vault is Sealed

# Check Vault status
kubectl exec -n discovery-stack vault-0 -- vault status

# Unseal Vault
kubectl exec -n discovery-stack vault-0 -- vault operator unseal <unseal-key>

Consul Cluster Not Forming

# Check Consul logs
kubectl logs -n discovery-stack consul-0

# Check cluster members
kubectl exec -n discovery-stack consul-0 -- consul members

# Check Consul peers
kubectl exec -n discovery-stack consul-0 -- consul operator raft list-peers

Application Can't Access Vault

# Test Kubernetes auth
kubectl exec -n applications prescription-service-xxx -- \
  curl -s http://vault.discovery-stack:8200/v1/auth/kubernetes/login \
  -d '{"jwt":"'$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)'","role":"ndp-app"}'

# Check Vault logs
kubectl logs -n discovery-stack vault-0

# Verify policy
kubectl exec -n discovery-stack vault-0 -- vault policy read ndp-app

Service Not Registering in Consul

# Check Consul catalog
kubectl exec -n discovery-stack consul-0 -- consul catalog services

# Check service health
kubectl exec -n discovery-stack consul-0 -- consul catalog nodes -service=<service-name>

# Check Consul agent logs
kubectl logs -n discovery-stack -l app=consul-agent

Security Considerations

Vault

• Seal Configuration: Use auto-unseal in production
• Access Control: Implement strict policies
• Audit Logging: Enable and monitor audit logs
• Encryption: Enable transit secrets engine for app-level encryption
• Backup: Regular snapshots of Vault data

Consul

• ACL System: Enable and configure ACL tokens
• TLS Encryption: Encrypt all Consul communication
• Intentions: Define service-to-service communication rules
• Gossip Encryption: Enable gossip protocol encryption
• Network Segmentation: Use Consul Enterprise namespaces

High Availability

Consul HA

• 3 or 5 server nodes for quorum
• Anti-affinity rules to spread across availability zones
• Health checks on all endpoints
• Automated backups of Consul state

Vault HA

• 3 or 5 Vault instances with Consul backend
• Load balancer for active/standby routing
• Auto-unseal for automatic recovery
• Regular snapshots of Consul backend

Next Steps

• Application Stack - Deploy NDP services
• Gateway Stack - Configure ingress for Consul/Vault UI
• Monitoring Stack - Monitor Consul and Vault