Monitoring¶

Monitor AegisAI platform health, performance, and security events using built-in health endpoints, status APIs, and external observability tools.

Health Endpoints¶

Liveness¶

Confirms the API process is running:

curl http://localhost:8000/api/v1/health

{
  "status": "healthy",
  "version": "1.1.0"
}

Readiness¶

Checks all backend dependencies:

curl http://localhost:8000/api/v1/health/ready

{
  "status": "ready",
  "checks": {
    "database": true,
    "redis": true,
    "opa": true
  }
}

Use readiness for Kubernetes/Docker health probes:

readinessProbe:
  httpGet:
    path: /api/v1/health/ready
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 15

livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 30

Platform Status¶

Detailed status for the dashboard connection indicator:

curl http://localhost:8000/api/v1/status \
  -H "X-API-Key: <your-api-key>"

Field	Description
`status`	`online` or `degraded`
`version`	Platform version
`tenant`	Tenant ID, name, masked API key
`services.database`	`online` or `offline`
`services.redis`	`online` or `offline`
`services.opa`	`online` or `offline`
`uptime`	Human-readable uptime
`timestamp`	UTC timestamp

Service Health Matrix¶

Service	Port	Health Check	Impact if Down
PostgreSQL	5432	`SELECT 1`	All data operations fail
Redis	6379	`PING`	Real-time events, pub/sub fail
OPA	8181	`/health`	Policy checks use fallback engine; Guard uses local regex fallback
FastAPI	8000	`/api/v1/health`	All API operations fail
Prompt Guard	8000	`/api/v1/guard/health`	Chat/action paths skip injection checks if `ENABLE_GUARD=false`
Frontend	5173	HTTP 200	Dashboard unavailable

Prompt Guard Metrics¶

curl http://localhost:8000/api/v1/guard/health
curl http://localhost:8000/api/v1/guard/stats \
  -H "X-API-Key: <your-api-key>"

Dashboard KPIs expose guard_blocked, guard_avg_ms, and guard_top_patterns via GET /api/v1/stats. See Prompt Guard.

Anomaly & Shadow AI Metrics¶

Metric	Description
`anomaly_ml_backend_total`	Inferences by backend (`sklearn`, `onnx`, `external`, `timeout`, …)
`anomaly_processing_seconds`	Sync anomaly detect latency (p95 budget ≤ 300 ms)
`anomaly_score_histogram`	Score distribution
`anomaly_false_positive_reports_total`	FP feedback from analysts
`anomaly_model_version_info`	Active model version labels
`shadow_candidates_total`	Discovery candidates by status
`shadow_time_to_confirm_seconds`	Seconds from candidate create → confirm/dismiss

Grafana: AegisAI Business Metrics includes Anomaly ML backend, latency p50/p95, score, FP reports, Shadow candidates, and time-to-confirm. Alerts: AnomalyLatencyHigh, ShadowAICandidateBacklog. See Anomalies and Agents.

Logging¶

AegisAI uses Python's standard logging module. Configure via LOG_LEVEL:

Level	Use Case
`DEBUG`	Development, verbose tracing
`INFO`	Production default
`WARNING`	Production (reduced noise)
`ERROR`	Errors only

Key log sources:

Component	Log Prefix	Events
`app.main`	Startup/shutdown	OPA loading, migrations
`app.api.v1.websocket`	WebSocket	Connections, disconnects
`app.services.event_publisher`	Events	Redis pub/sub
`policies.engine`	OPA	Policy evaluation, fallback
`security.anomaly_detector`	ML	Training, detection

Log Aggregation¶

Ship logs to your observability stack:

# Docker: JSON log driver
docker compose logs -f app --tail 100

# Or configure a log shipper (Fluentd, Filebeat, Vector)

Metrics to Monitor¶

Application Metrics¶

Metric	Source	Alert Threshold
API response time (p95)	Reverse proxy / APM	> 500ms
Error rate (5xx)	API logs	> 1% of requests
Active WebSocket connections	`/api/v1/status`	Sudden drops
Pending HITL count	`/api/v1/stats`	> SLA threshold
Anomaly count (24h)	`/api/v1/anomalies/stats`	Spike > 3x baseline

Infrastructure Metrics¶

Metric	Tool	Alert Threshold
PostgreSQL connections	`pg_stat_activity`	> 80% of max
Redis memory usage	`INFO memory`	> 80% of maxmemory
Disk usage (PostgreSQL volume)	OS metrics	> 85%
CPU / Memory (app container)	Docker stats	> 80% sustained

OPA Monitoring¶

# OPA health
curl http://localhost:8181/health

# Decision logs (if enabled)
curl http://localhost:8181/v1/data

If OPA is offline, AegisAI falls back to the in-process Python policy engine. Monitor for fallback activation in application logs.

Real-time Event Monitoring¶

Subscribe to security events via WebSocket for live dashboards:

ws://localhost:8000/api/v1/ws?token=<jwt>

Event	Severity	Action
`policy.violated`	error	Investigate agent and policy
`anomaly.detected`	warning	Review anomaly report
`hitl.requested`	info	Process approval
`system.status`	info	Connection health

Alerting Recommendations¶

Suggested alerts

Readiness check fails — any dependency offline
High error rate — 5xx responses > 1% over 5 minutes
Database connection pool exhausted
Disk space low on PostgreSQL volume
Pending HITL backlog exceeds SLA
Anomaly spike — 3x normal rate in 1 hour
OPA fallback active — policy engine degraded

External Integrations¶

Tool	Integration Point
Prometheus	Scrape `GET /metrics` (internal Docker network)
Grafana	Provisioned dashboards under `monitoring/grafana/`
Alertmanager	Telegram alerts via `monitoring/alertmanager/`
Blackbox	Probe `/api/v1/health`, `/ready`, public HTTPS
Datadog	APM agent on app container
Sentry	Python SDK for error tracking
PagerDuty	Alert on readiness failures

Full monitoring stack¶

See monitoring/README.md and deploy with:

# on VPS, after production stack is up
./scripts/deploy-monitoring.sh

Key env vars: GRAFANA_ADMIN_PASSWORD, GRAFANA_DOMAIN, optional TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID.

No Celery

AegisAI uses FastAPI BackgroundTasks / asyncio. Alert CeleryQueueBacklog is a compatibility stub; prefer background_tasks_in_progress.

Deployment — production setup
Troubleshooting — diagnose issues
Backup — data protection