Nova-Rewards

Nova Rewards Monitoring & Alerting

Complete observability stack for Nova Rewards platform with Prometheus, Grafana, and comprehensive alerting.

Architecture

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Backend   │────▶│  Prometheus  │────▶│   Grafana   │
│  (Metrics)  │     │  (Scraping)  │     │ (Dashboards)│
└─────────────┘     └──────┬───────┘     └─────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ Alertmanager │
                    │  (Routing)   │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         ┌────────┐  ┌─────────┐  ┌──────────┐
         │ Slack  │  │PagerDuty│  │  Email   │
         └────────┘  └─────────┘  └──────────┘

Components

Prometheus

Grafana

Alertmanager

Exporters

Quick Start

1. Setup Environment

cd monitoring
cp .env.example .env
# Edit .env with your configuration

2. Start Monitoring Stack

# Start all monitoring services
docker-compose -f docker-compose.monitoring.yml up -d

# Verify services are running
docker-compose -f docker-compose.monitoring.yml ps

3. Access Dashboards

4. Configure Alerts

Edit alertmanager/alertmanager.yml with your notification channels:

Metrics Collected

Application Metrics

System Metrics

Database Metrics

Cache Metrics

Alerts Configured

Critical Alerts

Warning Alerts

Runbooks

Detailed incident response procedures are available in the runbooks/ directory:

Grafana Dashboards

Nova Rewards Overview

System Metrics

Database Performance

Cache Performance

Integration with Existing Infrastructure

AWS CloudWatch

The monitoring stack complements existing CloudWatch monitoring:

Docker Compose Integration

# In your main docker-compose.yml, add monitoring network
networks:
  monitoring:
    external: true
    name: nova-rewards_monitoring

# Connect backend to monitoring network
services:
  backend:
    networks:
      - default
      - monitoring

Kubernetes Integration (if applicable)

# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nova-backend
spec:
  selector:
    matchLabels:
      app: nova-backend
  endpoints:
  - port: metrics
    path: /metrics

Maintenance

Backup Prometheus Data

# Backup Prometheus data
docker run --rm -v nova-rewards_prometheus_data:/data -v $(pwd):/backup \
  alpine tar czf /backup/prometheus-backup-$(date +%Y%m%d).tar.gz /data

Update Alert Rules

# Edit alert rules
vim prometheus/rules/alerts.yml

# Reload Prometheus configuration
curl -X POST http://localhost:9090/-/reload

Test Alerts

# Trigger test alert
curl -X POST http://localhost:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {"alertname": "TestAlert", "severity": "warning"},
    "annotations": {"summary": "Test alert"}
  }]'

Troubleshooting

Prometheus Not Scraping Targets

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check network connectivity
docker exec -it nova-prometheus wget -O- http://backend:4000/metrics

Grafana Not Showing Data

# Verify Prometheus datasource
curl http://localhost:3000/api/datasources

# Test Prometheus query
curl -G 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=up'

Alerts Not Firing

# Check alert rules
curl http://localhost:9090/api/v1/rules

# Check Alertmanager status
curl http://localhost:9093/api/v1/status

# View Alertmanager logs
docker logs nova-alertmanager

Best Practices

1. Alert Fatigue Prevention

2. Dashboard Organization

3. Metric Naming

4. Performance

Security

1. Access Control

# Enable Grafana authentication
GF_AUTH_ANONYMOUS_ENABLED=false
GF_AUTH_BASIC_ENABLED=true

2. Network Security

# Restrict Prometheus access
- "127.0.0.1:9090:9090"  # Only localhost

# Use reverse proxy with authentication

3. Secrets Management

# Use environment variables for sensitive data
# Never commit credentials to git
# Rotate credentials regularly

Support

For issues or questions:

References