Build the CHP Alert & Notification Engine
The Event System from Lesson 090 tells us:
something happened
The Alert Engine determines:
Does it matter?
How urgent is it?
Who needs to know?
Should we notify them now?
Has this already been reported?
The architecture becomes:
CHP EVENT
│
▼
ALERT ENGINE
│
┌──────────┴──────────┐
▼ ▼
NO ALERT ALERT
│
┌──────┴──────┐
▼ ▼
DEDUPLICATE CREATE
│ │
└──────┬──────┘
▼
NOTIFICATION
│
┌────────────┼────────────┐
▼ ▼ ▼
EMAIL DASHBOARD WEBHOOK
1. Event vs Alert vs Notification
These three should remain separate.
Event
BACKUP_FAILED
Alert
example.com backup has failed twice.
Severity: ERROR
Notification
Email sent to administrator.
So:
EVENT
↓
ALERT
↓
NOTIFICATION
One event does not necessarily mean an alert.
2. Example
Suppose a health check returns HTTP 500.
The event is:
HEALTH_FAILED
The alert policy may decide:
one failure:
don't notify
three consecutive failures:
create alert
Then:
ALERT_CREATED
and finally:
EMAIL_SENT
3. Why This Separation Matters
Without it, every event would trigger a notification.
Imagine:
HEALTH_FAILED
HEALTH_FAILED
HEALTH_FAILED
HEALTH_FAILED
HEALTH_FAILED
You don’t want:
5 emails
5 WhatsApp messages
5 webhooks
You want one meaningful incident.
4. Alert Table
Create:
CREATE TABLE alerts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
alert_code TEXT NOT NULL,
site_id INTEGER,
severity TEXT NOT NULL,
status TEXT NOT NULL,
title TEXT NOT NULL,
message TEXT,
first_event_id INTEGER,
latest_event_id INTEGER,
occurrence_count INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
acknowledged_at TEXT,
resolved_at TEXT,
FOREIGN KEY (site_id)
REFERENCES sites(id)
ON DELETE SET NULL,
FOREIGN KEY (first_event_id)
REFERENCES events(id),
FOREIGN KEY (latest_event_id)
REFERENCES events(id)
);
5. Alert States
Use:
OPEN
ACKNOWLEDGED
RESOLVED
SUPPRESSED
Flow:
OPEN
↓
ACKNOWLEDGED
↓
RESOLVED
Or:
OPEN
↓
RESOLVED
without acknowledgement.
6. Alert Severity
Use:
INFO
WARNING
ERROR
CRITICAL
Example:
Backup successful:
INFO
Usually this shouldn’t create an alert.
Backup failed:
ERROR
No verified backup:
CRITICAL
7. Alert Code
Use predictable identifiers:
BACKUP_FAILURE
BACKUP_STALE
NO_VERIFIED_BACKUP
SITE_DOWN
HEALTH_DEGRADED
SSL_EXPIRING
SSL_EXPIRED
DISK_USAGE_HIGH
DISK_USAGE_CRITICAL
REPAIR_FAILED
RESTORE_FAILED
SCHEDULER_FAILED
WORKER_FAILED
SECURITY_EVENT
8. Alert Title
Example:
Backup failure
Message:
The latest scheduled backup for example.com failed.
The title should be short.
The message contains context.
9. Alert Fingerprint
This is one of the most important concepts.
Suppose:
example.com
BACKUP_FAILURE
occurs 10 times.
All 10 events should map to the same alert:
fingerprint:
BACKUP_FAILURE:site:7
10. Why Fingerprints?
Without fingerprints:
10 events
↓
10 alerts
With fingerprints:
10 events
↓
1 alert
occurrence_count = 10
11. Add Fingerprint
Add:
ALTER TABLE alerts
ADD COLUMN fingerprint TEXT;
Then index it:
CREATE INDEX idx_alert_fingerprint
ON alerts(fingerprint);
12. Example Alert
Alert:
ALR-000042
Fingerprint:
BACKUP_FAILURE:site:7
Site:
example.com
Severity:
ERROR
Status:
OPEN
Occurrences:
4
First seen:
18:00
Last seen:
18:45
13. Alert Deduplication
When an event arrives:
EVENT
↓
calculate fingerprint
↓
find OPEN alert
If found:
increment occurrence_count
update latest_event
Don’t create a new alert.
14. If No Existing Alert
Create:
new alert
Then:
send notification
depending on policy.
15. Alert Resolution
Suppose:
BACKUP_FAILURE
creates:
BACKUP_FAILURE alert
Next day:
backup succeeds
The success event can resolve:
BACKUP_FAILURE
Result:
RESOLVED
16. Resolution Events
Create explicit events:
ALERT_CREATED
ALERT_UPDATED
ALERT_ACKNOWLEDGED
ALERT_RESOLVED
ALERT_SUPPRESSED
These become part of the event history.
17. Alert Policy Table
Create:
CREATE TABLE alert_policies (
id INTEGER PRIMARY KEY AUTOINCREMENT,
alert_code TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
severity TEXT NOT NULL,
threshold_count INTEGER DEFAULT 1,
threshold_window_minutes INTEGER DEFAULT 0,
notification_enabled INTEGER NOT NULL DEFAULT 1,
cooldown_minutes INTEGER DEFAULT 60,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
18. Example Policy
For site downtime:
alert:
SITE_DOWN
threshold:
3 failures
window:
5 minutes
Meaning:
3 failed checks
within 5 minutes
→ create alert
19. Backup Policy
Example:
BACKUP_FAILURE
threshold:
2
window:
6 hours
Meaning:
one failure:
monitor
two failures:
alert
20. No Verified Backup
This is different.
If:
last verified backup
older than 24h
create:
NO_VERIFIED_BACKUP
CRITICAL
This should be a state-based alert rather than requiring repeated failure events.
21. Alert Policy Types
There are two major types.
Event-based
BACKUP_FAILED
State-based
backup age > SLA
The alert engine should eventually support both.
22. Notification Channels
Start with:
EMAIL
DASHBOARD
WEBHOOK
Later:
SMS
PUSH
depending on infrastructure.
23. Notification Table
Create:
CREATE TABLE notifications (
id INTEGER PRIMARY KEY AUTOINCREMENT,
alert_id INTEGER NOT NULL,
channel TEXT NOT NULL,
recipient TEXT,
status TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
sent_at TEXT,
error_message TEXT,
FOREIGN KEY (alert_id)
REFERENCES alerts(id)
ON DELETE CASCADE
);
24. Notification States
QUEUED
SENDING
SENT
FAILED
CANCELLED
Again, don’t send directly from the event handler.
Use the CHP job queue.
25. Notification Flow
EVENT
↓
ALERT POLICY
↓
ALERT CREATED
↓
NOTIFICATION JOB
↓
JOB QUEUE
↓
WORKER
↓
EMAIL / WEBHOOK
This integrates the systems we built in Lessons 089–090.
26. Why Use the Job Queue?
Suppose an SMTP server is unavailable.
Without a queue:
backup process
↓
send email
↓
SMTP timeout
↓
backup operation delayed
Bad.
With the queue:
backup
↓
event
↓
alert
↓
notification job
The backup remains independent of email delivery.
27. Notification Job
Use:
NOTIFICATION
as another job type.
Eventually:
BACKUP
RESTORE
REPAIR
HEALTH
RETENTION
NOTIFICATION
all use the same worker architecture.
28. Notification Retry
Email delivery can safely retry.
Example:
attempt 1:
SMTP timeout
attempt 2:
SMTP timeout
attempt 3:
SUCCESS
Notification itself is generally retryable.
29. Exponential Backoff
Instead of:
retry
retry
retry
retry
use:
1 minute
5 minutes
15 minutes
30 minutes
This reduces pressure on failing services.
30. Notification Deduplication
Suppose:
BACKUP_FAILURE
remains open for 10 hours.
Don’t send:
one email every 15 minutes
unless the alert policy explicitly requires periodic reminders.
31. Cooldown
Use:
cooldown_minutes = 60
Meaning:
after notification
wait 60 minutes
before another notification
32. Escalation
A mature alert engine can escalate.
Example:
0 min:
dashboard
15 min:
email
60 min:
second email
4 hours:
critical escalation
But don’t implement all of this immediately.
Start with:
create
notify
cooldown
resolve
33. Notification Recipients
Create:
CREATE TABLE notification_recipients (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
channel TEXT NOT NULL,
destination TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL
);
Examples:
Administrator
EMAIL
admin@example.com
and:
Monitoring webhook
WEBHOOK
internal endpoint
34. Site-Specific Recipients
Eventually support:
global recipients
site recipients
customer recipients
For example:
example.com
→ customer@example.com
while platform-level failures go to:
hosting-admin@example.com
35. Alert Routing
Add a routing layer:
ALERT
↓
WHO SHOULD RECEIVE IT?
↓
CHANNEL
↓
NOTIFICATION
Example:
SITE_DOWN
→ customer + platform admin
But:
WORKER_FAILURE
→ platform admin only.
36. Customer Visibility
Alerts should have:
INTERNAL
CUSTOMER
Example:
SITE_DOWN:
CUSTOMER
while:
WORKER_CRASH:
INTERNAL
37. Alert Dashboard
The CHP dashboard can show:
ACTIVE ALERTS
=============
CRITICAL
1
ERROR
2
WARNING
4
Then:
CRITICAL
No verified backup
example.com
2h ago
38. Alert Detail
ALR-000042
==========
Site:
example.com
Alert:
BACKUP_FAILURE
Severity:
ERROR
Status:
OPEN
Occurrences:
4
First Seen:
18:00
Last Seen:
18:45
Latest Event:
EVT-001942
39. Acknowledge
An administrator can:
ACKNOWLEDGE
This means:
I know about this problem.
It does not mean:
The problem is fixed.
Therefore:
ACKNOWLEDGED
is still an active alert.
40. Resolve
Only when the underlying condition disappears:
RESOLVED
For example:
backup succeeds
↓
BACKUP_FAILURE resolved
41. Don’t Auto-Resolve on Acknowledgement
Incorrect:
admin acknowledges
↓
RESOLVED
Acknowledgement is not resolution.
42. Alert Reopening
Suppose:
BACKUP_FAILURE
RESOLVED
Then a new failure occurs later.
Create a new active alert or reopen according to a defined recurrence policy.
A simple first version:
resolved alert
+
new failure after resolution
=
new alert
This preserves incident history.
43. Notification Storm Protection
Suppose 100 sites all use the same database server.
Database fails.
You might get:
100 SITE_DOWN alerts
That’s legitimate information, but 100 separate emails may be excessive.
Future aggregation can produce:
INFRASTRUCTURE INCIDENT
Affected sites:
100
This is a later feature.
44. First Version
For CHP V1:
per-site alerts
+
fingerprint deduplication
+
cooldown
is enough.
45. Disk Alerts
The event system can produce:
DISK_USAGE_HIGH
Example policy:
WARNING:
80%
ERROR:
90%
CRITICAL:
95%
These thresholds should be configuration values.
46. SSL Alerts
Example:
SSL_EXPIRING
Policies:
30 days:
INFO
14 days:
WARNING
7 days:
ERROR
1 day:
CRITICAL
47. Site Availability Alerts
Example:
SITE_DOWN
Don’t alert after one failed HTTP request.
Use:
3 consecutive failures
to avoid transient false alarms.
48. Health Recovery
If:
SITE_DOWN
is open and the site returns:
HEALTHY
resolve the alert.
Create:
SITE_RECOVERED
event.
49. Alert Lifecycle
The full lifecycle is:
EVENT
↓
POLICY MATCH
↓
ALERT CREATED
↓
NOTIFICATION
↓
ACKNOWLEDGED
↓
CONDITION RECOVERS
↓
RESOLVED
50. Alert Engine Command
Create:
sudo nano /usr/local/bin/hosting-alerts
Commands:
hosting-alerts list
hosting-alerts show ALR-000042
hosting-alerts acknowledge ALR-000042
hosting-alerts resolve ALR-000042
hosting-alerts test
51. List Alerts
CresignSys Alerts
=================
ID SITE SEVERITY STATUS
------------------------------------------------
ALR-042 example.com CRITICAL OPEN
ALR-043 shop.com ERROR ACKNOWLEDGED
ALR-044 blog.com WARNING OPEN
52. Alert Test
Create:
hosting-alerts test BACKUP_FAILURE example.com
It should create a test event and verify:
event
↓
alert
↓
notification
without affecting the actual site.
53. Notification Test
Also:
hosting-notify-test EMAIL admin@example.com
This should verify notification configuration independently.
54. Alert Engine Worker
You can use the same central worker.
The workflow becomes:
event
↓
alert processor
↓
notification job
↓
worker
No separate daemon is strictly necessary initially.
55. Alert Processor
The processor can run every minute:
find new events
↓
evaluate policies
↓
create/update alerts
↓
queue notifications
Later this can become event-driven.
56. Avoid Processing the Same Event Twice
Add:
processed_at
to the event-processing mechanism, or maintain a separate processing table.
A simple approach is:
CREATE TABLE event_processing (
event_id INTEGER NOT NULL,
processor TEXT NOT NULL,
processed_at TEXT NOT NULL,
PRIMARY KEY (event_id, processor),
FOREIGN KEY (event_id)
REFERENCES events(id)
ON DELETE CASCADE
);
57. Why Processor Records?
Suppose there are processors:
alert_processor
metrics_processor
incident_processor
Each may process the same event independently.
This design supports multiple consumers.
58. Event → Alert Example
Event:
BACKUP_FAILED
example.com
Policy:
2 failures
within 6 hours
First event:
count = 1
no alert
Second event:
count = 2
→
ALR-00042
OPEN
→
notification job
59. Example Recovery
Next backup:
BACKUP_VERIFIED
Alert engine sees:
latest backup healthy
Then:
ALR-00042
RESOLVED
and emits:
ALERT_RESOLVED
60. Critical Safety Rule
Never let alert processing modify the underlying site automatically unless a specific, separately authorized remediation system exists.
For example:
SSL_EXPIRING
should create:
ALERT
not automatically:
change SSL configuration
The alert system is for detection and notification.
Remediation remains under the control/approval architecture.
61. Alert Engine Architecture
The complete system becomes:
EVENT
│
▼
EVENT PROCESSOR
│
▼
ALERT POLICY
│
┌────────┴────────┐
▼ ▼
NO MATCH MATCH
│
▼
FINGERPRINT
│
┌─────────┴─────────┐
▼ ▼
EXISTING ALERT NEW ALERT
│ │
└─────────┬─────────┘
▼
NOTIFICATION
│
▼
JOB QUEUE
│
▼
WORKER
62. CHP Architecture After Lesson 091
CRESIGNSYS CHP
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
OBSERVATION CONTROL RECOVERY
│ │ │
▼ ▼ ▼
HEALTH/DRIFT PLAN/APPROVAL BACKUP/RESTORE
│ │ │
└──────────────┬─────────┴──────────┬─────────────┘
▼ ▼
EVENTS JOB QUEUE
│ │
▼ ▼
ALERTS WORKERS
│
▼
NOTIFICATIONS
63. Lesson 091 — Core Principle
The important separation is:
EVENT
=
something happened
ALERT
=
something requires attention
NOTIFICATION
=
someone was informed
And:
one event
≠
one alert
≠
one notification
This separation prevents notification storms and gives CHP a proper monitoring architecture.
Next Lesson — 092
Build the CHP Site Health Monitoring Engine
The next layer will continuously determine whether each website is actually healthy.
It will combine:
HTTP
HTTPS
DNS
SSL
Nginx
PHP-FPM
MySQL
disk
CPU
RAM
WordPress
into a unified:
SITE HEALTH SCORE
with states:
HEALTHY
DEGRADED
WARNING
DOWN
UNKNOWN
The key design will be:
RAW CHECKS
↓
NORMALIZATION
↓
HEALTH STATE
↓
EVENT
↓
ALERT
so CHP can distinguish between a single failed check, a genuinely unhealthy website, and a temporary/transient problem.
Leave a Reply