Build the CHP Site Health Monitoring Engine
The Alert Engine from Lesson 091 can only work well if CHP has reliable health information.
We therefore need a dedicated health engine that answers:
Is this website actually healthy right now?
The architecture becomes:
SITE
│
┌─────────────┼─────────────┐
▼ ▼ ▼
DNS HTTP SSL
│ │ │
├─────────────┼─────────────┤
▼ ▼ ▼
NGINX PHP-FPM MYSQL
│ │ │
└─────────────┼─────────────┘
▼
HEALTH ENGINE
│
┌───────┴────────┐
▼ ▼
HEALTH STATE HEALTH SCORE
│
▼
EVENT
│
▼
ALERT
1. Health Is Not One Check
A website can have:
HTTP:
200 OK
while:
MySQL:
DOWN
or:
PHP-FPM:
DOWN
Therefore:
HTTP 200
≠
Website fully healthy
2. Health Dimensions
CHP should monitor at least:
DNS
HTTP
HTTPS
SSL
Nginx
PHP-FPM
MySQL
Filesystem
Disk
CPU
RAM
WordPress
Not every check needs to run at the same frequency.
3. Health Categories
Group checks into:
AVAILABILITY
APPLICATION
RUNTIME
INFRASTRUCTURE
SECURITY
Availability
DNS
HTTP
HTTPS
Application
WordPress
database connectivity
application response
Runtime
Nginx
PHP-FPM
MySQL
Infrastructure
disk
CPU
RAM
filesystem
Security
SSL
certificate expiry
HTTPS configuration
4. Health States
Use:
HEALTHY
DEGRADED
WARNING
DOWN
UNKNOWN
5. Meaning of HEALTHY
HEALTHY
means the important checks are passing and no critical dependency is currently failing.
Example:
DNS:
PASS
HTTPS:
PASS
SSL:
PASS
Nginx:
PASS
PHP-FPM:
PASS
MySQL:
PASS
WordPress:
PASS
6. DEGRADED
Example:
HTTP:
PASS
PHP-FPM:
PASS
MySQL:
PASS
Disk:
82%
Website still works.
But:
disk:
WARNING
Therefore:
overall:
DEGRADED
7. WARNING
A warning means the website may still be usable but there is a condition requiring attention.
Examples:
SSL expires in 14 days
disk usage > 90%
backup is approaching SLA
high resource usage
8. DOWN
A critical availability failure.
Example:
DNS:
PASS
HTTPS:
FAIL
HTTP:
FAIL
or:
Nginx:
DOWN
with the website unavailable.
9. UNKNOWN
Use:
UNKNOWN
when CHP cannot confidently determine health.
For example:
health worker:
not running
or:
database:
temporarily unreachable
without enough evidence to classify the site as down.
This prevents false certainty.
10. Don’t Use a Simple Average
Suppose:
HTTP:
100%
DNS:
100%
SSL:
100%
MySQL:
0%
A simple average might say:
75%
and classify the site as:
mostly healthy
That can be misleading.
MySQL may be a critical dependency.
11. Weighted Health Model
Instead use categories and weights.
Example:
HTTP/HTTPS:
30%
Application:
25%
Runtime:
20%
Database:
15%
Infrastructure:
10%
These are initial design values, not fixed production policy.
12. Critical Checks
Some checks should override the score.
For example:
DNS completely unavailable
may mean:
DOWN
even if:
disk:
healthy
CPU:
healthy
RAM:
healthy
13. Health Decision Tree
Conceptually:
HEALTH CHECK
│
DNS available?
/ \
NO YES
│ │
DOWN HTTPS available?
│
┌────┴────┐
NO YES
│ │
DOWN application?
│
┌────┴────┐
NO YES
│ │
DEGRADED HEALTHY
The actual implementation will consider more conditions.
14. Health Check Database
Create:
CREATE TABLE health_checks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_id INTEGER NOT NULL,
check_type TEXT NOT NULL,
status TEXT NOT NULL,
response_time_ms INTEGER,
message TEXT,
metadata TEXT,
checked_at TEXT NOT NULL,
FOREIGN KEY (site_id)
REFERENCES sites(id)
ON DELETE CASCADE
);
15. Check Status
Individual checks can use:
PASS
WARN
FAIL
UNKNOWN
Don’t use overall health states here.
16. Example Health Checks
DNS:
PASS
HTTP:
PASS
HTTPS:
PASS
SSL:
WARN
PHP-FPM:
PASS
MYSQL:
PASS
DISK:
WARN
Then the health engine calculates:
OVERALL:
WARNING
17. Store Response Time
For HTTP:
response_time_ms:
182
This allows CHP to detect performance degradation.
Example:
Normal:
180 ms
Current:
2400 ms
Website may technically be:
HTTP 200
but performance is poor.
18. HTTP Check
Create:
sudo nano /usr/local/bin/hosting-health
Usage:
sudo hosting-health example.com
The HTTP check should verify:
DNS resolution
TCP connection
TLS
HTTP response
status code
response time
19. Expected HTTP Status
For a normal website:
200
But don’t assume only 200 is valid.
Depending on the application:
301
302
may be expected.
CHP should follow redirects carefully and determine the final status.
20. Redirect Loop
Example:
HTTP
↓
HTTPS
↓
HTTP
↓
HTTPS
This should produce:
HTTP:
FAIL
with:
REDIRECT_LOOP
21. HTTP 500
If:
HTTP:
500
record:
HTTP:
FAIL
and:
metadata:
{
"status_code": 500
}
This should generate a health event.
22. HTTP 404
A root URL returning:
404
may indicate an application configuration problem.
But CHP should distinguish:
expected 404
from:
unexpected 404
A configurable health endpoint is therefore useful.
23. Health Endpoint
Eventually support:
https://example.com/health
or:
https://example.com/wp-json/
depending on application type.
For WordPress, CHP can use a safe application-level check rather than assuming /health exists.
24. DNS Check
Check:
A
AAAA
and optionally:
CNAME
depending on the site’s configuration.
Compare:
EXPECTED IP
against:
ACTUAL IP
where appropriate.
25. DNS Drift
Example:
Expected:
203.0.113.10
Actual:
203.0.113.25
This is not necessarily an outage.
It may be:
intentional DNS change
Therefore classify as:
DRIFT
rather than automatically:
DOWN
26. DNS Failure
If:
DNS resolution:
FAIL
and repeated checks confirm it:
overall:
DOWN
because the public website cannot normally be reached.
27. HTTPS Check
Verify:
TCP 443
TLS handshake
certificate
HTTP response
Example:
HTTPS:
PASS
28. SSL Certificate
Check:
subject
issuer
not_before
not_after
SAN
Calculate:
days_remaining
29. SSL Health
Example:
Certificate:
valid
Expires:
2026-09-20
Days remaining:
38
Status:
PASS
30. SSL Warning
Example:
Days remaining:
13
Status:
WARN
This should produce:
SSL_EXPIRING
event.
31. SSL Expired
If:
days_remaining < 0
then:
SSL:
FAIL
and likely:
HTTPS:
FAIL
depending on client behavior.
32. Nginx Check
Check:
systemctl is-active nginx
Expected:
active
But a running Nginx process alone does not prove the site works.
Therefore:
Nginx:
PASS
is only one dimension.
33. Nginx Configuration
Run:
nginx -t
Possible results:
PASS
or:
FAIL
This is especially important before configuration changes.
34. PHP-FPM
Check the configured PHP-FPM service.
Example:
systemctl is-active php8.3-fpm
Don’t hard-code the version into CHP.
The site configuration should determine:
PHP 8.2
PHP 8.3
PHP 8.4
where supported by the hosting environment.
35. PHP-FPM Socket
Also verify the configured socket exists.
Example:
/run/php/php8.3-fpm.sock
A service may be active while the expected socket is missing.
Therefore:
service:
PASS
socket:
FAIL
should result in:
PHP-FPM:
FAIL
36. MySQL
Check:
systemctl is-active mysql
Then perform an actual connection test.
Don’t treat:
systemctl:
active
as sufficient.
37. Database Connectivity
The site should be tested using its configured database.
Conceptually:
connect
↓
authenticate
↓
SELECT 1
↓
disconnect
Result:
MYSQL:
PASS
38. WordPress Check
For a WordPress site, verify:
WordPress files
wp-config.php
database connectivity
core integrity
site URL
Potentially:
wp core is-installed
where WP-CLI is available.
39. WordPress Core Integrity
Eventually:
wp core verify-checksums
can detect modified WordPress core files.
This is particularly useful for security monitoring.
40. Don’t Automatically Classify Checksum Failure as Site Down
Example:
HTTP:
PASS
WordPress:
PASS
Core integrity:
FAIL
The website is functioning but may have a security/integrity issue.
Overall:
WARNING
or:
DEGRADED
rather than:
DOWN
41. Filesystem Check
Check:
web root exists
and:
required directories
Example:
/storage/websites/example.com/public
42. Ownership Check
Check critical paths:
owner
group
permissions
Unexpected ownership can cause:
PHP failures
upload failures
cache failures
43. Disk Usage
Check:
df -P
For the relevant filesystem.
Don’t only check:
/var
if website data is stored on:
/storage
44. Disk Thresholds
Example:
< 80%:
PASS
80–90%:
WARN
90–95%:
ERROR
> 95%:
CRITICAL
Again, make thresholds configurable.
45. Inode Usage
Disk space can be available while inodes are exhausted.
Check:
df -i
Example:
disk:
60%
inodes:
99%
The site can still fail to create files.
Therefore:
INODE_USAGE
should be a separate check.
46. CPU
CPU should be monitored at the server level.
For example:
1-minute load average
Don’t classify a website as unhealthy merely because CPU is temporarily high.
Use sustained thresholds.
47. RAM
Likewise:
memory pressure
swap usage
available memory
are more meaningful than a single instantaneous percentage.
48. Swap
If:
swap:
95%
that may indicate memory pressure.
But swap usage alone does not necessarily mean a site is down.
Classify it as:
WARNING
unless application failures confirm a more serious problem.
49. Health Check Frequency
Not every check should run every minute.
Example:
HTTP:
1 minute
DNS:
5 minutes
SSL:
1 hour
WordPress:
15 minutes
disk:
5 minutes
CPU:
1 minute
50. Why Different Frequencies?
SSL expiration changes slowly.
HTTP availability can change quickly.
Therefore:
high-frequency:
availability
low-frequency:
configuration
This reduces unnecessary server load.
51. Health Policy Table
Create:
CREATE TABLE health_policies (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_id INTEGER NOT NULL,
check_type TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
interval_seconds INTEGER NOT NULL,
timeout_seconds INTEGER NOT NULL DEFAULT 10,
warning_threshold REAL,
critical_threshold REAL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
FOREIGN KEY (site_id)
REFERENCES sites(id)
ON DELETE CASCADE
);
52. Health Jobs
Health monitoring should use the central job queue.
Create:
HEALTH
jobs.
Scheduler:
find due health checks
↓
create HEALTH job
↓
worker
↓
perform checks
↓
store results
↓
calculate state
↓
emit events
53. Don’t Create One Job Per Check Initially
If a site has:
10 checks
you don’t necessarily need:
10 jobs
every minute.
Instead:
one HEALTH job
can execute the due checks.
54. Health Job Payload
Example:
{
"site_id": 7,
"checks": [
"HTTP",
"HTTPS",
"SSL",
"PHP_FPM",
"MYSQL"
]
}
55. Health Result
Store individual results:
HTTP:
PASS
182ms
HTTPS:
PASS
201ms
SSL:
PASS
38 days
PHP_FPM:
PASS
MYSQL:
PASS
Then calculate:
OVERALL:
HEALTHY
56. Health Snapshot
Create a current-state table:
CREATE TABLE site_health (
site_id INTEGER PRIMARY KEY,
overall_status TEXT NOT NULL,
health_score INTEGER,
last_checked_at TEXT NOT NULL,
status_changed_at TEXT,
summary TEXT,
FOREIGN KEY (site_id)
REFERENCES sites(id)
ON DELETE CASCADE
);
57. Why a Snapshot Table?
The raw check history may contain thousands of records.
The dashboard shouldn’t scan the entire history every time.
Instead:
site_health
contains:
current status
current score
last check
summary
while:
health_checks
contains historical data.
58. Current vs Historical Data
This is a recurring CHP architecture principle.
Current state
site_health
History
health_checks
events
Don’t confuse them.
59. Health Score
You can expose a score:
100
but don’t let the score replace the state.
Example:
Score:
92
Status:
WARNING
because:
SSL expires in 5 days
could be a warning even though most checks pass.
60. Example Scoring
Conceptually:
HTTP:
30 points
Application:
25 points
Runtime:
20 points
Database:
15 points
Infrastructure:
10 points
A failure reduces the score.
But critical overrides remain.
61. Critical Override
Example:
score:
95
DNS:
FAIL
Final:
DOWN
not:
HEALTHY
62. Consecutive Failure Logic
Don’t immediately declare:
DOWN
after one transient HTTP failure.
Use:
failure_count
Example:
1 failure:
suspect
2 failures:
degraded
3 consecutive failures:
DOWN
The exact thresholds should be configurable.
63. Recovery Logic
Similarly:
DOWN
should not necessarily become:
HEALTHY
after one successful check.
Use:
2–3 consecutive successes
to confirm recovery.
This reduces flapping.
64. Health Flapping
Example:
PASS
FAIL
PASS
FAIL
PASS
FAIL
This is called:
flapping
CHP should detect it.
65. Flapping Alert
If a site repeatedly changes:
HEALTHY
↕
DOWN
within a short period:
HEALTH_FLAPPING
can be generated.
66. Why Flapping Matters
A site that is:
up 95% of the time
may still be causing serious user impact if it repeatedly crashes.
Flapping should therefore be visible separately.
67. Health Event Generation
The health engine should generate events primarily for state changes.
Example:
HEALTH_CHANGED
Metadata:
{
"old_status": "HEALTHY",
"new_status": "DOWN",
"reason": "HTTP_FAILURE"
}
68. Don’t Generate Huge Event Volumes
Avoid:
HEALTHY
HEALTHY
HEALTHY
HEALTHY
every minute.
Generate:
HEALTH_CHANGED
only when meaningful state changes occur.
69. Health Summary
For each site:
Overall:
DEGRADED
Availability:
HEALTHY
Application:
HEALTHY
Runtime:
HEALTHY
Database:
HEALTHY
Infrastructure:
WARNING
Security:
WARNING
This is far more useful than one number.
70. CHP Site Health CLI
Use:
hosting-health example.com
Output:
CresignSys Site Health
======================
Site:
example.com
Overall:
HEALTHY
Score:
98/100
Availability:
DNS PASS
HTTP PASS 182ms
HTTPS PASS 201ms
Application:
WordPress PASS
Runtime:
Nginx PASS
PHP-FPM PASS
MySQL PASS
Security:
SSL PASS
Expiry 38 days
Infrastructure:
Disk PASS 62%
Inodes PASS 41%
RAM PASS 54%
Last checked:
2026-08-13 18:40
71. Warning Example
Overall:
DEGRADED
Score:
87/100
Security:
SSL:
WARNING
Expiry:
13 days
Infrastructure:
Disk:
WARNING
Usage:
87%
72. Down Example
Overall:
DOWN
Availability:
DNS:
PASS
HTTP:
FAIL
HTTPS:
FAIL
Runtime:
Nginx:
FAIL
Reason:
Nginx service unavailable.
73. Unknown Example
Overall:
UNKNOWN
Reason:
Health worker has not completed a successful check
within the configured monitoring window.
This is safer than claiming:
HEALTHY
when CHP has no recent evidence.
74. Health Dashboard
For all sites:
SITE STATUS SCORE
------------------------------------------------
example.com HEALTHY 98
shop.example.com DEGRADED 87
blog.example.com WARNING 71
client.example.com DOWN 20
newsite.com UNKNOWN -
75. Site Health Detail
The dashboard can show:
example.com
Overall:
DEGRADED
Availability:
Healthy
Application:
Healthy
Runtime:
Healthy
Security:
Warning
Infrastructure:
Warning
Active Alerts:
2
Last Verified Backup:
2h ago
This combines:
health
alerts
backup
into one operational view.
76. Health and Backup Are Different
A site can be:
HEALTHY
but:
BACKUP:
CRITICAL
Overall platform status should therefore show both.
Don’t hide backup problems just because the website works.
77. Health and Reconciliation Are Different
A site can be:
HEALTHY
but:
CONFIGURATION:
DRIFTED
For example:
PHP:
8.3
but desired:
8.4
The website works, but configuration differs from the intended state.
78. Four Operational Dimensions
CHP can now distinguish:
HEALTH
BACKUP
CONFIGURATION
SECURITY
Example:
Health:
HEALTHY
Backup:
WARNING
Configuration:
DRIFT
Security:
HEALTHY
This is much more informative than one overall “green/red” indicator.
79. Health Scheduler
The health scheduler can use the same central scheduler:
systemd timer
↓
CHP scheduler
↓
health policies
↓
HEALTH jobs
↓
worker
No second scheduler is required.
80. Health Job Priority
Health checks should usually be:
NORMAL
But availability checks for known failing sites can temporarily be:
HIGH
if required for rapid recovery detection.
81. Don’t Let Health Checks Starve Backups
Suppose:
100 sites
and every site needs health checks every minute.
Without limits:
health jobs
health jobs
health jobs
could consume all workers.
Use separate concurrency controls:
MAX_HEALTH_WORKERS=1
MAX_BACKUP_WORKERS=1
initially.
82. Health Timeout
Every check needs a timeout.
Example:
HTTP:
10 seconds
DNS:
5 seconds
MySQL:
5 seconds
WordPress:
15 seconds
A timeout should produce:
TIMEOUT
rather than hanging the worker indefinitely.
83. Health Check Security
Don’t allow the monitoring system to execute arbitrary URLs supplied by an untrusted user.
Validate:
domain
protocol
port
allowed destination
This prevents CHP from becoming a server-side request proxy.
84. Internal Checks
For local services:
systemctl
socket checks
local database
filesystem
For public checks:
DNS
HTTP
HTTPS
SSL
Keep these categories separate.
85. Health Data Retention
Raw health checks can become large.
For example:
10 checks
×
1 minute
×
100 sites
creates:
1,440,000 check records/day
Therefore do not retain every raw result indefinitely.
86. Health History Strategy
Keep:
current snapshot:
indefinitely
Detailed checks:
7–30 days
Aggregated history:
hourly/day summaries
for longer periods.
87. Health Aggregation
Instead of storing every result forever:
1-minute checks
can later become:
hourly:
availability %
average response time
failure count
and:
daily:
uptime %
average latency
incidents
88. Future SLA Reporting
This allows CHP to calculate:
Monthly uptime:
99.95%
Average response:
210ms
Downtime:
21 minutes
This can eventually feed hosting customer reports.
89. Health Engine Architecture
HEALTH SCHEDULER
│
▼
HEALTH JOB
│
▼
CHECK ENGINE
│
┌────────────────┼────────────────┐
▼ ▼ ▼
AVAILABILITY RUNTIME INFRASTRUCTURE
│ │ │
└────────────────┼────────────────┘
▼
NORMALIZATION
│
▼
HEALTH DECISION
│
┌─────────┴─────────┐
▼ ▼
CURRENT STATE EVENT
│ │
▼ ▼
DASHBOARD ALERT
90. Health Decision Algorithm
Conceptually:
run checks
↓
validate results
↓
apply consecutive failure rules
↓
apply critical overrides
↓
calculate category states
↓
calculate overall state
↓
update site_health
↓
emit state-change event
91. Example
Raw results:
DNS:
PASS
HTTP:
PASS
HTTPS:
PASS
SSL:
WARN
Nginx:
PASS
PHP-FPM:
PASS
MySQL:
PASS
Disk:
WARN
Category results:
Availability:
HEALTHY
Runtime:
HEALTHY
Database:
HEALTHY
Security:
WARNING
Infrastructure:
WARNING
Overall:
DEGRADED
92. Lesson 092 — Core Principle
The CHP health engine should never confuse:
"the server process is running"
with:
"the website is healthy."
Health must be evaluated across multiple layers:
DNS
↓
NETWORK
↓
HTTP/HTTPS
↓
WEB SERVER
↓
PHP
↓
DATABASE
↓
APPLICATION
↓
FILESYSTEM
↓
RESOURCES
↓
SECURITY
And the final architecture is:
RAW CHECK
↓
NORMALIZE
↓
STATE
↓
EVENT
↓
ALERT
↓
NOTIFICATION
Next Lesson — 093
Build the CHP Configuration Reconciliation Engine
The health engine tells us:
"Is the website working?"
The reconciliation engine will answer a different question:
“Is the website configured the way CHP expects it to be?”
It will compare:
DESIRED STATE
vs
ACTUAL STATE
across:
domain
web root
Nginx
PHP version
PHP-FPM socket
database
SSL
DNS
permissions
WordPress
backup policy
health policy
and classify differences as:
MATCH
DRIFT
UNKNOWN
CONFLICT
The key architecture will be:
DESIRED CONFIG
│
▼
RECONCILIATION
▲
│
ACTUAL CONFIG
│
▼
DIFF
│
┌────┴────┐
▼ ▼
MATCH DRIFT
│
▼
PLAN ENGINE
This is the layer that will eventually make CHP capable of safely detecting and correcting configuration drift across all hosted websites.
Leave a Reply