Build the CHP Health Engine

We now have two separate systems:

RECONCILIATION
    ↓
"Does the actual configuration match CHP?"

and:

HEALTH
    ↓
"Is the website actually working?"

These must remain separate.

---

1. Health Command

Create:

sudo nano /usr/local/bin/hosting-health

Usage:

sudo hosting-health example.com

Also:

sudo hosting-health example.com --json
sudo hosting-health example.com --verbose

---

2. Health Architecture

                 hosting-health
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
        SERVER       WEBSITE      DATA
          │            │            │
          ▼            ▼            ▼
        CPU          HTTP         MySQL
        RAM          HTTPS        WordPress
        Disk         TLS
        Load         DNS
        Nginx        Redirects
        PHP-FPM      Response

---

3. Health vs Reconciliation

Example:

CHP expects PHP 8.3
Actual PHP 8.2

Reconciliation:

DRIFT

But if PHP 8.2 is running and the website works:

Health:
HEALTHY

Therefore:

Configuration:
DRIFT

Runtime:
HEALTHY

Both can be true.

---

4. Health Result Categories

Use:

HEALTHY
WARNING
DEGRADED
DOWN
UNKNOWN
ERROR

HEALTHY

Everything important is functioning.

WARNING

Working, but an issue needs attention.

DEGRADED

Part of the service is impaired.

DOWN

The site/service is unavailable.

UNKNOWN

A reliable health determination cannot be made.

ERROR

The health checker itself failed.

---

5. Health Checks

The first version should check:

1. DNS
2. HTTP
3. HTTPS
4. TLS
5. Nginx
6. PHP-FPM
7. MySQL
8. WordPress
9. Disk
10. Memory
11. Load

---

6. Check 1 — DNS

Use:

dig +short A example.com

Expected:

SERVER_IPV4

Result:

DNS:
HEALTHY

If DNS doesn’t resolve:

DNS:
DOWN

But distinguish between:

DNS unavailable

and:

DNS points somewhere else

The second is configuration drift rather than necessarily a runtime outage.

---

7. DNS Health Function

Add to:

/etc/cresignsys/lib/health.sh

Function:

health_dns

Conceptually:

health_dns() {
    local domain="$1"

    if dig +short A "$domain" | grep -q .; then
        echo "HEALTHY"
    else
        echo "DOWN"
    fi
}

Later compare the result against CHP’s expected server IP.

---

8. HTTP Check

Use:

curl -I --max-time 10 http://example.com

Possible response:

HTTP/1.1 301 Moved Permanently

A redirect is not automatically a failure.

---

9. HTTP Status Categories

Treat:

2xx → HEALTHY
3xx → HEALTHY/WARNING
4xx → WARNING/DEGRADED
5xx → DOWN
timeout → DOWN
connection failure → DOWN

But don’t treat every 4xx identically.

For example:

401 Unauthorized

may be intentional.

---

10. HTTPS Check

Use:

curl -I --max-time 10 https://example.com

Expected:

HTTP/2 200

or:

HTTP/2 301

depending on the site’s design.

---

11. HTTP Redirect Chain

A common configuration:

http://example.com
       ↓
301
       ↓
https://example.com
       ↓
200

This is healthy.

Test:

curl -IL --max-time 10 http://example.com

The health engine should understand the chain.

---

12. Redirect Loop

Bad example:

HTTP
 ↓
HTTPS
 ↓
HTTP
 ↓
HTTPS

curl eventually fails.

Report:

HTTPS:
DOWN

Reason:
REDIRECT_LOOP

---

13. Excessive Redirects

Even if a redirect chain eventually succeeds:

HTTP
 ↓
HTTPS
 ↓
www
 ↓
non-www
 ↓
HTTPS

it may be inefficient.

Report:

WARNING

if the chain exceeds your configured threshold.

For example:

MAX_REDIRECTS=5

---

14. HTTP Response Time

Don’t only check status.

Measure:

curl -o /dev/null \
     -s \
     -w '%{http_code} %{time_total}\n' \
     https://example.com

Example:

200 0.342

Store:

HTTP status:
200

Response time:
342 ms

---

15. Response-Time Thresholds

Initial thresholds could be:

< 1 sec       HEALTHY
1–3 sec       WARNING
3–10 sec      DEGRADED
> 10 sec      DOWN/TIMEOUT

These should eventually be configurable.

Don’t treat these as universal performance benchmarks.

---

16. TLS Check

Use:

openssl s_client \
    -connect example.com:443 \
    -servername example.com

But don’t dump the entire output to logs.

Extract only:

certificate subject
issuer
expiry
verification result

---

17. TLS Verification

A healthy result:

TLS:
HEALTHY

Certificate:
VALID

Expires:
2026-11-20

If certificate verification fails:

TLS:
DOWN

---

18. Certificate Expiration Warning

Example:

30+ days:
HEALTHY

8–30 days:
WARNING

1–7 days:
DEGRADED

Expired:
DOWN

This should be configurable.

---

19. Nginx Check

Check:

systemctl is-active nginx

If active:

Nginx:
HEALTHY

If inactive:

Nginx:
DOWN

---

20. Nginx Configuration

Separately run:

nginx -t

Possible:

Service:
HEALTHY

Configuration:
ERROR

This is useful because a running Nginx process can still have a configuration problem waiting for the next reload.

---

21. PHP-FPM Check

Determine the site’s configured PHP-FPM service:

php8.3-fpm

Then:

systemctl is-active php8.3-fpm

Expected:

PHP-FPM:
HEALTHY

---

22. PHP Socket

Check:

test -S /run/php/php8.3-fpm.sock

If missing:

PHP-FPM:
DEGRADED

or:

DOWN

depending on whether the service itself is running.

---

23. PHP End-to-End Check

Service status alone isn’t enough.

You want:

Browser
   ↓
Nginx
   ↓
PHP-FPM
   ↓
PHP

A simple PHP health endpoint can eventually be useful.

For example:

/health.php

But don’t expose sensitive diagnostic information publicly.

---

24. Better Health Endpoint

Create a protected endpoint that returns:

OK

rather than:

PHP version
server variables
database password
filesystem paths

For example:

https://example.com/.well-known/chp-health

could eventually return:

OK

if appropriate for your architecture.

---

25. Don’t Expose Infrastructure Information

Avoid public endpoints displaying:

MYSQL_HOST
DB_USER
PHP_VERSION
SERVER_IP
filesystem paths
environment variables

A health endpoint should reveal as little as possible.

---

26. MySQL Health

For WordPress:

wp db check

is useful.

Possible result:

Database:
HEALTHY

If connection fails:

Database:
DOWN

---

27. MySQL Service

Also check:

systemctl is-active mysql

This gives:

MySQL Service:
HEALTHY

while:

WordPress Database:
DOWN

could indicate a connection/configuration problem.

---

28. WordPress Health

Check:

wp core is-installed

Then optionally:

wp db check

A WordPress site may have:

Nginx:
OK

PHP:
OK

MySQL:
OK

WordPress:
ERROR

For example, WordPress files may be incomplete.

---

29. WordPress HTTP Check

The most important application-level check is often:

HTTPS request
       ↓
WordPress response

If:

HTTP 200

the website is externally responding.

This is often more meaningful than checking individual processes.

---

30. WordPress Error Detection

A 200 response does not guarantee the page is healthy.

For example:

HTTP 200

could contain:

Fatal error

or:

Database connection error

Therefore application-specific checks can be added later.

---

31. Disk Health

Check:

df -h

For the website filesystem:

df -h /storage/websites/example.com

Example:

Filesystem      Size  Used Avail Use%
/dev/sdb        100G   65G   35G  65%

---

32. Disk Thresholds

Initial policy:

< 80%       HEALTHY
80–90%      WARNING
90–95%      DEGRADED
> 95%       CRITICAL

The exact thresholds should be configurable.

---

33. Inode Usage

Disk space can be available while inodes are exhausted.

Check:

df -i /storage/websites/example.com

This can detect:

100% inode usage

which can prevent new files from being created.

---

34. Memory

Check:

free -m

For overall server health, monitor:

total
used
available

Avoid using:

used / total

alone as the definition of memory pressure on Linux.

available is generally more useful for this purpose.

---

35. Load Average

Use:

uptime

or:

cat /proc/loadavg

Example:

0.25 0.31 0.28

Interpret load relative to the number of CPU cores.

---

36. CPU Count

nproc

If:

4 CPUs

and load is:

0.5

that’s generally light.

If load remains:

8+

on a 4-core system, that deserves investigation.

Don’t define a single universal load threshold.

---

37. Website-Level vs Server-Level Health

This distinction matters.

A server may have:

CPU:
HIGH

while:

example.com:
HTTP 200

Therefore:

Server health:
WARNING

Website health:
HEALTHY

Both can be true.

---

38. Site Health Model

The site health engine should primarily prioritize:

DNS
HTTP
HTTPS
TLS
Nginx
PHP
Database
Application

while server resource metrics provide:

WARNING

unless they actually cause service failure.

---

39. Health Score

Avoid an overly simplistic:

8 checks = 8 points

because some checks are much more important.

For example:

HTTPS DOWN

is more serious than:

Disk at 82%

Use severity instead.

---

40. Severity Model

Each check can produce:

OK
WARNING
CRITICAL
UNKNOWN

Example:

HTTP       OK
HTTPS      OK
TLS        OK
PHP        OK
MySQL      OK
Disk       WARNING

Overall:

WARNING

---

41. Critical Checks

For a normal public website:

DNS
HTTPS
HTTP

are usually critical.

For WordPress:

Database
PHP

are also critical.

---

42. Dependency Awareness

Don’t report five independent failures when one root failure caused them.

Example:

PHP-FPM DOWN

may cause:

HTTP 502
WordPress DOWN
Application DOWN

The health engine should understand this relationship.

---

43. Example

Instead of:

HTTP:
DOWN

PHP:
DOWN

WordPress:
DOWN

Application:
DOWN

say:

PHP-FPM:
CRITICAL

HTTP:
DEGRADED
Reason: upstream unavailable

WordPress:
NOT_REACHABLE

Likely root cause:
PHP-FPM failure

This is much more useful.

---

44. Dependency Graph

                    DNS
                     │
                     ▼
                  Nginx
                     │
              ┌──────┴──────┐
              ▼             ▼
           Static         PHP-FPM
                            │
                            ▼
                         MySQL
                            │
                            ▼
                        WordPress

If PHP-FPM fails:

PHP-FPM
   ↓
WordPress
   ↓
HTTP application response

can also fail.

---

45. Health Engine Library

Create:

sudo nano /etc/cresignsys/lib/health.sh

Functions:

health_dns
health_http
health_https
health_tls
health_nginx
health_php
health_mysql
health_wordpress
health_disk
health_memory
health_load
health_site

---

46. Don’t Mix Health and Reconciliation Functions

Keep:

reconcile_php()

separate from:

health_php()

because they answer different questions.

Reconciliation

Expected PHP:
8.3

Actual PHP:
8.2

Result:
DRIFT

Health

PHP-FPM:
running

Result:
HEALTHY

---

47. HTTP Health Function

Conceptually:

health_http() {
    local domain="$1"

    curl \
        --silent \
        --show-error \
        --location \
        --max-time 10 \
        --output /dev/null \
        --write-out '%{http_code} %{time_total}' \
        "https://${domain}"
}

Capture:

status
time

separately.

---

48. Curl Options

Useful options:

--max-time
--connect-timeout
--location
--silent
--show-error

For example:

curl \
  --connect-timeout 5 \
  --max-time 10 \
  --location \
  ...

This prevents a broken site from hanging the entire health process.

---

49. Health Timeout

Every external check should have a timeout.

Never allow:

curl
dig
openssl
mysql
wp

to hang indefinitely.

The health engine should finish predictably.

---

50. Parallel Health Checks

Later, checks can run concurrently:

DNS ──────┐
HTTP ─────┤
TLS ──────┤
PHP ──────┤──→ Result
MySQL ────┤
Disk ─────┘

But initially use sequential checks.

Reliability is more important than squeezing a few seconds out of the first implementation.

---

51. Health Output

Example:

CresignSys Site Health
======================

Site:
learn.cresignsys.com

DNS
---
Status:       HEALTHY
A Record:     CORRECT

HTTP
----
Status:       HEALTHY
HTTP Code:    301
Response:     0.18 sec

HTTPS
-----
Status:       HEALTHY
HTTP Code:    200
Response:     0.34 sec

TLS
---
Status:       HEALTHY
Certificate:  VALID
Expires:      2026-11-20

Nginx
-----
Service:      HEALTHY
Config:       VALID

PHP-FPM
-------
Service:      HEALTHY
Socket:       AVAILABLE

MySQL
-----
Service:      HEALTHY
Connection:   HEALTHY

WordPress
---------
Installation: HEALTHY
Database:     HEALTHY

Resources
---------
Disk:         HEALTHY
Memory:       HEALTHY
Load:         HEALTHY

Overall:
HEALTHY

---

52. Degraded Example

CresignSys Site Health
======================

HTTPS:
HEALTHY

PHP-FPM:
HEALTHY

MySQL:
HEALTHY

Disk:
WARNING
Usage: 87%

Overall:
WARNING

The website works, but the server needs attention.

---

53. Critical Example

HTTPS:
DOWN

PHP-FPM:
DOWN

MySQL:
HEALTHY

Likely cause:
PHP-FPM unavailable

Overall:
DOWN

---

54. Health JSON

Add:

sudo hosting-health example.com --json

Example:

{
  "domain": "example.com",
  "overall": "HEALTHY",
  "checks": {
    "dns": {
      "status": "HEALTHY"
    },
    "https": {
      "status": "HEALTHY",
      "http_code": 200,
      "response_time": 0.34
    },
    "tls": {
      "status": "HEALTHY"
    },
    "nginx": {
      "status": "HEALTHY"
    },
    "php": {
      "status": "HEALTHY"
    },
    "mysql": {
      "status": "HEALTHY"
    },
    "wordpress": {
      "status": "HEALTHY"
    }
  }
}

---

55. Store Health Results

The services table already has:

status
updated_at

but we should not overwrite everything with transient health data.

Eventually add a health-history table.

---

56. health_checks Table

Create in a future migration:

CREATE TABLE health_checks (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    site_id INTEGER NOT NULL,
    check_name TEXT NOT NULL,
    status TEXT NOT NULL,
    response_time_ms INTEGER,
    message TEXT,
    checked_at TEXT NOT NULL,

    FOREIGN KEY (site_id)
        REFERENCES sites(id)
        ON DELETE CASCADE
);

---

57. Why History?

Suppose:

14:00 HEALTHY
15:00 HEALTHY
16:00 WARNING
17:00 DOWN
18:00 HEALTHY

You can identify an outage window.

---

58. Don’t Store Every Check Forever

If health checks run every 5 minutes:

12 checks/hour
288 checks/day

for 100 sites:

28,800 checks/day

This grows quickly.

Use retention.

For example:

raw checks:
7 days

and later:

hourly aggregates:
90 days

---

59. Health Aggregation

Eventually:

Raw:
5-minute checks

       ↓

Hourly:
availability percentage

       ↓

Daily:
uptime percentage

Example:

August 13
Uptime:
99.87%

---

60. Uptime Calculation

If a site has:

288 checks/day

and:

2 failed checks

then approximate availability:

286 / 288 × 100
= 99.31%

For production-grade monitoring, account for check failures and measurement uncertainty rather than treating every failed probe as definitive downtime.

---

61. Health Endpoint for CHP

Later the central CHP control panel can display:

Sites
----------------------------------------
example.com          HEALTHY
shop.com             WARNING
blog.com              DOWN
medical.com           HEALTHY

Clicking a site:

example.com

shows:

Health
Reconciliation
Backups
SSL
Domains
Operations

---

62. Site Status Combines Both Systems

We now have:

hosting-reconcile

and:

hosting-health

Therefore:

hosting-site-status example.com

can combine them.

---

63. Combined Status

Example:

CresignSys Site Status
======================

Configuration:
DRIFT

Runtime:
HEALTHY

Overall:
WARNING

Configuration issue:
PHP expected 8.3, actual 8.2

This is much more informative than simply saying:

Site:
FAIL

---

64. Four Dimensions

A mature CHP status system can have:

CONFIGURATION
HEALTH
BACKUP
SECURITY

Example:

Configuration:
DRIFT

Health:
HEALTHY

Backup:
HEALTHY

Security:
WARNING

---

65. Future Security Checks

The health engine can eventually include:

certificate validity
open ports
file permissions
disk permissions
WordPress core integrity
plugin vulnerabilities
malware indicators

But security scanning should remain a separate subsystem where possible.

Don’t overload hosting-health.

---

66. Health vs Monitoring

hosting-health:

Run once

Monitoring:

Run repeatedly

So:

hosting-health

is the probe.

A future:

hosting-monitor

will schedule probes and record history.

---

67. Future Monitoring Architecture

             scheduler
                 │
                 ▼
         hosting-health
                 │
                 ▼
          health_checks
                 │
                 ▼
             alerting
                 │
        ┌────────┼────────┐
        ▼        ▼        ▼
      Email    Panel    WhatsApp*

The notification mechanism should be designed separately.

---

68. Don’t Alert on Every Failure

If a single probe fails because of a temporary network issue:

FAIL

you don’t necessarily want an immediate outage alert.

Use consecutive failures.

Example:

1 failure:
record

2 failures:
warning

3 consecutive failures:
DOWN/ALERT

The exact policy can be configured later.

---

69. Recovery Detection

If the site was:

DOWN

and then returns:

HEALTHY

record:

RECOVERED

This is useful for operations history.

---

70. Health State Machine

HEALTHY
   │
   ▼
WARNING
   │
   ▼
DEGRADED
   │
   ▼
DOWN
   │
   ▼
RECOVERING
   │
   ▼
HEALTHY

But don’t transition states on one noisy probe without considering consecutive failures.

---

71. First Version Scope

For now implement only:

DNS
HTTP
HTTPS
TLS
Nginx
PHP-FPM
MySQL
WordPress
Disk
Memory
Load

Don’t yet implement:

alerting
scheduler
uptime history
automatic restart
automatic repair

---

72. Automatic Restart Is Dangerous

Do not immediately implement:

systemctl restart php8.3-fpm

when health fails.

A failure may have a deeper cause:

memory exhaustion
configuration error
database problem
malicious traffic
disk full

Automatic restarts can hide the root problem.

---

73. Safe First Response

Health engine:

DETECT
   ↓
REPORT
   ↓
LOG

Later:

DETECT
   ↓
CLASSIFY
   ↓
REPAIR PLAN
   ↓
APPROVE
   ↓
APPLY

---

74. Test Sequence

After implementing:

sudo hosting-health learn.cresignsys.com

Then:

sudo hosting-health learn.cresignsys.com --json

Then compare with:

sudo hosting-reconcile learn.cresignsys.com

You should see the difference between:

configuration

and:

runtime

---

75. Example Combined Interpretation

Suppose:

Reconciliation:
DRIFT

Health:
HEALTHY

Interpretation:

The website is currently working, but its actual configuration differs from CHP’s recorded/desired configuration.

Another case:

Reconciliation:
MATCH

Health:
DOWN

Interpretation:

Configuration appears correct, but a runtime failure is occurring.

Another:

Reconciliation:
DRIFT

Health:
DOWN

Interpretation:

Configuration differs and the site is currently unavailable; investigate both, starting with the immediate runtime failure.

---

76. CHP Operational Model

We now have three important questions:

QUESTION 1
Does CHP know the site?
        ↓
DATABASE

QUESTION 2
Does the server match CHP?
        ↓
RECONCILIATION

QUESTION 3
Is the site working?
        ↓
HEALTH

Together:

DATABASE
   +
RECONCILIATION
   +
HEALTH
   ↓
SITE STATUS

---

77. Updated CHP Architecture

                         CRESIGNSYS
                              │
                ┌─────────────┴─────────────┐
                ▼                           ▼
          CONTROL PLANE                 DATA PLANE
                │                           │
                ▼                           ▼
           CHP DATABASE               Website Runtime
                │                           │
       ┌────────┼────────┐          ┌───────┼────────┐
       ▼        ▼        ▼          ▼       ▼        ▼
     Sites    Domains  Backups     Nginx    PHP     MySQL
       │
       ▼
Desired/known state
       │
       ├───────────────┐
       ▼               ▼
RECONCILIATION       HEALTH
       │               │
       │               ├── DNS
       │               ├── HTTP
       │               ├── HTTPS
       │               ├── TLS
       │               ├── PHP
       │               └── Database
       │
       └───────┬───────┘
               ▼
          SITE STATUS

---

78. Lesson 082 — Core Principle

The CHP health engine must be observational before it becomes corrective.

Its job is to reliably answer:

Is DNS working?
Is HTTP working?
Is HTTPS working?
Is TLS valid?
Is Nginx running?
Is PHP-FPM running?
Is MySQL available?
Is WordPress functional?
Is the filesystem healthy?
Is the server under resource pressure?

without changing the server.

That gives us a safe foundation for the next stage.

---

Next Lesson — 083

Build the Unified hosting-site-status

We now have:

hosting-db-import
hosting-reconcile
hosting-health

The next command will combine them:

sudo hosting-site-status example.com

It will produce a single operational view:

SITE
├── Identity
├── Domains
├── Configuration
├── Runtime Health
├── SSL
├── DNS
├── WordPress
├── Backups
├── Resources
└── Recent Operations

and ultimately calculate:

HEALTHY
WARNING
DEGRADED
DOWN
DRIFT
UNKNOWN

without confusing configuration drift with actual downtime.