CresignSys Learn

CresignSys Learn — Lesson 090

Written by

cresignsys

in

Build the CHP Event & Audit System

CHP now has:

  • Site discovery
  • Site management
  • Health monitoring
  • Reconciliation
  • Repair plans
  • Approval
  • Backup/restore
  • Job queue
  • Workers

But there is one major missing capability:

A single trustworthy history of what CHP did, when it happened, to which site, and why.

That is the purpose of the Event & Audit System.

1. Event vs Audit

These are related but not identical.

Event

Describes something that happened:

BACKUP_COMPLETED
HEALTH_CHECK_FAILED
DRIFT_DETECTED

Audit

Answers:

WHO performed it?
WHAT changed?
WHICH site?
WHICH object?
WHEN?
WHAT was the result?

So:

EVENT
  ↓
What happened?

AUDIT
  ↓
Who caused/authorized it?

2. Central Event Architecture

                 CHP COMPONENTS
                       │
       ┌───────────────┼────────────────┐
       ▼               ▼                ▼
    BACKUP           REPAIR          RESTORE
       │               │                │
       └───────────────┼────────────────┘
                       ▼
                 EVENT ENGINE
                       │
             ┌─────────┼─────────┐
             ▼         ▼         ▼
           AUDIT    TIMELINE   ALERTS

Every major subsystem publishes events to the same event store.

3. Create Event Table

Create:

CREATE TABLE events (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    event_code TEXT NOT NULL,
    event_category TEXT NOT NULL,
    severity TEXT NOT NULL DEFAULT 'INFO',

    site_id INTEGER,
    actor_id INTEGER,
    job_id INTEGER,
    plan_id INTEGER,
    backup_id INTEGER,

    message TEXT,
    metadata TEXT,

    created_at TEXT NOT NULL,

    FOREIGN KEY (site_id)
        REFERENCES sites(id)
        ON DELETE SET NULL,

    FOREIGN KEY (job_id)
        REFERENCES jobs(id)
        ON DELETE SET NULL
);

4. Event Code

Don’t rely only on free-form text.

Use a structured:

event_code

Examples:

SITE_DISCOVERED
SITE_IMPORTED
SITE_ENABLED
SITE_DISABLED

HEALTH_STARTED
HEALTH_PASSED
HEALTH_FAILED

DRIFT_DETECTED
PLAN_CREATED
PLAN_APPROVED
PLAN_REJECTED

JOB_CREATED
JOB_STARTED
JOB_COMPLETED
JOB_FAILED

BACKUP_STARTED
BACKUP_VERIFIED
BACKUP_FAILED

RESTORE_STARTED
RESTORE_COMPLETED
RESTORE_FAILED

REPAIR_STARTED
REPAIR_COMPLETED
REPAIR_ROLLED_BACK

5. Event Categories

Use:

SYSTEM
SITE
HEALTH
RECONCILIATION
REPAIR
BACKUP
RESTORE
JOB
SECURITY
AUTHORIZATION
USER

This makes filtering easier.

6. Severity

Start with:

DEBUG
INFO
WARNING
ERROR
CRITICAL

Example:

BACKUP_VERIFIED:
INFO

BACKUP_FAILED:
ERROR

NO_VERIFIED_BACKUP:
CRITICAL

7. Event Example

Event:
DRIFT_DETECTED

Category:
RECONCILIATION

Severity:
WARNING

Site:
example.com

Message:
PHP version differs from desired configuration.

Time:
2026-08-13 18:20

8. Metadata

Some events require additional structured information.

Example:

{
  "current_php": "8.2",
  "expected_php": "8.3",
  "drift_type": "PHP_VERSION"
}

Store it in:

metadata

as JSON.

9. Don’t Put Everything in message

Bad:

"PHP changed from 8.2 to 8.3 on example.com by admin at 18:22."

Better:

event_code:
PHP_CONFIGURATION_CHANGED

site_id:
7

actor_id:
4

metadata:
{
  "old": "8.2",
  "new": "8.3"
}

Then the UI can format the event however it wants.

10. Why Structured Events Matter

Later the dashboard can answer:

How many PHP changes occurred this month?

or:

Show all failed backups.

or:

Show everything admin did to example.com.

Structured data makes this possible.

11. Actor

Every event that involves a person should record:

actor_id

For example:

actor:
admin

event:
PLAN_APPROVED

12. System Events

Not every event has a human actor.

For example:

BACKUP_STARTED

may be initiated by:

scheduler

Use a system actor such as:

SYSTEM

or:

SCHEDULER

13. Actor Types

Eventually:

USER
SYSTEM
SCHEDULER
WORKER
API

This makes audit history clearer.

14. Example Timeline

For one repair:

18:20  DRIFT_DETECTED
18:21  PLAN_CREATED
18:22  PLAN_VALIDATED
18:25  PLAN_APPROVED
18:25  JOB_CREATED
18:26  BACKUP_STARTED
18:29  BACKUP_VERIFIED
18:30  REPAIR_STARTED
18:31  REPAIR_COMPLETED
18:31  HEALTH_PASSED
18:32  RECONCILIATION_PASSED

This is a complete operational story.

15. Site Timeline

The dashboard should eventually display:

example.com
────────────────────────────

18:32
Reconciliation passed

18:31
Health check passed

18:31
Repair completed

18:29
Backup verified

18:25
Repair approved

18:21
Repair plan created

18:20
PHP drift detected

16. Audit Trail

For security-sensitive actions, create more detailed audit records.

You already have:

audit_events

from the authorization system.

Keep that concept separate from general events.

17. Event vs Audit Storage

Use:

events

for:

system timeline
operations
health
backup
jobs

Use:

audit_events

for:

security-sensitive user actions
authorization
configuration changes
approvals
restores
permission changes

18. Why Separate Them?

An event might say:

HEALTH_CHECK_PASSED

An audit record might say:

admin changed PHP version

The second is security-sensitive and should receive stronger retention and access controls.

19. Audit Immutability

Audit records should ideally be append-only.

Don’t allow:

UPDATE audit_events

to rewrite history.

Instead:

new event

should record a correction if necessary.

20. Never Delete Audit History Through Normal UI

A normal administrator should not have:

Delete audit log

available.

Retention and archival should be separate administrative processes.

21. Event Library

Create:

sudo nano /etc/cresignsys/lib/events.sh

Functions:

emit_event
emit_audit
emit_security_event

22. emit_event

Conceptually:

emit_event \
    "BACKUP_VERIFIED" \
    "BACKUP" \
    "INFO" \
    "$SITE_ID" \
    "$MESSAGE" \
    "$METADATA"

The function handles:

timestamp
database insert
JSON validation
logging

23. Centralize Event Creation

Don’t write SQL manually in every script.

Bad:

hosting-backup:
INSERT INTO events...

hosting-repair:
INSERT INTO events...

hosting-restore:
INSERT INTO events...

Instead:

event library
     ↑
all components

24. Event IDs

Use:

event ID

as the immutable identifier.

Example:

EVT-000001

This is useful when referring to an event in support or debugging.

25. Add Event Code

Example:

EVT-000241

DRIFT_DETECTED

example.com

WARNING

26. Event Correlation

A single operation can create many events.

For example:

Repair:
RP-000021

creates:

PLAN_CREATED
PLAN_APPROVED
JOB_CREATED
BACKUP_STARTED
BACKUP_VERIFIED
REPAIR_STARTED
REPAIR_COMPLETED

We need to connect them.

27. Correlation ID

Add:

ALTER TABLE events
ADD COLUMN correlation_id TEXT;

Example:

CORR-20260813-00042

All events from the same operation use the same correlation ID.

28. Why Correlation IDs?

Then you can ask:

Show everything that happened during repair RP-000021.

Query:

correlation_id = ?

and receive the entire chain.

29. Example

CORR-00042

18:25 PLAN_APPROVED
18:25 JOB_CREATED
18:26 BACKUP_STARTED
18:29 BACKUP_VERIFIED
18:30 REPAIR_STARTED
18:31 REPAIR_COMPLETED
18:31 HEALTH_PASSED
18:32 RECONCILIATION_PASSED

This is extremely useful for debugging.

30. Parent Event

Eventually events can also reference:

parent_event_id

For example:

REPAIR_STARTED

can be associated with:

JOB_STARTED

But correlation IDs are enough for the first implementation.

31. Event Query Command

Create:

sudo nano /usr/local/bin/hosting-events

Usage:

hosting-events example.com

32. Example Output

CresignSys Events
=================

Site:
example.com

TIME                 EVENT                 SEVERITY
-------------------------------------------------------
18:32                 RECONCILIATION_PASS  INFO
18:31                 HEALTH_PASSED        INFO
18:31                 REPAIR_COMPLETED     INFO
18:29                 BACKUP_VERIFIED      INFO
18:25                 PLAN_APPROVED        INFO
18:21                 PLAN_CREATED         INFO
18:20                 DRIFT_DETECTED       WARNING

33. Filter by Severity

hosting-events example.com --severity ERROR

Output:

18:10
BACKUP_FAILED

18:15
HEALTH_FAILED

34. Filter by Event

hosting-events example.com --type BACKUP_FAILED

35. Filter by Time

hosting-events example.com --since "2026-08-01"

This becomes useful for monthly reports.

36. Event Categories

Example:

hosting-events example.com --category SECURITY

returns:

LOGIN
LOGIN_FAILED
PERMISSION_DENIED
PLAN_APPROVED
RESTORE_APPROVED

37. Security Events

Create standard security events:

LOGIN_SUCCESS
LOGIN_FAILED
SESSION_EXPIRED
PERMISSION_DENIED
UNAUTHORIZED_APPROVAL
UNAUTHORIZED_RESTORE
USER_CREATED
USER_DISABLED
ROLE_CHANGED

38. Why Log Failed Authorization?

Suppose someone repeatedly tries:

RESTORE client.com

without permission.

You want:

PERMISSION_DENIED

in the security audit trail.

This may indicate:

misconfiguration
or
compromised credentials

39. Don’t Log Secrets

Never write:

password
API key
private key
session token
database password

into events.

Even failed authentication events should contain safe information only.

40. Sensitive Metadata

Avoid:

{
  "password": "..."
}

Instead:

{
  "username": "admin",
  "result": "FAILED"
}

41. Event Retention

General events can have:

90 days

Audit events may require:

1 year

or longer depending on your operational and contractual requirements.

These should be configurable.

42. Don’t Delete Events Needed for Active Incidents

If an event is associated with:

open incident

it should be protected from ordinary cleanup.

43. Event Archive

Eventually:

ACTIVE EVENTS
      ↓
ARCHIVE
      ↓
COMPRESSED STORAGE

This keeps the operational database small.

44. Event Volume

Health monitoring may create many events.

For example:

HEALTH_PASSED

every 5 minutes creates:

288 events/day/site

For 100 sites:

28,800 events/day

This is excessive if every successful check is stored forever.

45. Event Deduplication

For repetitive healthy states, use state-transition events.

Instead of logging:

HEALTH_PASSED
HEALTH_PASSED
HEALTH_PASSED
HEALTH_PASSED

every time, log:

HEALTH_STATUS_CHANGED

UNHEALTHY → HEALTHY

and store periodic metrics separately.

46. Events vs Metrics

This distinction becomes important.

Event

Something happened:

HEALTH_FAILED

Metric

A measurement:

response_time_ms = 183

Don’t put high-frequency metrics into the event table.

47. Future Metrics System

Eventually CHP can have:

metrics

for:

CPU
RAM
disk
response time
backup duration
database size
traffic

while events contain significant state changes.

48. Event Severity Rules

Example:

INFO:
successful operations

WARNING:
drift
backup age approaching SLA

ERROR:
backup failure
health failure

CRITICAL:
no recovery point
restore failure + rollback failure
security compromise indicators

49. Alerts Use Events

The alert system should eventually subscribe to events.

Example:

BACKUP_FAILED
       ↓
alert policy
       ↓
send notification

Another:

NO_VERIFIED_BACKUP
       ↓
CRITICAL
       ↓
immediate alert

50. Don’t Alert Directly From Every Script

Bad:

hosting-backup
  └── send email

hosting-repair
  └── send email

hosting-restore
  └── send email

This duplicates notification logic.

Better:

operation
 ↓
event
 ↓
alert engine
 ↓
notification

51. Event-Driven Alert Architecture

                  EVENT
                    │
                    ▼
               EVENT STORE
                    │
                    ▼
              ALERT POLICY
                    │
             ┌──────┴──────┐
             ▼             ▼
          NO ALERT        ALERT
                           │
                  ┌────────┼────────┐
                  ▼        ▼        ▼
                EMAIL    PANEL    WEBHOOK

52. Example Alert Policy

Event:
BACKUP_FAILED

Condition:
2 failures within 6 hours

Action:
WARNING

Another:

Event:
NO_VERIFIED_BACKUP

Condition:
> 24 hours

Action:
CRITICAL

53. Event Correlation for Incidents

Suppose:

18:20
HEALTH_FAILED

18:21
PHP_DRIFT

18:25
REPAIR_STARTED

18:31
REPAIR_FAILED

18:32
ROLLBACK_COMPLETED

CHP can later automatically group these into:

INCIDENT
INC-00042

54. Incident System

This will eventually become:

EVENTS
  ↓
CORRELATION
  ↓
INCIDENT
  ↓
RESPONSE
  ↓
RESOLUTION

We won’t build incidents yet.

The event architecture should simply make them possible.

55. Event Integrity

Because audit history is important, consider adding:

previous_event_hash
event_hash

This can create a tamper-evident chain.

For example:

Event 1
  ↓ hash
Event 2
  ↓ hash
Event 3

If Event 2 is modified:

chain validation fails

56. Do We Need This Immediately?

Not necessarily.

For the first CHP version:

append-only audit records
+
restricted permissions
+
database backups

may be sufficient.

Hash chaining can be added later.

57. Event Database Indexes

Because the event table will grow, add indexes.

For example:

CREATE INDEX idx_events_site_time
ON events(site_id, created_at);

And:

CREATE INDEX idx_events_type
ON events(event_code);

And:

CREATE INDEX idx_events_correlation
ON events(correlation_id);

58. Audit Indexes

For audit events:

CREATE INDEX idx_audit_site_time
ON audit_events(site_id, created_at);

and:

CREATE INDEX idx_audit_actor
ON audit_events(actor_id);

59. Event Query Performance

The dashboard will frequently ask:

latest 50 events for site

So:

ORDER BY created_at DESC
LIMIT 50

should be indexed appropriately.

60. Event API

Eventually:

GET /api/sites/example.com/events

with:

?severity=ERROR
?category=BACKUP
?limit=50

The API should query structured events rather than parse log files.

61. Log Files Still Matter

Do not eliminate system logs.

Use:

EVENTS:
structured application history

LOGS:
diagnostic detail

For example:

Event:
BACKUP_FAILED

and the log contains:

mysqldump returned exit code 2
database connection timed out

62. Event + Log Correlation

Record:

correlation_id

in both the event and detailed log.

Then troubleshooting becomes:

Event
 ↓
Correlation ID
 ↓
Detailed logs

63. Example Complete Incident

CORR-00077

18:00
BACKUP_STARTED

18:04
BACKUP_FAILED

18:05
HEALTH_FAILED

18:06
DRIFT_DETECTED

18:08
PLAN_CREATED

18:10
PLAN_APPROVED

18:11
REPAIR_STARTED

18:12
REPAIR_FAILED

18:12
ROLLBACK_STARTED

18:14
ROLLBACK_COMPLETED

18:15
HEALTH_PASSED

This is exactly the kind of information an administrator needs.

64. Customer-Facing Timeline

Not every internal event should be shown to customers.

For example:

Customer may see:

Backup completed
Website health verified
Maintenance completed

But not necessarily:

internal worker IDs
database paths
security metadata

65. Internal vs Customer Events

Add visibility:

ALTER TABLE events
ADD COLUMN visibility TEXT NOT NULL DEFAULT 'INTERNAL';

Values:

INTERNAL
CUSTOMER

66. Example

BACKUP_VERIFIED
visibility:
CUSTOMER

But:

WORKER_HEARTBEAT_TIMEOUT
visibility:
INTERNAL

This prevents accidental information leakage.

67. Admin Dashboard

The administrator can see:

all events

Customer:

customer-safe events only

68. Event Permissions

Eventually:

VIEW_SYSTEM_EVENTS
VIEW_SECURITY_EVENTS
VIEW_SITE_EVENTS
VIEW_CUSTOMER_EVENTS

Sensitive security events should be restricted.

69. Audit Access Itself Should Be Audited

If:

admin views security audit logs

you may record:

AUDIT_LOG_ACCESSED

This is especially useful for sensitive environments.

70. Never Allow Audit Tampering Through UI

The dashboard should not expose:

Edit event
Delete event

for normal administrators.

The UI is for:

view
filter
search
export

71. Export

Eventually:

hosting-events example.com --export csv

or:

hosting-events example.com --export json

This is useful for:

support
compliance
incident analysis
customer reports

72. Export Security

Exports may contain sensitive operational information.

Therefore:

event export

should itself be:

audited

and protected by permissions.

73. Event Naming Convention

Use predictable names:

<OBJECT>_<ACTION>[_RESULT]

Examples:

SITE_IMPORTED
SITE_DISABLED

PLAN_CREATED
PLAN_APPROVED
PLAN_REJECTED

JOB_CREATED
JOB_STARTED
JOB_COMPLETED
JOB_FAILED

BACKUP_STARTED
BACKUP_VERIFIED
BACKUP_FAILED

RESTORE_STARTED
RESTORE_COMPLETED
RESTORE_FAILED

Avoid inconsistent names such as:

backupDone
BackupFinished
backup_ok

74. Event Result

For some events, metadata can contain:

{
  "result": "SUCCESS"
}

But don’t encode the result twice if the event code already clearly indicates it.

Keep event naming predictable.

75. Central Event Flow

Every subsystem should use:

ACTION
  ↓
EVENT
  ↓
EVENT STORE
  ↓
AUDIT / TIMELINE / ALERT

not:

ACTION
 ├── email
 ├── log
 ├── dashboard
 └── database

Centralization reduces complexity.

76. CHP Event Engine

The resulting architecture is:

                 EVENT ENGINE
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
    STORAGE         ALERTS        TIMELINE
       │              │              │
       ▼              ▼              ▼
    EVENTS          POLICIES       DASHBOARD
       │
       ▼
     AUDIT

77. Lesson 090 — Core Principle

The CHP event system should make it possible to answer five questions:

WHAT happened?
WHEN did it happen?
TO WHICH site?
WHO or WHAT caused it?
WHAT was the result?

For complex operations, add:

WHY?
WHICH PLAN?
WHICH JOB?
WHICH BACKUP?
WHICH CORRELATION?

The final operational chain is now:

OBSERVE
   ↓
DETECT
   ↓
PLAN
   ↓
AUTHORIZE
   ↓
APPROVE
   ↓
QUEUE
   ↓
BACKUP
   ↓
EXECUTE
   ↓
VERIFY
   ↓
EVENT
   ↓
AUDIT

CHP is now becoming a real hosting control plane, rather than simply a collection of hosting shell scripts.

Next Lesson — 091

Build the CHP Alert & Notification Engine

The event system can tell CHP:

something happened

but the administrator still needs to know:

Does this require attention?
How urgent is it?
Who should be notified?
How many times?
Through which channel?

Lesson 091 will build:

EVENT
  ↓
ALERT POLICY
  ↓
ALERT
  ↓
DEDUPLICATION
  ↓
NOTIFICATION

covering:

backup failures
no verified backup
SSL expiry
disk usage
site downtime
repair failures
restore failures
security events
scheduler failures

with:

INFO
WARNING
ERROR
CRITICAL

and notification channels such as:

email
dashboard
webhook

while preventing notification storms when the same problem occurs repeatedly.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts