Category: Uncategorized

  • CresignSys Learn — Lesson 094

    Build the CHP Desired-State & Configuration Management System

    Lesson 093 introduced reconciliation:

    DESIRED STATE
          vs
    ACTUAL STATE
          ↓
    MATCH / DRIFT / UNKNOWN / CONFLICT
    

    The next question is:

    Where does CHP get the Desired State?

    The answer should not be a collection of manually edited values scattered across scripts.

    Instead, CHP needs a proper Configuration Management System.


    1. Desired-State Architecture

    The configuration hierarchy should be:

                        CHP PLATFORM DEFAULTS
                                │
                                ▼
                         HOSTING PLAN
                                │
                                ▼
                           SITE PROFILE
                                │
                                ▼
                          SITE OVERRIDES
                                │
                                ▼
                         DESIRED STATE
                                │
                                ▼
                         RECONCILIATION
    

    For example:

    Platform:
    PHP 8.3
    
    Business Hosting Plan:
    PHP 8.3
    
    example.com:
    PHP 8.4 override
    
    FINAL DESIRED:
    PHP 8.4
    

    2. Why Hierarchy?

    Suppose you host:

    120 websites
    

    You don’t want to configure PHP, backups, SSL, and health monitoring individually 120 times.

    Instead:

    Business Plan
          │
          ├── site 1
          ├── site 2
          ├── site 3
          └── site 4
    

    All sites inherit the common configuration.


    3. Configuration Layers

    Use five conceptual layers:

    1. PLATFORM
    2. PLAN
    3. SITE PROFILE
    4. SITE OVERRIDE
    5. GENERATED DESIRED STATE
    

    The final desired state is calculated from these layers.


    4. Platform Defaults

    These are global defaults.

    Example:

    {
      "php_version": "8.3",
      "ssl_enabled": true,
      "backup_enabled": true,
      "health_monitoring": true
    }
    

    These apply unless something more specific overrides them.


    5. Hosting Plan

    Example plans:

    STARTER
    BUSINESS
    PROFESSIONAL
    MULTI_DOMAIN
    

    Each plan can define:

    PHP policy
    storage
    backup frequency
    health monitoring
    SSL
    resource limits
    

    6. Example Business Plan

    {
      "storage_gb": 5,
      "php_version": "8.3",
      "ssl_enabled": true,
      "backup_enabled": true,
      "backup_frequency": "daily",
      "health_monitoring": true
    }
    

    7. Site Profile

    A site profile contains configuration common to a class of websites.

    For example:

    WORDPRESS_STANDARD
    WORDPRESS_MANAGED
    PHP_CUSTOM
    STATIC_SITE
    

    A WordPress profile might define:

    {
      "application": "wordpress",
      "health_monitoring": true,
      "wordpress_core_monitoring": true
    }
    

    8. Site Override

    A specific customer may require:

    PHP 8.4
    

    even though the hosting plan uses:

    PHP 8.3
    

    The site override becomes:

    {
      "php_version": "8.4"
    }
    

    9. Final Desired State

    CHP combines everything:

    PLATFORM
       ↓
    PLAN
       ↓
    PROFILE
       ↓
    SITE OVERRIDE
       ↓
    FINAL DESIRED STATE
    

    Example:

    Platform PHP:
    8.3
    
    Plan PHP:
    8.3
    
    Profile:
    8.3
    
    Site Override:
    8.4
    
    ----------------
    Desired:
    8.4
    

    10. Configuration Precedence

    The rule should be deterministic:

    SITE OVERRIDE
          >
    SITE PROFILE
          >
    HOSTING PLAN
          >
    PLATFORM DEFAULT
    

    The most specific valid configuration wins.


    11. Never Use Random Precedence

    Avoid situations where:

    plan.sh
    site.sh
    repair.sh
    hosting-create.sh
    

    each decides independently which value wins.

    There should be one configuration resolver.


    12. Configuration Resolver

    Create:

    sudo nano /etc/cresignsys/lib/config-resolver.sh
    

    Main function:

    resolve_site_config
    

    Input:

    site_id
    

    Output:

    final desired state
    

    13. Resolver Example

    Conceptually:

    resolve_site_config 7
    

    returns:

    {
      "php_version": "8.4",
      "ssl_enabled": true,
      "backup_enabled": true,
      "backup_frequency": "daily",
      "health_monitoring": true
    }
    

    This becomes the authoritative desired state.


    14. Configuration Database

    Create a plan table if you don’t already have one:

    CREATE TABLE hosting_plans (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        name TEXT NOT NULL UNIQUE,
        description TEXT,
    
        configuration TEXT NOT NULL,
    
        enabled INTEGER NOT NULL DEFAULT 1,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL
    );
    

    15. Site Profile Table

    CREATE TABLE site_profiles (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        name TEXT NOT NULL UNIQUE,
        description TEXT,
    
        configuration TEXT NOT NULL,
    
        enabled INTEGER NOT NULL DEFAULT 1,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL
    );
    

    16. Assign Plan to Site

    Your sites table should contain something like:

    plan_id
    

    Example:

    example.com
        ↓
    Business Plan
    

    17. Assign Profile

    Also:

    profile_id
    

    Example:

    example.com
        ↓
    Business Plan
        ↓
    WORDPRESS_STANDARD
    

    18. Site Overrides

    Create:

    CREATE TABLE site_config_overrides (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
    
        config_key TEXT NOT NULL,
        config_value TEXT,
    
        management_mode TEXT NOT NULL DEFAULT 'MANAGED',
    
        actor_id INTEGER,
    
        reason TEXT,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    19. Example Overrides

    site:
    example.com
    
    php_version:
    8.4
    
    backup_frequency:
    hourly
    
    health_monitoring:
    true
    

    20. Configuration Keys

    Use standardized keys.

    For example:

    application.type
    php.version
    php.fpm_pool
    web.document_root
    
    ssl.enabled
    ssl.auto_renew
    
    backup.enabled
    backup.frequency
    backup.retention
    
    health.enabled
    health.interval
    
    dns.managed
    
    wordpress.management_mode
    wordpress.core_policy
    

    21. Avoid Free-Form Key Names

    Don’t allow:

    php
    PHP
    phpVersion
    php_version
    php-ver
    

    to represent the same thing.

    Use one canonical naming scheme.


    22. Configuration Schema

    CHP should define which keys are valid.

    For example:

    php.version
    type:
    STRING
    
    allowed:
    8.2
    8.3
    8.4
    

    Another:

    backup.enabled
    type:
    BOOLEAN
    

    Another:

    backup.retention_days
    type:
    INTEGER
    minimum:
    1
    

    23. Why Validation Matters

    Without validation someone could configure:

    php.version = "banana"
    

    The desired-state engine would accept invalid data.

    Configuration must be validated before becoming active desired state.


    24. Configuration Schema Table

    Create:

    CREATE TABLE config_schema (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        config_key TEXT NOT NULL UNIQUE,
        value_type TEXT NOT NULL,
    
        schema_definition TEXT,
    
        description TEXT,
    
        created_at TEXT NOT NULL
    );
    

    25. Example Schema

    {
      "config_key": "php.version",
      "value_type": "string",
      "allowed_values": [
        "8.2",
        "8.3",
        "8.4"
      ]
    }
    

    26. Configuration Management Modes

    Each configuration field should have a management mode.

    Use:

    MANAGED
    CUSTOMER_MANAGED
    READ_ONLY
    LOCKED
    

    27. MANAGED

    CHP owns the configuration.

    Example:

    Nginx virtual host
    

    If it drifts:

    DRIFT
    

    can generate a repair plan.


    28. CUSTOMER_MANAGED

    The customer owns the configuration.

    Example:

    WordPress plugins
    

    CHP can observe it but shouldn’t automatically change it.


    29. READ_ONLY

    CHP can inspect but cannot modify.

    Example:

    external DNS
    

    CHP may report:

    DNS:
    DRIFT
    

    but cannot fix it because DNS isn’t managed by CHP.


    30. LOCKED

    A locked configuration cannot be overridden by ordinary users.

    Example:

    security policy
    

    Platform administrators may control it.


    31. Configuration Scope

    A setting should also have a scope:

    PLATFORM
    PLAN
    PROFILE
    SITE
    

    Example:

    PHP default:
    PLATFORM
    
    PHP plan:
    PLAN
    
    PHP override:
    SITE
    

    32. Immutable Platform Defaults

    Some values should not be changed at the site level.

    Example:

    allowed PHP versions
    

    The platform can define:

    8.2
    8.3
    8.4
    

    A site cannot request:

    7.4
    

    if CHP has removed support for it.


    33. Allowed Configuration

    This gives us two concepts:

    DESIRED VALUE
    

    and:

    ALLOWED VALUE
    

    Desired state must always remain inside the allowed configuration space.


    34. Example

    Platform:

    Allowed PHP:
    8.3
    8.4
    

    Site requests:

    PHP:
    8.2
    

    Result:

    INVALID CONFIGURATION
    

    Not:

    DRIFT
    

    because the desired state itself is invalid.


    35. Configuration Lifecycle

    A configuration change should follow:

    REQUEST
     ↓
    VALIDATE
     ↓
    CALCULATE DESIRED STATE
     ↓
    COMPARE
     ↓
    PLAN
     ↓
    APPROVE
     ↓
    APPLY
     ↓
    VERIFY
    

    36. Configuration Status

    A site configuration can have:

    VALID
    INVALID
    PENDING
    APPLIED
    DRIFTED
    

    But keep this separate from reconciliation status.


    37. Configuration Version

    Every final desired state should have:

    config_version
    

    Example:

    example.com
    configuration version:
    17
    

    38. Why Versioning Matters

    Suppose an administrator requests:

    PHP 8.4
    

    and then another change:

    PHP 8.3
    

    before the first repair finishes.

    Versioning lets CHP identify stale plans.


    39. Configuration Revision

    Use a revision record:

    revision:
    17
    
    parent:
    16
    
    created by:
    admin
    
    reason:
    Upgrade PHP
    
    status:
    ACTIVE
    

    40. Configuration Revisions Table

    CREATE TABLE config_revisions (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
    
        revision_number INTEGER NOT NULL,
    
        configuration TEXT NOT NULL,
    
        actor_id INTEGER,
    
        reason TEXT,
    
        status TEXT NOT NULL,
    
        created_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE,
    
        UNIQUE(site_id, revision_number)
    );
    

    41. Revision Status

    Use:

    DRAFT
    ACTIVE
    SUPERSEDED
    CANCELLED
    

    Only one revision should normally be:

    ACTIVE
    

    for a site.


    42. Example History

    Revision 14
    PHP 8.2
    SUPERSEDED
    
    Revision 15
    PHP 8.3
    SUPERSEDED
    
    Revision 16
    PHP 8.4
    ACTIVE
    

    43. Configuration Diff

    When creating a new revision:

    OLD:
    PHP 8.3
    
    NEW:
    PHP 8.4
    

    The system should automatically produce:

    {
      "php.version": {
        "old": "8.3",
        "new": "8.4"
      }
    }
    

    44. Audit This Change

    Changing desired configuration is security-sensitive.

    Emit:

    CONFIGURATION_CHANGED
    

    Metadata:

    {
      "key": "php.version",
      "old": "8.3",
      "new": "8.4",
      "revision": 16
    }
    

    45. Configuration Source

    Every configuration value should ideally have a source.

    Example:

    php.version:
    8.4
    
    source:
    SITE_OVERRIDE
    

    Another:

    backup.enabled:
    true
    
    source:
    HOSTING_PLAN
    

    46. Effective Configuration

    The resolved output should contain:

    {
      "php.version": {
        "value": "8.4",
        "source": "SITE_OVERRIDE",
        "management": "MANAGED"
      },
    
      "backup.enabled": {
        "value": true,
        "source": "HOSTING_PLAN",
        "management": "MANAGED"
      }
    }
    

    This is extremely useful when debugging.


    47. Configuration Explanation

    The CLI should support:

    hosting-config explain example.com php.version
    

    Output:

    PHP Version
    ===========
    
    Platform:
    8.3
    
    Plan:
    8.3
    
    Profile:
    8.3
    
    Site Override:
    8.4
    
    Effective Value:
    8.4
    
    Source:
    SITE_OVERRIDE
    
    Management:
    MANAGED
    

    48. This Solves a Common Problem

    Without an explanation system, an administrator might ask:

    Why is this site using PHP 8.4 when the plan says 8.3?

    CHP can answer immediately:

    Site override:
    8.4
    

    49. Configuration Command

    Create:

    sudo nano /usr/local/bin/hosting-config
    

    Commands:

    hosting-config show example.com
    hosting-config explain example.com php.version
    hosting-config validate example.com
    hosting-config diff example.com
    hosting-config history example.com
    

    50. hosting-config show

    Example:

    CresignSys Configuration
    ========================
    
    Site:
    example.com
    
    Plan:
    Business
    
    Profile:
    WordPress Standard
    
    PHP:
    8.4
    
    SSL:
    Enabled
    
    Backup:
    Daily
    
    Health Monitoring:
    Enabled
    
    WordPress Management:
    Customer Managed
    
    Configuration Revision:
    16
    

    51. hosting-config diff

    This compares:

    desired
    vs
    actual
    

    Example:

    PHP:
    Desired 8.4
    Actual 8.3
    DRIFT
    
    SSL:
    Desired enabled
    Actual enabled
    MATCH
    
    Backup:
    Desired daily
    Actual daily
    MATCH
    

    52. hosting-config validate

    Before a desired state becomes active:

    schema validation
    +
    platform compatibility
    +
    plan limits
    +
    site constraints
    

    must pass.


    53. Plan Limits

    Suppose:

    Business Plan:
    5 GB
    

    A site override requesting:

    20 GB
    

    should not automatically be accepted.

    Possible result:

    PLAN_LIMIT_EXCEEDED
    

    The customer may need to upgrade.


    54. Configuration Policy

    Example:

    {
      "storage_gb": {
        "max": 5
      },
      "backup_retention_days": {
        "max": 30
      }
    }
    

    This allows plan-specific restrictions.


    55. Inheritance

    Example:

    Platform:
    backup enabled = true
    
    Plan:
    backup frequency = daily
    
    Profile:
    backup retention = 14 days
    
    Site:
    backup frequency = hourly
    

    Effective state:

    backup enabled:
    true
    source PLATFORM
    
    frequency:
    hourly
    source SITE
    
    retention:
    14 days
    source PROFILE
    

    56. Override Only What Is Needed

    Don’t duplicate the entire plan into every site.

    Bad:

    {
      "storage": 5,
      "php": "8.3",
      "ssl": true,
      "backup": true,
      "backup_frequency": "daily",
      "health": true,
      "php_override": "8.4"
    }
    

    Better:

    {
      "php.version": "8.4"
    }
    

    Everything else is inherited.


    57. Prevent Override Explosion

    A site could otherwise accumulate:

    100 overrides
    

    This becomes difficult to understand.

    Use a policy:

    Site overrides should be intentional exceptions, not a second configuration system.


    58. Override Review

    The dashboard should show:

    Site Overrides:
    3
    

    For example:

    PHP version:
    8.4
    
    Backup frequency:
    hourly
    
    Health interval:
    30 sec
    

    This makes exceptions visible.


    59. Locked Fields

    Suppose the platform requires:

    SSL:
    enabled
    

    The customer should not be able to override:

    ssl.enabled = false
    

    because the field is:

    LOCKED
    

    60. Configuration Capability Matrix

    Eventually CHP can determine:

                      PLATFORM PLAN SITE
    PHP VERSION         ✓       ✓     ✓
    SSL                 ✓       ✓     -
    BACKUP              ✓       ✓     ✓
    DNS                 ✓       -     -
    WORDPRESS PLUGINS   ✓       -     ✓
    

    This defines where each setting can be controlled.


    61. Configuration Templates

    Instead of manually constructing plans, use templates.

    Example:

    wordpress-standard.json
    wordpress-managed.json
    static-site.json
    php-custom.json
    

    A template can define:

    defaults
    allowed values
    management modes
    

    62. Template Example

    {
      "name": "WORDPRESS_STANDARD",
    
      "configuration": {
        "application.type": "wordpress",
        "health.enabled": true,
        "backup.enabled": true,
        "wordpress.management_mode": "CUSTOMER_MANAGED"
      }
    }
    

    63. Managed WordPress Template

    {
      "name": "WORDPRESS_MANAGED",
    
      "configuration": {
        "application.type": "wordpress",
        "health.enabled": true,
        "backup.enabled": true,
        "wordpress.management_mode": "MANAGED",
        "wordpress.core_monitoring": true
      }
    }
    

    64. Template Versioning

    Templates should also be versioned.

    Example:

    WORDPRESS_STANDARD v1
    WORDPRESS_STANDARD v2
    

    A template update should not silently alter every production website without a controlled rollout.


    65. Plan Changes

    Suppose Business Plan changes:

    PHP 8.3
    

    to:

    PHP 8.4
    

    Should all sites immediately change?

    No.

    The desired state resolver should identify affected sites and generate a controlled transition.


    66. Plan Migration

    Use:

    PLAN UPDATE
     ↓
    AFFECTED SITES
     ↓
    RECONCILIATION
     ↓
    MIGRATION PLANS
    

    This is safer than modifying production immediately.


    67. Configuration Rollout

    For large hosting environments:

    120 sites
    

    could be migrated:

    10 sites
     ↓
    monitor
     ↓
    25 sites
     ↓
    monitor
     ↓
    50 sites
     ↓
    120 sites
    

    This is a future feature, but the configuration architecture should allow it.


    68. Configuration Lock

    Before a repair:

    CONFIGURATION LOCK
    

    can prevent another process from changing the same configuration simultaneously.

    For example:

    PHP upgrade
    

    and:

    PHP downgrade
    

    should not run concurrently.


    69. Lock Scope

    Use locks at multiple levels:

    SITE LOCK
    CONFIGURATION DOMAIN LOCK
    JOB LOCK
    

    Example:

    example.com
    PHP
    LOCKED
    

    while PHP repair is running.


    70. Avoid Global Locks

    Don’t lock the entire CHP platform because one site is being repaired.

    Bad:

    example.com PHP repair
     ↓
    ALL SITES LOCKED
    

    Use narrow locks instead.


    71. Configuration State Machine

    A site configuration can move through:

    ACTIVE
       ↓
    CHANGE_REQUESTED
       ↓
    VALIDATED
       ↓
    PENDING_APPROVAL
       ↓
    APPROVED
       ↓
    APPLYING
       ↓
    VERIFYING
       ↓
    ACTIVE
    

    If something fails:

    APPLYING
       ↓
    FAILED
       ↓
    ROLLBACK
    

    72. Desired State Should Reflect Reality Carefully

    After a successful repair:

    actual:
    PHP 8.4
    

    and:

    desired:
    PHP 8.4
    

    now match.

    But don’t simply set desired state to whatever actual state happens to be.

    The desired state changes because an authorized configuration change occurred.


    73. Manual Changes

    Suppose an administrator manually changes:

    PHP 8.3 → 8.4
    

    outside CHP.

    Reconciliation detects:

    DRIFT
    

    If the administrator then explicitly accepts the change, CHP can create:

    CONFIGURATION_ACCEPTED
    

    and update desired state.

    This is called:

    adopt actual state
    

    74. Adopt Actual State

    Command:

    hosting-config adopt example.com php.version
    

    Potential output:

    Current desired:
    8.3
    
    Actual:
    8.4
    
    Accept actual value as desired?
    
    YES
    

    This should require appropriate authorization.


    75. Never Automatically Adopt Everything

    Otherwise:

    desired:
    PHP 8.3
    
    attacker changes:
    PHP 7.4
    
    CHP:
    "Actual must be desired!"
    

    That would destroy the purpose of reconciliation.

    Adoption must be explicit.


    76. Configuration Security

    Changes to:

    PHP
    Nginx
    SSL
    DNS
    database
    backup
    

    should generate audit records.

    At minimum:

    actor
    old value
    new value
    reason
    revision
    timestamp
    

    77. Configuration API

    Eventually:

    GET /api/sites/{site}/config
    

    and:

    POST /api/sites/{site}/config/changes
    

    But the API should submit a configuration change request, not directly modify production.


    78. Change Request

    The API should produce:

    CHANGE REQUEST
          ↓
    VALIDATE
          ↓
    PLAN
          ↓
    APPROVAL
          ↓
    EXECUTION
    

    This keeps the control plane safe.


    79. Configuration Explanation Is Essential

    Every final configuration value should be explainable:

    What is the value?
    Where did it come from?
    Why is it this value?
    Who can change it?
    Is it managed?
    What revision introduced it?
    

    For example:

    PHP:
    8.4
    
    Source:
    Site Override
    
    Management:
    Managed
    
    Revision:
    16
    
    Changed by:
    Administrator
    
    Reason:
    Application compatibility
    

    80. Final Configuration Model

    The final desired configuration should look conceptually like:

    {
      "site": "example.com",
    
      "revision": 16,
    
      "configuration": {
    
        "application": {
          "type": "wordpress"
        },
    
        "web": {
          "server": "nginx",
          "document_root": "/storage/websites/example.com/public"
        },
    
        "php": {
          "version": "8.4"
        },
    
        "ssl": {
          "enabled": true,
          "auto_renew": true
        },
    
        "backup": {
          "enabled": true,
          "frequency": "daily",
          "retention_days": 14
        },
    
        "health": {
          "enabled": true,
          "interval_seconds": 60
        },
    
        "wordpress": {
          "management_mode": "CUSTOMER_MANAGED"
        }
      }
    }
    

    This becomes the single desired-state document for the site.


    81. Complete Configuration Flow

    PLATFORM DEFAULT
           │
           ▼
    HOSTING PLAN
           │
           ▼
    SITE PROFILE
           │
           ▼
    SITE OVERRIDES
           │
           ▼
    VALIDATION
           │
           ▼
    CONFIG REVISION
           │
           ▼
    EFFECTIVE DESIRED STATE
           │
           ▼
    RECONCILIATION
           │
           ▼
    ACTUAL STATE
    

    82. CHP Architecture After Lesson 094

    The major components now connect:

                             CHP
                              │
           ┌──────────────────┼──────────────────┐
           │                  │                  │
           ▼                  ▼                  ▼
     CONFIGURATION         OBSERVATION        OPERATIONS
           │                  │                  │
           ▼                  ▼                  ▼
    DESIRED STATE          HEALTH             JOB QUEUE
           │                  │                  │
           └──────────┬───────┘                  │
                      ▼                          ▼
                 RECONCILIATION              WORKERS
                      │
                      ▼
                    DRIFT
                      │
                      ▼
                    PLAN
                      │
                      ▼
                  APPROVAL
                      │
                      ▼
                    BACKUP
                      │
                      ▼
                    APPLY
                      │
                      ▼
                   VERIFY
                      │
                      ▼
                 RECONCILE
    

    83. Lesson 094 — Core Principle

    The Desired-State System should answer:

    WHAT should this site look like?
    

    The Reconciliation Engine answers:

    IS it actually like that?
    

    The Repair Engine will answer:

    HOW can we safely make it like that?
    

    And the Approval System answers:

    ARE we authorized to do it?
    

    So the complete CHP control model becomes:

    DESIRED
       ↓
    OBSERVE
       ↓
    COMPARE
       ↓
    DRIFT
       ↓
    PLAN
       ↓
    AUTHORIZE
       ↓
    APPROVE
       ↓
    BACKUP
       ↓
    APPLY
       ↓
    VERIFY
       ↓
    RECONCILE
    

    Next Lesson — 095

    Build the CHP Repair Plan Engine

    Now that CHP has:

    Desired State
    Actual State
    Reconciliation
    Drift Detection
    Configuration Ownership
    Configuration Versioning
    

    we can finally construct the Repair Plan Engine.

    It will transform:

    DRIFT
    

    into a structured, reviewable plan:

    DRIFT
      ↓
    REPAIR PLAN
      ↓
    PRECONDITIONS
      ↓
    IMPACT ANALYSIS
      ↓
    BACKUP REQUIREMENT
      ↓
    ORDERED OPERATIONS
      ↓
    VERIFICATION
      ↓
    ROLLBACK STRATEGY
    

    For example:

    Detected:
    
    PHP desired = 8.4
    PHP actual  = 8.3
    

    CHP should generate:

    PLAN-00017
    
    1. Verify PHP 8.4 is installed
    2. Verify compatible PHP-FPM service
    3. Create configuration backup
    4. Update site PHP-FPM mapping
    5. Validate Nginx configuration
    6. Reload Nginx
    7. Restart/reload PHP-FPM if required
    8. Run health checks
    9. Reconcile configuration
    10. Mark plan successful
    

    It will also handle:

    PLAN_STALE
    DEPENDENCY_FAILURE
    CONFLICT
    ROLLBACK_REQUIRED
    VERIFICATION_FAILED
    

    and establish the rule that a repair plan describes what CHP intends to do; it does not itself authorize or execute the change.

  • CresignSys Learn — Lesson 093

    Build the CHP Configuration Reconciliation Engine

    The Health Engine answers:

    Is the website working?

    The Reconciliation Engine answers:

    Is the website configured the way CHP expects?

    These are different questions.

    A website can be healthy but incorrectly configured.

    Health:
    HEALTHY
    
    Configuration:
    DRIFTED
    

    For example:

    Desired PHP:
    8.3
    
    Actual PHP:
    8.2
    

    The website may still work, but the configuration is no longer what CHP expects.


    1. Desired State vs Actual State

    The central model is:

                  DESIRED STATE
                        │
                        ▼
                  RECONCILIATION
                        ▲
                        │
                  ACTUAL STATE
                        │
                        ▼
                      DIFF
    

    Result:

    MATCH
    DRIFT
    UNKNOWN
    CONFLICT
    

    2. What Is Desired State?

    Desired state is what CHP believes the site should look like.

    Example:

    {
      "domain": "example.com",
      "web_root": "/storage/websites/example.com/public",
      "php_version": "8.3",
      "php_fpm_socket": "/run/php/php8.3-fpm.sock",
      "ssl_enabled": true,
      "backup_enabled": true,
      "health_monitoring": true
    }
    

    3. What Is Actual State?

    Actual state is what CHP discovers from the server.

    Example:

    {
      "domain": "example.com",
      "web_root": "/storage/websites/example.com/public",
      "php_version": "8.2",
      "php_fpm_socket": "/run/php/php8.2-fpm.sock",
      "ssl_enabled": true,
      "backup_enabled": true,
      "health_monitoring": true
    }
    

    4. Reconciliation

    Compare:

    DESIRED                    ACTUAL
    --------                   -------
    PHP 8.3                    PHP 8.2
    SSL enabled                SSL enabled
    Backup enabled             Backup enabled
    Health enabled             Health enabled
    

    Result:

    PHP:
    DRIFT
    
    SSL:
    MATCH
    
    Backup:
    MATCH
    
    Health:
    MATCH
    

    5. Reconciliation Must Not Automatically Repair

    This is critical.

    The reconciliation engine should initially do:

    DISCOVER
       ↓
    COMPARE
       ↓
    REPORT
    

    not:

    DISCOVER
       ↓
    COMPARE
       ↓
    CHANGE PRODUCTION
    

    The repair system remains a separate controlled process.


    6. Architecture

                      SITE
                       │
              ┌────────┴────────┐
              ▼                 ▼
        DESIRED STATE       ACTUAL STATE
              │                 │
              └────────┬────────┘
                       ▼
                 RECONCILIATION
                       │
                       ▼
                      DIFF
                       │
            ┌──────────┼──────────┐
            ▼          ▼          ▼
          MATCH       DRIFT     UNKNOWN
                       │
                       ▼
                  REPAIR PLAN
    

    7. Configuration Domains

    Reconciliation should be divided into domains:

    DOMAIN
    WEB
    PHP
    DATABASE
    SSL
    DNS
    FILESYSTEM
    WORDPRESS
    BACKUP
    HEALTH
    

    8. Why Divide Configuration?

    Suppose:

    PHP:
    DRIFT
    

    while:

    SSL:
    MATCH
    

    The system can generate a precise repair plan instead of treating the whole site as broken.


    9. Desired State Table

    Create:

    CREATE TABLE desired_config (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
        config_domain TEXT NOT NULL,
    
        config_version INTEGER NOT NULL DEFAULT 1,
    
        desired_state TEXT NOT NULL,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    10. Example

    For:

    example.com
    

    you could have:

    WEB
    PHP
    SSL
    DATABASE
    BACKUP
    HEALTH
    

    as separate configuration records.


    11. Why config_version?

    Suppose:

    version 1:
    PHP 8.2
    

    then:

    version 2:
    PHP 8.3
    

    CHP can determine:

    desired configuration changed
    

    rather than confusing it with external drift.


    12. Configuration Fingerprint

    Create a normalized representation:

    {
      "php_version": "8.3",
      "ssl_enabled": true,
      "backup_enabled": true
    }
    

    Then calculate a hash.

    Example:

    desired fingerprint:
    abc123
    

    Actual:

    actual fingerprint:
    def456
    

    If:

    abc123 == def456
    

    the states match.


    13. Don’t Compare Raw JSON Text

    These two are logically identical:

    {
      "php": "8.3",
      "ssl": true
    }
    

    and:

    {
      "ssl": true,
      "php": "8.3"
    }
    

    String comparison would incorrectly report drift.

    Normalize the data first.


    14. Normalization

    Before comparison:

    RAW
     ↓
    NORMALIZE
     ↓
    SORT
     ↓
    STANDARDIZE
     ↓
    COMPARE
    

    For example:

    PHP:
    "8.3 "
    

    should become:

    "8.3"
    

    before comparison.


    15. Actual State Collector

    Create:

    sudo nano /etc/cresignsys/lib/state-collector.sh
    

    Functions:

    collect_domain_state
    collect_web_state
    collect_php_state
    collect_database_state
    collect_ssl_state
    collect_dns_state
    collect_filesystem_state
    collect_wordpress_state
    

    16. Domain State

    Collect:

    primary domain
    aliases
    www domain
    document root
    

    Example:

    {
      "domain": "example.com",
      "aliases": [
        "www.example.com"
      ],
      "web_root": "/storage/websites/example.com/public"
    }
    

    17. Web State

    Collect:

    Nginx site configuration
    enabled/disabled
    configuration test
    server blocks
    proxy configuration
    PHP handler
    

    Example:

    {
      "server": "nginx",
      "enabled": true,
      "config_test": true
    }
    

    18. PHP State

    Collect:

    PHP version
    PHP-FPM service
    PHP-FPM socket
    configured pool
    

    Example:

    {
      "version": "8.3",
      "fpm_service": "php8.3-fpm",
      "socket": "/run/php/php8.3-fpm.sock"
    }
    

    19. Database State

    Collect:

    engine
    database name
    host
    port
    connectivity
    

    Do not expose:

    password
    

    in the actual-state object.


    20. SSL State

    Collect:

    enabled
    certificate expiry
    issuer
    subject
    SAN
    certificate fingerprint
    

    Example:

    {
      "enabled": true,
      "expires_at": "2026-09-20T00:00:00",
      "issuer": "Let's Encrypt",
      "fingerprint": "..."
    }
    

    21. DNS State

    Collect:

    A
    AAAA
    CNAME
    

    where applicable.

    Example:

    {
      "A": [
        "203.0.113.10"
      ]
    }
    

    Don’t automatically classify a DNS difference as an error.

    It could be intentional.


    22. Filesystem State

    Collect:

    web root exists
    owner
    group
    permissions
    storage filesystem
    disk usage
    inode usage
    

    Example:

    {
      "exists": true,
      "owner": "www-data",
      "group": "www-data",
      "permissions": "0755"
    }
    

    23. WordPress State

    For WordPress:

    core version
    plugin list
    theme
    site URL
    database connection
    

    Example:

    {
      "installed": true,
      "core_version": "7.0.4"
    }
    

    24. WordPress Plugin State

    A desired configuration could specify:

    {
      "required_plugins": [
        "plugin-a",
        "plugin-b"
      ]
    }
    

    Actual state:

    {
      "installed_plugins": [
        "plugin-a",
        "plugin-c"
      ]
    }
    

    Reconciliation detects:

    plugin-b:
    MISSING
    
    plugin-c:
    UNEXPECTED
    

    But this should be treated carefully.

    An unexpected plugin isn’t automatically something CHP should delete.


    25. Configuration Ownership

    This leads to an important concept:

    CHP should only reconcile configuration that it owns.

    For example:

    CHP-managed:
    Nginx virtual host
    PHP-FPM mapping
    backup schedule
    health policy
    

    But perhaps:

    customer-managed:
    WordPress plugins
    WordPress themes
    application content
    

    unless the hosting plan explicitly enables CHP management.


    26. Managed Fields

    Add ownership:

    {
      "php_version": {
        "value": "8.3",
        "managed": true
      },
    
      "plugins": {
        "value": [],
        "managed": false
      }
    }
    

    This prevents CHP from fighting customer changes.


    27. Desired State Should Be Explicit

    Don’t infer:

    "Everything I discover is what the site should be."
    

    Instead:

    desired:
    explicitly configured
    

    and:

    actual:
    discovered
    

    28. Unknown Desired State

    If CHP doesn’t know:

    desired PHP version
    

    then:

    actual PHP:
    8.3
    

    should produce:

    UNKNOWN
    

    not:

    MATCH
    

    and not:

    DRIFT
    

    29. Unknown Actual State

    If CHP cannot determine:

    PHP-FPM socket
    

    then:

    actual:
    UNKNOWN
    

    The result becomes:

    UNKNOWN
    

    rather than assuming:

    DRIFT
    

    30. Conflict

    A conflict is different from ordinary drift.

    Example:

    Desired:
    PHP 8.3
    
    Actual:
    PHP 8.2
    
    Another active CHP job:
    upgrade PHP to 8.4
    

    Now CHP has conflicting desired information.

    Result:

    CONFLICT
    

    rather than simply:

    DRIFT
    

    31. Reconciliation Result Table

    Create:

    CREATE TABLE reconciliation_results (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
    
        config_domain TEXT NOT NULL,
    
        status TEXT NOT NULL,
    
        desired_state TEXT,
        actual_state TEXT,
    
        diff TEXT,
    
        desired_fingerprint TEXT,
        actual_fingerprint TEXT,
    
        checked_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    32. Result Status

    Use:

    MATCH
    DRIFT
    UNKNOWN
    CONFLICT
    ERROR
    

    33. Diff Object

    Example:

    {
      "php_version": {
        "desired": "8.3",
        "actual": "8.2",
        "change": "MODIFY"
      }
    }
    

    This can directly feed the repair planner.


    34. Multiple Drift Items

    Example:

    {
      "php_version": {
        "desired": "8.3",
        "actual": "8.2"
      },
      "web_root": {
        "desired": "/storage/websites/example.com/public",
        "actual": "/storage/websites/example.com/public_html"
      }
    }
    

    The repair planner can then determine whether these changes are related.


    35. Reconciliation Command

    Create:

    sudo nano /usr/local/bin/hosting-reconcile
    

    Usage:

    sudo hosting-reconcile example.com
    

    36. Example Output

    CresignSys Reconciliation
    =========================
    
    Site:
    example.com
    
    DOMAIN:
    MATCH
    
    WEB:
    MATCH
    
    PHP:
    DRIFT
    
      Desired:
      8.3
    
      Actual:
      8.2
    
    DATABASE:
    MATCH
    
    SSL:
    MATCH
    
    BACKUP:
    MATCH
    
    HEALTH:
    MATCH
    
    Overall:
    DRIFT
    

    37. All Sites

    sudo hosting-reconcile --all
    

    Output:

    SITE                    RESULT
    --------------------------------
    example.com             MATCH
    shop.example.com        DRIFT
    blog.example.com        MATCH
    client.example.com      UNKNOWN
    

    38. Reconciliation Scheduler

    The reconciliation engine should use the central job queue.

    Flow:

    scheduler
       ↓
    RECONCILE job
       ↓
    worker
       ↓
    collect actual state
       ↓
    load desired state
       ↓
    compare
       ↓
    store result
       ↓
    emit event
    

    39. Reconciliation Frequency

    Example:

    Nginx:
    15 minutes
    
    PHP:
    15 minutes
    
    SSL:
    1 hour
    
    DNS:
    30 minutes
    
    Backup policy:
    1 hour
    

    Again, these are examples.

    Configuration that changes rarely doesn’t need minute-by-minute reconciliation.


    40. Reconciliation Event

    If:

    MATCH
    

    and remains:

    MATCH
    

    don’t generate a new event every cycle.

    Only emit meaningful transitions.

    Example:

    MATCH
     ↓
    DRIFT
    

    generates:

    DRIFT_DETECTED
    

    Then:

    DRIFT
     ↓
    MATCH
    

    generates:

    DRIFT_RESOLVED
    

    41. Event Metadata

    Example:

    {
      "domain": "PHP",
      "field": "php_version",
      "desired": "8.3",
      "actual": "8.2"
    }
    

    This is much better than:

    "PHP mismatch"
    

    42. Drift Fingerprint

    Create:

    DRIFT:site7:PHP:php_version
    

    or a hash.

    This allows the alert system to deduplicate:

    DRIFT_DETECTED
    DRIFT_DETECTED
    DRIFT_DETECTED
    

    into one alert.


    43. Drift Alert

    The Alert Engine from Lesson 091 can now process:

    DRIFT_DETECTED
    

    Example:

    WARNING
    
    Configuration drift detected on example.com.
    
    PHP version:
    Expected 8.3
    Actual 8.2
    

    44. Drift Is Not Always an Error

    Suppose customer intentionally changed:

    PHP:
    8.3 → 8.2
    

    CHP should detect:

    DRIFT
    

    but shouldn’t necessarily repair it.

    Instead:

    DRIFT
    +
    customer-managed field
    

    could produce:

    INFO
    

    or no alert.


    45. Managed vs Unmanaged Drift

    This is one of the most important CHP concepts.

    Managed drift

    CHP owns field
    

    Example:

    Nginx config
    

    → alert/repair candidate.

    Unmanaged drift

    Customer owns field
    

    Example:

    WordPress plugin
    

    → report only.


    46. Configuration Ownership Table

    Create:

    CREATE TABLE config_ownership (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
        config_domain TEXT NOT NULL,
        config_key TEXT NOT NULL,
    
        management_mode TEXT NOT NULL,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    Modes:

    MANAGED
    UNMANAGED
    READ_ONLY
    

    47. Example

    example.com
    
    Nginx:
    MANAGED
    
    PHP:
    MANAGED
    
    SSL:
    MANAGED
    
    WordPress plugins:
    UNMANAGED
    
    WordPress content:
    UNMANAGED
    

    48. Why This Is Important for Reseller Hosting

    Different customers want different levels of management.

    Customer A:

    hosting only
    

    Customer B:

    managed WordPress
    

    Customer C:

    fully managed website
    

    CHP can use ownership policies to support all three.


    49. Desired State Sources

    Desired configuration can come from:

    hosting plan
    site settings
    customer request
    administrator configuration
    approved repair plan
    

    But there should be one canonical desired state.


    50. Desired State Precedence

    Example:

    PLAN DEFAULT
          ↓
    SITE OVERRIDE
          ↓
    APPROVED CHANGE
          ↓
    CURRENT DESIRED STATE
    

    This prevents conflicting configuration sources.


    51. Don’t Let the Repair Plan Become Desired State Automatically

    Suppose a repair plan proposes:

    PHP 8.3 → 8.4
    

    The plan should not change the desired state merely because it was created.

    Only after explicit approval/execution should the desired state be updated according to the defined policy.


    52. Desired State Versioning

    Every desired configuration change should create a new version:

    version 1:
    PHP 8.2
    
    version 2:
    PHP 8.3
    
    version 3:
    PHP 8.4
    

    This gives CHP configuration history.


    53. Configuration History Table

    You can eventually add:

    CREATE TABLE desired_config_history (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
        config_version INTEGER NOT NULL,
    
        config_state TEXT NOT NULL,
    
        actor_id INTEGER,
        reason TEXT,
    
        created_at TEXT NOT NULL
    );
    

    54. Example

    Version 1
    ---------
    PHP 8.2
    Created by:
    SYSTEM
    
    Version 2
    ---------
    PHP 8.3
    Created by:
    ADMIN
    Reason:
    Upgrade PHP
    

    This gives a clean audit trail.


    55. Reconciliation Does Not Mean Enforcement

    Think of it as:

    RECONCILIATION
    =
    measurement + comparison
    

    while:

    REPAIR
    =
    controlled enforcement
    

    Keeping them separate is safer.


    56. Reconciliation → Repair

    Once drift is detected:

    DRIFT
     ↓
    DIFF
     ↓
    REPAIR PLAN
    

    Example:

    Plan:
    Change PHP 8.2 → 8.3
    

    Then:

    PLAN
     ↓
    VALIDATE
     ↓
    APPROVE
     ↓
    JOB
     ↓
    BACKUP
     ↓
    REPAIR
    

    57. Never Create a Repair Plan From Raw Text

    Bad:

    "PHP is wrong"
    

    Good:

    {
      "operation": "SET_PHP_VERSION",
      "desired": "8.3",
      "actual": "8.2"
    }
    

    The repair engine can validate structured operations.


    58. Reconciliation Safety

    Before creating a repair plan, confirm:

    desired state is current
    actual state is current
    no conflicting job
    no active restore
    no maintenance lock
    

    59. Reconciliation Race Condition

    Example:

    18:00
    reconciliation sees PHP 8.2
    
    18:01
    admin manually changes PHP to 8.3
    
    18:02
    old repair plan says:
    change 8.2 → 8.3
    

    That plan is now stale.

    Therefore the repair planner must re-check actual state before execution.


    60. Desired/Actual Version Fingerprint

    A repair plan should record:

    desired fingerprint:
    ABC123
    
    actual fingerprint:
    XYZ789
    

    At execution:

    current actual fingerprint:
    NEW456
    

    If:

    XYZ789 != NEW456
    

    the plan is stale.

    Stop.


    61. Stale Plan

    Result:

    PLAN_STALE
    

    not:

    repair anyway
    

    This prevents CHP from applying outdated assumptions.


    62. Reconciliation Status

    The site can now have:

    CONFIGURATION:
    MATCH
    DRIFT
    UNKNOWN
    CONFLICT
    

    and the dashboard can show it separately from:

    HEALTH:
    HEALTHY
    

    63. Example Dashboard

    example.com
    
    Health
    ------
    HEALTHY
    98/100
    
    Configuration
    -------------
    DRIFT
    
    PHP:
    Expected 8.3
    Actual 8.2
    
    Backup
    ------
    HEALTHY
    
    Security
    --------
    HEALTHY
    
    Alerts
    ------
    1 WARNING
    

    This is a very useful hosting control-plane view.


    64. Reconciliation Summary

    For all sites:

    SITE                    HEALTH      CONFIG
    ------------------------------------------------
    example.com             HEALTHY     MATCH
    shop.example.com        HEALTHY     DRIFT
    blog.example.com        WARNING     MATCH
    client.example.com      DOWN        UNKNOWN
    

    65. Reconciliation Metrics

    Eventually track:

    sites matching desired state
    sites with drift
    sites unknown
    sites with conflicts
    

    Example:

    Managed Sites:
    120
    
    MATCH:
    108
    
    DRIFT:
    9
    
    UNKNOWN:
    2
    
    CONFLICT:
    1
    

    66. Configuration Compliance

    This becomes useful for hosting plans.

    Example:

    Configuration compliance:
    90%
    

    But again, don’t hide important individual drift behind a single percentage.


    67. Event Flow

    The complete configuration lifecycle is now:

    ACTUAL STATE
          │
          ▼
    RECONCILIATION
          │
          ▼
    COMPARE
          │
     ┌────┼──────────┐
     ▼    ▼          ▼
    MATCH DRIFT    UNKNOWN
           │
           ▼
       EVENT
           │
           ▼
        ALERT
           │
           ▼
    REPAIR PLAN
           │
           ▼
    APPROVAL
           │
           ▼
    JOB
           │
           ▼
    REPAIR
           │
           ▼
    RECONCILIATION
           │
           ▼
    MATCH
    

    This is the core control-loop architecture.


    68. The CHP Control Loop

    CHP is now moving toward a true reconciliation-based control plane:

               DESIRED
                  │
                  ▼
               OBSERVE
                  │
                  ▼
               COMPARE
                  │
                  ▼
                DRIFT
                  │
                  ▼
                 PLAN
                  │
                  ▼
              APPROVAL
                  │
                  ▼
                BACKUP
                  │
                  ▼
                APPLY
                  │
                  ▼
               VERIFY
                  │
                  ▼
              RECONCILE
                  │
              ┌───┴───┐
              ▼       ▼
            MATCH    DRIFT
              │       │
              │       └──→ investigate
              ▼
           COMPLETE
    

    69. Lesson 093 — Core Principle

    The most important architectural rule is:

    Observe first. Change later.

    Reconciliation should never assume that a difference automatically means something is broken.

    A difference can be:

    intentional
    unintentional
    customer-managed
    CHP-managed
    temporary
    unknown
    conflicting
    

    Therefore CHP should first establish:

    WHAT IS DESIRED?
    WHAT IS ACTUAL?
    WHO OWNS THE FIELD?
    WHAT EXACTLY IS DIFFERENT?
    

    Only then should the Repair Engine consider changing the site.


    Next Lesson — 094

    Build the CHP Desired-State & Configuration Management System

    The reconciliation engine can now detect:

    DESIRED ≠ ACTUAL
    

    But we need a reliable source for:

    What exactly is the desired configuration?

    Lesson 094 will build the configuration management layer:

    HOSTING PLAN
          │
          ▼
    SITE TEMPLATE
          │
          ▼
    SITE OVERRIDES
          │
          ▼
    DESIRED STATE
          │
          ▼
    RECONCILIATION
    

    It will define:

    site profiles
    PHP policies
    Nginx policies
    SSL policies
    backup policies
    health policies
    resource limits
    WordPress management mode
    customer overrides
    configuration versioning
    

    and establish which settings are:

    DEFAULT
    INHERITED
    OVERRIDDEN
    LOCKED
    MANAGED
    CUSTOMER-MANAGED
    

    That will provide the missing foundation for generating safe, deterministic Repair Plans.

  • CresignSys Learn — Lesson 092

    Build the CHP Site Health Monitoring Engine

    The Alert Engine from Lesson 091 can only work well if CHP has reliable health information.

    We therefore need a dedicated health engine that answers:

    Is this website actually healthy right now?

    The architecture becomes:

                         SITE
                          │
            ┌─────────────┼─────────────┐
            ▼             ▼             ▼
           DNS           HTTP          SSL
            │             │             │
            ├─────────────┼─────────────┤
            ▼             ▼             ▼
          NGINX         PHP-FPM        MYSQL
            │             │             │
            └─────────────┼─────────────┘
                          ▼
                     HEALTH ENGINE
                          │
                  ┌───────┴────────┐
                  ▼                ▼
              HEALTH STATE      HEALTH SCORE
                  │
                  ▼
                 EVENT
                  │
                  ▼
                 ALERT
    

    1. Health Is Not One Check

    A website can have:

    HTTP:
    200 OK
    

    while:

    MySQL:
    DOWN
    

    or:

    PHP-FPM:
    DOWN
    

    Therefore:

    HTTP 200
    ≠
    Website fully healthy
    

    2. Health Dimensions

    CHP should monitor at least:

    DNS
    HTTP
    HTTPS
    SSL
    Nginx
    PHP-FPM
    MySQL
    Filesystem
    Disk
    CPU
    RAM
    WordPress
    

    Not every check needs to run at the same frequency.


    3. Health Categories

    Group checks into:

    AVAILABILITY
    APPLICATION
    RUNTIME
    INFRASTRUCTURE
    SECURITY
    

    Availability

    DNS
    HTTP
    HTTPS
    

    Application

    WordPress
    database connectivity
    application response
    

    Runtime

    Nginx
    PHP-FPM
    MySQL
    

    Infrastructure

    disk
    CPU
    RAM
    filesystem
    

    Security

    SSL
    certificate expiry
    HTTPS configuration
    

    4. Health States

    Use:

    HEALTHY
    DEGRADED
    WARNING
    DOWN
    UNKNOWN
    

    5. Meaning of HEALTHY

    HEALTHY
    

    means the important checks are passing and no critical dependency is currently failing.

    Example:

    DNS:
    PASS
    
    HTTPS:
    PASS
    
    SSL:
    PASS
    
    Nginx:
    PASS
    
    PHP-FPM:
    PASS
    
    MySQL:
    PASS
    
    WordPress:
    PASS
    

    6. DEGRADED

    Example:

    HTTP:
    PASS
    
    PHP-FPM:
    PASS
    
    MySQL:
    PASS
    
    Disk:
    82%
    

    Website still works.

    But:

    disk:
    WARNING
    

    Therefore:

    overall:
    DEGRADED
    

    7. WARNING

    A warning means the website may still be usable but there is a condition requiring attention.

    Examples:

    SSL expires in 14 days
    disk usage > 90%
    backup is approaching SLA
    high resource usage
    

    8. DOWN

    A critical availability failure.

    Example:

    DNS:
    PASS
    
    HTTPS:
    FAIL
    
    HTTP:
    FAIL
    

    or:

    Nginx:
    DOWN
    

    with the website unavailable.


    9. UNKNOWN

    Use:

    UNKNOWN
    

    when CHP cannot confidently determine health.

    For example:

    health worker:
    not running
    

    or:

    database:
    temporarily unreachable
    

    without enough evidence to classify the site as down.

    This prevents false certainty.


    10. Don’t Use a Simple Average

    Suppose:

    HTTP:
    100%
    
    DNS:
    100%
    
    SSL:
    100%
    
    MySQL:
    0%
    

    A simple average might say:

    75%
    

    and classify the site as:

    mostly healthy
    

    That can be misleading.

    MySQL may be a critical dependency.


    11. Weighted Health Model

    Instead use categories and weights.

    Example:

    HTTP/HTTPS:
    30%
    
    Application:
    25%
    
    Runtime:
    20%
    
    Database:
    15%
    
    Infrastructure:
    10%
    

    These are initial design values, not fixed production policy.


    12. Critical Checks

    Some checks should override the score.

    For example:

    DNS completely unavailable
    

    may mean:

    DOWN
    

    even if:

    disk:
    healthy
    CPU:
    healthy
    RAM:
    healthy
    

    13. Health Decision Tree

    Conceptually:

                    HEALTH CHECK
                         │
                  DNS available?
                    /          \
                  NO            YES
                  │              │
                 DOWN        HTTPS available?
                                  │
                             ┌────┴────┐
                            NO         YES
                            │           │
                         DOWN       application?
                                        │
                                   ┌────┴────┐
                                  NO         YES
                                  │           │
                              DEGRADED      HEALTHY
    

    The actual implementation will consider more conditions.


    14. Health Check Database

    Create:

    CREATE TABLE health_checks (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
    
        check_type TEXT NOT NULL,
        status TEXT NOT NULL,
    
        response_time_ms INTEGER,
    
        message TEXT,
        metadata TEXT,
    
        checked_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    15. Check Status

    Individual checks can use:

    PASS
    WARN
    FAIL
    UNKNOWN
    

    Don’t use overall health states here.


    16. Example Health Checks

    DNS:
    PASS
    
    HTTP:
    PASS
    
    HTTPS:
    PASS
    
    SSL:
    WARN
    
    PHP-FPM:
    PASS
    
    MYSQL:
    PASS
    
    DISK:
    WARN
    

    Then the health engine calculates:

    OVERALL:
    WARNING
    

    17. Store Response Time

    For HTTP:

    response_time_ms:
    182
    

    This allows CHP to detect performance degradation.

    Example:

    Normal:
    180 ms
    
    Current:
    2400 ms
    

    Website may technically be:

    HTTP 200
    

    but performance is poor.


    18. HTTP Check

    Create:

    sudo nano /usr/local/bin/hosting-health
    

    Usage:

    sudo hosting-health example.com
    

    The HTTP check should verify:

    DNS resolution
    TCP connection
    TLS
    HTTP response
    status code
    response time
    

    19. Expected HTTP Status

    For a normal website:

    200
    

    But don’t assume only 200 is valid.

    Depending on the application:

    301
    302
    

    may be expected.

    CHP should follow redirects carefully and determine the final status.


    20. Redirect Loop

    Example:

    HTTP
     ↓
    HTTPS
     ↓
    HTTP
     ↓
    HTTPS
    

    This should produce:

    HTTP:
    FAIL
    

    with:

    REDIRECT_LOOP
    

    21. HTTP 500

    If:

    HTTP:
    500
    

    record:

    HTTP:
    FAIL
    

    and:

    metadata:
    {
      "status_code": 500
    }
    

    This should generate a health event.


    22. HTTP 404

    A root URL returning:

    404
    

    may indicate an application configuration problem.

    But CHP should distinguish:

    expected 404
    

    from:

    unexpected 404
    

    A configurable health endpoint is therefore useful.


    23. Health Endpoint

    Eventually support:

    https://example.com/health
    

    or:

    https://example.com/wp-json/
    

    depending on application type.

    For WordPress, CHP can use a safe application-level check rather than assuming /health exists.


    24. DNS Check

    Check:

    A
    AAAA
    

    and optionally:

    CNAME
    

    depending on the site’s configuration.

    Compare:

    EXPECTED IP
    

    against:

    ACTUAL IP
    

    where appropriate.


    25. DNS Drift

    Example:

    Expected:
    203.0.113.10
    
    Actual:
    203.0.113.25
    

    This is not necessarily an outage.

    It may be:

    intentional DNS change
    

    Therefore classify as:

    DRIFT
    

    rather than automatically:

    DOWN
    

    26. DNS Failure

    If:

    DNS resolution:
    FAIL
    

    and repeated checks confirm it:

    overall:
    DOWN
    

    because the public website cannot normally be reached.


    27. HTTPS Check

    Verify:

    TCP 443
    TLS handshake
    certificate
    HTTP response
    

    Example:

    HTTPS:
    PASS
    

    28. SSL Certificate

    Check:

    subject
    issuer
    not_before
    not_after
    SAN
    

    Calculate:

    days_remaining
    

    29. SSL Health

    Example:

    Certificate:
    valid
    
    Expires:
    2026-09-20
    
    Days remaining:
    38
    
    Status:
    PASS
    

    30. SSL Warning

    Example:

    Days remaining:
    13
    
    Status:
    WARN
    

    This should produce:

    SSL_EXPIRING
    

    event.


    31. SSL Expired

    If:

    days_remaining < 0
    

    then:

    SSL:
    FAIL
    

    and likely:

    HTTPS:
    FAIL
    

    depending on client behavior.


    32. Nginx Check

    Check:

    systemctl is-active nginx
    

    Expected:

    active
    

    But a running Nginx process alone does not prove the site works.

    Therefore:

    Nginx:
    PASS
    

    is only one dimension.


    33. Nginx Configuration

    Run:

    nginx -t
    

    Possible results:

    PASS
    

    or:

    FAIL
    

    This is especially important before configuration changes.


    34. PHP-FPM

    Check the configured PHP-FPM service.

    Example:

    systemctl is-active php8.3-fpm
    

    Don’t hard-code the version into CHP.

    The site configuration should determine:

    PHP 8.2
    PHP 8.3
    PHP 8.4
    

    where supported by the hosting environment.


    35. PHP-FPM Socket

    Also verify the configured socket exists.

    Example:

    /run/php/php8.3-fpm.sock
    

    A service may be active while the expected socket is missing.

    Therefore:

    service:
    PASS
    
    socket:
    FAIL
    

    should result in:

    PHP-FPM:
    FAIL
    

    36. MySQL

    Check:

    systemctl is-active mysql
    

    Then perform an actual connection test.

    Don’t treat:

    systemctl:
    active
    

    as sufficient.


    37. Database Connectivity

    The site should be tested using its configured database.

    Conceptually:

    connect
     ↓
    authenticate
     ↓
    SELECT 1
     ↓
    disconnect
    

    Result:

    MYSQL:
    PASS
    

    38. WordPress Check

    For a WordPress site, verify:

    WordPress files
    wp-config.php
    database connectivity
    core integrity
    site URL
    

    Potentially:

    wp core is-installed
    

    where WP-CLI is available.


    39. WordPress Core Integrity

    Eventually:

    wp core verify-checksums
    

    can detect modified WordPress core files.

    This is particularly useful for security monitoring.


    40. Don’t Automatically Classify Checksum Failure as Site Down

    Example:

    HTTP:
    PASS
    
    WordPress:
    PASS
    
    Core integrity:
    FAIL
    

    The website is functioning but may have a security/integrity issue.

    Overall:

    WARNING
    

    or:

    DEGRADED
    

    rather than:

    DOWN
    

    41. Filesystem Check

    Check:

    web root exists
    

    and:

    required directories
    

    Example:

    /storage/websites/example.com/public
    

    42. Ownership Check

    Check critical paths:

    owner
    group
    permissions
    

    Unexpected ownership can cause:

    PHP failures
    upload failures
    cache failures
    

    43. Disk Usage

    Check:

    df -P
    

    For the relevant filesystem.

    Don’t only check:

    /var
    

    if website data is stored on:

    /storage
    

    44. Disk Thresholds

    Example:

    < 80%:
    PASS
    
    80–90%:
    WARN
    
    90–95%:
    ERROR
    
    > 95%:
    CRITICAL
    

    Again, make thresholds configurable.


    45. Inode Usage

    Disk space can be available while inodes are exhausted.

    Check:

    df -i
    

    Example:

    disk:
    60%
    
    inodes:
    99%
    

    The site can still fail to create files.

    Therefore:

    INODE_USAGE
    

    should be a separate check.


    46. CPU

    CPU should be monitored at the server level.

    For example:

    1-minute load average
    

    Don’t classify a website as unhealthy merely because CPU is temporarily high.

    Use sustained thresholds.


    47. RAM

    Likewise:

    memory pressure
    swap usage
    available memory
    

    are more meaningful than a single instantaneous percentage.


    48. Swap

    If:

    swap:
    95%
    

    that may indicate memory pressure.

    But swap usage alone does not necessarily mean a site is down.

    Classify it as:

    WARNING
    

    unless application failures confirm a more serious problem.


    49. Health Check Frequency

    Not every check should run every minute.

    Example:

    HTTP:
    1 minute
    
    DNS:
    5 minutes
    
    SSL:
    1 hour
    
    WordPress:
    15 minutes
    
    disk:
    5 minutes
    
    CPU:
    1 minute
    

    50. Why Different Frequencies?

    SSL expiration changes slowly.

    HTTP availability can change quickly.

    Therefore:

    high-frequency:
    availability
    
    low-frequency:
    configuration
    

    This reduces unnecessary server load.


    51. Health Policy Table

    Create:

    CREATE TABLE health_policies (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        site_id INTEGER NOT NULL,
        check_type TEXT NOT NULL,
    
        enabled INTEGER NOT NULL DEFAULT 1,
    
        interval_seconds INTEGER NOT NULL,
    
        timeout_seconds INTEGER NOT NULL DEFAULT 10,
    
        warning_threshold REAL,
        critical_threshold REAL,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    52. Health Jobs

    Health monitoring should use the central job queue.

    Create:

    HEALTH
    

    jobs.

    Scheduler:

    find due health checks
     ↓
    create HEALTH job
     ↓
    worker
     ↓
    perform checks
     ↓
    store results
     ↓
    calculate state
     ↓
    emit events
    

    53. Don’t Create One Job Per Check Initially

    If a site has:

    10 checks
    

    you don’t necessarily need:

    10 jobs
    

    every minute.

    Instead:

    one HEALTH job
    

    can execute the due checks.


    54. Health Job Payload

    Example:

    {
      "site_id": 7,
      "checks": [
        "HTTP",
        "HTTPS",
        "SSL",
        "PHP_FPM",
        "MYSQL"
      ]
    }
    

    55. Health Result

    Store individual results:

    HTTP:
    PASS
    182ms
    
    HTTPS:
    PASS
    201ms
    
    SSL:
    PASS
    38 days
    
    PHP_FPM:
    PASS
    
    MYSQL:
    PASS
    

    Then calculate:

    OVERALL:
    HEALTHY
    

    56. Health Snapshot

    Create a current-state table:

    CREATE TABLE site_health (
        site_id INTEGER PRIMARY KEY,
    
        overall_status TEXT NOT NULL,
        health_score INTEGER,
    
        last_checked_at TEXT NOT NULL,
        status_changed_at TEXT,
    
        summary TEXT,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );
    

    57. Why a Snapshot Table?

    The raw check history may contain thousands of records.

    The dashboard shouldn’t scan the entire history every time.

    Instead:

    site_health
    

    contains:

    current status
    current score
    last check
    summary
    

    while:

    health_checks
    

    contains historical data.


    58. Current vs Historical Data

    This is a recurring CHP architecture principle.

    Current state

    site_health
    

    History

    health_checks
    events
    

    Don’t confuse them.


    59. Health Score

    You can expose a score:

    100
    

    but don’t let the score replace the state.

    Example:

    Score:
    92
    
    Status:
    WARNING
    

    because:

    SSL expires in 5 days
    

    could be a warning even though most checks pass.


    60. Example Scoring

    Conceptually:

    HTTP:
    30 points
    
    Application:
    25 points
    
    Runtime:
    20 points
    
    Database:
    15 points
    
    Infrastructure:
    10 points
    

    A failure reduces the score.

    But critical overrides remain.


    61. Critical Override

    Example:

    score:
    95
    
    DNS:
    FAIL
    

    Final:

    DOWN
    

    not:

    HEALTHY
    

    62. Consecutive Failure Logic

    Don’t immediately declare:

    DOWN
    

    after one transient HTTP failure.

    Use:

    failure_count
    

    Example:

    1 failure:
    suspect
    
    2 failures:
    degraded
    
    3 consecutive failures:
    DOWN
    

    The exact thresholds should be configurable.


    63. Recovery Logic

    Similarly:

    DOWN
    

    should not necessarily become:

    HEALTHY
    

    after one successful check.

    Use:

    2–3 consecutive successes
    

    to confirm recovery.

    This reduces flapping.


    64. Health Flapping

    Example:

    PASS
    FAIL
    PASS
    FAIL
    PASS
    FAIL
    

    This is called:

    flapping
    

    CHP should detect it.


    65. Flapping Alert

    If a site repeatedly changes:

    HEALTHY
    ↕
    DOWN
    

    within a short period:

    HEALTH_FLAPPING
    

    can be generated.


    66. Why Flapping Matters

    A site that is:

    up 95% of the time
    

    may still be causing serious user impact if it repeatedly crashes.

    Flapping should therefore be visible separately.


    67. Health Event Generation

    The health engine should generate events primarily for state changes.

    Example:

    HEALTH_CHANGED
    

    Metadata:

    {
      "old_status": "HEALTHY",
      "new_status": "DOWN",
      "reason": "HTTP_FAILURE"
    }
    

    68. Don’t Generate Huge Event Volumes

    Avoid:

    HEALTHY
    HEALTHY
    HEALTHY
    HEALTHY
    

    every minute.

    Generate:

    HEALTH_CHANGED
    

    only when meaningful state changes occur.


    69. Health Summary

    For each site:

    Overall:
    DEGRADED
    
    Availability:
    HEALTHY
    
    Application:
    HEALTHY
    
    Runtime:
    HEALTHY
    
    Database:
    HEALTHY
    
    Infrastructure:
    WARNING
    
    Security:
    WARNING
    

    This is far more useful than one number.


    70. CHP Site Health CLI

    Use:

    hosting-health example.com
    

    Output:

    CresignSys Site Health
    ======================
    
    Site:
    example.com
    
    Overall:
    HEALTHY
    
    Score:
    98/100
    
    Availability:
      DNS       PASS
      HTTP      PASS  182ms
      HTTPS     PASS  201ms
    
    Application:
      WordPress PASS
    
    Runtime:
      Nginx     PASS
      PHP-FPM   PASS
      MySQL     PASS
    
    Security:
      SSL       PASS
      Expiry    38 days
    
    Infrastructure:
      Disk      PASS  62%
      Inodes    PASS  41%
      RAM       PASS  54%
    
    Last checked:
    2026-08-13 18:40
    

    71. Warning Example

    Overall:
    DEGRADED
    
    Score:
    87/100
    
    Security:
      SSL:
      WARNING
    
      Expiry:
      13 days
    
    Infrastructure:
      Disk:
      WARNING
    
      Usage:
      87%
    

    72. Down Example

    Overall:
    DOWN
    
    Availability:
      DNS:
      PASS
    
      HTTP:
      FAIL
    
      HTTPS:
      FAIL
    
    Runtime:
      Nginx:
      FAIL
    
    Reason:
    Nginx service unavailable.
    

    73. Unknown Example

    Overall:
    UNKNOWN
    
    Reason:
    Health worker has not completed a successful check
    within the configured monitoring window.
    

    This is safer than claiming:

    HEALTHY
    

    when CHP has no recent evidence.


    74. Health Dashboard

    For all sites:

    SITE                    STATUS       SCORE
    ------------------------------------------------
    example.com             HEALTHY      98
    shop.example.com        DEGRADED     87
    blog.example.com        WARNING      71
    client.example.com      DOWN         20
    newsite.com             UNKNOWN      -
    

    75. Site Health Detail

    The dashboard can show:

    example.com
    
    Overall:
    DEGRADED
    
    Availability:
    Healthy
    
    Application:
    Healthy
    
    Runtime:
    Healthy
    
    Security:
    Warning
    
    Infrastructure:
    Warning
    
    Active Alerts:
    2
    
    Last Verified Backup:
    2h ago
    

    This combines:

    health
    alerts
    backup
    

    into one operational view.


    76. Health and Backup Are Different

    A site can be:

    HEALTHY
    

    but:

    BACKUP:
    CRITICAL
    

    Overall platform status should therefore show both.

    Don’t hide backup problems just because the website works.


    77. Health and Reconciliation Are Different

    A site can be:

    HEALTHY
    

    but:

    CONFIGURATION:
    DRIFTED
    

    For example:

    PHP:
    8.3
    

    but desired:

    8.4
    

    The website works, but configuration differs from the intended state.


    78. Four Operational Dimensions

    CHP can now distinguish:

    HEALTH
    BACKUP
    CONFIGURATION
    SECURITY
    

    Example:

    Health:
    HEALTHY
    
    Backup:
    WARNING
    
    Configuration:
    DRIFT
    
    Security:
    HEALTHY
    

    This is much more informative than one overall “green/red” indicator.


    79. Health Scheduler

    The health scheduler can use the same central scheduler:

    systemd timer
          ↓
    CHP scheduler
          ↓
    health policies
          ↓
    HEALTH jobs
          ↓
    worker
    

    No second scheduler is required.


    80. Health Job Priority

    Health checks should usually be:

    NORMAL
    

    But availability checks for known failing sites can temporarily be:

    HIGH
    

    if required for rapid recovery detection.


    81. Don’t Let Health Checks Starve Backups

    Suppose:

    100 sites
    

    and every site needs health checks every minute.

    Without limits:

    health jobs
    health jobs
    health jobs
    

    could consume all workers.

    Use separate concurrency controls:

    MAX_HEALTH_WORKERS=1
    MAX_BACKUP_WORKERS=1
    

    initially.


    82. Health Timeout

    Every check needs a timeout.

    Example:

    HTTP:
    10 seconds
    
    DNS:
    5 seconds
    
    MySQL:
    5 seconds
    
    WordPress:
    15 seconds
    

    A timeout should produce:

    TIMEOUT
    

    rather than hanging the worker indefinitely.


    83. Health Check Security

    Don’t allow the monitoring system to execute arbitrary URLs supplied by an untrusted user.

    Validate:

    domain
    protocol
    port
    allowed destination
    

    This prevents CHP from becoming a server-side request proxy.


    84. Internal Checks

    For local services:

    systemctl
    socket checks
    local database
    filesystem
    

    For public checks:

    DNS
    HTTP
    HTTPS
    SSL
    

    Keep these categories separate.


    85. Health Data Retention

    Raw health checks can become large.

    For example:

    10 checks
    ×
    1 minute
    ×
    100 sites
    

    creates:

    1,440,000 check records/day
    

    Therefore do not retain every raw result indefinitely.


    86. Health History Strategy

    Keep:

    current snapshot:
    indefinitely
    

    Detailed checks:

    7–30 days
    

    Aggregated history:

    hourly/day summaries
    

    for longer periods.


    87. Health Aggregation

    Instead of storing every result forever:

    1-minute checks
    

    can later become:

    hourly:
    availability %
    average response time
    failure count
    

    and:

    daily:
    uptime %
    average latency
    incidents
    

    88. Future SLA Reporting

    This allows CHP to calculate:

    Monthly uptime:
    99.95%
    
    Average response:
    210ms
    
    Downtime:
    21 minutes
    

    This can eventually feed hosting customer reports.


    89. Health Engine Architecture

                     HEALTH SCHEDULER
                            │
                            ▼
                        HEALTH JOB
                            │
                            ▼
                       CHECK ENGINE
                            │
           ┌────────────────┼────────────────┐
           ▼                ▼                ▼
       AVAILABILITY      RUNTIME         INFRASTRUCTURE
           │                │                │
           └────────────────┼────────────────┘
                            ▼
                      NORMALIZATION
                            │
                            ▼
                      HEALTH DECISION
                            │
                  ┌─────────┴─────────┐
                  ▼                   ▼
             CURRENT STATE          EVENT
                  │                   │
                  ▼                   ▼
             DASHBOARD              ALERT
    

    90. Health Decision Algorithm

    Conceptually:

    run checks
        ↓
    validate results
        ↓
    apply consecutive failure rules
        ↓
    apply critical overrides
        ↓
    calculate category states
        ↓
    calculate overall state
        ↓
    update site_health
        ↓
    emit state-change event
    

    91. Example

    Raw results:

    DNS:
    PASS
    
    HTTP:
    PASS
    
    HTTPS:
    PASS
    
    SSL:
    WARN
    
    Nginx:
    PASS
    
    PHP-FPM:
    PASS
    
    MySQL:
    PASS
    
    Disk:
    WARN
    

    Category results:

    Availability:
    HEALTHY
    
    Runtime:
    HEALTHY
    
    Database:
    HEALTHY
    
    Security:
    WARNING
    
    Infrastructure:
    WARNING
    

    Overall:

    DEGRADED
    

    92. Lesson 092 — Core Principle

    The CHP health engine should never confuse:

    "the server process is running"
    

    with:

    "the website is healthy."
    

    Health must be evaluated across multiple layers:

    DNS
     ↓
    NETWORK
     ↓
    HTTP/HTTPS
     ↓
    WEB SERVER
     ↓
    PHP
     ↓
    DATABASE
     ↓
    APPLICATION
     ↓
    FILESYSTEM
     ↓
    RESOURCES
     ↓
    SECURITY
    

    And the final architecture is:

    RAW CHECK
        ↓
    NORMALIZE
        ↓
    STATE
        ↓
    EVENT
        ↓
    ALERT
        ↓
    NOTIFICATION
    

    Next Lesson — 093

    Build the CHP Configuration Reconciliation Engine

    The health engine tells us:

    "Is the website working?"
    

    The reconciliation engine will answer a different question:

    “Is the website configured the way CHP expects it to be?”

    It will compare:

    DESIRED STATE
           vs
    ACTUAL STATE
    

    across:

    domain
    web root
    Nginx
    PHP version
    PHP-FPM socket
    database
    SSL
    DNS
    permissions
    WordPress
    backup policy
    health policy
    

    and classify differences as:

    MATCH
    DRIFT
    UNKNOWN
    CONFLICT
    

    The key architecture will be:

    DESIRED CONFIG
          │
          ▼
    RECONCILIATION
          ▲
          │
    ACTUAL CONFIG
          │
          ▼
    DIFF
          │
     ┌────┴────┐
     ▼         ▼
    MATCH     DRIFT
               │
               ▼
            PLAN ENGINE
    

    This is the layer that will eventually make CHP capable of safely detecting and correcting configuration drift across all hosted websites.

  • CresignSys Learn — Lesson 091

    Build the CHP Alert & Notification Engine

    The Event System from Lesson 090 tells us:

    something happened
    

    The Alert Engine determines:

    Does it matter?
    How urgent is it?
    Who needs to know?
    Should we notify them now?
    Has this already been reported?
    

    The architecture becomes:

                     CHP EVENT
                         │
                         ▼
                    ALERT ENGINE
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
           NO ALERT                ALERT
                                    │
                             ┌──────┴──────┐
                             ▼             ▼
                        DEDUPLICATE      CREATE
                             │             │
                             └──────┬──────┘
                                    ▼
                               NOTIFICATION
                                    │
                       ┌────────────┼────────────┐
                       ▼            ▼            ▼
                     EMAIL       DASHBOARD     WEBHOOK
    

    1. Event vs Alert vs Notification

    These three should remain separate.

    Event

    BACKUP_FAILED
    

    Alert

    example.com backup has failed twice.
    Severity: ERROR
    

    Notification

    Email sent to administrator.
    

    So:

    EVENT
      ↓
    ALERT
      ↓
    NOTIFICATION
    

    One event does not necessarily mean an alert.


    2. Example

    Suppose a health check returns HTTP 500.

    The event is:

    HEALTH_FAILED
    

    The alert policy may decide:

    one failure:
    don't notify
    
    three consecutive failures:
    create alert
    

    Then:

    ALERT_CREATED
    

    and finally:

    EMAIL_SENT
    

    3. Why This Separation Matters

    Without it, every event would trigger a notification.

    Imagine:

    HEALTH_FAILED
    HEALTH_FAILED
    HEALTH_FAILED
    HEALTH_FAILED
    HEALTH_FAILED
    

    You don’t want:

    5 emails
    5 WhatsApp messages
    5 webhooks
    

    You want one meaningful incident.


    4. Alert Table

    Create:

    CREATE TABLE alerts (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        alert_code TEXT NOT NULL,
        site_id INTEGER,
    
        severity TEXT NOT NULL,
        status TEXT NOT NULL,
    
        title TEXT NOT NULL,
        message TEXT,
    
        first_event_id INTEGER,
        latest_event_id INTEGER,
    
        occurrence_count INTEGER NOT NULL DEFAULT 1,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
        acknowledged_at TEXT,
        resolved_at TEXT,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE SET NULL,
    
        FOREIGN KEY (first_event_id)
            REFERENCES events(id),
    
        FOREIGN KEY (latest_event_id)
            REFERENCES events(id)
    );
    

    5. Alert States

    Use:

    OPEN
    ACKNOWLEDGED
    RESOLVED
    SUPPRESSED
    

    Flow:

    OPEN
     ↓
    ACKNOWLEDGED
     ↓
    RESOLVED
    

    Or:

    OPEN
     ↓
    RESOLVED
    

    without acknowledgement.


    6. Alert Severity

    Use:

    INFO
    WARNING
    ERROR
    CRITICAL
    

    Example:

    Backup successful:
    INFO
    

    Usually this shouldn’t create an alert.

    Backup failed:
    ERROR
    
    No verified backup:
    CRITICAL
    

    7. Alert Code

    Use predictable identifiers:

    BACKUP_FAILURE
    BACKUP_STALE
    NO_VERIFIED_BACKUP
    
    SITE_DOWN
    HEALTH_DEGRADED
    
    SSL_EXPIRING
    SSL_EXPIRED
    
    DISK_USAGE_HIGH
    DISK_USAGE_CRITICAL
    
    REPAIR_FAILED
    RESTORE_FAILED
    
    SCHEDULER_FAILED
    WORKER_FAILED
    
    SECURITY_EVENT
    

    8. Alert Title

    Example:

    Backup failure
    

    Message:

    The latest scheduled backup for example.com failed.
    

    The title should be short.

    The message contains context.


    9. Alert Fingerprint

    This is one of the most important concepts.

    Suppose:

    example.com
    BACKUP_FAILURE
    

    occurs 10 times.

    All 10 events should map to the same alert:

    fingerprint:
    BACKUP_FAILURE:site:7
    

    10. Why Fingerprints?

    Without fingerprints:

    10 events
     ↓
    10 alerts
    

    With fingerprints:

    10 events
     ↓
    1 alert
    occurrence_count = 10
    

    11. Add Fingerprint

    Add:

    ALTER TABLE alerts
    ADD COLUMN fingerprint TEXT;
    

    Then index it:

    CREATE INDEX idx_alert_fingerprint
    ON alerts(fingerprint);
    

    12. Example Alert

    Alert:
    ALR-000042
    
    Fingerprint:
    BACKUP_FAILURE:site:7
    
    Site:
    example.com
    
    Severity:
    ERROR
    
    Status:
    OPEN
    
    Occurrences:
    4
    
    First seen:
    18:00
    
    Last seen:
    18:45
    

    13. Alert Deduplication

    When an event arrives:

    EVENT
     ↓
    calculate fingerprint
     ↓
    find OPEN alert
    

    If found:

    increment occurrence_count
    update latest_event
    

    Don’t create a new alert.


    14. If No Existing Alert

    Create:

    new alert
    

    Then:

    send notification
    

    depending on policy.


    15. Alert Resolution

    Suppose:

    BACKUP_FAILURE
    

    creates:

    BACKUP_FAILURE alert
    

    Next day:

    backup succeeds
    

    The success event can resolve:

    BACKUP_FAILURE
    

    Result:

    RESOLVED
    

    16. Resolution Events

    Create explicit events:

    ALERT_CREATED
    ALERT_UPDATED
    ALERT_ACKNOWLEDGED
    ALERT_RESOLVED
    ALERT_SUPPRESSED
    

    These become part of the event history.


    17. Alert Policy Table

    Create:

    CREATE TABLE alert_policies (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        alert_code TEXT NOT NULL,
        enabled INTEGER NOT NULL DEFAULT 1,
    
        severity TEXT NOT NULL,
    
        threshold_count INTEGER DEFAULT 1,
        threshold_window_minutes INTEGER DEFAULT 0,
    
        notification_enabled INTEGER NOT NULL DEFAULT 1,
    
        cooldown_minutes INTEGER DEFAULT 60,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL
    );
    

    18. Example Policy

    For site downtime:

    alert:
    SITE_DOWN
    
    threshold:
    3 failures
    
    window:
    5 minutes
    

    Meaning:

    3 failed checks
    within 5 minutes
    → create alert
    

    19. Backup Policy

    Example:

    BACKUP_FAILURE
    
    threshold:
    2
    
    window:
    6 hours
    

    Meaning:

    one failure:
    monitor
    
    two failures:
    alert
    

    20. No Verified Backup

    This is different.

    If:

    last verified backup
    older than 24h
    

    create:

    NO_VERIFIED_BACKUP
    CRITICAL
    

    This should be a state-based alert rather than requiring repeated failure events.


    21. Alert Policy Types

    There are two major types.

    Event-based

    BACKUP_FAILED
    

    State-based

    backup age > SLA
    

    The alert engine should eventually support both.


    22. Notification Channels

    Start with:

    EMAIL
    DASHBOARD
    WEBHOOK
    

    Later:

    SMS
    PUSH
    

    depending on infrastructure.


    23. Notification Table

    Create:

    CREATE TABLE notifications (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        alert_id INTEGER NOT NULL,
    
        channel TEXT NOT NULL,
        recipient TEXT,
    
        status TEXT NOT NULL,
    
        attempts INTEGER NOT NULL DEFAULT 0,
    
        created_at TEXT NOT NULL,
        sent_at TEXT,
        error_message TEXT,
    
        FOREIGN KEY (alert_id)
            REFERENCES alerts(id)
            ON DELETE CASCADE
    );
    

    24. Notification States

    QUEUED
    SENDING
    SENT
    FAILED
    CANCELLED
    

    Again, don’t send directly from the event handler.

    Use the CHP job queue.


    25. Notification Flow

    EVENT
     ↓
    ALERT POLICY
     ↓
    ALERT CREATED
     ↓
    NOTIFICATION JOB
     ↓
    JOB QUEUE
     ↓
    WORKER
     ↓
    EMAIL / WEBHOOK
    

    This integrates the systems we built in Lessons 089–090.


    26. Why Use the Job Queue?

    Suppose an SMTP server is unavailable.

    Without a queue:

    backup process
     ↓
    send email
     ↓
    SMTP timeout
     ↓
    backup operation delayed
    

    Bad.

    With the queue:

    backup
     ↓
    event
     ↓
    alert
     ↓
    notification job
    

    The backup remains independent of email delivery.


    27. Notification Job

    Use:

    NOTIFICATION
    

    as another job type.

    Eventually:

    BACKUP
    RESTORE
    REPAIR
    HEALTH
    RETENTION
    NOTIFICATION
    

    all use the same worker architecture.


    28. Notification Retry

    Email delivery can safely retry.

    Example:

    attempt 1:
    SMTP timeout
    
    attempt 2:
    SMTP timeout
    
    attempt 3:
    SUCCESS
    

    Notification itself is generally retryable.


    29. Exponential Backoff

    Instead of:

    retry
    retry
    retry
    retry
    

    use:

    1 minute
    5 minutes
    15 minutes
    30 minutes
    

    This reduces pressure on failing services.


    30. Notification Deduplication

    Suppose:

    BACKUP_FAILURE
    

    remains open for 10 hours.

    Don’t send:

    one email every 15 minutes
    

    unless the alert policy explicitly requires periodic reminders.


    31. Cooldown

    Use:

    cooldown_minutes = 60
    

    Meaning:

    after notification
    wait 60 minutes
    before another notification
    

    32. Escalation

    A mature alert engine can escalate.

    Example:

    0 min:
    dashboard
    
    15 min:
    email
    
    60 min:
    second email
    
    4 hours:
    critical escalation
    

    But don’t implement all of this immediately.

    Start with:

    create
    notify
    cooldown
    resolve
    

    33. Notification Recipients

    Create:

    CREATE TABLE notification_recipients (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
    
        name TEXT NOT NULL,
        channel TEXT NOT NULL,
        destination TEXT NOT NULL,
    
        enabled INTEGER NOT NULL DEFAULT 1,
    
        created_at TEXT NOT NULL
    );
    

    Examples:

    Administrator
    EMAIL
    admin@example.com
    

    and:

    Monitoring webhook
    WEBHOOK
    internal endpoint
    

    34. Site-Specific Recipients

    Eventually support:

    global recipients
    site recipients
    customer recipients
    

    For example:

    example.com
    → customer@example.com
    

    while platform-level failures go to:

    hosting-admin@example.com
    

    35. Alert Routing

    Add a routing layer:

    ALERT
     ↓
    WHO SHOULD RECEIVE IT?
     ↓
    CHANNEL
     ↓
    NOTIFICATION
    

    Example:

    SITE_DOWN
    

    → customer + platform admin

    But:

    WORKER_FAILURE
    

    → platform admin only.


    36. Customer Visibility

    Alerts should have:

    INTERNAL
    CUSTOMER
    

    Example:

    SITE_DOWN:
    CUSTOMER
    

    while:

    WORKER_CRASH:
    INTERNAL
    

    37. Alert Dashboard

    The CHP dashboard can show:

    ACTIVE ALERTS
    =============
    
    CRITICAL
    1
    
    ERROR
    2
    
    WARNING
    4
    

    Then:

    CRITICAL
    No verified backup
    example.com
    2h ago
    

    38. Alert Detail

    ALR-000042
    ==========
    
    Site:
    example.com
    
    Alert:
    BACKUP_FAILURE
    
    Severity:
    ERROR
    
    Status:
    OPEN
    
    Occurrences:
    4
    
    First Seen:
    18:00
    
    Last Seen:
    18:45
    
    Latest Event:
    EVT-001942
    

    39. Acknowledge

    An administrator can:

    ACKNOWLEDGE
    

    This means:

    I know about this problem.

    It does not mean:

    The problem is fixed.

    Therefore:

    ACKNOWLEDGED
    

    is still an active alert.


    40. Resolve

    Only when the underlying condition disappears:

    RESOLVED
    

    For example:

    backup succeeds
     ↓
    BACKUP_FAILURE resolved
    

    41. Don’t Auto-Resolve on Acknowledgement

    Incorrect:

    admin acknowledges
     ↓
    RESOLVED
    

    Acknowledgement is not resolution.


    42. Alert Reopening

    Suppose:

    BACKUP_FAILURE
    RESOLVED
    

    Then a new failure occurs later.

    Create a new active alert or reopen according to a defined recurrence policy.

    A simple first version:

    resolved alert
    +
    new failure after resolution
    =
    new alert
    

    This preserves incident history.


    43. Notification Storm Protection

    Suppose 100 sites all use the same database server.

    Database fails.

    You might get:

    100 SITE_DOWN alerts
    

    That’s legitimate information, but 100 separate emails may be excessive.

    Future aggregation can produce:

    INFRASTRUCTURE INCIDENT
    
    Affected sites:
    100
    

    This is a later feature.


    44. First Version

    For CHP V1:

    per-site alerts
    +
    fingerprint deduplication
    +
    cooldown
    

    is enough.


    45. Disk Alerts

    The event system can produce:

    DISK_USAGE_HIGH
    

    Example policy:

    WARNING:
    80%
    
    ERROR:
    90%
    
    CRITICAL:
    95%
    

    These thresholds should be configuration values.


    46. SSL Alerts

    Example:

    SSL_EXPIRING
    

    Policies:

    30 days:
    INFO
    
    14 days:
    WARNING
    
    7 days:
    ERROR
    
    1 day:
    CRITICAL
    

    47. Site Availability Alerts

    Example:

    SITE_DOWN
    

    Don’t alert after one failed HTTP request.

    Use:

    3 consecutive failures
    

    to avoid transient false alarms.


    48. Health Recovery

    If:

    SITE_DOWN
    

    is open and the site returns:

    HEALTHY
    

    resolve the alert.

    Create:

    SITE_RECOVERED
    

    event.


    49. Alert Lifecycle

    The full lifecycle is:

    EVENT
     ↓
    POLICY MATCH
     ↓
    ALERT CREATED
     ↓
    NOTIFICATION
     ↓
    ACKNOWLEDGED
     ↓
    CONDITION RECOVERS
     ↓
    RESOLVED
    

    50. Alert Engine Command

    Create:

    sudo nano /usr/local/bin/hosting-alerts
    

    Commands:

    hosting-alerts list
    hosting-alerts show ALR-000042
    hosting-alerts acknowledge ALR-000042
    hosting-alerts resolve ALR-000042
    hosting-alerts test
    

    51. List Alerts

    CresignSys Alerts
    =================
    
    ID          SITE          SEVERITY   STATUS
    ------------------------------------------------
    ALR-042     example.com   CRITICAL   OPEN
    ALR-043     shop.com      ERROR      ACKNOWLEDGED
    ALR-044     blog.com      WARNING    OPEN
    

    52. Alert Test

    Create:

    hosting-alerts test BACKUP_FAILURE example.com
    

    It should create a test event and verify:

    event
     ↓
    alert
     ↓
    notification
    

    without affecting the actual site.


    53. Notification Test

    Also:

    hosting-notify-test EMAIL admin@example.com
    

    This should verify notification configuration independently.


    54. Alert Engine Worker

    You can use the same central worker.

    The workflow becomes:

    event
     ↓
    alert processor
     ↓
    notification job
     ↓
    worker
    

    No separate daemon is strictly necessary initially.


    55. Alert Processor

    The processor can run every minute:

    find new events
     ↓
    evaluate policies
     ↓
    create/update alerts
     ↓
    queue notifications
    

    Later this can become event-driven.


    56. Avoid Processing the Same Event Twice

    Add:

    processed_at
    

    to the event-processing mechanism, or maintain a separate processing table.

    A simple approach is:

    CREATE TABLE event_processing (
        event_id INTEGER NOT NULL,
        processor TEXT NOT NULL,
        processed_at TEXT NOT NULL,
    
        PRIMARY KEY (event_id, processor),
    
        FOREIGN KEY (event_id)
            REFERENCES events(id)
            ON DELETE CASCADE
    );
    

    57. Why Processor Records?

    Suppose there are processors:

    alert_processor
    metrics_processor
    incident_processor
    

    Each may process the same event independently.

    This design supports multiple consumers.


    58. Event → Alert Example

    Event:

    BACKUP_FAILED
    example.com
    

    Policy:

    2 failures
    within 6 hours
    

    First event:

    count = 1
    no alert
    

    Second event:

    count = 2
    

    →

    ALR-00042
    OPEN
    

    →

    notification job
    

    59. Example Recovery

    Next backup:

    BACKUP_VERIFIED
    

    Alert engine sees:

    latest backup healthy
    

    Then:

    ALR-00042
    RESOLVED
    

    and emits:

    ALERT_RESOLVED
    

    60. Critical Safety Rule

    Never let alert processing modify the underlying site automatically unless a specific, separately authorized remediation system exists.

    For example:

    SSL_EXPIRING
    

    should create:

    ALERT
    

    not automatically:

    change SSL configuration
    

    The alert system is for detection and notification.

    Remediation remains under the control/approval architecture.


    61. Alert Engine Architecture

    The complete system becomes:

                        EVENT
                          │
                          ▼
                    EVENT PROCESSOR
                          │
                          ▼
                     ALERT POLICY
                          │
                 ┌────────┴────────┐
                 ▼                 ▼
              NO MATCH            MATCH
                                   │
                                   ▼
                               FINGERPRINT
                                   │
                         ┌─────────┴─────────┐
                         ▼                   ▼
                    EXISTING ALERT       NEW ALERT
                         │                   │
                         └─────────┬─────────┘
                                   ▼
                              NOTIFICATION
                                   │
                                   ▼
                                JOB QUEUE
                                   │
                                   ▼
                                 WORKER
    

    62. CHP Architecture After Lesson 091

                             CRESIGNSYS CHP
                                    │
           ┌────────────────────────┼────────────────────────┐
           ▼                        ▼                        ▼
       OBSERVATION               CONTROL                 RECOVERY
           │                        │                        │
           ▼                        ▼                        ▼
     HEALTH/DRIFT             PLAN/APPROVAL            BACKUP/RESTORE
           │                        │                        │
           └──────────────┬─────────┴──────────┬─────────────┘
                          ▼                    ▼
                       EVENTS               JOB QUEUE
                          │                    │
                          ▼                    ▼
                        ALERTS              WORKERS
                          │
                          ▼
                    NOTIFICATIONS
    

    63. Lesson 091 — Core Principle

    The important separation is:

    EVENT
    =
    something happened
    
    ALERT
    =
    something requires attention
    
    NOTIFICATION
    =
    someone was informed
    

    And:

    one event
    ≠
    one alert
    ≠
    one notification
    

    This separation prevents notification storms and gives CHP a proper monitoring architecture.


    Next Lesson — 092

    Build the CHP Site Health Monitoring Engine

    The next layer will continuously determine whether each website is actually healthy.

    It will combine:

    HTTP
    HTTPS
    DNS
    SSL
    Nginx
    PHP-FPM
    MySQL
    disk
    CPU
    RAM
    WordPress
    

    into a unified:

    SITE HEALTH SCORE
    

    with states:

    HEALTHY
    DEGRADED
    WARNING
    DOWN
    UNKNOWN
    

    The key design will be:

    RAW CHECKS
        ↓
    NORMALIZATION
        ↓
    HEALTH STATE
        ↓
    EVENT
        ↓
    ALERT
    

    so CHP can distinguish between a single failed check, a genuinely unhealthy website, and a temporary/transient problem.

  • CresignSys Learn — Lesson 090

    Build the CHP Event & Audit System

    CHP now has:

    • Site discovery
    • Site management
    • Health monitoring
    • Reconciliation
    • Repair plans
    • Approval
    • Backup/restore
    • Job queue
    • Workers

    But there is one major missing capability:

    A single trustworthy history of what CHP did, when it happened, to which site, and why.

    That is the purpose of the Event & Audit System.


    1. Event vs Audit

    These are related but not identical.

    Event

    Describes something that happened:

    BACKUP_COMPLETED
    HEALTH_CHECK_FAILED
    DRIFT_DETECTED
    

    Audit

    Answers:

    WHO performed it?
    WHAT changed?
    WHICH site?
    WHICH object?
    WHEN?
    WHAT was the result?
    

    So:

    EVENT
      ↓
    What happened?
    
    AUDIT
      ↓
    Who caused/authorized it?
    

    2. Central Event Architecture

                     CHP COMPONENTS
                           │
           ┌───────────────┼────────────────┐
           ▼               ▼                ▼
        BACKUP           REPAIR          RESTORE
           │               │                │
           └───────────────┼────────────────┘
                           ▼
                     EVENT ENGINE
                           │
                 ┌─────────┼─────────┐
                 ▼         ▼         ▼
               AUDIT    TIMELINE   ALERTS
    

    Every major subsystem publishes events to the same event store.


    3. Create Event Table

    Create:

    CREATE TABLE events (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        event_code TEXT NOT NULL,
        event_category TEXT NOT NULL,
        severity TEXT NOT NULL DEFAULT 'INFO',
    
        site_id INTEGER,
        actor_id INTEGER,
        job_id INTEGER,
        plan_id INTEGER,
        backup_id INTEGER,
    
        message TEXT,
        metadata TEXT,
    
        created_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE SET NULL,
    
        FOREIGN KEY (job_id)
            REFERENCES jobs(id)
            ON DELETE SET NULL
    );
    

    4. Event Code

    Don’t rely only on free-form text.

    Use a structured:

    event_code
    

    Examples:

    SITE_DISCOVERED
    SITE_IMPORTED
    SITE_ENABLED
    SITE_DISABLED
    
    HEALTH_STARTED
    HEALTH_PASSED
    HEALTH_FAILED
    
    DRIFT_DETECTED
    PLAN_CREATED
    PLAN_APPROVED
    PLAN_REJECTED
    
    JOB_CREATED
    JOB_STARTED
    JOB_COMPLETED
    JOB_FAILED
    
    BACKUP_STARTED
    BACKUP_VERIFIED
    BACKUP_FAILED
    
    RESTORE_STARTED
    RESTORE_COMPLETED
    RESTORE_FAILED
    
    REPAIR_STARTED
    REPAIR_COMPLETED
    REPAIR_ROLLED_BACK
    

    5. Event Categories

    Use:

    SYSTEM
    SITE
    HEALTH
    RECONCILIATION
    REPAIR
    BACKUP
    RESTORE
    JOB
    SECURITY
    AUTHORIZATION
    USER
    

    This makes filtering easier.


    6. Severity

    Start with:

    DEBUG
    INFO
    WARNING
    ERROR
    CRITICAL
    

    Example:

    BACKUP_VERIFIED:
    INFO
    
    BACKUP_FAILED:
    ERROR
    
    NO_VERIFIED_BACKUP:
    CRITICAL
    

    7. Event Example

    Event:
    DRIFT_DETECTED
    
    Category:
    RECONCILIATION
    
    Severity:
    WARNING
    
    Site:
    example.com
    
    Message:
    PHP version differs from desired configuration.
    
    Time:
    2026-08-13 18:20
    

    8. Metadata

    Some events require additional structured information.

    Example:

    {
      "current_php": "8.2",
      "expected_php": "8.3",
      "drift_type": "PHP_VERSION"
    }
    

    Store it in:

    metadata
    

    as JSON.


    9. Don’t Put Everything in message

    Bad:

    "PHP changed from 8.2 to 8.3 on example.com by admin at 18:22."
    

    Better:

    event_code:
    PHP_CONFIGURATION_CHANGED
    
    site_id:
    7
    
    actor_id:
    4
    
    metadata:
    {
      "old": "8.2",
      "new": "8.3"
    }
    

    Then the UI can format the event however it wants.


    10. Why Structured Events Matter

    Later the dashboard can answer:

    How many PHP changes occurred this month?
    

    or:

    Show all failed backups.
    

    or:

    Show everything admin did to example.com.
    

    Structured data makes this possible.


    11. Actor

    Every event that involves a person should record:

    actor_id
    

    For example:

    actor:
    admin
    
    event:
    PLAN_APPROVED
    

    12. System Events

    Not every event has a human actor.

    For example:

    BACKUP_STARTED
    

    may be initiated by:

    scheduler
    

    Use a system actor such as:

    SYSTEM
    

    or:

    SCHEDULER
    

    13. Actor Types

    Eventually:

    USER
    SYSTEM
    SCHEDULER
    WORKER
    API
    

    This makes audit history clearer.


    14. Example Timeline

    For one repair:

    18:20  DRIFT_DETECTED
    18:21  PLAN_CREATED
    18:22  PLAN_VALIDATED
    18:25  PLAN_APPROVED
    18:25  JOB_CREATED
    18:26  BACKUP_STARTED
    18:29  BACKUP_VERIFIED
    18:30  REPAIR_STARTED
    18:31  REPAIR_COMPLETED
    18:31  HEALTH_PASSED
    18:32  RECONCILIATION_PASSED
    

    This is a complete operational story.


    15. Site Timeline

    The dashboard should eventually display:

    example.com
    ────────────────────────────
    
    18:32
    Reconciliation passed
    
    18:31
    Health check passed
    
    18:31
    Repair completed
    
    18:29
    Backup verified
    
    18:25
    Repair approved
    
    18:21
    Repair plan created
    
    18:20
    PHP drift detected
    

    16. Audit Trail

    For security-sensitive actions, create more detailed audit records.

    You already have:

    audit_events
    

    from the authorization system.

    Keep that concept separate from general events.


    17. Event vs Audit Storage

    Use:

    events
    

    for:

    system timeline
    operations
    health
    backup
    jobs
    

    Use:

    audit_events
    

    for:

    security-sensitive user actions
    authorization
    configuration changes
    approvals
    restores
    permission changes
    

    18. Why Separate Them?

    An event might say:

    HEALTH_CHECK_PASSED
    

    An audit record might say:

    admin changed PHP version
    

    The second is security-sensitive and should receive stronger retention and access controls.


    19. Audit Immutability

    Audit records should ideally be append-only.

    Don’t allow:

    UPDATE audit_events
    

    to rewrite history.

    Instead:

    new event
    

    should record a correction if necessary.


    20. Never Delete Audit History Through Normal UI

    A normal administrator should not have:

    Delete audit log
    

    available.

    Retention and archival should be separate administrative processes.


    21. Event Library

    Create:

    sudo nano /etc/cresignsys/lib/events.sh
    

    Functions:

    emit_event
    emit_audit
    emit_security_event
    

    22. emit_event

    Conceptually:

    emit_event \
        "BACKUP_VERIFIED" \
        "BACKUP" \
        "INFO" \
        "$SITE_ID" \
        "$MESSAGE" \
        "$METADATA"
    

    The function handles:

    timestamp
    database insert
    JSON validation
    logging
    

    23. Centralize Event Creation

    Don’t write SQL manually in every script.

    Bad:

    hosting-backup:
    INSERT INTO events...
    
    hosting-repair:
    INSERT INTO events...
    
    hosting-restore:
    INSERT INTO events...
    

    Instead:

    event library
         ↑
    all components
    

    24. Event IDs

    Use:

    event ID
    

    as the immutable identifier.

    Example:

    EVT-000001
    

    This is useful when referring to an event in support or debugging.


    25. Add Event Code

    Example:

    EVT-000241
    
    DRIFT_DETECTED
    
    example.com
    
    WARNING
    

    26. Event Correlation

    A single operation can create many events.

    For example:

    Repair:
    RP-000021
    

    creates:

    PLAN_CREATED
    PLAN_APPROVED
    JOB_CREATED
    BACKUP_STARTED
    BACKUP_VERIFIED
    REPAIR_STARTED
    REPAIR_COMPLETED
    

    We need to connect them.


    27. Correlation ID

    Add:

    ALTER TABLE events
    ADD COLUMN correlation_id TEXT;
    

    Example:

    CORR-20260813-00042
    

    All events from the same operation use the same correlation ID.


    28. Why Correlation IDs?

    Then you can ask:

    Show everything that happened during repair RP-000021.
    

    Query:

    correlation_id = ?
    

    and receive the entire chain.


    29. Example

    CORR-00042
    
    18:25 PLAN_APPROVED
    18:25 JOB_CREATED
    18:26 BACKUP_STARTED
    18:29 BACKUP_VERIFIED
    18:30 REPAIR_STARTED
    18:31 REPAIR_COMPLETED
    18:31 HEALTH_PASSED
    18:32 RECONCILIATION_PASSED
    

    This is extremely useful for debugging.


    30. Parent Event

    Eventually events can also reference:

    parent_event_id
    

    For example:

    REPAIR_STARTED
    

    can be associated with:

    JOB_STARTED
    

    But correlation IDs are enough for the first implementation.


    31. Event Query Command

    Create:

    sudo nano /usr/local/bin/hosting-events
    

    Usage:

    hosting-events example.com
    

    32. Example Output

    CresignSys Events
    =================
    
    Site:
    example.com
    
    TIME                 EVENT                 SEVERITY
    -------------------------------------------------------
    18:32                 RECONCILIATION_PASS  INFO
    18:31                 HEALTH_PASSED        INFO
    18:31                 REPAIR_COMPLETED     INFO
    18:29                 BACKUP_VERIFIED      INFO
    18:25                 PLAN_APPROVED        INFO
    18:21                 PLAN_CREATED         INFO
    18:20                 DRIFT_DETECTED       WARNING
    

    33. Filter by Severity

    hosting-events example.com --severity ERROR
    

    Output:

    18:10
    BACKUP_FAILED
    
    18:15
    HEALTH_FAILED
    

    34. Filter by Event

    hosting-events example.com --type BACKUP_FAILED
    

    35. Filter by Time

    hosting-events example.com --since "2026-08-01"
    

    This becomes useful for monthly reports.


    36. Event Categories

    Example:

    hosting-events example.com --category SECURITY
    

    returns:

    LOGIN
    LOGIN_FAILED
    PERMISSION_DENIED
    PLAN_APPROVED
    RESTORE_APPROVED
    

    37. Security Events

    Create standard security events:

    LOGIN_SUCCESS
    LOGIN_FAILED
    SESSION_EXPIRED
    PERMISSION_DENIED
    UNAUTHORIZED_APPROVAL
    UNAUTHORIZED_RESTORE
    USER_CREATED
    USER_DISABLED
    ROLE_CHANGED
    

    38. Why Log Failed Authorization?

    Suppose someone repeatedly tries:

    RESTORE client.com
    

    without permission.

    You want:

    PERMISSION_DENIED
    

    in the security audit trail.

    This may indicate:

    misconfiguration
    or
    compromised credentials
    

    39. Don’t Log Secrets

    Never write:

    password
    API key
    private key
    session token
    database password
    

    into events.

    Even failed authentication events should contain safe information only.


    40. Sensitive Metadata

    Avoid:

    {
      "password": "..."
    }
    

    Instead:

    {
      "username": "admin",
      "result": "FAILED"
    }
    

    41. Event Retention

    General events can have:

    90 days
    

    Audit events may require:

    1 year
    

    or longer depending on your operational and contractual requirements.

    These should be configurable.


    42. Don’t Delete Events Needed for Active Incidents

    If an event is associated with:

    open incident
    

    it should be protected from ordinary cleanup.


    43. Event Archive

    Eventually:

    ACTIVE EVENTS
          ↓
    ARCHIVE
          ↓
    COMPRESSED STORAGE
    

    This keeps the operational database small.


    44. Event Volume

    Health monitoring may create many events.

    For example:

    HEALTH_PASSED
    

    every 5 minutes creates:

    288 events/day/site
    

    For 100 sites:

    28,800 events/day
    

    This is excessive if every successful check is stored forever.


    45. Event Deduplication

    For repetitive healthy states, use state-transition events.

    Instead of logging:

    HEALTH_PASSED
    HEALTH_PASSED
    HEALTH_PASSED
    HEALTH_PASSED
    

    every time, log:

    HEALTH_STATUS_CHANGED
    
    UNHEALTHY → HEALTHY
    

    and store periodic metrics separately.


    46. Events vs Metrics

    This distinction becomes important.

    Event

    Something happened:

    HEALTH_FAILED
    

    Metric

    A measurement:

    response_time_ms = 183
    

    Don’t put high-frequency metrics into the event table.


    47. Future Metrics System

    Eventually CHP can have:

    metrics
    

    for:

    CPU
    RAM
    disk
    response time
    backup duration
    database size
    traffic
    

    while events contain significant state changes.


    48. Event Severity Rules

    Example:

    INFO:
    successful operations
    
    WARNING:
    drift
    backup age approaching SLA
    
    ERROR:
    backup failure
    health failure
    
    CRITICAL:
    no recovery point
    restore failure + rollback failure
    security compromise indicators
    

    49. Alerts Use Events

    The alert system should eventually subscribe to events.

    Example:

    BACKUP_FAILED
           ↓
    alert policy
           ↓
    send notification
    

    Another:

    NO_VERIFIED_BACKUP
           ↓
    CRITICAL
           ↓
    immediate alert
    

    50. Don’t Alert Directly From Every Script

    Bad:

    hosting-backup
      └── send email
    
    hosting-repair
      └── send email
    
    hosting-restore
      └── send email
    

    This duplicates notification logic.

    Better:

    operation
     ↓
    event
     ↓
    alert engine
     ↓
    notification
    

    51. Event-Driven Alert Architecture

                      EVENT
                        │
                        ▼
                   EVENT STORE
                        │
                        ▼
                  ALERT POLICY
                        │
                 ┌──────┴──────┐
                 ▼             ▼
              NO ALERT        ALERT
                               │
                      ┌────────┼────────┐
                      ▼        ▼        ▼
                    EMAIL    PANEL    WEBHOOK
    

    52. Example Alert Policy

    Event:
    BACKUP_FAILED
    
    Condition:
    2 failures within 6 hours
    
    Action:
    WARNING
    

    Another:

    Event:
    NO_VERIFIED_BACKUP
    
    Condition:
    > 24 hours
    
    Action:
    CRITICAL
    

    53. Event Correlation for Incidents

    Suppose:

    18:20
    HEALTH_FAILED
    
    18:21
    PHP_DRIFT
    
    18:25
    REPAIR_STARTED
    
    18:31
    REPAIR_FAILED
    
    18:32
    ROLLBACK_COMPLETED
    

    CHP can later automatically group these into:

    INCIDENT
    INC-00042
    

    54. Incident System

    This will eventually become:

    EVENTS
      ↓
    CORRELATION
      ↓
    INCIDENT
      ↓
    RESPONSE
      ↓
    RESOLUTION
    

    We won’t build incidents yet.

    The event architecture should simply make them possible.


    55. Event Integrity

    Because audit history is important, consider adding:

    previous_event_hash
    event_hash
    

    This can create a tamper-evident chain.

    For example:

    Event 1
      ↓ hash
    Event 2
      ↓ hash
    Event 3
    

    If Event 2 is modified:

    chain validation fails
    

    56. Do We Need This Immediately?

    Not necessarily.

    For the first CHP version:

    append-only audit records
    +
    restricted permissions
    +
    database backups
    

    may be sufficient.

    Hash chaining can be added later.


    57. Event Database Indexes

    Because the event table will grow, add indexes.

    For example:

    CREATE INDEX idx_events_site_time
    ON events(site_id, created_at);
    

    And:

    CREATE INDEX idx_events_type
    ON events(event_code);
    

    And:

    CREATE INDEX idx_events_correlation
    ON events(correlation_id);
    

    58. Audit Indexes

    For audit events:

    CREATE INDEX idx_audit_site_time
    ON audit_events(site_id, created_at);
    

    and:

    CREATE INDEX idx_audit_actor
    ON audit_events(actor_id);
    

    59. Event Query Performance

    The dashboard will frequently ask:

    latest 50 events for site
    

    So:

    ORDER BY created_at DESC
    LIMIT 50
    

    should be indexed appropriately.


    60. Event API

    Eventually:

    GET /api/sites/example.com/events
    

    with:

    ?severity=ERROR
    ?category=BACKUP
    ?limit=50
    

    The API should query structured events rather than parse log files.


    61. Log Files Still Matter

    Do not eliminate system logs.

    Use:

    EVENTS:
    structured application history
    
    LOGS:
    diagnostic detail
    

    For example:

    Event:
    BACKUP_FAILED
    

    and the log contains:

    mysqldump returned exit code 2
    database connection timed out
    

    62. Event + Log Correlation

    Record:

    correlation_id
    

    in both the event and detailed log.

    Then troubleshooting becomes:

    Event
     ↓
    Correlation ID
     ↓
    Detailed logs
    

    63. Example Complete Incident

    CORR-00077
    
    18:00
    BACKUP_STARTED
    
    18:04
    BACKUP_FAILED
    
    18:05
    HEALTH_FAILED
    
    18:06
    DRIFT_DETECTED
    
    18:08
    PLAN_CREATED
    
    18:10
    PLAN_APPROVED
    
    18:11
    REPAIR_STARTED
    
    18:12
    REPAIR_FAILED
    
    18:12
    ROLLBACK_STARTED
    
    18:14
    ROLLBACK_COMPLETED
    
    18:15
    HEALTH_PASSED
    

    This is exactly the kind of information an administrator needs.


    64. Customer-Facing Timeline

    Not every internal event should be shown to customers.

    For example:

    Customer may see:

    Backup completed
    Website health verified
    Maintenance completed
    

    But not necessarily:

    internal worker IDs
    database paths
    security metadata
    

    65. Internal vs Customer Events

    Add visibility:

    ALTER TABLE events
    ADD COLUMN visibility TEXT NOT NULL DEFAULT 'INTERNAL';
    

    Values:

    INTERNAL
    CUSTOMER
    

    66. Example

    BACKUP_VERIFIED
    visibility:
    CUSTOMER
    

    But:

    WORKER_HEARTBEAT_TIMEOUT
    visibility:
    INTERNAL
    

    This prevents accidental information leakage.


    67. Admin Dashboard

    The administrator can see:

    all events
    

    Customer:

    customer-safe events only
    

    68. Event Permissions

    Eventually:

    VIEW_SYSTEM_EVENTS
    VIEW_SECURITY_EVENTS
    VIEW_SITE_EVENTS
    VIEW_CUSTOMER_EVENTS
    

    Sensitive security events should be restricted.


    69. Audit Access Itself Should Be Audited

    If:

    admin views security audit logs
    

    you may record:

    AUDIT_LOG_ACCESSED
    

    This is especially useful for sensitive environments.


    70. Never Allow Audit Tampering Through UI

    The dashboard should not expose:

    Edit event
    Delete event
    

    for normal administrators.

    The UI is for:

    view
    filter
    search
    export
    

    71. Export

    Eventually:

    hosting-events example.com --export csv
    

    or:

    hosting-events example.com --export json
    

    This is useful for:

    support
    compliance
    incident analysis
    customer reports
    

    72. Export Security

    Exports may contain sensitive operational information.

    Therefore:

    event export
    

    should itself be:

    audited
    

    and protected by permissions.


    73. Event Naming Convention

    Use predictable names:

    <OBJECT>_<ACTION>[_RESULT]
    

    Examples:

    SITE_IMPORTED
    SITE_DISABLED
    
    PLAN_CREATED
    PLAN_APPROVED
    PLAN_REJECTED
    
    JOB_CREATED
    JOB_STARTED
    JOB_COMPLETED
    JOB_FAILED
    
    BACKUP_STARTED
    BACKUP_VERIFIED
    BACKUP_FAILED
    
    RESTORE_STARTED
    RESTORE_COMPLETED
    RESTORE_FAILED
    

    Avoid inconsistent names such as:

    backupDone
    BackupFinished
    backup_ok
    

    74. Event Result

    For some events, metadata can contain:

    {
      "result": "SUCCESS"
    }
    

    But don’t encode the result twice if the event code already clearly indicates it.

    Keep event naming predictable.


    75. Central Event Flow

    Every subsystem should use:

    ACTION
      ↓
    EVENT
      ↓
    EVENT STORE
      ↓
    AUDIT / TIMELINE / ALERT
    

    not:

    ACTION
     ├── email
     ├── log
     ├── dashboard
     └── database
    

    Centralization reduces complexity.


    76. CHP Event Engine

    The resulting architecture is:

                     EVENT ENGINE
                          │
           ┌──────────────┼──────────────┐
           ▼              ▼              ▼
        STORAGE         ALERTS        TIMELINE
           │              │              │
           ▼              ▼              ▼
        EVENTS          POLICIES       DASHBOARD
           │
           ▼
         AUDIT
    

    77. Lesson 090 — Core Principle

    The CHP event system should make it possible to answer five questions:

    WHAT happened?
    WHEN did it happen?
    TO WHICH site?
    WHO or WHAT caused it?
    WHAT was the result?
    

    For complex operations, add:

    WHY?
    WHICH PLAN?
    WHICH JOB?
    WHICH BACKUP?
    WHICH CORRELATION?
    

    The final operational chain is now:

    OBSERVE
       ↓
    DETECT
       ↓
    PLAN
       ↓
    AUTHORIZE
       ↓
    APPROVE
       ↓
    QUEUE
       ↓
    BACKUP
       ↓
    EXECUTE
       ↓
    VERIFY
       ↓
    EVENT
       ↓
    AUDIT
    

    CHP is now becoming a real hosting control plane, rather than simply a collection of hosting shell scripts.


    Next Lesson — 091

    Build the CHP Alert & Notification Engine

    The event system can tell CHP:

    something happened
    

    but the administrator still needs to know:

    Does this require attention?
    How urgent is it?
    Who should be notified?
    How many times?
    Through which channel?
    

    Lesson 091 will build:

    EVENT
      ↓
    ALERT POLICY
      ↓
    ALERT
      ↓
    DEDUPLICATION
      ↓
    NOTIFICATION
    

    covering:

    backup failures
    no verified backup
    SSL expiry
    disk usage
    site downtime
    repair failures
    restore failures
    security events
    scheduler failures
    

    with:

    INFO
    WARNING
    ERROR
    CRITICAL
    

    and notification channels such as:

    email
    dashboard
    webhook
    

    while preventing notification storms when the same problem occurs repeatedly.

  • CresignSys Learn — Lesson 089

    Build the CHP Central Job Queue

    CHP now has several operations:

    BACKUP
    RESTORE
    REPAIR
    HEALTH CHECK
    RECONCILIATION
    RETENTION
    

    If every command executes independently, conflicts will eventually occur.

    For example:

    BACKUP example.com
            +
    REPAIR example.com
    

    could run simultaneously.

    That is exactly what we want to prevent.

    The next layer is therefore:

                        CHP JOB ENGINE
                             │
                 ┌───────────┼───────────┐
                 ▼           ▼           ▼
              BACKUP       REPAIR      RESTORE
                 │           │           │
                 └───────────┼───────────┘
                             ▼
                           QUEUE
                             │
                             ▼
                          WORKER
                             │
                 ┌───────────┴───────────┐
                 ▼                       ▼
              SUCCESS                   FAILURE
    

    1. What the Job Queue Does

    Instead of every command immediately performing work:

    hosting-backup
    hosting-repair
    hosting-restore
    

    they submit a job.

    Example:

    hosting-backup example.com
            ↓
    CREATE JOB
            ↓
    QUEUED
            ↓
    WORKER
            ↓
    RUNNING
            ↓
    SUCCESS
    

    2. Job States

    Create a standard state machine:

    QUEUED
       ↓
    RUNNING
       ↓
    SUCCESS
    

    Failure:

    RUNNING
       ↓
    FAILED
    

    Cancellation:

    QUEUED
       ↓
    CANCELLED
    

    3. Job Types

    Use:

    BACKUP
    RESTORE
    REPAIR
    HEALTH
    RECONCILE
    RETENTION
    

    Later:

    SSL_RENEW
    DNS_CHECK
    WORDPRESS_UPDATE
    PACKAGE_UPDATE
    

    can be added without redesigning the queue.


    4. Create the Job Table

    Create:

    CREATE TABLE jobs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        job_code TEXT NOT NULL UNIQUE,
        site_id INTEGER,
        job_type TEXT NOT NULL,
        priority TEXT NOT NULL DEFAULT 'NORMAL',
        status TEXT NOT NULL,
        payload TEXT,
        created_at TEXT NOT NULL,
        scheduled_at TEXT,
        started_at TEXT,
        completed_at TEXT,
        worker_id TEXT,
        attempt INTEGER NOT NULL DEFAULT 0,
        error_message TEXT,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE SET NULL
    );
    

    5. Why payload?

    The job needs parameters.

    For example:

    {
      "domain": "example.com"
    }
    

    A restore job might contain:

    {
      "domain": "example.com",
      "backup_id": "BK-20260813-170001"
    }
    

    A repair job:

    {
      "plan_id": "RP-000021"
    }
    

    6. Never Put Arbitrary Commands in Payload

    Bad:

    {
      "command": "rm -rf /storage/websites/example.com"
    }
    

    Good:

    {
      "job_type": "BACKUP",
      "site_id": 7
    }
    

    The worker interprets structured data.


    7. Job Code

    Every job gets a unique code:

    JOB-000001
    JOB-000002
    JOB-000003
    

    This is easier for operators than:

    database row 281
    

    8. Job Example

    JOB-000241
    
    Type:
    BACKUP
    
    Site:
    example.com
    
    Priority:
    NORMAL
    
    Status:
    QUEUED
    
    Created:
    2026-08-13 18:00
    

    9. Job Priority

    Use:

    CRITICAL
    HIGH
    NORMAL
    LOW
    

    Examples:

    PRE-RESTORE:
    CRITICAL
    
    RESTORE:
    CRITICAL
    
    REPAIR:
    HIGH
    
    HEALTH:
    NORMAL
    
    BACKUP:
    NORMAL
    
    RETENTION:
    LOW
    

    10. Why Priority Matters

    Suppose:

    20 backups queued
    

    and an operator needs an urgent restore.

    The restore shouldn’t wait behind all 20 normal backups.

    Instead:

    RESTORE
    CRITICAL
    

    moves ahead of:

    BACKUP
    NORMAL
    

    11. But Priority Alone Is Not Enough

    Suppose:

    BACKUP example.com
    RUNNING
    

    and:

    RESTORE example.com
    CRITICAL
    

    The restore must not simply start.

    The queue also needs resource and site conflict rules.


    12. Job Conflict Model

    Define operation relationships.

    For example:

    ExistingRequestedResult
    BACKUPBACKUPQueue
    BACKUPREPAIRWait
    BACKUPRESTOREWait
    REPAIRREPAIRWait
    REPAIRRESTOREWait
    RESTOREBACKUPWait
    HEALTHBACKUPUsually allowed
    HEALTHREPAIRUsually allowed
    RECONCILEREPAIRQueue/wait

    The exact policy can evolve.


    13. Site Lock Remains

    The central queue does not replace the site lock.

    You need both:

    JOB QUEUE
       ↓
    site conflict check
       ↓
    SITE LOCK
       ↓
    execution
    

    The queue prevents jobs from unnecessarily competing.

    The lock provides the final runtime protection.


    14. Why Both?

    Imagine two separate processes:

    Worker A:
    repair example.com
    
    Worker B:
    restore example.com
    

    Both query the database at nearly the same moment.

    Both may conclude:

    No conflicting job.
    

    The database-level queue logic alone may not be enough.

    The site lock provides the final serialization point.


    15. Job Worker

    Create:

    sudo nano /usr/local/bin/hosting-worker
    

    Its job is:

    find next job
          ↓
    claim job
          ↓
    check conflicts
          ↓
    acquire site lock
          ↓
    execute
          ↓
    record result
          ↓
    release lock
    

    16. Worker Identity

    Each worker should have an identity:

    worker-01
    

    Store:

    worker_id
    

    in the job record.

    This allows the system to answer:

    Which worker is processing this job?
    

    17. Claiming a Job

    This is a concurrency problem.

    Two workers could see:

    JOB-000241
    QUEUED
    

    simultaneously.

    Both might attempt to execute it.

    The worker therefore needs an atomic claim.


    18. Atomic Job Claim

    Conceptually:

    UPDATE jobs
    SET
        status = 'RUNNING',
        worker_id = ?,
        started_at = ?,
        attempt = attempt + 1
    WHERE id = ?
      AND status = 'QUEUED';
    

    Then check:

    rows affected == 1
    

    If:

    0
    

    another worker already claimed the job.


    19. Why This Matters

    Without atomic claiming:

    Worker A:
    sees QUEUED
    
    Worker B:
    sees QUEUED
    
    Worker A:
    runs
    
    Worker B:
    also runs
    

    This can cause duplicate backups or dangerous duplicate repairs.


    20. SQLite Concurrency

    Your current CHP database is SQLite.

    SQLite can handle multiple readers well, but writes need careful transaction handling.

    For job claiming:

    BEGIN IMMEDIATE
    

    can be useful to serialize the short claim transaction.

    Keep transactions short.

    Don’t hold a database transaction while:

    backup
    restore
    repair
    

    is running.


    21. Important Rule

    Never do:

    BEGIN TRANSACTION
    
    backup files for 20 minutes
    
    COMMIT
    

    That holds the database transaction far too long.

    Instead:

    BEGIN
    claim job
    COMMIT
    
    run backup
    
    BEGIN
    record result
    COMMIT
    

    22. Worker Loop

    Conceptually:

    while true:
    
        job = find_next_queued_job()
    
        if no job:
            sleep
    
        claim(job)
    
        if conflict:
            release/requeue
    
        execute(job)
    
        record_result()
    

    23. Don’t Busy-Loop

    Avoid:

    while true; do
        query database
    done
    

    with no delay.

    Use a small sleep:

    sleep 2
    

    or use a service/queue notification mechanism later.


    24. Worker as systemd Service

    Create:

    sudo nano /etc/systemd/system/cresignsys-worker.service
    

    Conceptually:

    [Unit]
    Description=CresignSys Job Worker
    After=network-online.target
    
    [Service]
    Type=simple
    ExecStart=/usr/local/bin/hosting-worker
    Restart=always
    RestartSec=5
    

    25. Start Worker

    Eventually:

    sudo systemctl daemon-reload
    sudo systemctl enable --now cresignsys-worker
    

    Then:

    systemctl status cresignsys-worker
    

    26. Multiple Workers

    Initially:

    1 worker
    

    Later:

    worker-01
    worker-02
    worker-03
    

    This allows parallel processing.

    But don’t increase concurrency without understanding:

    disk I/O
    CPU
    RAM
    MySQL
    network
    

    27. Worker Capacity

    Suppose the server has:

    8 CPU cores
    16 GB RAM
    

    You might eventually permit:

    2 backup workers
    

    rather than 8.

    Backup operations can be heavily disk-bound.


    28. Site-Level Concurrency

    Even with multiple workers:

    example.com:
    max concurrent mutating jobs = 1
    

    while:

    example.com:
    health checks
    

    may run separately.


    29. Operation Classes

    Define:

    READ_ONLY
    

    and:

    MUTATING
    

    Read-only

    HEALTH
    RECONCILE
    

    Mutating

    BACKUP
    RESTORE
    REPAIR
    RETENTION
    

    This classification helps determine conflicts.


    30. But Backup Is Special

    Backup doesn’t normally modify the website, but it writes heavily to storage.

    Therefore it is:

    site_read_only:
    YES
    
    resource_heavy:
    YES
    

    This distinction is useful.


    31. Job Resource Profile

    Each job can have:

    site_lock:
    YES/NO
    
    disk_heavy:
    YES/NO
    
    database_heavy:
    YES/NO
    
    network_heavy:
    YES/NO
    
    mutating:
    YES/NO
    

    For example:

    BACKUP:
    site_lock YES
    disk_heavy YES
    database_heavy YES
    mutating NO
    

    32. Repair

    REPAIR:
    site_lock YES
    disk_heavy LOW
    database_heavy NO
    mutating YES
    

    33. Restore

    RESTORE:
    site_lock YES
    disk_heavy HIGH
    database_heavy HIGH
    mutating YES
    

    34. Health Check

    HEALTH:
    site_lock NO
    disk_heavy NO
    database_heavy LOW
    mutating NO
    

    This lets CHP make smarter scheduling decisions.


    35. Job Handler Architecture

    Don’t build one giant script:

    hosting-worker
      └── 3000 lines
    

    Instead:

    hosting-worker
          │
          ├── handle-backup
          ├── handle-repair
          ├── handle-restore
          ├── handle-health
          ├── handle-reconcile
          └── handle-retention
    

    36. Job Dispatcher

    Conceptually:

    case "$JOB_TYPE" in
        BACKUP)
            handle_backup
            ;;
        REPAIR)
            handle_repair
            ;;
        RESTORE)
            handle_restore
            ;;
        HEALTH)
            handle_health
            ;;
        RECONCILE)
            handle_reconcile
            ;;
        RETENTION)
            handle_retention
            ;;
        *)
            fail_job "Unknown job type"
            ;;
    esac
    

    37. Unknown Job Type

    Never silently ignore it.

    Output:

    JOB FAILED
    
    Unknown job type:
    WORDPRESS_UPDATE
    

    until the handler exists.


    38. Job Retry

    Some jobs can safely retry.

    For example:

    HEALTH:
    YES
    
    BACKUP:
    YES, limited
    
    REPAIR:
    NO automatic retry
    
    RESTORE:
    NO automatic retry
    

    This is critical.


    39. Why Repair Should Not Auto-Retry

    Suppose repair did:

    change configuration
     ↓
    health failure
     ↓
    rollback
    

    Automatically retrying could repeat the same failure.

    Instead:

    FAILED / ROLLED_BACK
    

    and require investigation.


    40. Restore Should Not Auto-Retry

    A restore may be partially destructive.

    Never:

    restore failed
     ↓
    retry automatically
    

    without analyzing the failure.

    Mark:

    FAILED
    

    and stop.


    41. Health Retry

    Health checks can safely retry transient network conditions.

    Example:

    attempt 1:
    HTTP timeout
    
    attempt 2:
    HTTP timeout
    
    attempt 3:
    HTTP 200
    

    Result:

    HEALTHY
    

    42. Backup Retry

    A backup can have:

    attempt 1:
    database timeout
    
    attempt 2:
    SUCCESS
    

    But use a limited number of attempts.


    43. Retry Configuration

    Eventually:

    HEALTH:
    3 attempts
    
    BACKUP:
    2 attempts
    
    REPAIR:
    0
    
    RESTORE:
    0
    

    Store this as policy rather than scattering values across scripts.


    44. Job Timeout

    Every job should have a maximum runtime.

    Examples:

    HEALTH:
    2 minutes
    
    RECONCILE:
    5 minutes
    
    BACKUP:
    2 hours
    
    REPAIR:
    30 minutes
    
    RESTORE:
    3 hours
    

    These are examples, not final production values.


    45. Why Timeouts Matter

    Suppose:

    backup:
    RUNNING
    

    for:

    18 hours
    

    The worker may have crashed.

    The job must eventually become:

    STALE
    

    or:

    FAILED
    

    after appropriate recovery logic.


    46. Don’t Kill Jobs Blindly

    A timeout doesn’t automatically mean:

    kill -9
    

    Some operations need cleanup.

    For example:

    mysqldump
    tar
    restore
    

    may need controlled termination.


    47. Worker Heartbeat

    Add:

    ALTER TABLE jobs
    ADD COLUMN heartbeat_at TEXT;
    

    While running:

    heartbeat:
    18:32
    

    Then the scheduler can detect stale workers.


    48. Worker Heartbeat

    The worker periodically updates:

    heartbeat_at
    

    For example:

    every 30 seconds
    

    A job with:

    status=RUNNING
    heartbeat=2 hours ago
    

    may be abandoned.


    49. Worker Recovery

    A future recovery process can detect:

    RUNNING
    +
    heartbeat stale
    

    and mark:

    ORPHANED
    

    Then an operator can inspect it.

    Don’t automatically assume the underlying operation stopped.


    50. Why ORPHANED Matters

    Suppose the worker process died but:

    nginx configuration
    

    was already modified.

    Automatically retrying could create a second modification.

    Therefore:

    ORPHANED
    

    requires investigation.


    51. Job Event Log

    Create:

    CREATE TABLE job_events (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        job_id INTEGER NOT NULL,
        event_type TEXT NOT NULL,
        message TEXT,
        created_at TEXT NOT NULL,
    
        FOREIGN KEY (job_id)
            REFERENCES jobs(id)
            ON DELETE CASCADE
    );
    

    Events:

    CREATED
    CLAIMED
    STARTED
    LOCK_ACQUIRED
    ACTION_STARTED
    ACTION_COMPLETED
    RETRY
    FAILED
    SUCCESS
    CANCELLED
    

    52. Why Events?

    The job table tells you:

    status = FAILED
    

    The event history tells you:

    18:01 created
    18:02 claimed
    18:02 backup started
    18:10 database dump completed
    18:12 file archive failed
    18:12 job failed
    

    Much better for debugging.


    53. Job CLI

    Create:

    sudo nano /usr/local/bin/hosting-job
    

    Usage:

    hosting-job list
    hosting-job show JOB-000241
    hosting-job cancel JOB-000241
    hosting-job retry JOB-000241
    

    Be careful with:

    retry
    

    because not every job is retryable.


    54. Job List

    CresignSys Jobs
    ==============
    
    JOB         TYPE       SITE          STATUS    PRIORITY
    JOB-241     BACKUP     example.com   RUNNING   NORMAL
    JOB-242     HEALTH     shop.com      QUEUED    NORMAL
    JOB-243     REPAIR     blog.com      QUEUED    HIGH
    JOB-244     RETENTION  example.com   QUEUED    LOW
    

    55. Job Details

    JOB-000241
    ==========
    
    Type:
    BACKUP
    
    Site:
    example.com
    
    Status:
    RUNNING
    
    Priority:
    NORMAL
    
    Worker:
    worker-01
    
    Attempt:
    1
    
    Started:
    18:02
    
    Heartbeat:
    18:07
    

    56. Cancellation

    Cancellation is safe only when the job is:

    QUEUED
    

    For a running job:

    CANCEL REQUESTED
    

    may be possible.

    Don’t immediately kill the process.


    57. Cancellation State

    Use:

    CANCEL_REQUESTED
    

    Worker sees:

    cancel requested
    

    and exits at a safe checkpoint.


    58. Safe Checkpoints

    For backup:

    after database dump
    after file archive
    before finalization
    

    For repair:

    between actions
    

    But never interrupt halfway through an atomic configuration operation.


    59. Central Queue and Scheduler

    The backup scheduler should no longer execute:

    hosting-backup
    

    directly.

    Instead:

    Backup scheduler
          ↓
    create BACKUP job
          ↓
    job queue
          ↓
    worker
          ↓
    hosting-backup handler
    

    60. Repair Integration

    Instead of:

    hosting-repair RP-000021
    

    being directly executed by arbitrary processes, eventually:

    approved repair
          ↓
    create REPAIR job
          ↓
    worker
          ↓
    repair handler
    

    The worker becomes the controlled execution boundary.


    61. Restore Integration

    Likewise:

    approved restore
          ↓
    RESTORE job
          ↓
    worker
          ↓
    restore handler
    

    This gives all operations a common framework.


    62. Job Creation API

    Create a central function:

    create_job()
    

    Inputs:

    site_id
    job_type
    priority
    payload
    scheduled_at
    

    Output:

    JOB-000241
    

    63. Don’t Duplicate Job Creation Logic

    Avoid:

    backup script:
    own INSERT
    
    repair script:
    different INSERT
    
    restore script:
    third INSERT
    

    Instead:

    lib/jobs.sh
         │
         ├── create_job
         ├── claim_job
         ├── complete_job
         ├── fail_job
         └── cancel_job
    

    64. Create Job Library

    sudo nano /etc/cresignsys/lib/jobs.sh
    

    Functions:

    create_job
    get_next_job
    claim_job
    update_heartbeat
    complete_job
    fail_job
    cancel_job
    record_job_event
    

    65. Job Claiming Rules

    get_next_job should consider:

    status = QUEUED
    scheduled_at <= now
    priority
    site conflicts
    global concurrency
    

    Then:

    claim atomically
    

    66. Queue Ordering

    Basic ordering:

    CRITICAL
    HIGH
    NORMAL
    LOW
    

    Within the same priority:

    oldest scheduled job first
    

    This gives:

    priority DESC
    scheduled_at ASC
    

    conceptually.


    67. Fairness

    Priority can create starvation.

    Suppose:

    HIGH
    HIGH
    HIGH
    HIGH
    HIGH
    

    arrive continuously.

    Then:

    LOW
    

    might never run.

    Later, use aging:

    older low-priority jobs gradually gain priority
    

    Don’t implement this complexity yet.


    68. Global Concurrency

    Create configuration:

    /etc/cresignsys/hosting.conf
    

    Example:

    MAX_WORKERS=2
    MAX_BACKUP_WORKERS=1
    MAX_RESTORE_WORKERS=1
    MAX_REPAIR_WORKERS=1
    

    This prevents resource exhaustion.


    69. Resource-Specific Limits

    A server may allow:

    2 health checks
    1 backup
    1 repair
    

    simultaneously.

    But:

    restore
    

    could require:

    exclusive site access
    

    and perhaps exclusive database access.


    70. Job Categories

    Define:

    OBSERVATION
    MAINTENANCE
    RECOVERY
    

    Observation

    HEALTH
    RECONCILE
    

    Maintenance

    BACKUP
    RETENTION
    REPAIR
    

    Recovery

    RESTORE
    

    This classification helps future policy.


    71. Central Job Flow

                        REQUEST
                           │
                           ▼
                       CREATE JOB
                           │
                           ▼
                        QUEUED
                           │
                           ▼
                      SCHEDULER
                           │
                           ▼
                        WORKER
                           │
                           ▼
                     CLAIM + LOCK
                           │
                           ▼
                        EXECUTE
                           │
                  ┌────────┴────────┐
                  ▼                 ▼
               SUCCESS             FAIL
                  │                 │
                  ▼                 ▼
              COMPLETE             ERROR
    

    72. Security Boundary

    The job worker should be one of the most protected CHP components.

    The architecture should eventually be:

    WEB USER
       ↓
    API
       ↓
    AUTHORIZATION
       ↓
    JOB CREATION
       ↓
    QUEUE
       ↓
    WORKER
       ↓
    CONTROLLED OPERATION
    

    Not:

    WEB USER
       ↓
    shell command
    

    73. Job Payload Validation

    Before executing:

    validate job type
    validate site ID
    validate referenced plan
    validate backup ID
    validate required fields
    

    For example:

    REPAIR
    requires:
    plan_id
    

    while:

    BACKUP
    requires:
    site_id
    

    74. Never Trust Payload

    Even though the job was created internally, validate it again at execution time.

    Example:

    RESTORE job
    backup_id:
    BK-001
    

    Worker must verify:

    backup exists
    backup verified
    backup belongs to site
    backup not expired
    

    before restoration.


    75. Job Idempotency

    This is an important concept.

    A job should avoid performing the same irreversible action twice.

    For example:

    BACKUP
    

    can often safely be retried.

    But:

    RESTORE
    

    cannot automatically be repeated safely.


    76. Idempotency Key

    Add:

    ALTER TABLE jobs
    ADD COLUMN idempotency_key TEXT;
    

    Example:

    backup:
    site=7
    schedule=DAILY
    date=2026-08-13
    

    could produce:

    BACKUP:7:DAILY:2026-08-13
    

    The scheduler can then avoid duplicate jobs.


    77. Example

    Scheduler runs:

    02:00
    

    creates:

    BACKUP:7:DAILY:2026-08-13
    

    Scheduler runs again:

    02:15
    

    sees the same idempotency key already exists.

    Result:

    NO DUPLICATE JOB
    

    78. Idempotency for Repair

    Repair jobs can use:

    REPAIR:RP-000021
    

    If a job already exists:

    don't create another
    

    This is another protection against accidental duplicate execution.


    79. Idempotency for Restore

    Use:

    RESTORE:site7:backup123
    

    But because restores are destructive, the system should additionally require explicit approval and current-state checks.


    80. Job Queue Dashboard

    Eventually the CHP dashboard can show:

    Job Queue
    ---------
    
    Running:
    2
    
    Queued:
    7
    
    Failed:
    1
    
    Completed today:
    42
    

    And:

    JOB        TYPE       SITE        STATUS
    241        BACKUP     example     RUNNING
    242        HEALTH     shop        QUEUED
    243        REPAIR     blog        QUEUED
    244        RESTORE    client      FAILED
    

    81. Central Queue Changes CHP

    Before:

    many independent scripts
    

    After:

    one controlled execution system
    

    This gives CHP:

    concurrency control
    auditability
    retry policy
    timeouts
    priorities
    locks
    worker management
    

    82. Final CHP Architecture

    We now have:

                             CHP
                              │
            ┌─────────────────┼─────────────────┐
            ▼                 ▼                 ▼
        DISCOVERY        OBSERVATION          CONTROL
            │                 │                 │
            ▼                 ▼                 ▼
       DB IMPORT         HEALTH             PLAN
                            │                 │
                            ▼                 ▼
                      RECONCILIATION       APPROVAL
                                              │
                                              ▼
                                            JOB
                                              │
                                              ▼
                                           WORKER
                                              │
                             ┌────────────────┼────────────────┐
                             ▼                ▼                ▼
                          BACKUP            REPAIR           RESTORE
                             │                │                │
                             └────────────────┼────────────────┘
                                              ▼
                                           VERIFY
    

    83. The Most Important Design Rule

    The central queue should control execution, not authorization.

    Correct:

    Authorization
         ↓
    Can this operation happen?
         ↓
    Job created
         ↓
    Queue
         ↓
    Worker
    

    Incorrect:

    Queue
     ↓
    Maybe authorization
    

    Authorization must happen before the job becomes executable.


    84. Another Important Rule

    The worker should never blindly trust:

    status=APPROVED
    

    for sensitive jobs.

    It must revalidate:

    approval
    plan
    site state
    fingerprint
    expiration
    permissions
    

    at execution time.

    This gives:

    DEFENSE IN DEPTH
    

    85. Lesson 089 — Core Principle

    CHP is evolving from a collection of scripts into an actual control plane:

                    REQUEST
                       ↓
                  AUTHORIZATION
                       ↓
                     PLAN
                       ↓
                   APPROVAL
                       ↓
                     JOB
                       ↓
                    QUEUE
                       ↓
                    WORKER
                       ↓
                     LOCK
                       ↓
                   EXECUTION
                       ↓
                    VERIFY
                       ↓
                     AUDIT
    

    Each stage has one responsibility.

    That separation is what makes the platform safer, easier to debug, and eventually scalable to many hosted websites.


    Next Lesson — 090

    Build the CHP Event & Audit System

    The next missing foundation is a unified history of everything CHP does.

    We will build:

    SITE DISCOVERED
    SITE IMPORTED
    HEALTH CHECK
    RECONCILIATION
    DRIFT DETECTED
    PLAN CREATED
    PLAN APPROVED
    JOB CREATED
    BACKUP CREATED
    REPAIR EXECUTED
    RESTORE EXECUTED
    ROLLBACK
    FAILURE
    

    into a central event stream:

                        CHP EVENTS
                            │
            ┌───────────────┼────────────────┐
            ▼               ▼                ▼
          AUDIT          TIMELINE         ALERTS
            │               │                │
            ▼               ▼                ▼
       WHO DID IT?      WHAT HAPPENED?    WHAT NEEDS
       WHEN?            WHEN?             ATTENTION?
       WHY?             TO WHICH SITE?
    

    This will become the foundation for the future CHP dashboard activity timeline, customer activity history, security auditing, incident investigation, and automated alerts.

  • CresignSys Learn — Lesson 088

    Build the CHP Backup Scheduler & Retention Engine

    We now have:

    hosting-backup
    hosting-backup-list
    hosting-backup-verify
    hosting-restore

    The next step is to make backups automatic.

    The architecture becomes:

                     BACKUP POLICY
                          │
                          ▼
                      SCHEDULER
                          │
                          ▼
                     BACKUP JOB
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
           SUCCESS                   FAIL
              │                       │
              ▼                       ▼
          RETENTION                 ALERT
              │
              ▼
         SAFE PRUNING

    The critical rule is:

    Retention must never delete the last usable recovery point merely to satisfy a numerical retention policy.


    1. What We Need to Build

    Create these components:

    hosting-backup-schedule
    hosting-backup-run
    hosting-backup-prune
    hosting-backup-status

    The scheduler should eventually support:

    daily
    weekly
    monthly
    on-demand
    repair
    pre-restore

    2. Don’t Use One Cron Entry Per Website

    Avoid:

    cron:
    example.com
    cron:
    shop.example.com
    cron:
    blog.example.com
    cron:
    medical.example.com

    That becomes difficult to manage.

    Instead:

    SYSTEM SCHEDULER
           ↓
    CHP BACKUP SCHEDULER
           ↓
    QUERY DATABASE
           ↓
    FIND DUE SITES
           ↓
    CREATE BACKUP JOBS

    This centralizes scheduling.


    3. Scheduler Architecture

                  systemd timer
                        │
                        ▼
              hosting-backup-scheduler
                        │
                 ┌──────┴──────┐
                 ▼             ▼
              Database       Policy
                 │             │
                 └──────┬──────┘
                        ▼
                    DUE JOBS
                        │
                        ▼
                 BACKUP ENGINE
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
           SUCCESS               FAIL
              │                   │
              ▼                   ▼
           RETENTION             ALERT

    4. Why systemd Timer?

    Since the server is Ubuntu, use systemd for the scheduler rather than relying entirely on traditional cron.

    Advantages:

    logging
    service state
    failure tracking
    restart behavior
    central management

    5. Scheduler Service

    Create:

    sudo nano /etc/systemd/system/cresignsys-backup-scheduler.service

    Conceptually:

    [Unit]
    Description=CresignSys Backup Scheduler
    After=network-online.target
    
    [Service]
    Type=oneshot
    ExecStart=/usr/local/bin/hosting-backup-scheduler

    6. Scheduler Timer

    Create:

    sudo nano /etc/systemd/system/cresignsys-backup-scheduler.timer

    Example:

    [Unit]
    Description=Run CresignSys Backup Scheduler
    
    [Timer]
    OnCalendar=*:0/15
    Persistent=true
    
    [Install]
    WantedBy=timers.target

    This checks every 15 minutes.

    The scheduler itself determines whether a particular site’s backup is actually due.


    7. Why Check Frequently?

    Suppose a site is configured:

    Daily:
    02:00

    If the scheduler runs only once at exactly 02:00 and the server is temporarily unavailable, the backup could be missed.

    With:

    every 15 minutes

    the scheduler can determine:

    backup due?
    yes

    and execute it.


    8. Persistent=true

    This is useful when the server is offline.

    For example:

    02:00
    server offline
    
    05:00
    server starts

    The timer knows it missed a scheduled run.

    The scheduler can then decide:

    run missed backup

    subject to policy.


    9. Backup Policy Table

    Create:

    CREATE TABLE backup_policies (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        site_id INTEGER NOT NULL,
        enabled INTEGER NOT NULL DEFAULT 1,
    
        daily_enabled INTEGER NOT NULL DEFAULT 1,
        daily_time TEXT,
    
        weekly_enabled INTEGER NOT NULL DEFAULT 0,
        weekly_day INTEGER,
        weekly_time TEXT,
    
        monthly_enabled INTEGER NOT NULL DEFAULT 0,
        monthly_day INTEGER,
        monthly_time TEXT,
    
        retention_daily INTEGER NOT NULL DEFAULT 7,
        retention_weekly INTEGER NOT NULL DEFAULT 4,
        retention_monthly INTEGER NOT NULL DEFAULT 12,
    
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );

    10. Example Policy

    For:

    example.com

    store:

    Daily:
    YES
    
    Daily time:
    02:00
    
    Weekly:
    YES
    
    Weekly day:
    Sunday
    
    Weekly time:
    03:00
    
    Monthly:
    YES
    
    Monthly day:
    1
    
    Monthly time:
    04:00

    11. Retention Policy

    Example:

    Daily:
    7
    
    Weekly:
    4
    
    Monthly:
    12

    This does not mean:

    delete everything older than 12 months

    It means:

    retain selected recovery points according to policy

    12. Backup Classification

    Every scheduled backup should be classified:

    DAILY
    WEEKLY
    MONTHLY

    Example:

    BK-001
    DAILY
    
    BK-002
    DAILY
    
    BK-003
    WEEKLY

    The retention engine uses these classifications.


    13. Backup Schedule Table

    It can be useful to distinguish:

    policy

    from:

    execution history

    The policy says:

    "Run every day at 02:00."

    The backup record says:

    "Actually ran at 02:04 and succeeded."

    Don’t mix those concepts.


    14. Add Scheduled Metadata

    Add to backups:

    ALTER TABLE backups
    ADD COLUMN schedule_type TEXT;

    Possible:

    DAILY
    WEEKLY
    MONTHLY
    MANUAL
    REPAIR
    PRE_RESTORE

    15. Scheduler Command

    Create:

    sudo nano /usr/local/bin/hosting-backup-scheduler

    Start:

    #!/usr/bin/env bash
    
    set -Eeuo pipefail

    Load:

    source /etc/cresignsys/hosting.conf
    source /etc/cresignsys/lib/common.sh
    source /etc/cresignsys/lib/logging.sh
    source /etc/cresignsys/lib/database.sh
    source /etc/cresignsys/lib/backup.sh
    source /etc/cresignsys/lib/backup-policy.sh

    16. Scheduler Algorithm

    The scheduler should:

    1. Find enabled sites
    2. Load backup policy
    3. Determine whether backup is due
    4. Check whether another backup is running
    5. Check storage availability
    6. Create backup job
    7. Execute backup
    8. Record result
    9. Trigger retention
    10. Record scheduler result

    17. Find Due Sites

    Conceptually:

    for each managed site:
        load backup policy
    
        if backup is due:
            create backup job

    Don’t execute backups for:

    READ_ONLY

    sites unless their backup policy explicitly permits monitoring-only backups.


    18. Management State and Backups

    This requires a distinction.

    A site can be:

    READ_ONLY

    for configuration changes but still have:

    BACKUP:
    ENABLED

    Therefore don’t automatically equate:

    READ_ONLY
    =
    NO BACKUP

    Backup is not necessarily a configuration modification.


    19. Better Permission Model

    Use separate capabilities:

    MONITOR
    BACKUP
    MANAGE
    RESTORE
    APPROVE

    Then:

    READ_ONLY

    could mean:

    MONITOR:
    YES
    
    BACKUP:
    YES
    
    MANAGE:
    NO
    
    RESTORE:
    NO

    This is more flexible.


    20. Backup Job Table

    Create:

    CREATE TABLE backup_jobs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        site_id INTEGER NOT NULL,
        schedule_type TEXT NOT NULL,
        status TEXT NOT NULL,
        scheduled_for TEXT NOT NULL,
        started_at TEXT,
        completed_at TEXT,
        backup_id INTEGER,
        error_message TEXT,
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id),
    
        FOREIGN KEY (backup_id)
            REFERENCES backups(id)
    );

    21. Job States

    Use:

    SCHEDULED
    RUNNING
    SUCCESS
    FAILED
    SKIPPED
    CANCELLED

    Example:

    02:00
    SCHEDULED
    
    02:03
    RUNNING
    
    02:12
    SUCCESS

    22. Why a Job Table?

    Without it, you only know:

    Backup exists.

    With it, you know:

    Was it scheduled?
    Did it start?
    Was it skipped?
    Why did it fail?
    How long did it take?
    Which backup resulted?

    This is essential for operations.


    23. Duplicate Prevention

    Suppose the scheduler runs at:

    02:00
    02:15

    and the 02:00 backup is still running.

    The 02:15 execution must detect:

    BACKUP ALREADY RUNNING

    and skip creating another backup.


    24. Site Backup Lock

    Use:

    /var/lock/cresignsys/example.com.lock

    The backup engine already uses this.

    The scheduler should not bypass it.


    25. Job Lock vs Site Lock

    These solve different problems.

    Job lock

    Prevents duplicate scheduler execution.

    Site lock

    Prevents conflicting operations on the website.

    You can have:

    scheduler lock
    +
    site lock

    26. Example

    Scheduler A:
    creates job 101
    
    Scheduler B:
    sees job 101 RUNNING
    
    Scheduler B:
    SKIPPED

    This prevents duplicate backups.


    27. Missed Backup Policy

    Suppose:

    scheduled:
    02:00
    
    server:
    offline
    
    server returns:
    07:00

    The scheduler sees:

    MISSED

    What should it do?

    For the first version:

    run once

    if the backup has not already occurred that day.


    28. Don’t Create Multiple Catch-Up Backups

    If the server was offline for:

    3 days

    don’t create:

    3 immediate backups

    one after another.

    Instead:

    one catch-up backup

    then return to the normal schedule.


    29. Missed Backup Status

    Record:

    schedule_type:
    DAILY
    
    scheduled_for:
    2026-08-13 02:00
    
    actual:
    2026-08-13 07:05
    
    result:
    SUCCESS
    
    execution_reason:
    MISSED_SCHEDULE

    This is useful for reporting.


    30. Backup Freshness

    Site status can then calculate:

    latest successful backup

    and:

    backup age

    Example:

    Latest:
    07:05
    
    Age:
    2 hours

    31. Backup SLA

    A hosting plan can define:

    backup freshness:
    24 hours

    Then:

    age < 24h
    HEALTHY

    and:

    age > 24h
    WARNING

    32. Backup SLA by Hosting Plan

    This fits naturally with your hosting platform.

    For example:

    Starter:
    daily
    
    Business:
    daily
    
    Professional:
    daily + weekly
    
    Business Plus:
    daily + weekly + monthly

    The exact commercial policy can be configured later.


    33. Storage Check

    Before creating a backup:

    df -P /var/lib/cresignsys/backups

    Determine available storage.

    If:

    available:
    5 GB

    but estimated backup:

    8 GB

    don’t begin.


    34. Estimate Backup Size

    Use the site root:

    du -sb "$WEB_ROOT"

    plus:

    database estimate
    configuration
    compression overhead

    Then:

    required space

    should include a safety margin.


    35. Safety Margin

    Don’t require:

    available >= estimated

    Use something like:

    available >= estimated × 1.5

    for the first version.

    This avoids failure when compression or temporary files require extra space.


    36. What If Storage Is Low?

    Don’t automatically delete backups before creating the new backup.

    Instead:

    CHECK
     ↓
    RETENTION CANDIDATES
     ↓
    VERIFY SAFE TO DELETE
     ↓
    PRUNE
     ↓
    RECHECK STORAGE
     ↓
    BACKUP

    But only if pruning does not violate recovery requirements.


    37. Never Delete First, Think Later

    Bad:

    disk full
     ↓
    delete oldest backups
     ↓
    backup

    The deletion could remove the only usable recovery point.

    Better:

    disk full
     ↓
    calculate protected backups
     ↓
    calculate safe candidates
     ↓
    prune only safe candidates
     ↓
    recheck

    38. Protected Backup

    A backup is protected if it is:

    latest verified backup

    or:

    required by weekly policy

    or:

    required by monthly policy

    or:

    associated with an active repair

    or:

    currently being restored

    39. Retention Algorithm

    Suppose backups are:

    D1
    D2
    D3
    D4
    D5
    D6
    D7
    D8
    D9

    Policy:

    retain 7 daily

    Keep:

    D3–D9

    Candidates:

    D1
    D2

    But if D2 is also the weekly recovery point:

    D2:
    PROTECTED

    then D1 becomes the only deletion candidate.


    40. Retention Is Not Simple FIFO

    Never implement:

    sort by date
    delete oldest N

    without classification.

    Retention needs to understand:

    daily
    weekly
    monthly
    protected
    verified
    active

    41. Example

    Policy:

    Daily:
    7
    
    Weekly:
    4
    
    Monthly:
    12

    A single backup can satisfy more than one role.

    For example:

    Sunday backup

    can be:

    DAILY
    +
    WEEKLY

    The retention engine should avoid storing unnecessary duplicate copies where possible.


    42. Retention Selection

    Conceptually:

    1. Select newest 7 daily backups
    2. Select newest 4 weekly backups
    3. Select newest 12 monthly backups
    4. Combine selections
    5. Mark all selected backups PROTECTED
    6. Everything else becomes a candidate

    Then apply safety checks.


    43. Candidate States

    Don’t immediately delete.

    Use:

    CANDIDATE

    first.

    Then:

    SAFE_TO_DELETE

    Then:

    DELETED

    This makes the retention process auditable.


    44. Retention Table

    Create:

    CREATE TABLE backup_retention_actions (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        backup_id INTEGER NOT NULL,
        action TEXT NOT NULL,
        reason TEXT,
        created_at TEXT NOT NULL,
        executed_at TEXT,
    
        FOREIGN KEY (backup_id)
            REFERENCES backups(id)
    );

    Example:

    backup:
    BK-001
    
    action:
    PROTECT
    
    reason:
    Newest daily backup

    or:

    backup:
    BK-002
    
    action:
    DELETE
    
    reason:
    Older than daily retention

    45. Never Delete Unverified Backups First

    It might seem logical to delete:

    FAILED

    backups first.

    But even a failed backup could contain useful partial data for forensic purposes.

    Instead define a cleanup policy.

    For example:

    FAILED:
    retain 24–72 hours

    then remove automatically.


    46. Corrupted Backup

    If a previously verified backup later fails checksum verification:

    VERIFIED
     ↓
    CORRUPTED

    It must immediately lose its protected status.

    Then find another valid recovery point.


    47. Last Known-Good Backup

    For every site, maintain:

    last_verified_backup_id

    Conceptually:

    site
     ↓
    last known good backup

    This can be used by:

    repair rollback
    restore recommendation
    disaster recovery
    site status

    48. Add to Sites

    Eventually:

    ALTER TABLE sites
    ADD COLUMN last_verified_backup_id INTEGER;

    However, because SQLite foreign-key handling may require care when altering tables, implement this through your normal CHP database migration system rather than manually changing production tables repeatedly.


    49. Better: Derived Query

    Initially you don’t even need to store it.

    Query:

    latest backup
    WHERE
    site_id = ?
    AND status = VERIFIED
    ORDER BY verified_at DESC
    LIMIT 1

    This avoids duplicated state.


    50. Backup Status

    The site dashboard can show:

    Backup:
    HEALTHY
    
    Last verified:
    2026-08-13 07:05
    
    Age:
    2h 15m

    If no verified backup exists:

    Backup:
    CRITICAL

    51. Scheduler Logging

    Every scheduled execution should log:

    site
    schedule
    scheduled time
    actual time
    result
    backup ID
    duration

    Example:

    2026-08-13 07:05
    example.com
    DAILY
    SUCCESS
    BK-20260813-070501
    duration=7m32s

    52. Backup Duration

    Record:

    started_at
    completed_at
    duration_seconds

    This allows detection of unusual changes.

    For example:

    Normal:
    7 minutes
    
    Current:
    48 minutes

    could indicate:

    site growth
    storage problem
    database issue
    network issue

    53. Backup Performance

    Over time, CHP can calculate:

    average backup duration
    backup size growth
    database growth
    storage consumption

    This becomes useful for capacity planning.


    54. Backup Job Failure

    If:

    hosting-backup

    returns failure:

    job:
    FAILED

    Do not run retention as though the backup succeeded.

    Instead:

    FAILED
     ↓
    record failure
     ↓
    alert

    Retention can run independently later.


    55. Retry Policy

    A backup failure can be retried carefully.

    For example:

    attempt 1:
    FAILED
    
    wait:
    30 minutes
    
    attempt 2:
    FAILED
    
    wait:
    60 minutes
    
    attempt 3:
    FAILED

    Then:

    FINAL FAILURE

    Don’t retry indefinitely.


    56. Why Limited Retry?

    Possible causes include:

    disk full
    database locked
    permission error
    network storage unavailable
    corrupt filesystem

    Repeated retries can make the situation worse.


    57. Backup Retry Table

    The job record can include:

    attempt_number

    Example:

    Job 101
    Attempt 1:
    FAILED
    
    Attempt 2:
    SUCCESS

    58. Scheduler Concurrency

    Suppose 20 websites are due at 02:00.

    Don’t necessarily run:

    20 backups
    simultaneously

    That could overload:

    CPU
    disk I/O
    RAM
    MySQL

    59. Global Backup Concurrency

    Configure:

    max concurrent backups:
    2

    Initially.

    Then:

    Site A → RUNNING
    Site B → RUNNING
    Site C → QUEUED
    Site D → QUEUED

    60. Backup Queue

    Eventually:

    BACKUP QUEUE
    -------------
    A RUNNING
    B RUNNING
    C QUEUED
    D QUEUED
    E QUEUED

    This becomes important when CHP hosts many sites.


    61. Site Lock Still Applies

    Even with a global queue:

    site A

    cannot have:

    backup
    +
    restore

    at the same time.

    The site lock remains the final protection.


    62. Scheduler Priorities

    Eventually jobs can have:

    CRITICAL
    HIGH
    NORMAL
    LOW

    For example:

    PRE_RESTORE:
    CRITICAL
    
    REPAIR:
    HIGH
    
    CUSTOMER DAILY:
    NORMAL

    This prevents routine backups from blocking urgent recovery operations.


    63. Backup Scheduling Database

    The overall structure now becomes:

    sites
     │
     ├── backup_policies
     │
     ├── backup_jobs
     │
     └── backups
            │
            └── retention_actions

    This cleanly separates:

    policy
    execution
    artifact
    retention

    64. Backup Scheduler Status

    Create:

    sudo nano /usr/local/bin/hosting-backup-status

    Output:

    CresignSys Backup Status
    ========================
    
    Site:
    example.com
    
    Policy:
    Daily 02:00
    Weekly Sunday 03:00
    Monthly 1st 04:00
    
    Last Backup:
    2026-08-13 07:05
    
    Status:
    VERIFIED
    
    Next Scheduled:
    2026-08-14 02:00
    
    Retention:
    7 daily
    4 weekly
    12 monthly
    
    Storage:
    42%
    
    Scheduler:
    HEALTHY

    65. Site-Level Backup Status

    For all sites:

    sudo hosting-backup-status --all

    Output:

    SITE                    LAST BACKUP        STATUS
    -----------------------------------------------------
    example.com             2h                 HEALTHY
    shop.example.com        5h                 HEALTHY
    blog.example.com        31h                WARNING
    client.example.com      4d                 CRITICAL

    This will eventually feed the CHP dashboard.


    66. Scheduler Health

    The scheduler itself must be monitored.

    Suppose:

    systemd timer:
    stopped

    Then:

    all sites

    could silently stop receiving backups.

    Therefore:

    CHP scheduler health

    must be part of the platform health model.


    67. Scheduler Heartbeat

    Record:

    last_scheduler_run

    For example:

    2026-08-13 15:00

    If current time is:

    18:00

    and no scheduler run has occurred:

    SCHEDULER:
    WARNING

    68. Scheduler Failure

    If the timer repeatedly fails:

    SCHEDULER:
    CRITICAL

    This is different from:

    individual backup:
    FAILED

    The distinction matters.


    69. Three Levels

    PLATFORM
       │
       ├── Scheduler health
       │
       └── Site backup health

    Example:

    Scheduler:
    HEALTHY
    
    Site A:
    HEALTHY
    
    Site B:
    FAILED
    
    Site C:
    HEALTHY

    Only Site B has a backup problem.


    70. Backup Alert Conditions

    Eventually alert when:

    no verified backup within SLA

    or:

    backup failed repeatedly

    or:

    storage critically low

    or:

    scheduler not running

    or:

    last verified backup corrupted

    71. Do Not Alert on Every Scheduler Run

    Bad:

    Backup succeeded
    Backup succeeded
    Backup succeeded

    The system should notify only when there is something requiring attention.


    72. Backup Recovery Objective

    Now we can introduce two important concepts.

    RPO

    Recovery Point Objective:

    How much recent data can the customer afford to lose?

    Example:

    Daily backup:
    RPO ≈ 24 hours

    RTO

    Recovery Time Objective:

    How quickly can the website be restored?

    These concepts will eventually influence hosting plans.


    73. Example

    Starter:

    RPO:
    24h
    
    RTO:
    best effort

    Professional:

    RPO:
    24h
    
    RTO:
    priority restore

    Premium:

    RPO:
    shorter interval
    
    RTO:
    priority recovery

    Exact commercial values should be defined separately.


    74. Backup Scheduler and CHP Repairs

    Now consider:

    02:00:
    scheduled backup
    
    02:10:
    repair starts

    The repair engine can use:

    02:00 backup

    if it meets the required safety conditions.

    But a repair should preferably create its own:

    REPAIR

    backup immediately before modification.


    75. Why Scheduled Backup Is Not Enough

    A scheduled backup at:

    02:00

    does not protect against:

    repair at 16:00

    because the site may have changed significantly during the day.

    Therefore:

    scheduled backup
    +
    pre-repair backup

    are different safety mechanisms.


    76. Backup Types Summary

    DAILY
       customer recovery
    
    WEEKLY
       longer retention
    
    MONTHLY
       long-term recovery
    
    REPAIR
       immediate rollback
    
    PRE_RESTORE
       protection before restoration
    
    MANUAL
       operator-created recovery point

    77. Retention Protection

    The retention engine should protect:

    latest verified DAILY
    latest verified WEEKLY
    latest verified MONTHLY
    active REPAIR backup
    active PRE_RESTORE backup

    Only then can it identify deletion candidates.


    78. Deletion Algorithm

    LOAD ALL BACKUPS
           ↓
    REMOVE ACTIVE/PROTECTED
           ↓
    SELECT RETENTION SET
           ↓
    MARK CANDIDATES
           ↓
    VERIFY AT LEAST ONE GOOD RECOVERY POINT
           ↓
    DELETE
           ↓
    VERIFY STORAGE

    79. Final Safety Check

    Before deleting any backup:

    verified_count_after_delete >= minimum_required_recovery_points

    For example:

    minimum:
    1

    Then CHP must never produce:

    VERIFIED BACKUPS:
    0

    unless an administrator explicitly overrides the policy.


    80. Never Delete the Last Verified Backup

    This deserves its own rule:

    if deleting backup would leave zero VERIFIED backups:
        DENY deletion

    Even if:

    retention policy says delete

    the safety rule wins.


    81. Example

    Suppose:

    Backup A:
    VERIFIED
    
    Backup B:
    FAILED
    
    Backup C:
    CORRUPTED

    Retention says:

    delete old backups

    The engine must preserve:

    Backup A

    because it is the only known-good recovery point.


    82. If Storage Is 100%

    Suppose:

    disk:
    100%

    and:

    new backup:
    cannot start

    The system should:

    1. identify safe deletion candidates
    2. protect last verified backup
    3. prune only safe candidates
    4. retry backup

    If insufficient space remains:

    BACKUP FAILED

    and alert.

    Never delete the last recovery point just to make the new backup fit.


    83. Backup Scheduler Final Architecture

                       SYSTEMD TIMER
                             │
                             ▼
                  BACKUP SCHEDULER
                             │
                   ┌─────────┴─────────┐
                   ▼                   ▼
               DUE JOBS             MISSED JOBS
                   │                   │
                   └─────────┬─────────┘
                             ▼
                        JOB QUEUE
                             │
                             ▼
                       BACKUP ENGINE
                             │
                  ┌──────────┴──────────┐
                  ▼                     ▼
               SUCCESS                 FAIL
                  │                     │
                  ▼                     ▼
             RETENTION               RETRY
                  │                     │
                  ▼                     ▼
              SAFE PRUNE              ALERT

    84. Complete CHP Architecture So Far

                             CRESIGNSYS CHP
                                   │
           ┌───────────────────────┼───────────────────────┐
           ▼                       ▼                       ▼
       DISCOVERY              OBSERVATION                CONTROL
           │                       │                       │
           ▼                       ▼                       ▼
     DB IMPORT              RECONCILIATION           REPAIR PLAN
                                   │                       │
                                   ▼                       ▼
                                HEALTH                 APPROVAL
                                                           │
                                                           ▼
                                                        REPAIR
                                                           │
                                                           ▼
                                                        BACKUP
                                                           │
                                                           ▼
                                                       VERIFY

    And backup itself now has:

    POLICY
      ↓
    SCHEDULER
      ↓
    JOB
      ↓
    BACKUP
      ↓
    VERIFY
      ↓
    RETENTION

    85. Lesson 088 — Core Principle

    A professional backup scheduler does not simply say:

    "Run backup every day."

    It must understand:

    WHEN should it run?
    WAS it actually run?
    DID it succeed?
    IS the backup verified?
    IS it still useful?
    IS it protected by retention?
    IS there enough storage?
    IS another backup running?
    IS the scheduler healthy?
    IS there at least one recovery point?

    The critical retention rule remains:

                 NEVER
                  ↓
    delete the last
    known-good backup

    Next Lesson — 089

    Build the CHP Central Job Queue

    We now have several independent operations:

    BACKUP
    RESTORE
    REPAIR
    HEALTH CHECK
    RECONCILIATION
    RETENTION

    Running all of them independently will eventually create conflicts.

    The next step is therefore a central:

    CHP JOB ENGINE

    with:

    QUEUED
    RUNNING
    SUCCESS
    FAILED
    CANCELLED

    and controlled concurrency:

                  CHP JOB QUEUE
                        │
           ┌────────────┼────────────┐
           ▼            ▼            ▼
         BACKUP       REPAIR       RESTORE
           │            │            │
           └────────────┼────────────┘
                        ▼
                  WORKER ENGINE
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
           RUNNING              QUEUED
              │
              ▼
           RESULT

    This will allow CHP to prevent dangerous situations such as:

    BACKUP + RESTORE
    REPAIR + RESTORE
    REPAIR + REPAIR

    running against the same website simultaneously.

  • CresignSys Learn — Lesson 087

    Build the CHP Backup & Restore Engine

    The repair system now depends on one critical capability:

    Before CHP changes a website, it must be able to restore the previous state.

    Therefore backup is not merely a customer feature.

    It is also a safety mechanism for CHP itself.


    1. Backup Architecture

    We will build:

    hosting-backup
    hosting-restore
    hosting-backup-list
    hosting-backup-verify

    The flow becomes:

                        WEBSITE
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
            FILES        MYSQL        CONFIG
              │            │            │
              └────────────┼────────────┘
                           ▼
                        MANIFEST
                           │
                           ▼
                       CHECKSUMS
                           │
                           ▼
                    VERIFIED BACKUP

    2. Create Backup Command

    sudo nano /usr/local/bin/hosting-backup

    Usage:

    sudo hosting-backup example.com

    Optional:

    sudo hosting-backup example.com --full
    sudo hosting-backup example.com --config-only
    sudo hosting-backup example.com --json

    For the first implementation, focus on a full site backup.


    3. What a Full Backup Contains

    A full backup should contain:

    Website files
    MySQL database
    Nginx site configuration
    Backup manifest
    Checksums
    Metadata

    Conceptually:

    BACKUP
    ├── files/
    ├── database/
    ├── config/
    ├── manifest.json
    └── checksums.sha256

    4. Backup Directory

    Use:

    /var/lib/cresignsys/backups/

    Structure:

    /var/lib/cresignsys/backups/
    └── example.com/
        └── BK-20260813-170001/
            ├── files/
            ├── database/
            ├── config/
            ├── manifest.json
            └── checksums.sha256

    5. Never Use a Predictable Temporary Directory

    Don’t build backups directly in:

    /tmp/example.com-backup

    with a predictable name.

    Use a unique working directory:

    /var/lib/cresignsys/tmp/

    with a generated identifier.


    6. Backup ID

    Every backup gets a unique ID:

    BK-20260813-170001

    or preferably:

    BK-20260813-170001-a8f3

    The ID becomes the permanent reference.

    Example:

    sudo hosting-restore example.com BK-20260813-170001-a8f3

    7. Backup Database Table

    Create:

    CREATE TABLE backups (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        site_id INTEGER NOT NULL,
        backup_code TEXT NOT NULL UNIQUE,
        backup_type TEXT NOT NULL,
        storage_path TEXT NOT NULL,
        status TEXT NOT NULL,
        size_bytes INTEGER,
        created_at TEXT NOT NULL,
        verified_at TEXT,
        expires_at TEXT,
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
            ON DELETE CASCADE
    );

    8. Backup States

    Use:

    CREATING
    CREATED
    VERIFYING
    VERIFIED
    FAILED
    RESTORING
    RESTORED
    CORRUPTED
    EXPIRED
    DELETED

    The important distinction is:

    CREATED

    does not necessarily mean:

    VERIFIED

    9. Backup Lifecycle

    CREATING
       ↓
    CREATED
       ↓
    VERIFYING
       ↓
    VERIFIED

    Failure:

    CREATING
       ↓
    FAILED

    or:

    VERIFIED
       ↓
    CORRUPTED

    if a later integrity check fails.


    10. Acquire Site Lock

    Backup must use the same site lock:

    /var/lock/cresignsys/example.com.lock

    Why?

    Because you don’t want:

    BACKUP
       +
    REPAIR

    changing the same files simultaneously.


    11. Backup and Repair

    Correct:

    REPAIR
     ↓
    lock
     ↓
    backup
     ↓
    change

    Incorrect:

    backup
    repair
    both running simultaneously

    The lock serializes site-changing operations.


    12. Database Consistency

    A website backup contains two major states:

    FILES
    DATABASE

    These should represent approximately the same application state.

    For WordPress:

    wp-content/
    wp-config.php

    and:

    MySQL database

    must correspond.


    13. Why Files Alone Are Not Enough

    If you restore only:

    wp-content/

    but not the database, you may have:

    old files
    new database

    or:

    new files
    old database

    which can produce inconsistent application state.


    14. Why Database Alone Is Not Enough

    A WordPress database does not contain:

    themes
    plugins
    uploads
    wp-config.php

    Therefore a real WordPress backup requires both.


    15. Database Backup

    For MySQL:

    mysqldump

    should create:

    database/example.com.sql

    Use credentials securely.

    Do not put the database password directly into:

    ps
    command history
    logs

    16. Avoid Passwords on the Command Line

    Don’t use:

    mysqldump -u root -pPASSWORD

    because command-line arguments may be visible to other processes or shell history.

    Use an appropriate protected MySQL configuration or credential mechanism.


    17. Backup Database Name

    Retrieve the WordPress database name from the site’s configuration.

    For example:

    DB_NAME

    But don’t expose:

    DB_PASSWORD

    in the backup manifest.


    18. Database Dump

    Conceptually:

    mysqldump \
        --single-transaction \
        --routines \
        --triggers \
        "$DB_NAME" \
        > "$BACKUP_DIR/database/database.sql"

    For InnoDB-heavy WordPress databases, --single-transaction helps create a consistent logical dump without taking a full table lock.


    19. Verify Database Dump

    After creation:

    test -s "$DATABASE_DUMP"

    Then:

    Database backup:
    CREATED

    A zero-byte dump must be treated as failure.


    20. Database Dump Integrity

    At minimum:

    mysql --execute="..." 

    or a suitable validation process should confirm the dump is structurally usable.

    A future version can perform a test import into a temporary database.


    21. Files Backup

    The site root might be:

    /storage/websites/example.com/public

    Archive it:

    files/site.tar.zst

    or:

    files/site.tar.gz

    For modern Linux systems, tar with compression is appropriate.


    22. Example

    tar \
        --create \
        --gzip \
        --file="$BACKUP_DIR/files/site.tar.gz" \
        -C "$WEB_ROOT" \
        .

    This stores the contents of the site root rather than embedding the entire absolute path.


    23. Why -C Matters

    Instead of:

    /storage/websites/example.com/public/wp-content

    inside the archive, you get:

    wp-content/

    This makes restore safer and more portable.


    24. Don’t Archive the Backup Directory

    Never put:

    /var/lib/cresignsys/backups/

    inside the website backup if it resides under the site root.

    Otherwise:

    backup
     → includes backup
     → includes backup
     → includes backup

    could occur.

    Keep backup storage outside website roots.


    25. Nginx Configuration

    Store:

    config/nginx.conf

    for the specific site.

    For example:

    /etc/nginx/sites-enabled/example.com.conf

    Copy it to:

    config/nginx-example.com.conf

    26. Preserve Metadata

    Record:

    owner
    group
    permissions

    for configuration files.

    Example:

    root:root
    0644

    27. SSL Metadata

    Don’t initially copy private keys into a customer backup unless your security model explicitly supports encrypted key storage.

    Instead record:

    certificate subject
    issuer
    expiry
    SANs
    certificate fingerprint

    in the manifest.


    28. Why Not Automatically Copy Private Keys?

    TLS private keys are highly sensitive.

    A backup containing:

    privkey.pem

    must have significantly stronger security controls.

    For the first CHP backup engine:

    SSL:
    metadata only

    is safer.


    29. Backup Manifest

    Create:

    manifest.json

    Example:

    {
      "backup_code": "BK-20260813-170001-a8f3",
      "domain": "example.com",
      "site_id": 7,
      "backup_type": "FULL",
      "created_at": "2026-08-13T17:00:01+05:30",
      "application": "WORDPRESS",
      "web_root": "/storage/websites/example.com/public",
      "database": "example_wp",
      "files_archive": "files/site.tar.gz",
      "database_dump": "database/database.sql",
      "nginx_config": "config/nginx-example.com.conf"
    }

    30. Don’t Put Secrets in the Manifest

    Never include:

    DB_PASSWORD
    API_TOKEN
    SSH_PRIVATE_KEY
    TLS_PRIVATE_KEY

    The manifest should describe the backup, not contain credentials.


    31. Add Version

    Include:

    {
      "format_version": 1
    }

    This is important.

    Future CHP versions may change the backup format.


    32. Example Future

    Today:

    format_version:
    1

    Later:

    format_version:
    2

    The restore engine can determine how to interpret the backup.


    33. Checksums

    After all backup files are created:

    sha256sum

    Generate:

    checksums.sha256

    Example:

    abc123...  files/site.tar.gz
    def456...  database/database.sql
    789abc...  config/nginx-example.com.conf

    34. Why SHA-256?

    It allows CHP to detect:

    corruption
    partial copy
    unexpected modification

    before restoration.


    35. Verify the Backup

    Run:

    sha256sum --check checksums.sha256

    Expected:

    files/site.tar.gz: OK
    database/database.sql: OK
    config/nginx-example.com.conf: OK

    Then:

    Backup:
    VERIFIED

    36. Backup Verification

    The process is:

    Create files
         ↓
    Create database dump
         ↓
    Create config backup
         ↓
    Create manifest
         ↓
    Generate checksums
         ↓
    Verify checksums
         ↓
    Mark VERIFIED

    37. Backup Size

    Calculate:

    du -sb "$BACKUP_DIR"

    Store:

    size_bytes

    in the database.

    This allows the dashboard to show:

    Backup size:
    1.8 GB

    38. Backup List Command

    Create:

    sudo nano /usr/local/bin/hosting-backup-list

    Usage:

    sudo hosting-backup-list example.com

    Output:

    CresignSys Backups
    ==================
    
    Site:
    example.com
    
    ID                         DATE                 SIZE      STATUS
    BK-20260813-170001-a8f3    2026-08-13 17:00     1.8 GB    VERIFIED
    BK-20260812-170001-c2a7    2026-08-12 17:00     1.7 GB    VERIFIED
    BK-20260811-170001-91fd    2026-08-11 17:00     1.7 GB    VERIFIED

    39. Backup Verify Command

    Create:

    sudo nano /usr/local/bin/hosting-backup-verify

    Usage:

    sudo hosting-backup-verify example.com BK-20260813-170001-a8f3

    It should:

    load manifest
     ↓
    check required files
     ↓
    verify SHA-256
     ↓
    validate database dump
     ↓
    report result

    40. Backup Verification Result

    CresignSys Backup Verification
    ==============================
    
    Backup:
    BK-20260813-170001-a8f3
    
    Manifest:
    OK
    
    Files archive:
    OK
    
    Database dump:
    OK
    
    Nginx configuration:
    OK
    
    Checksums:
    OK
    
    Overall:
    VERIFIED

    41. Restore Command

    Create:

    sudo nano /usr/local/bin/hosting-restore

    Usage:

    sudo hosting-restore example.com BK-20260813-170001-a8f3

    But this must be extremely cautious.

    Restore is potentially destructive.


    42. Restore Should Require Explicit Confirmation

    Don’t allow:

    hosting-restore example.com BACKUP

    to immediately overwrite a production site.

    Use:

    hosting-restore example.com BACKUP --confirm

    or an equivalent explicit confirmation mechanism.


    43. Even Better: Restore Plan

    Eventually:

    hosting-restore-plan
            ↓
    review
            ↓
    approve
            ↓
    hosting-restore

    This is safer for production.

    For the first implementation, at minimum require an explicit confirmation flag.


    44. Restore Workflow

    LOAD BACKUP
         ↓
    VERIFY BACKUP
         ↓
    ACQUIRE LOCK
         ↓
    CURRENT SITE BACKUP
         ↓
    STOP/QUIESCE IF REQUIRED
         ↓
    RESTORE DATABASE
         ↓
    RESTORE FILES
         ↓
    RESTORE CONFIG
         ↓
    VALIDATE
         ↓
    HEALTH CHECK
         ↓
    RECONCILE

    45. Never Restore Without a Safety Backup

    Suppose the operator wants:

    BK-20260810

    restored.

    Before doing that, CHP should create:

    PRE-RESTORE-20260813

    of the current state.

    Therefore:

    CURRENT
       ↓
    SAFETY BACKUP
       ↓
    RESTORE OLD BACKUP

    If the restore fails:

    SAFETY BACKUP
       ↓
    ROLLBACK

    46. Restore Is Therefore a Transaction

    Conceptually:

    CURRENT STATE
         │
         ▼
    SAFETY BACKUP
         │
         ▼
    RESTORE TARGET
         │
         ▼
    VALIDATE
         │
     ┌───┴────┐
     ▼        ▼
    PASS     FAIL
     │         │
     ▼         ▼
    DONE     RESTORE
              SAFETY
              BACKUP

    47. Database Restore

    For MySQL:

    mysql "$DB_NAME" < database/database.sql

    But this can be destructive.

    Before restoring:

    current database
           ↓
    temporary safety backup

    must exist.


    48. Don’t Blindly Drop Production Database

    Avoid automatically executing:

    DROP DATABASE ...

    in the first restore implementation.

    A safer first version can restore into a controlled temporary database and validate before switching, or require a stronger manual workflow for destructive replacement.


    49. WordPress Restore Complexity

    A WordPress restore involves:

    FILES
    DATABASE
    CONFIG

    and potentially:

    CACHE
    PHP VERSION
    DOMAIN
    SSL

    Restoring only files and database does not necessarily restore the entire runtime environment.

    Therefore the manifest should record the environment.


    50. Environment Metadata

    Add:

    {
      "environment": {
        "php_version": "8.3",
        "web_server": "nginx",
        "database_engine": "mysql",
        "database_version": "8.0"
      }
    }

    This helps CHP determine compatibility during restore.


    51. Restore Compatibility Check

    Before restore:

    Backup PHP:
    8.3
    
    Current PHP:
    8.2

    The system should warn:

    WARNING
    
    Backup was created under PHP 8.3.
    Current site uses PHP 8.2.
    
    Restore may not be compatible.

    Don’t silently continue.


    52. Backup Manifest Should Record CHP Version

    Add:

    {
      "chp_version": "1.0",
      "backup_format_version": 1
    }

    This makes future migration easier.


    53. Backup Integrity vs Backup Usability

    A checksum tells you:

    "The file hasn't changed."

    It does not prove:

    "The backup can successfully restore a working website."

    These are different.


    54. Three Backup Levels

    Use:

    INTEGRITY

    means:

    checksums valid
    STRUCTURAL

    means:

    archive/database dump can be opened
    RESTORABLE

    means:

    successfully restored and validated

    55. Backup Verification Model

    Created
      ↓
    Integrity verified
      ↓
    Structure verified
      ↓
    Restore tested
      ↓
    RESTORABLE

    The first CHP version can initially support:

    INTEGRITY
    STRUCTURAL

    and later implement automated restore testing.


    56. Restore Testing

    For production-quality hosting:

    backup
      ↓
    temporary environment
      ↓
    restore
      ↓
    health check
      ↓
    destroy temporary environment

    This proves the backup is actually usable.


    57. Don’t Test by Destroying Production

    Never test a backup with:

    restore → production

    unless an actual restoration is intended.

    Use:

    staging
    temporary VM
    isolated container

    for restoration testing.


    58. Retention Policy

    Backups should eventually follow a policy.

    Example:

    Daily:
    7 copies
    
    Weekly:
    4 copies
    
    Monthly:
    12 copies

    But don’t delete the only remaining backup.


    59. Retention Engine

    A future command:

    hosting-backup-prune example.com

    will determine:

    which backups are protected
    which are expired
    which can be deleted

    60. Never Delete an Unverified Backup Automatically

    Suppose:

    Backup A:
    VERIFIED
    
    Backup B:
    FAILED
    
    Backup C:
    VERIFIED

    Retention should not treat them identically.

    A failed backup should be handled separately.


    61. Backup Storage Capacity

    The backup system itself needs monitoring.

    Check:

    df -h /var/lib/cresignsys/backups

    If backup storage is full:

    BACKUP:
    BLOCKED

    Do not partially create a backup and report success.


    62. Partial Backup Cleanup

    If backup creation fails:

    creating
     ↓
    failure

    the incomplete directory should be marked:

    FAILED

    and cleaned up or quarantined.

    Never present it as:

    VERIFIED

    63. Backup Locking

    The backup command should acquire the site lock before:

    database dump
    file archive
    config copy

    and release it afterward.

    Use:

    trap cleanup EXIT

    to guarantee cleanup.


    64. Database and Files Consistency

    For busy WordPress sites, files can change during backup.

    For example:

    12:00
    database dump starts
    
    12:01
    new upload occurs
    
    12:02
    files archive starts

    The backup may contain:

    database:
    before upload
    
    files:
    after upload

    This is a subtle consistency problem.


    65. First Version Policy

    For the first CHP version:

    Best-effort application-consistent backup

    is acceptable.

    But record:

    backup consistency:
    BEST_EFFORT

    in the manifest.

    Don’t claim:

    ATOMIC

    unless you actually guarantee it.


    66. Future Consistent Backup

    A mature system could:

    enable maintenance mode
           ↓
    flush application state
           ↓
    database snapshot/dump
           ↓
    files snapshot
           ↓
    disable maintenance mode

    or use storage/database snapshot mechanisms.

    That is a later feature.


    67. Repair Backup vs Customer Backup

    This distinction is important.

    Customer backup

    Long-term:

    daily
    weekly
    monthly

    Repair safety backup

    Short-term:

    before repair

    Example:

    RP-000021

    The repair backup may only need to be retained for:

    7–30 days

    depending on policy.


    68. Backup Types

    Use:

    FULL
    DATABASE
    FILES
    CONFIG
    REPAIR
    PRE_RESTORE

    Example:

    BK-20260813-170001:
    FULL

    and:

    BK-20260813-170500:
    REPAIR

    69. Repair Backup Relationship

    Add:

    ALTER TABLE backups
    ADD COLUMN source_operation_id INTEGER;

    Then:

    REPAIR
      ↓
    BACKUP
      ↓
    BK-20260813-170500

    can be traced back to:

    operation:
    REPAIR

    70. Restore History

    Record:

    restore_operations

    Eventually:

    CREATE TABLE restore_operations (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        site_id INTEGER NOT NULL,
        backup_id INTEGER NOT NULL,
        status TEXT NOT NULL,
        started_at TEXT NOT NULL,
        completed_at TEXT,
        result TEXT,
        FOREIGN KEY (site_id) REFERENCES sites(id),
        FOREIGN KEY (backup_id) REFERENCES backups(id)
    );

    71. Restore Status

    Use:

    PLANNED
    VERIFYING
    BACKING_UP_CURRENT
    RESTORING
    VALIDATING
    RESTORED
    FAILED
    ROLLED_BACK

    72. Restore Output

    Successful:

    CresignSys Restore
    ==================
    
    Site:
    example.com
    
    Backup:
    BK-20260813-170001-a8f3
    
    Backup verification:
    OK
    
    Current-state safety backup:
    OK
    
    Files:
    RESTORED
    
    Database:
    RESTORED
    
    Configuration:
    RESTORED
    
    Health:
    HEALTHY
    
    Reconciliation:
    MATCH
    
    Result:
    RESTORED

    73. Failed Restore

    CresignSys Restore
    ==================
    
    Backup:
    BK-20260813-170001-a8f3
    
    Backup verification:
    OK
    
    Safety backup:
    OK
    
    Files:
    RESTORED
    
    Database:
    FAILED
    
    Rollback:
    SUCCESS
    
    Health:
    HEALTHY
    
    Result:
    ROLLED_BACK

    74. Restore Critical Failure

    Database:
    FAILED
    
    Rollback:
    FAILED
    
    Current site:
    UNKNOWN
    
    Result:
    CRITICAL
    
    Manual intervention required.

    Again, don’t hide the failure.


    75. Backup Security

    Backup directories should not be publicly accessible.

    Never place:

    /var/lib/cresignsys/backups

    under:

    /storage/websites/

    or a public Nginx document root.


    76. Backup Permissions

    Use restrictive permissions.

    For example:

    directory:
    0700
    
    sensitive database dumps:
    0600

    The exact ownership should match the CHP service architecture.


    77. Encrypt Sensitive Backups

    Eventually, customer backups should support:

    encryption at rest

    especially for:

    database dumps
    private application data
    customer uploads

    Possible architecture:

    Backup
      ↓
    Compress
      ↓
    Encrypt
      ↓
    Store

    Do not treat a plain SQL dump as safe merely because the filesystem is protected.


    78. Encryption Key Management

    Do not store:

    encryption key

    inside:

    backup directory

    Otherwise:

    backup + key

    means compromise of one location compromises both.

    Key management should be a separate CHP security subsystem.


    79. Off-Site Backups

    Eventually:

    LOCAL
      +
    REMOTE

    For example:

    Server
     ↓
    Local backup
     ↓
    Object storage

    The remote copy protects against:

    disk failure
    VM failure
    server compromise
    accidental deletion

    80. Backup Architecture

    The mature system becomes:

                       CHP BACKUP
                           │
                 ┌─────────┴─────────┐
                 ▼                   ▼
               LOCAL               REMOTE
                 │                   │
                 ▼                   ▼
            Fast Restore         Disaster Recovery

    81. Backup Status in Site Status

    Our previous:

    hosting-site-status

    can now display:

    Backups
    -------
    
    Latest:
    BK-20260813-170001
    
    Status:
    VERIFIED
    
    Age:
    2 hours
    
    Local:
    YES
    
    Remote:
    YES
    
    Overall:
    HEALTHY

    82. Backup Warning

    If:

    Latest backup:
    3 days old

    then:

    Backups:
    WARNING

    Even if:

    Website:
    HEALTHY

    83. Backup Failure

    If:

    Last backup:
    FAILED

    and no newer backup exists:

    Backups:
    CRITICAL

    This should affect the overall site status.


    84. Updated Overall Model

    We now have:

    Runtime
    Configuration
    Backups
    Resources

    Example:

    Runtime:
    HEALTHY
    
    Configuration:
    MATCH
    
    Backups:
    WARNING
    
    Resources:
    HEALTHY
    
    Overall:
    WARNING

    85. Repair Engine Integration

    The repair engine can now do:

    REPAIR PLAN
         ↓
    APPROVAL
         ↓
    PRE-REPAIR BACKUP
         ↓
    REPAIR
         ↓
    VERIFY

    This is much safer than the previous architecture.


    86. Restore Integration

    If repair fails:

    REPAIR
     ↓
    FAILED
     ↓
    ROLLBACK
     ↓
    BACKUP RESTORE
     ↓
    HEALTH CHECK
     ↓
    RECONCILE

    The backup engine therefore becomes part of the repair engine’s rollback infrastructure.


    87. Complete CHP Safety Chain

                      REPAIR
                         │
                         ▼
                    APPROVAL
                         │
                         ▼
                    SAFETY BACKUP
                         │
                         ▼
                      CHANGE
                         │
                         ▼
                    VALIDATION
                         │
                    ┌────┴────┐
                    ▼         ▼
                  PASS      FAIL
                    │         │
                    ▼         ▼
                 HEALTH    RESTORE
                    │         │
                    ▼         ▼
              RECONCILE    HEALTH
                    │         │
                    ▼         ▼
                 VERIFIED  ROLLED_BACK

    88. Lesson 087 — Core Principle

    A backup is not just:

    "some files copied somewhere."

    A CHP backup must answer:

    What site is this?
    When was it created?
    What exactly does it contain?
    Can the files be verified?
    Can the database be verified?
    What environment created it?
    Is it restorable?
    Where is it stored?
    How long should it be retained?

    The most important distinction is:

    CREATED
    ≠
    VERIFIED
    ≠
    RESTORABLE

    That distinction will prevent CHP from falsely believing it has a usable recovery point.


    Next Lesson — 088

    Build the CHP Backup Scheduler & Retention Engine

    Now that backup and restore exist, the next layer is automatic protection:

    hosting-backup-schedule

    with policies such as:

    Daily:
    02:00
    
    Weekly:
    Sunday
    
    Monthly:
    1st day

    and retention:

    7 daily
    4 weekly
    12 monthly

    The scheduler will need to handle:

    site-specific schedules
    server-wide schedules
    missed backups
    concurrent backups
    backup locks
    retention
    failed backups
    storage limits
    local + remote copies

    and, importantly, it must never delete the last known-good recovery point just to satisfy a retention count.

  • CresignSys Learn — Lesson 086

    Build the CHP Approval & Authorization Engine

    The repair engine is now capable of changing a website, but it must not decide by itself whether a repair is authorized.

    We therefore add:

    REPAIR PLAN
        ↓
    VALIDATION
        ↓
    AUTHORIZATION
        ↓
    APPROVAL
        ↓
    REPAIR EXECUTION

    This creates a clear security boundary between proposing a change and allowing the change.


    1. Why Approval Is Separate

    Consider:

    PHP:
    8.2
    
    Expected:
    8.3

    CHP can safely determine:

    DRIFT

    and generate:

    RP-000021

    But it should not assume:

    DRIFT = permission to modify

    The operator may have intentionally configured PHP 8.2.

    Therefore:

    Detection
    ≠
    Authorization

    2. Approval Architecture

                        DRIFT
                          │
                          ▼
                    REPAIR PLAN
                          │
                          ▼
                     VALIDATION
                          │
                          ▼
                    RISK ANALYSIS
                          │
                          ▼
                  APPROVAL REQUIRED
                          │
                  ┌───────┴───────┐
                  ▼               ▼
               APPROVE          REJECT
                  │               │
                  ▼               ▼
               APPROVED        CANCELLED
                  │
                  ▼
           hosting-repair

    3. New Command

    Create:

    sudo nano /usr/local/bin/hosting-repair-approve

    Usage:

    sudo hosting-repair-approve RP-000021

    Reject:

    sudo hosting-repair-reject RP-000021

    For now, approval should be an explicit action.


    4. Approval Requirements

    Before approval, verify:

    Plan exists
    Plan is VALID
    Site is MANAGED
    Plan is not expired
    Plan fingerprint is current
    Risk policy permits approval
    Operator is authorized

    Only then:

    VALID
     ↓
    APPROVED

    5. Approval Database

    Create:

    CREATE TABLE repair_approvals (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        plan_id INTEGER NOT NULL,
        operator_id INTEGER,
        decision TEXT NOT NULL,
        comment TEXT,
        approved_at TEXT NOT NULL,
    
        FOREIGN KEY (plan_id)
            REFERENCES repair_plans(id)
            ON DELETE CASCADE
    );

    6. Decision Values

    Use:

    APPROVED
    REJECTED

    Later you can add:

    DEFERRED
    EXPIRED

    but don’t need them initially.


    7. Why Keep Rejected Approvals?

    Suppose:

    RP-000021

    was rejected.

    Don’t simply delete it.

    Record:

    Plan:
    RP-000021
    
    Decision:
    REJECTED
    
    Operator:
    admin
    
    Comment:
    Keep PHP 8.2 for compatibility.
    
    Time:
    2026-08-13 17:20

    This creates an audit trail.


    8. Approval History

    A plan can potentially have multiple approval events:

    17:00
    REJECTED
    
    17:30
    new plan created
    
    18:00
    APPROVED

    Don’t overwrite history.

    Each decision should be recorded.


    9. But Only One Active Decision

    For execution, CHP should determine the latest valid decision.

    For example:

    APPROVED

    followed by:

    REJECTED

    means the plan must no longer be executable.

    The latest decision wins, subject to plan state.


    10. Approval State

    The repair plan itself can contain:

    approval_status

    For example:

    PENDING
    APPROVED
    REJECTED

    However, keep the detailed history in:

    repair_approvals

    11. Update repair_plans

    Add:

    ALTER TABLE repair_plans
    ADD COLUMN approval_status TEXT NOT NULL DEFAULT 'PENDING';

    Then:

    DRAFT
    VALID

    can have:

    approval_status:
    PENDING

    After approval:

    approval_status:
    APPROVED

    12. Don’t Confuse Plan Status and Approval Status

    These are different.

    Example:

    Plan status:
    VALID
    
    Approval:
    PENDING

    means:

    The plan is technically valid but nobody has authorized it yet.

    Another:

    Plan status:
    APPROVED

    could mean the approval has been accepted and execution may proceed.

    A cleaner model is:

    plan status:
    VALID
    
    approval status:
    APPROVED

    Then the execution engine checks both.


    13. Authorization vs Approval

    These are also different.

    Authorization

    Is this operator allowed to approve this type of repair?

    Approval

    Did this operator actually approve this specific repair?

    Therefore:

    Authorization
         ↓
    Can this user approve?
         ↓
    Approval
         ↓
    Did this user approve?

    14. Operator Roles

    Create a simple role model.

    Start with:

    VIEWER
    OPERATOR
    ADMIN

    15. Viewer

    Can:

    view sites
    view health
    view reconciliation
    view repair plans
    view operation history

    Cannot:

    approve
    execute
    rollback

    16. Operator

    Can:

    view
    create repair plans
    approve LOW/MEDIUM repairs
    execute approved repairs
    view operations

    depending on the site’s permissions.


    17. Admin

    Can:

    everything

    including:

    HIGH-risk approval
    management-state changes
    user authorization
    platform settings

    Critical operations should still require stronger controls.


    18. Risk-Based Approval

    Use:

    LOW
    MEDIUM
    HIGH
    CRITICAL

    and map them to approval requirements.

    Example:

    RiskMinimum approval
    LOWOperator
    MEDIUMOperator
    HIGHAdmin
    CRITICALManual / restricted

    19. PHP Repair

    Our current PHP configuration repair:

    Risk:
    MEDIUM

    Therefore:

    OPERATOR

    may approve it.


    20. DNS Repair

    A future DNS change:

    Risk:
    HIGH

    requires:

    ADMIN

    This prevents an ordinary operator from accidentally redirecting a production domain.


    21. Database Repair

    A database modification:

    Risk:
    CRITICAL

    should not be automatically approved through the basic operator workflow.

    Require:

    manual administrator authorization

    and eventually possibly:

    two-person approval

    22. Approval Command

    For the first CLI implementation:

    sudo hosting-repair-approve RP-000021

    The command should:

    load plan
     ↓
    verify plan
     ↓
    verify operator
     ↓
    verify risk
     ↓
    revalidate fingerprint
     ↓
    record approval
     ↓
    update plan

    23. Revalidate Before Approval

    Suppose the plan was created at:

    16:00

    and approval happens at:

    17:00

    The server may have changed.

    Therefore:

    hosting-repair-approve

    must perform another reconciliation.

    If state changed:

    PLAN STALE

    and approval must stop.


    24. Example

    Plan:

    RP-000021
    
    PHP:
    8.2 → 8.3

    Current server:

    PHP:
    8.2

    Approval:

    VALID

    But if the server now has:

    PHP:
    8.4

    then:

    Approval:
    BLOCKED
    
    Reason:
    Plan is stale.

    25. Why Approval Must Recheck

    Otherwise:

    Plan created
        ↓
    Server changes
        ↓
    Old plan approved
        ↓
    Old plan executed

    could overwrite a newer administrator decision.


    26. Approval Expiration

    Approved plans should also have an expiration.

    Example:

    Approved:
    17:00
    
    Expires:
    19:00

    If execution happens after:

    19:00

    then:

    APPROVAL EXPIRED

    Generate a fresh plan.


    27. Approval TTL

    Add:

    approval_expires_at

    to the plan.

    ALTER TABLE repair_plans
    ADD COLUMN approval_expires_at TEXT;

    28. Why Approval Expiry?

    Consider:

    Monday:
    approve PHP change

    but the operator doesn’t execute it until:

    Friday

    The server may have changed dramatically.

    An old approval should not remain valid indefinitely.


    29. Approval Comment

    Require a comment for:

    HIGH
    CRITICAL

    Example:

    Reason:
    Approved after confirming application compatibility.

    For LOW-risk operations, a comment can be optional.


    30. Approval Example

    CresignSys Repair Approval
    ==========================
    
    Plan:
    RP-000021
    
    Site:
    example.com
    
    Repair:
    PHP 8.2 → 8.3
    
    Risk:
    MEDIUM
    
    Plan:
    VALID
    
    Fingerprint:
    CURRENT
    
    Operator:
    admin
    
    Decision:
    APPROVED
    
    Expires:
    2026-08-13 19:00

    31. Rejection Example

    CresignSys Repair Approval
    ==========================
    
    Plan:
    RP-000021
    
    Decision:
    REJECTED
    
    Reason:
    Application currently depends on PHP 8.2.
    
    No changes have been made.

    Plan becomes:

    CANCELLED

    or:

    REJECTED

    depending on your state model.


    32. Better State Model

    Use:

    DRAFT
    VALID
    APPROVED
    APPLYING
    VERIFIED
    ROLLED_BACK
    FAILED
    REJECTED
    EXPIRED

    Then:

    VALID
     ↓
    APPROVED

    or:

    VALID
     ↓
    REJECTED

    33. Plan Lifecycle

    DRAFT
      │
      ▼
    VALID
      │
      ├─────────────┐
      ▼             ▼
    APPROVED      REJECTED
      │
      ▼
    APPLYING
      │
     ┌┴─────────────┐
     ▼              ▼
    VERIFIED      ROLLED_BACK
      │
      ▼
    COMPLETED

    34. Plan Expiration

    A plan can move from:

    VALID

    to:

    EXPIRED

    if it is not approved within the configured time.

    An approved plan can also expire if:

    approval_expires_at

    has passed.


    35. Don’t Reuse Expired Plans

    If:

    RP-000021

    expires:

    hosting-repair RP-000021

    must reject it.

    Correct process:

    new reconcile
     ↓
    new plan
     ↓
    new approval

    36. CLI Identity

    For the first local implementation, you may use:

    id -un

    to identify the operating-system user.

    Example:

    operator:
    root

    But this is only temporary.

    A production web control panel needs its own authenticated identity.


    37. Future User Model

    Eventually:

    users

    might contain:

    id
    username
    email
    role
    status
    created_at

    and:

    site_permissions

    could determine which sites an operator can manage.


    38. Site-Level Authorization

    An operator might be allowed to manage:

    example.com
    shop.example.com

    but only view:

    customer-site.com

    Therefore authorization needs two dimensions:

    WHO
    +
    WHICH SITE

    39. Example

    Operator:
    John
    
    Role:
    OPERATOR
    
    Permissions:
    example.com      MANAGE
    shop.example.com MANAGE
    client.com       VIEW

    John can approve:

    example.com

    but not:

    client.com

    40. Future Permission Table

    CREATE TABLE site_permissions (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        user_id INTEGER NOT NULL,
        site_id INTEGER NOT NULL,
        permission TEXT NOT NULL,
    
        FOREIGN KEY (user_id)
            REFERENCES users(id),
    
        FOREIGN KEY (site_id)
            REFERENCES sites(id)
    );

    Possible permissions:

    VIEW
    MANAGE
    APPROVE
    ADMIN

    41. Don’t Use Only Global Roles

    Avoid:

    ADMIN = can modify every site
    OPERATOR = can modify every site

    as the only authorization mechanism.

    Hosting platforms often need:

    global role
    +
    site permission

    42. Approval Check

    Conceptually:

    operator
       ↓
    global role
       ↓
    site permission
       ↓
    risk permission
       ↓
    approval allowed?

    All four must pass.


    43. Approval Decision

    For:

    RP-000021

    the engine should determine:

    Operator:
    admin
    
    Role:
    OPERATOR
    
    Site:
    example.com
    
    Permission:
    MANAGE
    
    Risk:
    MEDIUM
    
    Can approve:
    YES

    Then:

    APPROVED

    44. Unauthorized Example

    Operator:
    viewer
    
    Role:
    VIEWER
    
    Site:
    example.com
    
    Risk:
    MEDIUM
    
    Can approve:
    NO

    Output:

    Approval denied.
    
    Operator does not have permission to approve this repair.

    No state change.


    45. Audit Everything

    Every approval attempt should be logged.

    Successful:

    APPROVAL:
    APPROVED

    Denied:

    APPROVAL:
    DENIED

    Even rejected authorization attempts can be valuable security information.


    46. Audit Event

    Eventually create:

    CREATE TABLE audit_events (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        event_type TEXT NOT NULL,
        actor_id INTEGER,
        site_id INTEGER,
        plan_id INTEGER,
        result TEXT NOT NULL,
        message TEXT,
        created_at TEXT NOT NULL
    );

    Examples:

    PLAN_CREATED
    PLAN_VALIDATED
    APPROVAL_REQUESTED
    APPROVAL_GRANTED
    APPROVAL_DENIED
    REPAIR_STARTED
    REPAIR_COMPLETED
    REPAIR_ROLLED_BACK

    47. Approval and Repair Are Different Actors

    This is useful even if you initially use one operator.

    Future model:

    Operator A:
    creates plan
    
    Admin B:
    approves
    
    Repair engine:
    executes

    This provides separation of duties.


    48. Two-Person Approval

    For high-risk operations:

    Operator A:
    proposes
    
    Admin B:
    approves
    
    Admin C:
    second approval

    Then:

    HIGH-RISK REPAIR
            │
            ▼
     APPROVAL #1
            │
            ▼
     APPROVAL #2
            │
            ▼
          APPLY

    Do not implement this yet, but design the database so it remains possible.


    49. Multiple Approvals

    The existing:

    repair_approvals

    table supports multiple records:

    plan_id | operator | decision
    --------|----------|---------
    21      | 5        | APPROVED
    21      | 7        | APPROVED

    Then policy determines whether enough approvals exist.


    50. Approval Policy

    Eventually:

    risk = LOW
    required approvals = 1
    
    risk = MEDIUM
    required approvals = 1
    
    risk = HIGH
    required approvals = 2
    
    risk = CRITICAL
    manual intervention

    This should be configuration, not hard-coded everywhere.


    51. Approval Policy Table

    Eventually:

    CREATE TABLE approval_policies (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        risk_level TEXT NOT NULL,
        required_approvals INTEGER NOT NULL,
        minimum_role TEXT NOT NULL
    );

    Example:

    LOW       1  OPERATOR
    MEDIUM    1  OPERATOR
    HIGH      2  ADMIN
    CRITICAL  manual

    52. Avoid Hard-Coded Permission Logic

    Don’t scatter:

    if [[ "$RISK" == "HIGH" ]]; then ...

    through dozens of scripts.

    Use a central function:

    check_repair_authorization

    and eventually:

    get_approval_policy

    53. Authorization Library

    Create:

    sudo nano /etc/cresignsys/lib/authorization.sh

    Functions:

    get_current_operator
    get_operator_role
    check_site_permission
    get_risk_policy
    can_create_plan
    can_approve_plan
    can_execute_plan

    54. Approval Function

    Conceptually:

    can_approve_plan() {
        local operator="$1"
        local plan_id="$2"
    
        # verify identity
        # verify site permission
        # verify role
        # verify risk
        # verify plan state
        # verify approval policy
    }

    Return:

    ALLOW
    DENY

    55. Approval Is Not Execution Permission

    This distinction is important.

    An operator may be allowed to:

    APPROVE

    but the system may require:

    REPAIR EXECUTOR

    to be a restricted service account.

    The repair engine itself should run with controlled privileges.


    56. Don’t Give the Web UI Root Shell Access

    Eventually the web dashboard should not execute:

    sudo bash

    or arbitrary commands.

    Instead:

    Web UI
      ↓
    CHP API
      ↓
    Authorization
      ↓
    Structured repair request
      ↓
    Repair engine

    57. Never Accept Shell Commands From Browser

    Bad:

    {
      "command": "systemctl restart nginx"
    }

    Good:

    {
      "plan_id": "RP-000021"
    }

    The backend loads the approved plan.


    58. Approval API

    Eventually:

    POST /api/repair-plans/RP-000021/approve

    The server determines:

    current user
    plan
    risk
    site
    authorization

    The browser never tells the server:

    who is allowed

    The server determines that itself.


    59. CSRF and Session Security

    When the web interface is introduced, approval operations require:

    authenticated session
    CSRF protection
    authorization check
    audit logging

    These are not optional for write operations.


    60. CLI Safety

    For local CLI use, require explicit command:

    sudo hosting-repair-approve RP-000021

    Do not make:

    hosting-repair-plan

    automatically approve anything.


    61. No Approval Through Environment Variable

    Avoid:

    CHP_APPROVED=true hosting-repair ...

    as a production authorization mechanism.

    Approval must exist in the authoritative CHP state.


    62. No --force

    Avoid:

    hosting-repair RP-000021 --force

    in the initial system.

    A force option tends to become a bypass around safety checks.

    If an exceptional override is eventually needed, make it:

    explicit
    audited
    role-restricted
    time-limited

    63. Approval Expiration Check

    Before execution:

    current_time < approval_expires_at

    If false:

    APPROVAL EXPIRED

    Then:

    REPAIR BLOCKED

    64. Approval Fingerprint

    Store the fingerprint when approval occurs:

    approved_fingerprint

    Then the repair engine checks:

    current_fingerprint
    ==
    approved_fingerprint

    If not:

    APPROVAL INVALIDATED

    This provides another safety layer.


    65. Add to Database

    ALTER TABLE repair_plans
    ADD COLUMN approved_fingerprint TEXT;

    When approval occurs:

    approved_fingerprint = current_fingerprint

    66. Full Approval Validation

    At approval:

    Plan:
    VALID
    
    Site:
    MANAGED
    
    Operator:
    AUTHORIZED
    
    Risk:
    ALLOWED
    
    Fingerprint:
    CURRENT
    
    Approval:
    RECORDED

    At execution:

    Plan:
    APPROVED
    
    Approval:
    NOT EXPIRED
    
    Fingerprint:
    MATCHES
    
    Site:
    MANAGED
    
    Reconciliation:
    EXPECTED
    
    Proceed.

    67. Example End-to-End

    16:00
    DRIFT detected
    
    16:02
    RP-000021 created
    
    16:03
    Plan validated
    
    16:05
    Operator reviews plan
    
    16:06
    Operator approves
    
    16:06
    Approval fingerprint recorded
    
    16:07
    Repair starts
    
    16:07
    Current state rechecked
    
    16:08
    Backup created
    
    16:08
    Configuration changed
    
    16:08
    Nginx validated
    
    16:08
    Nginx reloaded
    
    16:09
    Health check passed
    
    16:09
    Reconciliation passed
    
    16:09
    Plan VERIFIED

    68. If Something Changes

    Suppose:

    16:05
    Approved
    
    16:06
    Administrator manually changes PHP
    
    16:07
    Repair starts

    The repair engine detects:

    fingerprint mismatch

    and stops:

    REPAIR BLOCKED
    PLAN STALE

    This prevents CHP from overwriting the administrator’s new configuration.


    69. Approval Is a Snapshot

    Think of approval as:

    "I authorize this exact proposed change to this exact site state."

    Not:

    "I authorize whatever this plan becomes later."

    This is why fingerprints are essential.


    70. Lesson 086 — Core Principle

    The CHP authorization model is:

    WHO
      +
    WHAT
      +
    WHICH SITE
      +
    WHICH RISK
      +
    WHICH PLAN
      +
    WHICH STATE
      +
    WHEN

    must all be validated before a repair is executed.

    The final flow becomes:

    DRIFT
     ↓
    REPAIR PLAN
     ↓
    VALIDATE
     ↓
    AUTHORIZE
     ↓
    APPROVE
     ↓
    RECHECK STATE
     ↓
    BACKUP
     ↓
    REPAIR
     ↓
    VERIFY

    This gives CHP a proper separation between observation, planning, authorization, execution, and verification.


    Next Lesson — 087

    Build the CHP Backup & Restore Engine

    The repair engine now depends on a reliable rollback mechanism.

    We will build:

    sudo hosting-backup example.com

    and:

    sudo hosting-restore example.com BACKUP-ID

    covering:

    Website files
    MySQL database
    Nginx configuration
    SSL metadata
    Backup manifest
    SHA-256 verification
    Retention
    Backup integrity
    Restore validation

    The key architecture will be:

                      BACKUP
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
           FILES                DATABASE
              │                   │
              └─────────┬─────────┘
                        ▼
                    MANIFEST
                        │
                        ▼
                   CHECKSUMS
                        │
                        ▼
                  VERIFIED BACKUP
                        │
                        ▼
                     RESTORE
                        │
                        ▼
                  VALIDATE + HEALTH

    This backup layer will become the foundation not only for customer backups, but also for safe CHP repairs and automatic rollback.

  • CresignSys Learn — Lesson 085

    Build the CHP Repair Execution Engine

    We now have:

    hosting-reconcile
            ↓
    detect drift
    
    hosting-repair-plan
            ↓
    create validated plan

    Now we build:

    sudo hosting-repair RP-000021

    This is the first CHP command that is allowed to modify a live website.

    Therefore, the design must be much stricter.


    1. Repair Execution Model

    The repair engine follows:

    PLAN
     ↓
    VERIFY
     ↓
    LOCK
     ↓
    RECONCILE
     ↓
    BACKUP
     ↓
    CHANGE
     ↓
    VALIDATE
     ↓
    RELOAD
     ↓
    HEALTH CHECK
     ↓
    RECONCILE
     ↓
    VERIFIED

    If something fails:

    CHANGE
     ↓
    FAIL
     ↓
    ROLLBACK
     ↓
    VERIFY
     ↓
    ROLLED_BACK

    2. Never Execute an Old Plan

    The first operation must be:

    Load plan

    Then:

    Is plan APPROVED?

    If not:

    REPAIR BLOCKED
    
    Reason:
    Plan has not been approved.

    3. Allowed Plan States

    Only:

    APPROVED

    can enter the execution engine.

    These must be rejected:

    DRAFT
    VALID
    FAILED
    CANCELLED
    EXPIRED
    APPLIED
    ROLLED_BACK

    4. Create the Command

    sudo nano /usr/local/bin/hosting-repair

    Start:

    #!/usr/bin/env bash
    
    set -Eeuo pipefail

    5. Load Libraries

    source /etc/cresignsys/hosting.conf
    source /etc/cresignsys/lib/common.sh
    source /etc/cresignsys/lib/logging.sh
    source /etc/cresignsys/lib/database.sh
    source /etc/cresignsys/lib/domain.sh
    source /etc/cresignsys/lib/discovery.sh
    source /etc/cresignsys/lib/reconcile.sh
    source /etc/cresignsys/lib/health.sh
    source /etc/cresignsys/lib/status.sh
    source /etc/cresignsys/lib/repair-plan.sh

    6. Require Root

    require_root

    The repair engine should never run as an ordinary user.


    7. Load the Plan

    Usage:

    sudo hosting-repair RP-000021

    Retrieve:

    plan_id
    site_id
    plan_code
    status
    risk_level
    fingerprint

    8. Verify Plan Status

    Expected:

    APPROVED

    Anything else:

    exit 1

    Output:

    Repair blocked.
    
    Plan RP-000021 is not approved.
    Current status:
    VALID

    9. Load Site

    Retrieve:

    site_id
    primary_domain
    web_root
    site_user
    application_type
    management_state

    10. Management Permission

    A site marked:

    READ_ONLY

    must not be modified.

    The repair engine should stop:

    Repair blocked.
    
    Site management state:
    READ_ONLY

    Only:

    MANAGED

    should permit modification.


    11. Why This Matters

    A discovered site may have been imported into CHP only for:

    monitoring

    without permission for:

    automatic configuration changes

    Therefore:

    DISCOVERED
    ≠
    MANAGED

    12. Acquire Site Lock

    Before touching anything:

    /var/lock/cresignsys/example.com.lock

    Use the existing CHP lock mechanism.

    The lock protects against:

    repair
    backup
    restore
    reconcile
    other site-changing operations

    running simultaneously.


    13. Lock Failure

    If another operation owns the lock:

    Repair blocked.
    
    Another operation is currently active for:
    example.com

    Do not force the lock.


    14. Record Operation

    Create an operation:

    operation:
    REPAIR
    
    site_id:
    7
    
    plan_id:
    RP-000021
    
    status:
    STARTED

    This gives the repair an audit trail before modification begins.


    15. Reconcile Again

    This is one of the most important safety checks.

    The plan was generated earlier.

    The server may have changed since then.

    Run:

    hosting-reconcile example.com

    again.


    16. Example

    Plan says:

    Current:
    8.2
    
    Expected:
    8.3

    Reconciliation now says:

    Current:
    8.4

    The plan is stale.

    Stop:

    REPAIR BLOCKED
    
    Reason:
    Current state differs from plan.

    17. Fingerprint Verification

    Compare:

    plan_fingerprint

    with:

    current_fingerprint

    If:

    same

    continue.

    If:

    different

    mark:

    STALE

    18. Never Force a Stale Plan

    Do not provide:

    hosting-repair RP-000021 --force

    in the first implementation.

    A stale plan must be regenerated.

    Correct process:

    STALE
     ↓
    new reconciliation
     ↓
    new repair plan
     ↓
    approval
     ↓
    apply

    19. Verify Preconditions Again

    For PHP 8.3 repair:

    php8.3-fpm installed
    php8.3-fpm running
    socket exists
    Nginx configuration exists
    backup directory writable

    If any prerequisite fails:

    Repair:
    BLOCKED

    20. Create Repair Backup

    Before changing anything:

    /var/lib/cresignsys/repair-backups/

    Create:

    example.com/
    └── RP-000021/

    21. Backup the Configuration

    Example:

    Original:
    /etc/nginx/sites-enabled/example.com.conf
    
    Backup:
    /var/lib/cresignsys/repair-backups/example.com/RP-000021/example.com.conf

    Verify:

    backup exists

    22. Verify Backup Hash

    Calculate:

    SHA-256(original)
    SHA-256(backup)

    They must match.

    Example:

    Original:
    abc123...
    
    Backup:
    abc123...
    
    Backup:
    VERIFIED

    Do not proceed if the hashes differ.


    23. Record Backup

    The repair journal should contain:

    Action:
    BACKUP_FILE
    
    Status:
    SUCCESS
    
    Original:
    ...
    
    Backup:
    ...
    
    SHA256:
    ...

    24. Important Rule

    Never modify the live file first and create the backup afterward.

    Wrong:

    CHANGE
     ↓
    BACKUP

    Correct:

    BACKUP
     ↓
    VERIFY BACKUP
     ↓
    CHANGE

    25. File Safety Check

    Before modification:

    Is target a regular file?

    Check:

    test -f "$TARGET"

    Reject:

    directory
    symlink
    device
    missing target

    unless that specific action explicitly supports it.


    26. Verify Path

    For Nginx configuration:

    /etc/nginx/sites-enabled/

    must be an allowed repair directory.

    Don’t allow a plan to modify:

    /etc/passwd
    /etc/shadow
    /root/.ssh/

    27. Apply First Action

    For PHP repair:

    UPDATE_NGINX_PHP_TARGET

    The repair engine should use a dedicated function:

    apply_update_nginx_php_target

    Do not execute arbitrary SQL or shell commands stored in the database.


    28. Structured Action

    The database might contain:

    action_type:
    UPDATE_NGINX_PHP_TARGET
    
    target:
    example.com
    
    old_value:
    php8.2-fpm.sock
    
    new_value:
    php8.3-fpm.sock

    The application interprets that structured action.


    29. Verify Old Value Before Replacing

    Before modifying:

    Does the file actually contain:
    php8.2-fpm.sock?

    If not:

    Repair blocked.
    
    Expected old value not found.

    This protects against overwriting unrelated changes.


    30. Never Blindly Replace

    Avoid:

    sed -i 's/php8.2/php8.3/g' file

    because it may modify unrelated configuration.

    Instead identify the exact configuration directive.

    For example:

    fastcgi_pass

    and modify only its expected target.


    31. Write Temporary File

    Don’t directly overwrite the production configuration.

    Use:

    temporary file
          ↓
    validate
          ↓
    atomic replacement

    Example conceptual path:

    /etc/nginx/sites-enabled/.example.com.conf.chp.tmp

    32. Validate Temporary Configuration

    Run:

    nginx -t

    If validation fails:

    DO NOT replace live configuration.

    Delete the temporary file.


    33. Successful Validation

    If:

    nginx -t

    succeeds:

    temporary configuration:
    VALID

    Then perform the controlled replacement.


    34. Atomic Replacement

    The goal is:

    OLD CONFIG
         │
         │
         ▼
    VALIDATED NEW CONFIG

    rather than:

    OLD CONFIG
         ↓
    PARTIALLY WRITTEN FILE

    Atomic replacement reduces the chance of leaving a corrupted configuration.


    35. Preserve File Metadata

    When replacing a configuration, preserve:

    owner
    group
    permissions

    For example:

    root:root
    0644

    The repair engine should not accidentally create:

    www-data:www-data
    0777

    36. Recalculate Hash

    After replacement:

    SHA-256(new file)

    Store it in the journal.

    Example:

    Old:
    abc123...
    
    New:
    def456...

    37. Run nginx -t Again

    Even after replacement:

    nginx -t

    must pass.

    If it fails:

    ROLLBACK

    38. Reload, Don’t Restart

    For an Nginx configuration change:

    systemctl reload nginx

    is preferable to:

    systemctl restart nginx

    because reload normally avoids unnecessarily terminating existing connections.


    39. Verify Nginx

    Immediately check:

    systemctl is-active nginx

    Expected:

    active

    If inactive:

    ROLLBACK

    40. Run Health Check

    Now:

    hosting-health example.com

    Expected:

    HTTPS:
    HEALTHY
    
    PHP:
    HEALTHY
    
    WordPress:
    HEALTHY

    41. Health Failure

    Suppose:

    HTTPS:
    DOWN
    
    HTTP:
    502

    The repair must not simply mark:

    APPLIED

    Instead:

    VERIFICATION FAILED

    then rollback.


    42. Reconcile Again

    After successful health:

    hosting-reconcile example.com

    Expected:

    PHP:
    MATCH

    This is important because health alone doesn’t prove configuration correctness.


    43. Successful Repair

    Only when:

    backup verified
    +
    change applied
    +
    nginx valid
    +
    nginx healthy
    +
    site healthy
    +
    reconciliation MATCH

    should the plan become:

    VERIFIED

    44. Full Successful Flow

    APPROVED
       ↓
    LOCK
       ↓
    RECONCILE
       ↓
    FINGERPRINT MATCH
       ↓
    BACKUP
       ↓
    BACKUP VERIFIED
       ↓
    MODIFY
       ↓
    NGINX TEST
       ↓
    RELOAD
       ↓
    HEALTH CHECK
       ↓
    RECONCILE
       ↓
    VERIFIED

    45. Failure Flow

    APPROVED
       ↓
    LOCK
       ↓
    BACKUP
       ↓
    MODIFY
       ↓
    VALIDATION FAIL
       ↓
    ROLLBACK
       ↓
    HEALTH CHECK
       ↓
    RECONCILE
       ↓
    ROLLED_BACK

    46. Rollback

    Rollback should restore the exact backed-up file.

    Verify:

    backup hash

    before restoring.

    Then:

    restore
     ↓
    nginx -t
     ↓
    reload
     ↓
    health

    47. Rollback Must Also Be Verified

    Never assume:

    restore succeeded

    means:

    website recovered

    After rollback:

    nginx -t
    hosting-health
    hosting-reconcile

    must run.


    48. Rollback Failure

    Worst-case scenario:

    Original:
    backup exists
    
    Repair:
    failed
    
    Rollback:
    failed

    Then:

    CRITICAL
    
    Site may require manual intervention.

    Do not hide this.


    49. Repair Result States

    Use:

    VERIFIED
    ROLLED_BACK
    FAILED

    Example:

    Plan:
    RP-000021
    
    Execution:
    SUCCESS
    
    Verification:
    SUCCESS
    
    Final:
    VERIFIED

    50. Failed Repair

    Plan:
    RP-000021
    
    Execution:
    FAILED
    
    Rollback:
    SUCCESS
    
    Verification:
    SUCCESS
    
    Final:
    ROLLED_BACK

    51. Critical Failure

    Plan:
    RP-000021
    
    Execution:
    FAILED
    
    Rollback:
    FAILED
    
    Verification:
    FAILED
    
    Final:
    CRITICAL

    This should trigger immediate operator attention in the future monitoring system.


    52. Operation Journal

    Create:

    CREATE TABLE repair_actions (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        plan_id INTEGER NOT NULL,
        sequence_no INTEGER NOT NULL,
        action_type TEXT NOT NULL,
        target TEXT,
        status TEXT NOT NULL,
        started_at TEXT NOT NULL,
        completed_at TEXT,
        message TEXT,
        old_hash TEXT,
        new_hash TEXT,
        FOREIGN KEY (plan_id)
            REFERENCES repair_plans(id)
            ON DELETE CASCADE
    );

    53. Why Action History?

    Suppose a repair fails.

    You need:

    Action 1:
    SUCCESS
    
    Action 2:
    SUCCESS
    
    Action 3:
    FAILED
    
    Rollback:
    SUCCESS

    This is much more useful than:

    Repair failed.

    54. Update Repair Plan

    During execution:

    APPROVED

    becomes:

    APPLYING

    Then:

    VERIFIED

    or:

    ROLLED_BACK

    or:

    FAILED

    55. Update Operation

    The operation record can contain:

    operation:
    REPAIR
    
    status:
    SUCCESS
    
    result:
    VERIFIED

    or:

    status:
    FAILED
    
    result:
    ROLLED_BACK

    56. Don’t Confuse Operation Status

    Example:

    Operation:
    SUCCESS
    
    Repair result:
    ROLLED_BACK

    This means:

    The repair engine executed correctly, detected a failed verification, and successfully restored the original state.

    That is different from:

    Operation:
    FAILED
    
    Repair result:
    FAILED

    where the repair process itself encountered an unrecoverable problem.


    57. Repair Command Output

    A successful repair:

    CresignSys Repair
    =================
    
    Plan:
    RP-000021
    
    Site:
    example.com
    
    [1/6] Verify plan             OK
    [2/6] Acquire lock            OK
    [3/6] Reconcile               OK
    [4/6] Backup                  OK
    [5/6] Apply changes           OK
    [6/6] Verify                  OK
    
    Configuration:
    MATCH
    
    Health:
    HEALTHY
    
    Result:
    VERIFIED

    58. Rolled-Back Output

    CresignSys Repair
    =================
    
    Plan:
    RP-000021
    
    [1/6] Verify plan             OK
    [2/6] Acquire lock            OK
    [3/6] Reconcile               OK
    [4/6] Backup                  OK
    [5/6] Apply changes           OK
    [6/6] Verify                  FAILED
    
    Reason:
    HTTP 502
    
    Rollback:
    SUCCESS
    
    Post-rollback health:
    HEALTHY
    
    Result:
    ROLLED_BACK

    59. Critical Output

    CresignSys Repair
    =================
    
    Plan:
    RP-000021
    
    Repair:
    FAILED
    
    Rollback:
    FAILED
    
    Current site state:
    UNKNOWN
    
    ACTION REQUIRED:
    Manual investigation required.

    This is the correct place to stop.

    Do not attempt endless automatic repairs.


    60. Maximum Repair Attempts

    A repair plan should execute only once.

    Don’t do:

    retry forever

    If it fails:

    FAILED

    or:

    ROLLED_BACK

    Generate a new plan after investigation.


    61. No Automatic Repair Loops

    Never build:

    health DOWN
     ↓
    repair
     ↓
    health DOWN
     ↓
    repair
     ↓
    health DOWN
     ↓
    repair

    This can create destructive behavior.

    Instead:

    failure
     ↓
    stop
     ↓
    record
     ↓
    alert

    62. Site Lock Release

    The lock must always be released.

    Use Bash cleanup:

    cleanup() {
        release_site_lock "$DOMAIN"
    }
    
    trap cleanup EXIT

    This is essential.

    Even if the script fails unexpectedly:

    lock

    should not remain indefinitely.


    63. Signals

    Handle:

    SIGINT
    SIGTERM
    EXIT

    A controlled shutdown should:

    stop new actions
    record interruption
    attempt safe rollback if required
    release lock

    Do not simply kill the process halfway through a modification.


    64. Repair Journal as State Machine

    ACTION START
         ↓
    RUNNING
         │
     ┌───┴────┐
     ▼        ▼
    SUCCESS   FAILED
     │          │
     ▼          ▼
    NEXT      ROLLBACK

    Every action has a known state.


    65. Never Skip the Backup

    Even if an action seems harmless:

    configuration modification

    the repair engine should require a rollback strategy.

    If no rollback is possible:

    Risk:
    HIGH

    and manual approval should be required.


    66. Repairing Files vs Services

    Not every repair needs the same rollback.

    File change

    backup file
    modify
    restore file

    Service state

    record previous state
    change service
    restore previous state

    Database change

    much higher risk

    Database repair should not be included in this first engine.


    67. First Supported Repair

    Keep Lesson 085 limited to:

    PHP-FPM target in Nginx configuration

    This gives us one complete vertical slice:

    DRIFT
     ↓
    PLAN
     ↓
    APPROVAL
     ↓
    BACKUP
     ↓
    CHANGE
     ↓
    VALIDATE
     ↓
    RELOAD
     ↓
    HEALTH
     ↓
    RECONCILE
     ↓
    VERIFIED

    68. Do Not Add These Yet

    Avoid implementing:

    automatic WordPress updates
    automatic plugin updates
    database repairs
    DNS changes
    SSL issuance
    recursive ownership repair
    package installation
    OS upgrades

    Those need separate safety models.


    69. The First Complete CHP Automation

    After this lesson, CHP can conceptually do:

    Site discovered
          ↓
    Site registered
          ↓
    Configuration checked
          ↓
    Drift detected
          ↓
    Repair plan created
          ↓
    Plan validated
          ↓
    Operator approves
          ↓
    Backup created
          ↓
    Repair applied
          ↓
    Health checked
          ↓
    Configuration reconciled
          ↓
    Repair VERIFIED

    This is the foundation of a real hosting control plane.


    70. Final Architecture

                        CRESIGNSYS CHP
                              │
            ┌─────────────────┼─────────────────┐
            ▼                 ▼                 ▼
        DISCOVERY        OBSERVATION          CONTROL
            │                 │                 │
            ▼                 ▼                 ▼
       DB IMPORT        RECONCILIATION      REPAIR PLAN
                             │                 │
                             ▼                 ▼
                           HEALTH          APPROVAL
                                               │
                                               ▼
                                             REPAIR
                                               │
                                   ┌───────────┴───────────┐
                                   ▼                       ▼
                                SUCCESS                  FAILURE
                                   │                       │
                                   ▼                       ▼
                               VERIFY                  ROLLBACK
                                   │                       │
                                   └───────────┬───────────┘
                                               ▼
                                          FINAL STATUS

    71. Lesson 085 — Core Principle

    The repair engine should behave like a transaction coordinator, even though Linux configuration changes are not automatically transactional.

    The safe pattern is:

    VERIFY
    → LOCK
    → RECHECK
    → BACKUP
    → CHANGE
    → VALIDATE
    → RELOAD
    → HEALTH CHECK
    → RECONCILE
    → VERIFY

    If validation fails:

    ROLLBACK
    → HEALTH CHECK
    → RECONCILE
    → REPORT

    The next major component is therefore not another repair type. It is the CHP approval and authorization layer, which determines who is allowed to approve a repair, what level of approval is required, and how that approval is recorded before the repair engine is allowed to execute.