Build the CHP Repair Execution Engine

We now have:

hosting-reconcile
        ↓
detect drift

hosting-repair-plan
        ↓
create validated plan

Now we build:

sudo hosting-repair RP-000021

This is the first CHP command that is allowed to modify a live website.

Therefore, the design must be much stricter.

---

1. Repair Execution Model

The repair engine follows:

PLAN
 ↓
VERIFY
 ↓
LOCK
 ↓
RECONCILE
 ↓
BACKUP
 ↓
CHANGE
 ↓
VALIDATE
 ↓
RELOAD
 ↓
HEALTH CHECK
 ↓
RECONCILE
 ↓
VERIFIED

If something fails:

CHANGE
 ↓
FAIL
 ↓
ROLLBACK
 ↓
VERIFY
 ↓
ROLLED_BACK

---

2. Never Execute an Old Plan

The first operation must be:

Load plan

Then:

Is plan APPROVED?

If not:

REPAIR BLOCKED

Reason:
Plan has not been approved.

---

3. Allowed Plan States

Only:

APPROVED

can enter the execution engine.

These must be rejected:

DRAFT
VALID
FAILED
CANCELLED
EXPIRED
APPLIED
ROLLED_BACK

---

4. Create the Command

sudo nano /usr/local/bin/hosting-repair

Start:

#!/usr/bin/env bash

set -Eeuo pipefail

---

5. Load Libraries

source /etc/cresignsys/hosting.conf
source /etc/cresignsys/lib/common.sh
source /etc/cresignsys/lib/logging.sh
source /etc/cresignsys/lib/database.sh
source /etc/cresignsys/lib/domain.sh
source /etc/cresignsys/lib/discovery.sh
source /etc/cresignsys/lib/reconcile.sh
source /etc/cresignsys/lib/health.sh
source /etc/cresignsys/lib/status.sh
source /etc/cresignsys/lib/repair-plan.sh

---

6. Require Root

require_root

The repair engine should never run as an ordinary user.

---

7. Load the Plan

Usage:

sudo hosting-repair RP-000021

Retrieve:

plan_id
site_id
plan_code
status
risk_level
fingerprint

---

8. Verify Plan Status

Expected:

APPROVED

Anything else:

exit 1

Output:

Repair blocked.

Plan RP-000021 is not approved.
Current status:
VALID

---

9. Load Site

Retrieve:

site_id
primary_domain
web_root
site_user
application_type
management_state

---

10. Management Permission

A site marked:

READ_ONLY

must not be modified.

The repair engine should stop:

Repair blocked.

Site management state:
READ_ONLY

Only:

MANAGED

should permit modification.

---

11. Why This Matters

A discovered site may have been imported into CHP only for:

monitoring

without permission for:

automatic configuration changes

Therefore:

DISCOVERED
≠
MANAGED

---

12. Acquire Site Lock

Before touching anything:

/var/lock/cresignsys/example.com.lock

Use the existing CHP lock mechanism.

The lock protects against:

repair
backup
restore
reconcile
other site-changing operations

running simultaneously.

---

13. Lock Failure

If another operation owns the lock:

Repair blocked.

Another operation is currently active for:
example.com

Do not force the lock.

---

14. Record Operation

Create an operation:

operation:
REPAIR

site_id:
7

plan_id:
RP-000021

status:
STARTED

This gives the repair an audit trail before modification begins.

---

15. Reconcile Again

This is one of the most important safety checks.

The plan was generated earlier.

The server may have changed since then.

Run:

hosting-reconcile example.com

again.

---

16. Example

Plan says:

Current:
8.2

Expected:
8.3

Reconciliation now says:

Current:
8.4

The plan is stale.

Stop:

REPAIR BLOCKED

Reason:
Current state differs from plan.

---

17. Fingerprint Verification

Compare:

plan_fingerprint

with:

current_fingerprint

If:

same

continue.

If:

different

mark:

STALE

---

18. Never Force a Stale Plan

Do not provide:

hosting-repair RP-000021 --force

in the first implementation.

A stale plan must be regenerated.

Correct process:

STALE
 ↓
new reconciliation
 ↓
new repair plan
 ↓
approval
 ↓
apply

---

19. Verify Preconditions Again

For PHP 8.3 repair:

php8.3-fpm installed
php8.3-fpm running
socket exists
Nginx configuration exists
backup directory writable

If any prerequisite fails:

Repair:
BLOCKED

---

20. Create Repair Backup

Before changing anything:

/var/lib/cresignsys/repair-backups/

Create:

example.com/
└── RP-000021/

---

21. Backup the Configuration

Example:

Original:
/etc/nginx/sites-enabled/example.com.conf

Backup:
/var/lib/cresignsys/repair-backups/example.com/RP-000021/example.com.conf

Verify:

backup exists

---

22. Verify Backup Hash

Calculate:

SHA-256(original)
SHA-256(backup)

They must match.

Example:

Original:
abc123...

Backup:
abc123...

Backup:
VERIFIED

Do not proceed if the hashes differ.

---

23. Record Backup

The repair journal should contain:

Action:
BACKUP_FILE

Status:
SUCCESS

Original:
...

Backup:
...

SHA256:
...

---

24. Important Rule

Never modify the live file first and create the backup afterward.

Wrong:

CHANGE
 ↓
BACKUP

Correct:

BACKUP
 ↓
VERIFY BACKUP
 ↓
CHANGE

---

25. File Safety Check

Before modification:

Is target a regular file?

Check:

test -f "$TARGET"

Reject:

directory
symlink
device
missing target

unless that specific action explicitly supports it.

---

26. Verify Path

For Nginx configuration:

/etc/nginx/sites-enabled/

must be an allowed repair directory.

Don’t allow a plan to modify:

/etc/passwd
/etc/shadow
/root/.ssh/

---

27. Apply First Action

For PHP repair:

UPDATE_NGINX_PHP_TARGET

The repair engine should use a dedicated function:

apply_update_nginx_php_target

Do not execute arbitrary SQL or shell commands stored in the database.

---

28. Structured Action

The database might contain:

action_type:
UPDATE_NGINX_PHP_TARGET

target:
example.com

old_value:
php8.2-fpm.sock

new_value:
php8.3-fpm.sock

The application interprets that structured action.

---

29. Verify Old Value Before Replacing

Before modifying:

Does the file actually contain:
php8.2-fpm.sock?

If not:

Repair blocked.

Expected old value not found.

This protects against overwriting unrelated changes.

---

30. Never Blindly Replace

Avoid:

sed -i 's/php8.2/php8.3/g' file

because it may modify unrelated configuration.

Instead identify the exact configuration directive.

For example:

fastcgi_pass

and modify only its expected target.

---

31. Write Temporary File

Don’t directly overwrite the production configuration.

Use:

temporary file
      ↓
validate
      ↓
atomic replacement

Example conceptual path:

/etc/nginx/sites-enabled/.example.com.conf.chp.tmp

---

32. Validate Temporary Configuration

Run:

nginx -t

If validation fails:

DO NOT replace live configuration.

Delete the temporary file.

---

33. Successful Validation

If:

nginx -t

succeeds:

temporary configuration:
VALID

Then perform the controlled replacement.

---

34. Atomic Replacement

The goal is:

OLD CONFIG
     │
     │
     ▼
VALIDATED NEW CONFIG

rather than:

OLD CONFIG
     ↓
PARTIALLY WRITTEN FILE

Atomic replacement reduces the chance of leaving a corrupted configuration.

---

35. Preserve File Metadata

When replacing a configuration, preserve:

owner
group
permissions

For example:

root:root
0644

The repair engine should not accidentally create:

www-data:www-data
0777

---

36. Recalculate Hash

After replacement:

SHA-256(new file)

Store it in the journal.

Example:

Old:
abc123...

New:
def456...

---

37. Run nginx -t Again

Even after replacement:

nginx -t

must pass.

If it fails:

ROLLBACK

---

38. Reload, Don’t Restart

For an Nginx configuration change:

systemctl reload nginx

is preferable to:

systemctl restart nginx

because reload normally avoids unnecessarily terminating existing connections.

---

39. Verify Nginx

Immediately check:

systemctl is-active nginx

Expected:

active

If inactive:

ROLLBACK

---

40. Run Health Check

Now:

hosting-health example.com

Expected:

HTTPS:
HEALTHY

PHP:
HEALTHY

WordPress:
HEALTHY

---

41. Health Failure

Suppose:

HTTPS:
DOWN

HTTP:
502

The repair must not simply mark:

APPLIED

Instead:

VERIFICATION FAILED

then rollback.

---

42. Reconcile Again

After successful health:

hosting-reconcile example.com

Expected:

PHP:
MATCH

This is important because health alone doesn’t prove configuration correctness.

---

43. Successful Repair

Only when:

backup verified
+
change applied
+
nginx valid
+
nginx healthy
+
site healthy
+
reconciliation MATCH

should the plan become:

VERIFIED

---

44. Full Successful Flow

APPROVED
   ↓
LOCK
   ↓
RECONCILE
   ↓
FINGERPRINT MATCH
   ↓
BACKUP
   ↓
BACKUP VERIFIED
   ↓
MODIFY
   ↓
NGINX TEST
   ↓
RELOAD
   ↓
HEALTH CHECK
   ↓
RECONCILE
   ↓
VERIFIED

---

45. Failure Flow

APPROVED
   ↓
LOCK
   ↓
BACKUP
   ↓
MODIFY
   ↓
VALIDATION FAIL
   ↓
ROLLBACK
   ↓
HEALTH CHECK
   ↓
RECONCILE
   ↓
ROLLED_BACK

---

46. Rollback

Rollback should restore the exact backed-up file.

Verify:

backup hash

before restoring.

Then:

restore
 ↓
nginx -t
 ↓
reload
 ↓
health

---

47. Rollback Must Also Be Verified

Never assume:

restore succeeded

means:

website recovered

After rollback:

nginx -t
hosting-health
hosting-reconcile

must run.

---

48. Rollback Failure

Worst-case scenario:

Original:
backup exists

Repair:
failed

Rollback:
failed

Then:

CRITICAL

Site may require manual intervention.

Do not hide this.

---

49. Repair Result States

Use:

VERIFIED
ROLLED_BACK
FAILED

Example:

Plan:
RP-000021

Execution:
SUCCESS

Verification:
SUCCESS

Final:
VERIFIED

---

50. Failed Repair

Plan:
RP-000021

Execution:
FAILED

Rollback:
SUCCESS

Verification:
SUCCESS

Final:
ROLLED_BACK

---

51. Critical Failure

Plan:
RP-000021

Execution:
FAILED

Rollback:
FAILED

Verification:
FAILED

Final:
CRITICAL

This should trigger immediate operator attention in the future monitoring system.

---

52. Operation Journal

Create:

CREATE TABLE repair_actions (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    plan_id INTEGER NOT NULL,
    sequence_no INTEGER NOT NULL,
    action_type TEXT NOT NULL,
    target TEXT,
    status TEXT NOT NULL,
    started_at TEXT NOT NULL,
    completed_at TEXT,
    message TEXT,
    old_hash TEXT,
    new_hash TEXT,
    FOREIGN KEY (plan_id)
        REFERENCES repair_plans(id)
        ON DELETE CASCADE
);

---

53. Why Action History?

Suppose a repair fails.

You need:

Action 1:
SUCCESS

Action 2:
SUCCESS

Action 3:
FAILED

Rollback:
SUCCESS

This is much more useful than:

Repair failed.

---

54. Update Repair Plan

During execution:

APPROVED

becomes:

APPLYING

Then:

VERIFIED

or:

ROLLED_BACK

or:

FAILED

---

55. Update Operation

The operation record can contain:

operation:
REPAIR

status:
SUCCESS

result:
VERIFIED

or:

status:
FAILED

result:
ROLLED_BACK

---

56. Don’t Confuse Operation Status

Example:

Operation:
SUCCESS

Repair result:
ROLLED_BACK

This means:

The repair engine executed correctly, detected a failed verification, and successfully restored the original state.

That is different from:

Operation:
FAILED

Repair result:
FAILED

where the repair process itself encountered an unrecoverable problem.

---

57. Repair Command Output

A successful repair:

CresignSys Repair
=================

Plan:
RP-000021

Site:
example.com

[1/6] Verify plan             OK
[2/6] Acquire lock            OK
[3/6] Reconcile               OK
[4/6] Backup                  OK
[5/6] Apply changes           OK
[6/6] Verify                  OK

Configuration:
MATCH

Health:
HEALTHY

Result:
VERIFIED

---

58. Rolled-Back Output

CresignSys Repair
=================

Plan:
RP-000021

[1/6] Verify plan             OK
[2/6] Acquire lock            OK
[3/6] Reconcile               OK
[4/6] Backup                  OK
[5/6] Apply changes           OK
[6/6] Verify                  FAILED

Reason:
HTTP 502

Rollback:
SUCCESS

Post-rollback health:
HEALTHY

Result:
ROLLED_BACK

---

59. Critical Output

CresignSys Repair
=================

Plan:
RP-000021

Repair:
FAILED

Rollback:
FAILED

Current site state:
UNKNOWN

ACTION REQUIRED:
Manual investigation required.

This is the correct place to stop.

Do not attempt endless automatic repairs.

---

60. Maximum Repair Attempts

A repair plan should execute only once.

Don’t do:

retry forever

If it fails:

FAILED

or:

ROLLED_BACK

Generate a new plan after investigation.

---

61. No Automatic Repair Loops

Never build:

health DOWN
 ↓
repair
 ↓
health DOWN
 ↓
repair
 ↓
health DOWN
 ↓
repair

This can create destructive behavior.

Instead:

failure
 ↓
stop
 ↓
record
 ↓
alert

---

62. Site Lock Release

The lock must always be released.

Use Bash cleanup:

cleanup() {
    release_site_lock "$DOMAIN"
}

trap cleanup EXIT

This is essential.

Even if the script fails unexpectedly:

lock

should not remain indefinitely.

---

63. Signals

Handle:

SIGINT
SIGTERM
EXIT

A controlled shutdown should:

stop new actions
record interruption
attempt safe rollback if required
release lock

Do not simply kill the process halfway through a modification.

---

64. Repair Journal as State Machine

ACTION START
     ↓
RUNNING
     │
 ┌───┴────┐
 ▼        ▼
SUCCESS   FAILED
 │          │
 ▼          ▼
NEXT      ROLLBACK

Every action has a known state.

---

65. Never Skip the Backup

Even if an action seems harmless:

configuration modification

the repair engine should require a rollback strategy.

If no rollback is possible:

Risk:
HIGH

and manual approval should be required.

---

66. Repairing Files vs Services

Not every repair needs the same rollback.

File change

backup file
modify
restore file

Service state

record previous state
change service
restore previous state

Database change

much higher risk

Database repair should not be included in this first engine.

---

67. First Supported Repair

Keep Lesson 085 limited to:

PHP-FPM target in Nginx configuration

This gives us one complete vertical slice:

DRIFT
 ↓
PLAN
 ↓
APPROVAL
 ↓
BACKUP
 ↓
CHANGE
 ↓
VALIDATE
 ↓
RELOAD
 ↓
HEALTH
 ↓
RECONCILE
 ↓
VERIFIED

---

68. Do Not Add These Yet

Avoid implementing:

automatic WordPress updates
automatic plugin updates
database repairs
DNS changes
SSL issuance
recursive ownership repair
package installation
OS upgrades

Those need separate safety models.

---

69. The First Complete CHP Automation

After this lesson, CHP can conceptually do:

Site discovered
      ↓
Site registered
      ↓
Configuration checked
      ↓
Drift detected
      ↓
Repair plan created
      ↓
Plan validated
      ↓
Operator approves
      ↓
Backup created
      ↓
Repair applied
      ↓
Health checked
      ↓
Configuration reconciled
      ↓
Repair VERIFIED

This is the foundation of a real hosting control plane.

---

70. Final Architecture

                    CRESIGNSYS CHP
                          │
        ┌─────────────────┼─────────────────┐
        ▼                 ▼                 ▼
    DISCOVERY        OBSERVATION          CONTROL
        │                 │                 │
        ▼                 ▼                 ▼
   DB IMPORT        RECONCILIATION      REPAIR PLAN
                         │                 │
                         ▼                 ▼
                       HEALTH          APPROVAL
                                           │
                                           ▼
                                         REPAIR
                                           │
                               ┌───────────┴───────────┐
                               ▼                       ▼
                            SUCCESS                  FAILURE
                               │                       │
                               ▼                       ▼
                           VERIFY                  ROLLBACK
                               │                       │
                               └───────────┬───────────┘
                                           ▼
                                      FINAL STATUS

---

71. Lesson 085 — Core Principle

The repair engine should behave like a transaction coordinator, even though Linux configuration changes are not automatically transactional.

The safe pattern is:

VERIFY
→ LOCK
→ RECHECK
→ BACKUP
→ CHANGE
→ VALIDATE
→ RELOAD
→ HEALTH CHECK
→ RECONCILE
→ VERIFY

If validation fails:

ROLLBACK
→ HEALTH CHECK
→ RECONCILE
→ REPORT

The next major component is therefore not another repair type. It is the CHP approval and authorization layer, which determines who is allowed to approve a repair, what level of approval is required, and how that approval is recorded before the repair engine is allowed to execute.