Build the CHP Central Job Queue
CHP now has several operations:
BACKUP
RESTORE
REPAIR
HEALTH CHECK
RECONCILIATION
RETENTION
If every command executes independently, conflicts will eventually occur.
For example:
BACKUP example.com
+
REPAIR example.com
could run simultaneously.
That is exactly what we want to prevent.
The next layer is therefore:
CHP JOB ENGINE
│
┌───────────┼───────────┐
▼ ▼ ▼
BACKUP REPAIR RESTORE
│ │ │
└───────────┼───────────┘
▼
QUEUE
│
▼
WORKER
│
┌───────────┴───────────┐
▼ ▼
SUCCESS FAILURE
1. What the Job Queue Does
Instead of every command immediately performing work:
hosting-backup
hosting-repair
hosting-restore
they submit a job.
Example:
hosting-backup example.com
↓
CREATE JOB
↓
QUEUED
↓
WORKER
↓
RUNNING
↓
SUCCESS
2. Job States
Create a standard state machine:
QUEUED
↓
RUNNING
↓
SUCCESS
Failure:
RUNNING
↓
FAILED
Cancellation:
QUEUED
↓
CANCELLED
3. Job Types
Use:
BACKUP
RESTORE
REPAIR
HEALTH
RECONCILE
RETENTION
Later:
SSL_RENEW
DNS_CHECK
WORDPRESS_UPDATE
PACKAGE_UPDATE
can be added without redesigning the queue.
4. Create the Job Table
Create:
CREATE TABLE jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_code TEXT NOT NULL UNIQUE,
site_id INTEGER,
job_type TEXT NOT NULL,
priority TEXT NOT NULL DEFAULT 'NORMAL',
status TEXT NOT NULL,
payload TEXT,
created_at TEXT NOT NULL,
scheduled_at TEXT,
started_at TEXT,
completed_at TEXT,
worker_id TEXT,
attempt INTEGER NOT NULL DEFAULT 0,
error_message TEXT,
FOREIGN KEY (site_id)
REFERENCES sites(id)
ON DELETE SET NULL
);
5. Why payload?
The job needs parameters.
For example:
{
"domain": "example.com"
}
A restore job might contain:
{
"domain": "example.com",
"backup_id": "BK-20260813-170001"
}
A repair job:
{
"plan_id": "RP-000021"
}
6. Never Put Arbitrary Commands in Payload
Bad:
{
"command": "rm -rf /storage/websites/example.com"
}
Good:
{
"job_type": "BACKUP",
"site_id": 7
}
The worker interprets structured data.
7. Job Code
Every job gets a unique code:
JOB-000001
JOB-000002
JOB-000003
This is easier for operators than:
database row 281
8. Job Example
JOB-000241
Type:
BACKUP
Site:
example.com
Priority:
NORMAL
Status:
QUEUED
Created:
2026-08-13 18:00
9. Job Priority
Use:
CRITICAL
HIGH
NORMAL
LOW
Examples:
PRE-RESTORE:
CRITICAL
RESTORE:
CRITICAL
REPAIR:
HIGH
HEALTH:
NORMAL
BACKUP:
NORMAL
RETENTION:
LOW
10. Why Priority Matters
Suppose:
20 backups queued
and an operator needs an urgent restore.
The restore shouldn’t wait behind all 20 normal backups.
Instead:
RESTORE
CRITICAL
moves ahead of:
BACKUP
NORMAL
11. But Priority Alone Is Not Enough
Suppose:
BACKUP example.com
RUNNING
and:
RESTORE example.com
CRITICAL
The restore must not simply start.
The queue also needs resource and site conflict rules.
12. Job Conflict Model
Define operation relationships.
For example:
| Existing | Requested | Result |
|---|---|---|
| BACKUP | BACKUP | Queue |
| BACKUP | REPAIR | Wait |
| BACKUP | RESTORE | Wait |
| REPAIR | REPAIR | Wait |
| REPAIR | RESTORE | Wait |
| RESTORE | BACKUP | Wait |
| HEALTH | BACKUP | Usually allowed |
| HEALTH | REPAIR | Usually allowed |
| RECONCILE | REPAIR | Queue/wait |
The exact policy can evolve.
13. Site Lock Remains
The central queue does not replace the site lock.
You need both:
JOB QUEUE
↓
site conflict check
↓
SITE LOCK
↓
execution
The queue prevents jobs from unnecessarily competing.
The lock provides the final runtime protection.
14. Why Both?
Imagine two separate processes:
Worker A:
repair example.com
Worker B:
restore example.com
Both query the database at nearly the same moment.
Both may conclude:
No conflicting job.
The database-level queue logic alone may not be enough.
The site lock provides the final serialization point.
15. Job Worker
Create:
sudo nano /usr/local/bin/hosting-worker
Its job is:
find next job
↓
claim job
↓
check conflicts
↓
acquire site lock
↓
execute
↓
record result
↓
release lock
16. Worker Identity
Each worker should have an identity:
worker-01
Store:
worker_id
in the job record.
This allows the system to answer:
Which worker is processing this job?
17. Claiming a Job
This is a concurrency problem.
Two workers could see:
JOB-000241
QUEUED
simultaneously.
Both might attempt to execute it.
The worker therefore needs an atomic claim.
18. Atomic Job Claim
Conceptually:
UPDATE jobs
SET
status = 'RUNNING',
worker_id = ?,
started_at = ?,
attempt = attempt + 1
WHERE id = ?
AND status = 'QUEUED';
Then check:
rows affected == 1
If:
0
another worker already claimed the job.
19. Why This Matters
Without atomic claiming:
Worker A:
sees QUEUED
Worker B:
sees QUEUED
Worker A:
runs
Worker B:
also runs
This can cause duplicate backups or dangerous duplicate repairs.
20. SQLite Concurrency
Your current CHP database is SQLite.
SQLite can handle multiple readers well, but writes need careful transaction handling.
For job claiming:
BEGIN IMMEDIATE
can be useful to serialize the short claim transaction.
Keep transactions short.
Don’t hold a database transaction while:
backup
restore
repair
is running.
21. Important Rule
Never do:
BEGIN TRANSACTION
backup files for 20 minutes
COMMIT
That holds the database transaction far too long.
Instead:
BEGIN
claim job
COMMIT
run backup
BEGIN
record result
COMMIT
22. Worker Loop
Conceptually:
while true:
job = find_next_queued_job()
if no job:
sleep
claim(job)
if conflict:
release/requeue
execute(job)
record_result()
23. Don’t Busy-Loop
Avoid:
while true; do
query database
done
with no delay.
Use a small sleep:
sleep 2
or use a service/queue notification mechanism later.
24. Worker as systemd Service
Create:
sudo nano /etc/systemd/system/cresignsys-worker.service
Conceptually:
[Unit]
Description=CresignSys Job Worker
After=network-online.target
[Service]
Type=simple
ExecStart=/usr/local/bin/hosting-worker
Restart=always
RestartSec=5
25. Start Worker
Eventually:
sudo systemctl daemon-reload
sudo systemctl enable --now cresignsys-worker
Then:
systemctl status cresignsys-worker
26. Multiple Workers
Initially:
1 worker
Later:
worker-01
worker-02
worker-03
This allows parallel processing.
But don’t increase concurrency without understanding:
disk I/O
CPU
RAM
MySQL
network
27. Worker Capacity
Suppose the server has:
8 CPU cores
16 GB RAM
You might eventually permit:
2 backup workers
rather than 8.
Backup operations can be heavily disk-bound.
28. Site-Level Concurrency
Even with multiple workers:
example.com:
max concurrent mutating jobs = 1
while:
example.com:
health checks
may run separately.
29. Operation Classes
Define:
READ_ONLY
and:
MUTATING
Read-only
HEALTH
RECONCILE
Mutating
BACKUP
RESTORE
REPAIR
RETENTION
This classification helps determine conflicts.
30. But Backup Is Special
Backup doesn’t normally modify the website, but it writes heavily to storage.
Therefore it is:
site_read_only:
YES
resource_heavy:
YES
This distinction is useful.
31. Job Resource Profile
Each job can have:
site_lock:
YES/NO
disk_heavy:
YES/NO
database_heavy:
YES/NO
network_heavy:
YES/NO
mutating:
YES/NO
For example:
BACKUP:
site_lock YES
disk_heavy YES
database_heavy YES
mutating NO
32. Repair
REPAIR:
site_lock YES
disk_heavy LOW
database_heavy NO
mutating YES
33. Restore
RESTORE:
site_lock YES
disk_heavy HIGH
database_heavy HIGH
mutating YES
34. Health Check
HEALTH:
site_lock NO
disk_heavy NO
database_heavy LOW
mutating NO
This lets CHP make smarter scheduling decisions.
35. Job Handler Architecture
Don’t build one giant script:
hosting-worker
└── 3000 lines
Instead:
hosting-worker
│
├── handle-backup
├── handle-repair
├── handle-restore
├── handle-health
├── handle-reconcile
└── handle-retention
36. Job Dispatcher
Conceptually:
case "$JOB_TYPE" in
BACKUP)
handle_backup
;;
REPAIR)
handle_repair
;;
RESTORE)
handle_restore
;;
HEALTH)
handle_health
;;
RECONCILE)
handle_reconcile
;;
RETENTION)
handle_retention
;;
*)
fail_job "Unknown job type"
;;
esac
37. Unknown Job Type
Never silently ignore it.
Output:
JOB FAILED
Unknown job type:
WORDPRESS_UPDATE
until the handler exists.
38. Job Retry
Some jobs can safely retry.
For example:
HEALTH:
YES
BACKUP:
YES, limited
REPAIR:
NO automatic retry
RESTORE:
NO automatic retry
This is critical.
39. Why Repair Should Not Auto-Retry
Suppose repair did:
change configuration
↓
health failure
↓
rollback
Automatically retrying could repeat the same failure.
Instead:
FAILED / ROLLED_BACK
and require investigation.
40. Restore Should Not Auto-Retry
A restore may be partially destructive.
Never:
restore failed
↓
retry automatically
without analyzing the failure.
Mark:
FAILED
and stop.
41. Health Retry
Health checks can safely retry transient network conditions.
Example:
attempt 1:
HTTP timeout
attempt 2:
HTTP timeout
attempt 3:
HTTP 200
Result:
HEALTHY
42. Backup Retry
A backup can have:
attempt 1:
database timeout
attempt 2:
SUCCESS
But use a limited number of attempts.
43. Retry Configuration
Eventually:
HEALTH:
3 attempts
BACKUP:
2 attempts
REPAIR:
0
RESTORE:
0
Store this as policy rather than scattering values across scripts.
44. Job Timeout
Every job should have a maximum runtime.
Examples:
HEALTH:
2 minutes
RECONCILE:
5 minutes
BACKUP:
2 hours
REPAIR:
30 minutes
RESTORE:
3 hours
These are examples, not final production values.
45. Why Timeouts Matter
Suppose:
backup:
RUNNING
for:
18 hours
The worker may have crashed.
The job must eventually become:
STALE
or:
FAILED
after appropriate recovery logic.
46. Don’t Kill Jobs Blindly
A timeout doesn’t automatically mean:
kill -9
Some operations need cleanup.
For example:
mysqldump
tar
restore
may need controlled termination.
47. Worker Heartbeat
Add:
ALTER TABLE jobs
ADD COLUMN heartbeat_at TEXT;
While running:
heartbeat:
18:32
Then the scheduler can detect stale workers.
48. Worker Heartbeat
The worker periodically updates:
heartbeat_at
For example:
every 30 seconds
A job with:
status=RUNNING
heartbeat=2 hours ago
may be abandoned.
49. Worker Recovery
A future recovery process can detect:
RUNNING
+
heartbeat stale
and mark:
ORPHANED
Then an operator can inspect it.
Don’t automatically assume the underlying operation stopped.
50. Why ORPHANED Matters
Suppose the worker process died but:
nginx configuration
was already modified.
Automatically retrying could create a second modification.
Therefore:
ORPHANED
requires investigation.
51. Job Event Log
Create:
CREATE TABLE job_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id INTEGER NOT NULL,
event_type TEXT NOT NULL,
message TEXT,
created_at TEXT NOT NULL,
FOREIGN KEY (job_id)
REFERENCES jobs(id)
ON DELETE CASCADE
);
Events:
CREATED
CLAIMED
STARTED
LOCK_ACQUIRED
ACTION_STARTED
ACTION_COMPLETED
RETRY
FAILED
SUCCESS
CANCELLED
52. Why Events?
The job table tells you:
status = FAILED
The event history tells you:
18:01 created
18:02 claimed
18:02 backup started
18:10 database dump completed
18:12 file archive failed
18:12 job failed
Much better for debugging.
53. Job CLI
Create:
sudo nano /usr/local/bin/hosting-job
Usage:
hosting-job list
hosting-job show JOB-000241
hosting-job cancel JOB-000241
hosting-job retry JOB-000241
Be careful with:
retry
because not every job is retryable.
54. Job List
CresignSys Jobs
==============
JOB TYPE SITE STATUS PRIORITY
JOB-241 BACKUP example.com RUNNING NORMAL
JOB-242 HEALTH shop.com QUEUED NORMAL
JOB-243 REPAIR blog.com QUEUED HIGH
JOB-244 RETENTION example.com QUEUED LOW
55. Job Details
JOB-000241
==========
Type:
BACKUP
Site:
example.com
Status:
RUNNING
Priority:
NORMAL
Worker:
worker-01
Attempt:
1
Started:
18:02
Heartbeat:
18:07
56. Cancellation
Cancellation is safe only when the job is:
QUEUED
For a running job:
CANCEL REQUESTED
may be possible.
Don’t immediately kill the process.
57. Cancellation State
Use:
CANCEL_REQUESTED
Worker sees:
cancel requested
and exits at a safe checkpoint.
58. Safe Checkpoints
For backup:
after database dump
after file archive
before finalization
For repair:
between actions
But never interrupt halfway through an atomic configuration operation.
59. Central Queue and Scheduler
The backup scheduler should no longer execute:
hosting-backup
directly.
Instead:
Backup scheduler
↓
create BACKUP job
↓
job queue
↓
worker
↓
hosting-backup handler
60. Repair Integration
Instead of:
hosting-repair RP-000021
being directly executed by arbitrary processes, eventually:
approved repair
↓
create REPAIR job
↓
worker
↓
repair handler
The worker becomes the controlled execution boundary.
61. Restore Integration
Likewise:
approved restore
↓
RESTORE job
↓
worker
↓
restore handler
This gives all operations a common framework.
62. Job Creation API
Create a central function:
create_job()
Inputs:
site_id
job_type
priority
payload
scheduled_at
Output:
JOB-000241
63. Don’t Duplicate Job Creation Logic
Avoid:
backup script:
own INSERT
repair script:
different INSERT
restore script:
third INSERT
Instead:
lib/jobs.sh
│
├── create_job
├── claim_job
├── complete_job
├── fail_job
└── cancel_job
64. Create Job Library
sudo nano /etc/cresignsys/lib/jobs.sh
Functions:
create_job
get_next_job
claim_job
update_heartbeat
complete_job
fail_job
cancel_job
record_job_event
65. Job Claiming Rules
get_next_job should consider:
status = QUEUED
scheduled_at <= now
priority
site conflicts
global concurrency
Then:
claim atomically
66. Queue Ordering
Basic ordering:
CRITICAL
HIGH
NORMAL
LOW
Within the same priority:
oldest scheduled job first
This gives:
priority DESC
scheduled_at ASC
conceptually.
67. Fairness
Priority can create starvation.
Suppose:
HIGH
HIGH
HIGH
HIGH
HIGH
arrive continuously.
Then:
LOW
might never run.
Later, use aging:
older low-priority jobs gradually gain priority
Don’t implement this complexity yet.
68. Global Concurrency
Create configuration:
/etc/cresignsys/hosting.conf
Example:
MAX_WORKERS=2
MAX_BACKUP_WORKERS=1
MAX_RESTORE_WORKERS=1
MAX_REPAIR_WORKERS=1
This prevents resource exhaustion.
69. Resource-Specific Limits
A server may allow:
2 health checks
1 backup
1 repair
simultaneously.
But:
restore
could require:
exclusive site access
and perhaps exclusive database access.
70. Job Categories
Define:
OBSERVATION
MAINTENANCE
RECOVERY
Observation
HEALTH
RECONCILE
Maintenance
BACKUP
RETENTION
REPAIR
Recovery
RESTORE
This classification helps future policy.
71. Central Job Flow
REQUEST
│
▼
CREATE JOB
│
▼
QUEUED
│
▼
SCHEDULER
│
▼
WORKER
│
▼
CLAIM + LOCK
│
▼
EXECUTE
│
┌────────┴────────┐
▼ ▼
SUCCESS FAIL
│ │
▼ ▼
COMPLETE ERROR
72. Security Boundary
The job worker should be one of the most protected CHP components.
The architecture should eventually be:
WEB USER
↓
API
↓
AUTHORIZATION
↓
JOB CREATION
↓
QUEUE
↓
WORKER
↓
CONTROLLED OPERATION
Not:
WEB USER
↓
shell command
73. Job Payload Validation
Before executing:
validate job type
validate site ID
validate referenced plan
validate backup ID
validate required fields
For example:
REPAIR
requires:
plan_id
while:
BACKUP
requires:
site_id
74. Never Trust Payload
Even though the job was created internally, validate it again at execution time.
Example:
RESTORE job
backup_id:
BK-001
Worker must verify:
backup exists
backup verified
backup belongs to site
backup not expired
before restoration.
75. Job Idempotency
This is an important concept.
A job should avoid performing the same irreversible action twice.
For example:
BACKUP
can often safely be retried.
But:
RESTORE
cannot automatically be repeated safely.
76. Idempotency Key
Add:
ALTER TABLE jobs
ADD COLUMN idempotency_key TEXT;
Example:
backup:
site=7
schedule=DAILY
date=2026-08-13
could produce:
BACKUP:7:DAILY:2026-08-13
The scheduler can then avoid duplicate jobs.
77. Example
Scheduler runs:
02:00
creates:
BACKUP:7:DAILY:2026-08-13
Scheduler runs again:
02:15
sees the same idempotency key already exists.
Result:
NO DUPLICATE JOB
78. Idempotency for Repair
Repair jobs can use:
REPAIR:RP-000021
If a job already exists:
don't create another
This is another protection against accidental duplicate execution.
79. Idempotency for Restore
Use:
RESTORE:site7:backup123
But because restores are destructive, the system should additionally require explicit approval and current-state checks.
80. Job Queue Dashboard
Eventually the CHP dashboard can show:
Job Queue
---------
Running:
2
Queued:
7
Failed:
1
Completed today:
42
And:
JOB TYPE SITE STATUS
241 BACKUP example RUNNING
242 HEALTH shop QUEUED
243 REPAIR blog QUEUED
244 RESTORE client FAILED
81. Central Queue Changes CHP
Before:
many independent scripts
After:
one controlled execution system
This gives CHP:
concurrency control
auditability
retry policy
timeouts
priorities
locks
worker management
82. Final CHP Architecture
We now have:
CHP
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
DISCOVERY OBSERVATION CONTROL
│ │ │
▼ ▼ ▼
DB IMPORT HEALTH PLAN
│ │
▼ ▼
RECONCILIATION APPROVAL
│
▼
JOB
│
▼
WORKER
│
┌────────────────┼────────────────┐
▼ ▼ ▼
BACKUP REPAIR RESTORE
│ │ │
└────────────────┼────────────────┘
▼
VERIFY
83. The Most Important Design Rule
The central queue should control execution, not authorization.
Correct:
Authorization
↓
Can this operation happen?
↓
Job created
↓
Queue
↓
Worker
Incorrect:
Queue
↓
Maybe authorization
Authorization must happen before the job becomes executable.
84. Another Important Rule
The worker should never blindly trust:
status=APPROVED
for sensitive jobs.
It must revalidate:
approval
plan
site state
fingerprint
expiration
permissions
at execution time.
This gives:
DEFENSE IN DEPTH
85. Lesson 089 — Core Principle
CHP is evolving from a collection of scripts into an actual control plane:
REQUEST
↓
AUTHORIZATION
↓
PLAN
↓
APPROVAL
↓
JOB
↓
QUEUE
↓
WORKER
↓
LOCK
↓
EXECUTION
↓
VERIFY
↓
AUDIT
Each stage has one responsibility.
That separation is what makes the platform safer, easier to debug, and eventually scalable to many hosted websites.
Next Lesson — 090
Build the CHP Event & Audit System
The next missing foundation is a unified history of everything CHP does.
We will build:
SITE DISCOVERED
SITE IMPORTED
HEALTH CHECK
RECONCILIATION
DRIFT DETECTED
PLAN CREATED
PLAN APPROVED
JOB CREATED
BACKUP CREATED
REPAIR EXECUTED
RESTORE EXECUTED
ROLLBACK
FAILURE
into a central event stream:
CHP EVENTS
│
┌───────────────┼────────────────┐
▼ ▼ ▼
AUDIT TIMELINE ALERTS
│ │ │
▼ ▼ ▼
WHO DID IT? WHAT HAPPENED? WHAT NEEDS
WHEN? WHEN? ATTENTION?
WHY? TO WHICH SITE?
This will become the foundation for the future CHP dashboard activity timeline, customer activity history, security auditing, incident investigation, and automated alerts.
Leave a Reply