CresignSys Learn

CresignSys Learn — Lesson 088

Written by

cresignsys

in

Build the CHP Backup Scheduler & Retention Engine

We now have:

hosting-backup
hosting-backup-list
hosting-backup-verify
hosting-restore

The next step is to make backups automatic.

The architecture becomes:

                 BACKUP POLICY
                      │
                      ▼
                  SCHEDULER
                      │
                      ▼
                 BACKUP JOB
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
       SUCCESS                   FAIL
          │                       │
          ▼                       ▼
      RETENTION                 ALERT
          │
          ▼
     SAFE PRUNING

The critical rule is:

Retention must never delete the last usable recovery point merely to satisfy a numerical retention policy.

1. What We Need to Build

Create these components:

hosting-backup-schedule
hosting-backup-run
hosting-backup-prune
hosting-backup-status

The scheduler should eventually support:

daily
weekly
monthly
on-demand
repair
pre-restore

2. Don’t Use One Cron Entry Per Website

Avoid:

cron:
example.com
cron:
shop.example.com
cron:
blog.example.com
cron:
medical.example.com

That becomes difficult to manage.

Instead:

SYSTEM SCHEDULER
       ↓
CHP BACKUP SCHEDULER
       ↓
QUERY DATABASE
       ↓
FIND DUE SITES
       ↓
CREATE BACKUP JOBS

This centralizes scheduling.

3. Scheduler Architecture

              systemd timer
                    │
                    ▼
          hosting-backup-scheduler
                    │
             ┌──────┴──────┐
             ▼             ▼
          Database       Policy
             │             │
             └──────┬──────┘
                    ▼
                DUE JOBS
                    │
                    ▼
             BACKUP ENGINE
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
       SUCCESS               FAIL
          │                   │
          ▼                   ▼
       RETENTION             ALERT

4. Why systemd Timer?

Since the server is Ubuntu, use systemd for the scheduler rather than relying entirely on traditional cron.

Advantages:

logging
service state
failure tracking
restart behavior
central management

5. Scheduler Service

Create:

sudo nano /etc/systemd/system/cresignsys-backup-scheduler.service

Conceptually:

[Unit]
Description=CresignSys Backup Scheduler
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/hosting-backup-scheduler

6. Scheduler Timer

Create:

sudo nano /etc/systemd/system/cresignsys-backup-scheduler.timer

Example:

[Unit]
Description=Run CresignSys Backup Scheduler

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target

This checks every 15 minutes.

The scheduler itself determines whether a particular site’s backup is actually due.

7. Why Check Frequently?

Suppose a site is configured:

Daily:
02:00

If the scheduler runs only once at exactly 02:00 and the server is temporarily unavailable, the backup could be missed.

With:

every 15 minutes

the scheduler can determine:

backup due?
yes

and execute it.

8. Persistent=true

This is useful when the server is offline.

For example:

02:00
server offline

05:00
server starts

The timer knows it missed a scheduled run.

The scheduler can then decide:

run missed backup

subject to policy.

9. Backup Policy Table

Create:

CREATE TABLE backup_policies (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    site_id INTEGER NOT NULL,
    enabled INTEGER NOT NULL DEFAULT 1,

    daily_enabled INTEGER NOT NULL DEFAULT 1,
    daily_time TEXT,

    weekly_enabled INTEGER NOT NULL DEFAULT 0,
    weekly_day INTEGER,
    weekly_time TEXT,

    monthly_enabled INTEGER NOT NULL DEFAULT 0,
    monthly_day INTEGER,
    monthly_time TEXT,

    retention_daily INTEGER NOT NULL DEFAULT 7,
    retention_weekly INTEGER NOT NULL DEFAULT 4,
    retention_monthly INTEGER NOT NULL DEFAULT 12,

    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL,

    FOREIGN KEY (site_id)
        REFERENCES sites(id)
        ON DELETE CASCADE
);

10. Example Policy

For:

example.com

store:

Daily:
YES

Daily time:
02:00

Weekly:
YES

Weekly day:
Sunday

Weekly time:
03:00

Monthly:
YES

Monthly day:
1

Monthly time:
04:00

11. Retention Policy

Example:

Daily:
7

Weekly:
4

Monthly:
12

This does not mean:

delete everything older than 12 months

It means:

retain selected recovery points according to policy

12. Backup Classification

Every scheduled backup should be classified:

DAILY
WEEKLY
MONTHLY

Example:

BK-001
DAILY

BK-002
DAILY

BK-003
WEEKLY

The retention engine uses these classifications.

13. Backup Schedule Table

It can be useful to distinguish:

policy

from:

execution history

The policy says:

"Run every day at 02:00."

The backup record says:

"Actually ran at 02:04 and succeeded."

Don’t mix those concepts.

14. Add Scheduled Metadata

Add to backups:

ALTER TABLE backups
ADD COLUMN schedule_type TEXT;

Possible:

DAILY
WEEKLY
MONTHLY
MANUAL
REPAIR
PRE_RESTORE

15. Scheduler Command

Create:

sudo nano /usr/local/bin/hosting-backup-scheduler

Start:

#!/usr/bin/env bash

set -Eeuo pipefail

Load:

source /etc/cresignsys/hosting.conf
source /etc/cresignsys/lib/common.sh
source /etc/cresignsys/lib/logging.sh
source /etc/cresignsys/lib/database.sh
source /etc/cresignsys/lib/backup.sh
source /etc/cresignsys/lib/backup-policy.sh

16. Scheduler Algorithm

The scheduler should:

1. Find enabled sites
2. Load backup policy
3. Determine whether backup is due
4. Check whether another backup is running
5. Check storage availability
6. Create backup job
7. Execute backup
8. Record result
9. Trigger retention
10. Record scheduler result

17. Find Due Sites

Conceptually:

for each managed site:
    load backup policy

    if backup is due:
        create backup job

Don’t execute backups for:

READ_ONLY

sites unless their backup policy explicitly permits monitoring-only backups.

18. Management State and Backups

This requires a distinction.

A site can be:

READ_ONLY

for configuration changes but still have:

BACKUP:
ENABLED

Therefore don’t automatically equate:

READ_ONLY
=
NO BACKUP

Backup is not necessarily a configuration modification.

19. Better Permission Model

Use separate capabilities:

MONITOR
BACKUP
MANAGE
RESTORE
APPROVE

Then:

READ_ONLY

could mean:

MONITOR:
YES

BACKUP:
YES

MANAGE:
NO

RESTORE:
NO

This is more flexible.

20. Backup Job Table

Create:

CREATE TABLE backup_jobs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    site_id INTEGER NOT NULL,
    schedule_type TEXT NOT NULL,
    status TEXT NOT NULL,
    scheduled_for TEXT NOT NULL,
    started_at TEXT,
    completed_at TEXT,
    backup_id INTEGER,
    error_message TEXT,

    FOREIGN KEY (site_id)
        REFERENCES sites(id),

    FOREIGN KEY (backup_id)
        REFERENCES backups(id)
);

21. Job States

Use:

SCHEDULED
RUNNING
SUCCESS
FAILED
SKIPPED
CANCELLED

Example:

02:00
SCHEDULED

02:03
RUNNING

02:12
SUCCESS

22. Why a Job Table?

Without it, you only know:

Backup exists.

With it, you know:

Was it scheduled?
Did it start?
Was it skipped?
Why did it fail?
How long did it take?
Which backup resulted?

This is essential for operations.

23. Duplicate Prevention

Suppose the scheduler runs at:

02:00
02:15

and the 02:00 backup is still running.

The 02:15 execution must detect:

BACKUP ALREADY RUNNING

and skip creating another backup.

24. Site Backup Lock

Use:

/var/lock/cresignsys/example.com.lock

The backup engine already uses this.

The scheduler should not bypass it.

25. Job Lock vs Site Lock

These solve different problems.

Job lock

Prevents duplicate scheduler execution.

Site lock

Prevents conflicting operations on the website.

You can have:

scheduler lock
+
site lock

26. Example

Scheduler A:
creates job 101

Scheduler B:
sees job 101 RUNNING

Scheduler B:
SKIPPED

This prevents duplicate backups.

27. Missed Backup Policy

Suppose:

scheduled:
02:00

server:
offline

server returns:
07:00

The scheduler sees:

MISSED

What should it do?

For the first version:

run once

if the backup has not already occurred that day.

28. Don’t Create Multiple Catch-Up Backups

If the server was offline for:

3 days

don’t create:

3 immediate backups

one after another.

Instead:

one catch-up backup

then return to the normal schedule.

29. Missed Backup Status

Record:

schedule_type:
DAILY

scheduled_for:
2026-08-13 02:00

actual:
2026-08-13 07:05

result:
SUCCESS

execution_reason:
MISSED_SCHEDULE

This is useful for reporting.

30. Backup Freshness

Site status can then calculate:

latest successful backup

and:

backup age

Example:

Latest:
07:05

Age:
2 hours

31. Backup SLA

A hosting plan can define:

backup freshness:
24 hours

Then:

age < 24h
HEALTHY

and:

age > 24h
WARNING

32. Backup SLA by Hosting Plan

This fits naturally with your hosting platform.

For example:

Starter:
daily

Business:
daily

Professional:
daily + weekly

Business Plus:
daily + weekly + monthly

The exact commercial policy can be configured later.

33. Storage Check

Before creating a backup:

df -P /var/lib/cresignsys/backups

Determine available storage.

If:

available:
5 GB

but estimated backup:

8 GB

don’t begin.

34. Estimate Backup Size

Use the site root:

du -sb "$WEB_ROOT"

plus:

database estimate
configuration
compression overhead

Then:

required space

should include a safety margin.

35. Safety Margin

Don’t require:

available >= estimated

Use something like:

available >= estimated × 1.5

for the first version.

This avoids failure when compression or temporary files require extra space.

36. What If Storage Is Low?

Don’t automatically delete backups before creating the new backup.

Instead:

CHECK
 ↓
RETENTION CANDIDATES
 ↓
VERIFY SAFE TO DELETE
 ↓
PRUNE
 ↓
RECHECK STORAGE
 ↓
BACKUP

But only if pruning does not violate recovery requirements.

37. Never Delete First, Think Later

Bad:

disk full
 ↓
delete oldest backups
 ↓
backup

The deletion could remove the only usable recovery point.

Better:

disk full
 ↓
calculate protected backups
 ↓
calculate safe candidates
 ↓
prune only safe candidates
 ↓
recheck

38. Protected Backup

A backup is protected if it is:

latest verified backup

or:

required by weekly policy

or:

required by monthly policy

or:

associated with an active repair

or:

currently being restored

39. Retention Algorithm

Suppose backups are:

D1
D2
D3
D4
D5
D6
D7
D8
D9

Policy:

retain 7 daily

Keep:

D3–D9

Candidates:

D1
D2

But if D2 is also the weekly recovery point:

D2:
PROTECTED

then D1 becomes the only deletion candidate.

40. Retention Is Not Simple FIFO

Never implement:

sort by date
delete oldest N

without classification.

Retention needs to understand:

daily
weekly
monthly
protected
verified
active

41. Example

Policy:

Daily:
7

Weekly:
4

Monthly:
12

A single backup can satisfy more than one role.

For example:

Sunday backup

can be:

DAILY
+
WEEKLY

The retention engine should avoid storing unnecessary duplicate copies where possible.

42. Retention Selection

Conceptually:

1. Select newest 7 daily backups
2. Select newest 4 weekly backups
3. Select newest 12 monthly backups
4. Combine selections
5. Mark all selected backups PROTECTED
6. Everything else becomes a candidate

Then apply safety checks.

43. Candidate States

Don’t immediately delete.

Use:

CANDIDATE

first.

Then:

SAFE_TO_DELETE

Then:

DELETED

This makes the retention process auditable.

44. Retention Table

Create:

CREATE TABLE backup_retention_actions (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    backup_id INTEGER NOT NULL,
    action TEXT NOT NULL,
    reason TEXT,
    created_at TEXT NOT NULL,
    executed_at TEXT,

    FOREIGN KEY (backup_id)
        REFERENCES backups(id)
);

Example:

backup:
BK-001

action:
PROTECT

reason:
Newest daily backup

or:

backup:
BK-002

action:
DELETE

reason:
Older than daily retention

45. Never Delete Unverified Backups First

It might seem logical to delete:

FAILED

backups first.

But even a failed backup could contain useful partial data for forensic purposes.

Instead define a cleanup policy.

For example:

FAILED:
retain 24–72 hours

then remove automatically.

46. Corrupted Backup

If a previously verified backup later fails checksum verification:

VERIFIED
 ↓
CORRUPTED

It must immediately lose its protected status.

Then find another valid recovery point.

47. Last Known-Good Backup

For every site, maintain:

last_verified_backup_id

Conceptually:

site
 ↓
last known good backup

This can be used by:

repair rollback
restore recommendation
disaster recovery
site status

48. Add to Sites

Eventually:

ALTER TABLE sites
ADD COLUMN last_verified_backup_id INTEGER;

However, because SQLite foreign-key handling may require care when altering tables, implement this through your normal CHP database migration system rather than manually changing production tables repeatedly.

49. Better: Derived Query

Initially you don’t even need to store it.

Query:

latest backup
WHERE
site_id = ?
AND status = VERIFIED
ORDER BY verified_at DESC
LIMIT 1

This avoids duplicated state.

50. Backup Status

The site dashboard can show:

Backup:
HEALTHY

Last verified:
2026-08-13 07:05

Age:
2h 15m

If no verified backup exists:

Backup:
CRITICAL

51. Scheduler Logging

Every scheduled execution should log:

site
schedule
scheduled time
actual time
result
backup ID
duration

Example:

2026-08-13 07:05
example.com
DAILY
SUCCESS
BK-20260813-070501
duration=7m32s

52. Backup Duration

Record:

started_at
completed_at
duration_seconds

This allows detection of unusual changes.

For example:

Normal:
7 minutes

Current:
48 minutes

could indicate:

site growth
storage problem
database issue
network issue

53. Backup Performance

Over time, CHP can calculate:

average backup duration
backup size growth
database growth
storage consumption

This becomes useful for capacity planning.

54. Backup Job Failure

If:

hosting-backup

returns failure:

job:
FAILED

Do not run retention as though the backup succeeded.

Instead:

FAILED
 ↓
record failure
 ↓
alert

Retention can run independently later.

55. Retry Policy

A backup failure can be retried carefully.

For example:

attempt 1:
FAILED

wait:
30 minutes

attempt 2:
FAILED

wait:
60 minutes

attempt 3:
FAILED

Then:

FINAL FAILURE

Don’t retry indefinitely.

56. Why Limited Retry?

Possible causes include:

disk full
database locked
permission error
network storage unavailable
corrupt filesystem

Repeated retries can make the situation worse.

57. Backup Retry Table

The job record can include:

attempt_number

Example:

Job 101
Attempt 1:
FAILED

Attempt 2:
SUCCESS

58. Scheduler Concurrency

Suppose 20 websites are due at 02:00.

Don’t necessarily run:

20 backups
simultaneously

That could overload:

CPU
disk I/O
RAM
MySQL

59. Global Backup Concurrency

Configure:

max concurrent backups:
2

Initially.

Then:

Site A → RUNNING
Site B → RUNNING
Site C → QUEUED
Site D → QUEUED

60. Backup Queue

Eventually:

BACKUP QUEUE
-------------
A RUNNING
B RUNNING
C QUEUED
D QUEUED
E QUEUED

This becomes important when CHP hosts many sites.

61. Site Lock Still Applies

Even with a global queue:

site A

cannot have:

backup
+
restore

at the same time.

The site lock remains the final protection.

62. Scheduler Priorities

Eventually jobs can have:

CRITICAL
HIGH
NORMAL
LOW

For example:

PRE_RESTORE:
CRITICAL

REPAIR:
HIGH

CUSTOMER DAILY:
NORMAL

This prevents routine backups from blocking urgent recovery operations.

63. Backup Scheduling Database

The overall structure now becomes:

sites
 │
 ├── backup_policies
 │
 ├── backup_jobs
 │
 └── backups
        │
        └── retention_actions

This cleanly separates:

policy
execution
artifact
retention

64. Backup Scheduler Status

Create:

sudo nano /usr/local/bin/hosting-backup-status

Output:

CresignSys Backup Status
========================

Site:
example.com

Policy:
Daily 02:00
Weekly Sunday 03:00
Monthly 1st 04:00

Last Backup:
2026-08-13 07:05

Status:
VERIFIED

Next Scheduled:
2026-08-14 02:00

Retention:
7 daily
4 weekly
12 monthly

Storage:
42%

Scheduler:
HEALTHY

65. Site-Level Backup Status

For all sites:

sudo hosting-backup-status --all

Output:

SITE                    LAST BACKUP        STATUS
-----------------------------------------------------
example.com             2h                 HEALTHY
shop.example.com        5h                 HEALTHY
blog.example.com        31h                WARNING
client.example.com      4d                 CRITICAL

This will eventually feed the CHP dashboard.

66. Scheduler Health

The scheduler itself must be monitored.

Suppose:

systemd timer:
stopped

Then:

all sites

could silently stop receiving backups.

Therefore:

CHP scheduler health

must be part of the platform health model.

67. Scheduler Heartbeat

Record:

last_scheduler_run

For example:

2026-08-13 15:00

If current time is:

18:00

and no scheduler run has occurred:

SCHEDULER:
WARNING

68. Scheduler Failure

If the timer repeatedly fails:

SCHEDULER:
CRITICAL

This is different from:

individual backup:
FAILED

The distinction matters.

69. Three Levels

PLATFORM
   │
   ├── Scheduler health
   │
   └── Site backup health

Example:

Scheduler:
HEALTHY

Site A:
HEALTHY

Site B:
FAILED

Site C:
HEALTHY

Only Site B has a backup problem.

70. Backup Alert Conditions

Eventually alert when:

no verified backup within SLA

or:

backup failed repeatedly

or:

storage critically low

or:

scheduler not running

or:

last verified backup corrupted

71. Do Not Alert on Every Scheduler Run

Bad:

Backup succeeded
Backup succeeded
Backup succeeded

The system should notify only when there is something requiring attention.

72. Backup Recovery Objective

Now we can introduce two important concepts.

RPO

Recovery Point Objective:

How much recent data can the customer afford to lose?

Example:

Daily backup:
RPO ≈ 24 hours

RTO

Recovery Time Objective:

How quickly can the website be restored?

These concepts will eventually influence hosting plans.

73. Example

Starter:

RPO:
24h

RTO:
best effort

Professional:

RPO:
24h

RTO:
priority restore

Premium:

RPO:
shorter interval

RTO:
priority recovery

Exact commercial values should be defined separately.

74. Backup Scheduler and CHP Repairs

Now consider:

02:00:
scheduled backup

02:10:
repair starts

The repair engine can use:

02:00 backup

if it meets the required safety conditions.

But a repair should preferably create its own:

REPAIR

backup immediately before modification.

75. Why Scheduled Backup Is Not Enough

A scheduled backup at:

02:00

does not protect against:

repair at 16:00

because the site may have changed significantly during the day.

Therefore:

scheduled backup
+
pre-repair backup

are different safety mechanisms.

76. Backup Types Summary

DAILY
   customer recovery

WEEKLY
   longer retention

MONTHLY
   long-term recovery

REPAIR
   immediate rollback

PRE_RESTORE
   protection before restoration

MANUAL
   operator-created recovery point

77. Retention Protection

The retention engine should protect:

latest verified DAILY
latest verified WEEKLY
latest verified MONTHLY
active REPAIR backup
active PRE_RESTORE backup

Only then can it identify deletion candidates.

78. Deletion Algorithm

LOAD ALL BACKUPS
       ↓
REMOVE ACTIVE/PROTECTED
       ↓
SELECT RETENTION SET
       ↓
MARK CANDIDATES
       ↓
VERIFY AT LEAST ONE GOOD RECOVERY POINT
       ↓
DELETE
       ↓
VERIFY STORAGE

79. Final Safety Check

Before deleting any backup:

verified_count_after_delete >= minimum_required_recovery_points

For example:

minimum:
1

Then CHP must never produce:

VERIFIED BACKUPS:
0

unless an administrator explicitly overrides the policy.

80. Never Delete the Last Verified Backup

This deserves its own rule:

if deleting backup would leave zero VERIFIED backups:
    DENY deletion

Even if:

retention policy says delete

the safety rule wins.

81. Example

Suppose:

Backup A:
VERIFIED

Backup B:
FAILED

Backup C:
CORRUPTED

Retention says:

delete old backups

The engine must preserve:

Backup A

because it is the only known-good recovery point.

82. If Storage Is 100%

Suppose:

disk:
100%

and:

new backup:
cannot start

The system should:

1. identify safe deletion candidates
2. protect last verified backup
3. prune only safe candidates
4. retry backup

If insufficient space remains:

BACKUP FAILED

and alert.

Never delete the last recovery point just to make the new backup fit.

83. Backup Scheduler Final Architecture

                   SYSTEMD TIMER
                         │
                         ▼
              BACKUP SCHEDULER
                         │
               ┌─────────┴─────────┐
               ▼                   ▼
           DUE JOBS             MISSED JOBS
               │                   │
               └─────────┬─────────┘
                         ▼
                    JOB QUEUE
                         │
                         ▼
                   BACKUP ENGINE
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
           SUCCESS                 FAIL
              │                     │
              ▼                     ▼
         RETENTION               RETRY
              │                     │
              ▼                     ▼
          SAFE PRUNE              ALERT

84. Complete CHP Architecture So Far

                         CRESIGNSYS CHP
                               │
       ┌───────────────────────┼───────────────────────┐
       ▼                       ▼                       ▼
   DISCOVERY              OBSERVATION                CONTROL
       │                       │                       │
       ▼                       ▼                       ▼
 DB IMPORT              RECONCILIATION           REPAIR PLAN
                               │                       │
                               ▼                       ▼
                            HEALTH                 APPROVAL
                                                       │
                                                       ▼
                                                    REPAIR
                                                       │
                                                       ▼
                                                    BACKUP
                                                       │
                                                       ▼
                                                   VERIFY

And backup itself now has:

POLICY
  ↓
SCHEDULER
  ↓
JOB
  ↓
BACKUP
  ↓
VERIFY
  ↓
RETENTION

85. Lesson 088 — Core Principle

A professional backup scheduler does not simply say:

"Run backup every day."

It must understand:

WHEN should it run?
WAS it actually run?
DID it succeed?
IS the backup verified?
IS it still useful?
IS it protected by retention?
IS there enough storage?
IS another backup running?
IS the scheduler healthy?
IS there at least one recovery point?

The critical retention rule remains:

             NEVER
              ↓
delete the last
known-good backup

Next Lesson — 089

Build the CHP Central Job Queue

We now have several independent operations:

BACKUP
RESTORE
REPAIR
HEALTH CHECK
RECONCILIATION
RETENTION

Running all of them independently will eventually create conflicts.

The next step is therefore a central:

CHP JOB ENGINE

with:

QUEUED
RUNNING
SUCCESS
FAILED
CANCELLED

and controlled concurrency:

              CHP JOB QUEUE
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
     BACKUP       REPAIR       RESTORE
       │            │            │
       └────────────┼────────────┘
                    ▼
              WORKER ENGINE
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
       RUNNING              QUEUED
          │
          ▼
       RESULT

This will allow CHP to prevent dangerous situations such as:

BACKUP + RESTORE
REPAIR + RESTORE
REPAIR + REPAIR

running against the same website simultaneously.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts