Skip to content

ZSM: ZFS Storage Management and Backup Framework

( A comprehensive ZFS management and ZFS-based backup and disaster recovery system, designed to keep all systems fully configured with regular snapshots, remote replication, retention management, bulkhead bare-metal restore via network boot, and autonomous hardware adaptation during recovery )

ZSM is a custom storage-management and backup framework built around ZFS replication, snapshot retention, incremental transfer, recovery, and operational automation.

It was developed to manage large numbers of ZFS datasets and snapshots across distributed systems while providing controlled retention, replication, verification, cleanup, monitoring, and recovery workflows.

Rather than treating ZFS replication as a collection of independent zfs send and zfs receive operations, ZSM maintains enough state to understand the relationship between snapshots, bookmarks, transfers, retention policies, and remote datasets. This allows routine synchronization and cleanup to be performed repeatedly and safely as part of an automated operational system.

The system was designed for real infrastructure rather than as a demonstration project. Particular attention is given to failure recovery, partial transfers, snapshot dependencies, incremental replication, concurrency, controlled destruction, and the practical limitations of ZFS itself.

What ZSM manages

Snapshot lifecycle

Snapshot creation, retention, holds, bookmarks, and controlled cleanup.

Replication state

Relationships between snapshots and replication targets are tracked so incremental replication can continue across successive operations.

Replication concurrency

Multiple independent replication streams can operate concurrently, subject to configured resource limits.

Multiple targets

A source can maintain independent backup copies on multiple ZFS systems.

Chained replication

A backup target can itself participate in a subsequent replication stage, allowing arbitrarily deep backup topologies.

Failure handling

Interrupted or incomplete replication operations can be detected and handled without blindly treating the target as a clean replication state.

Recovery

Backup datasets can be used as the basis for bare-metal restoration, physical-to-virtual recovery, and snapshot exploration.

Core technologies: ZFS, snapshots, bookmarks, send/receive, channel programs, replication, TFTP, PXE, NFS, IPMI, KVM, retention policies, holds, hardware adaption

Engineering focus: data integrity, disaster recovery, automation, storage architecture, operational control, remote operations

Overview

ZSM is an operational layer built around ZFS. It coordinates snapshots, replication, retention, recovery, and the state required to manage those operations across local and remote systems.

ZSM does not replace ZFS. It uses ZFS's snapshot and replication primitives as the foundation and adds the orchestration, policy, state management, and recovery workflow needed to operate them as a backup system.

This framework addresses the full lifecycle of backup and recovery: capturing consistent snapshots of every system, replicating them across a chained network of backup servers, and — critically — restoring a complete system onto replacement hardware that may differ from the original, entirely over the network, typically with no physical access required.

The design goal is a workflow that is simple for the common case (a few guided clicks) and capable for the rare case (significant hardware divergence requiring driver and configuration adaptation).

flowchart LR
    A["Production System<br/>ZFS Storage"]
    B["ZSM<br/>Backup & Recovery<br/>Management"]
    C["Backup Systems<br/>ZFS Storage"]
    D["Recovery<br/>Restore / Explore"]

    A -->|"Snapshots & Replication"| B
    B -->|"Managed Backup Data"| C
    C -->|"Recovery Data"| D
More Detailed Diagram 
flowchart TB
    subgraph SOURCE["SOURCE SYSTEM"]
        FS["Production ZFS Pool"]
        SNAP["Snapshot Capture"]
        HOOK["Pre / Post Snapshot Hooks"]
        FS --> SNAP
        SNAP --> HOOK
    end

    subgraph ZSM["ZSM"]
        CORE["ZSM Orchestration Layer"]

        RET["Retention & Cleanup"]
        STATE["Replication State"]
        BOOK["Bookmarks"]
        HOLD["Snapshot Holds"]
        META["Persistent Metadata"]

        CORE --> RET
        CORE --> STATE
        CORE --> BOOK
        CORE --> HOLD
        CORE --> META
    end

    subgraph REPLICATION["REPLICATION"]
        R1["Incremental Stream"]
        R2["Incremental Stream"]
        R3["Incremental Stream"]
    end

    subgraph TARGETS["BACKUP TARGETS"]
        T1["Backup Server A"]
        T2["Backup Server B"]
        T3["Backup Server C"]
    end

    HOOK --> CORE

    CORE --> R1
    CORE --> R2
    CORE --> R3

    R1 --> T1
    R2 --> T2
    R3 --> T3

    T1 -->|"Optional chained replication"| T2
    T2 -->|"Optional chained replication"| T3

    subgraph RECOVERY["RECOVERY / RESTORE"]
        NET["Network Boot"]
        RESTORE["Guided Restore"]
        ADAPT["Hardware Adaptation"]
        P2V["Physical → Virtual"]
        EXPLORE["Snapshot Exploration"]
    end

    T1 --> NET
    T2 --> NET
    T3 --> NET

    NET --> RESTORE
    RESTORE --> ADAPT
    RESTORE --> P2V
    RESTORE --> EXPLORE

Why Built It

ZFS provides excellent primitives for snapshots and replication, but building a reliable backup system around those primitives introduces problems that are not solved by a single replication command.

A production backup environment has to answer questions such as:

  • Which snapshot represents the correct replication point?
  • Can a snapshot safely be destroyed?
  • What happens when a transfer is interrupted?
  • How long should replication state survive?
  • How much cleanup should happen in one operation?
  • How should large numbers of datasets be processed concurrently?
  • How can an operator determine what the system is doing?
  • How can destructive operations be tested before they are executed?
  • How can the system recover when the environment no longer looks exactly as expected?

ZSM exists to provide the operational layer around those problems.

The underlying philosophy is to use ZFS for the storage primitives while putting the policy, state management, orchestration, safety controls, and operational visibility in a layer designed specifically for the environment.

Existing ZFS tools/systems that didn't work for me

While there are several other ZFS snapshot scripts and programs out there, none of them met all of my needs or matched my ambitious goals. Here are a few other tools that, while useful to a certain extent, didn't quite satisfy my requirements:

Sanoid and Syncoid

These are widely considered the gold standard for command-line users. Sanoid is a policy-driven snapshot manager that allows you to define how often snapshots are taken and when they should be pruned (e.g., keep 24 hourly, 30 daily, and 6 monthly snapshots). Syncoid is the companion tool used to perform the actual replication between datasets, whether on the same machine or across a network using SSH. They are lightweight, highly reliable, and follow the Unix philosophy of doing one thing well.

ZRepl

ZRepl is a daemon-based replication tool designed for more complex topologies. Unlike Syncoid, which is typically run as a cron job or manual command, ZRepl runs as a background service that monitors datasets and automatically pushes or pulls snapshots based on a configuration file. It is particularly useful for maintaining a mirror of a production server on a backup target with minimal manual intervention.

ZFS-Autobackup

This is a simpler script-based approach often used by home lab enthusiasts. It provides a streamlined way to automate the snapshot-and-send process to external drives or remote servers without the overhead of a full daemon like ZRepl or a complete OS like TrueNAS.

Proxmox VE

While primarily a virtualization platform, Proxmox integrates ZFS natively. It provides a built-in replication feature that allows you to synchronize ZFS-based VM and container disks between nodes in a cluster every few minutes. This is specifically optimized for high-availability clusters rather than long-term archival backups.

TrueNAS

For those who prefer a GUI, TrueNAS provides the most comprehensive integrated ZFS management suite. It abstracts the underlying zfs send and zfs receive commands into a visual interface where you can create "Periodic Snapshot Tasks" and "Replication Tasks." This is the ideal choice for users who want a professional-grade NAS experience without writing configuration files or scripts.

Architecture

ZSM sits between the operational environment and the underlying ZFS tooling. It coordinates snapshot creation, state tracking, replication, retention, cleanup, and recovery rather than attempting to replace ZFS itself.

At a high level, the system manages several related pieces of state:

  • Snapshots - point-in-time filesystem state used for backup and recovery.
  • Bookmarks - persistent replication references that can outlive their corresponding snapshots and provide resume points for incremental transfers.
  • Snapshot holds - protection against premature destruction while data is still required by an outstanding operation.
  • Remote dataset state - information required to determine what can be incrementally transferred to another system.
  • Retention state - independent policies governing how long snapshots and replication metadata remain available.
  • Operational state - information used to coordinate long-running or concurrent work and expose its progress.

This allows ZSM to treat backup as an ongoing state-management problem rather than as a sequence of unrelated filesystem commands.

Snapshot Capture and Retention

  • Automated snapshot scheduling across all systems with per-dataset, configurable retention policies
  • Intelligent snapshot lifecycle management — precise control over when and how old snapshots are purged
  • Critical snapshots can be locked with ZFS holds to prevent accidental or policy-driven destruction
  • Full snapshot history browsing for any system and any dataset

Replication Topology

flowchart LR
    P["Production<br/>ZFS"]

    B1["Backup<br/>Server 1"]
    B2["Backup<br/>Server 2"]
    B3["Backup<br/>Server 3"]

    D1["Downstream<br/>Backup"]
    D2["Downstream<br/>Backup"]

    P -->|"Replication"| B1
    P -->|"Replication"| B2
    P -->|"Replication"| B3

    B1 -.->|"Chained replication"| D1
    B2 -.->|"Chained replication"| D2
More Detailed Diagram 
flowchart LR
    SRC["Production System<br/>ZFS Storage"]

    ZSM["ZSM<br/>Backup & Recovery<br/>Orchestration"]

    T1["Backup Target"]
    T2["Backup Target"]
    TN["Additional<br/>Backup Targets"]

    SRC -->|"Snapshots"| ZSM

    ZSM -->|"Incremental replication"| T1
    ZSM -->|"Incremental replication"| T2
    ZSM -->|"Incremental replication"| TN

    T1 -.->|"Optional chained replication"| T2
    T2 -.->|"Optional chained replication"| TN

    T1 --> REC["Recovery<br/>Restore / P2V /<br/>Snapshot Exploration"]
    T2 --> REC
    TN --> REC
  • Snapshot mirroring via a highly intelligent and optimized replication tool
  • Reliable, fast, optimal replication to keep one or more backup servers current
  • Backup servers can be chained: a local backup server provides fast on-site restores and mirrors to remote off-site storage for geographic redundancy
  • Any system with remote access capability on the network can be a replication source or target
More 'Replication Topology' Details

Snapshot-Based Incremental Replication

ZSM coordinates incremental ZFS replication between systems using snapshot relationships and persistent replication state. Rather than repeatedly transferring complete datasets, the system identifies appropriate points in snapshot history and transfers only the changes required to bring a destination up to date.

ZSM distinguishes between the lifetime of a snapshot and the lifetime of the replication state derived from it.

Snapshots may be removed according to normal retention policies while replication bookmarks remain available as incremental-send reference points. This allows the system to reclaim snapshot storage without unnecessarily losing the state required to continue efficient incremental replication.

Bookmark retention is therefore intentionally independent from snapshot retention.

This distinction is important in long-running backup systems: storage retention and replication continuity are related, but they are not the same problem.

Multiple Parallel Replication Streams

ZSM can process multiple replication operations concurrently, allowing independent datasets to be synchronized in parallel. Concurrency is configurable so replication throughput can be increased when resources permit without requiring the entire system to operate at maximum parallelism.

Multiple Backup Target Support

A source system can replicate its data to multiple backup destinations, allowing local and remote copies to be maintained independently. This supports architectures where one destination provides fast local recovery while another provides geographic or otherwise independent protection.

Infinite Backup Server Chaining

Replication does not require every backup system to communicate directly with the original source. A backup system can itself serve as the source for another replication stage, allowing backup data to propagate through an arbitrarily deep chain of systems.

This makes it possible to build replication topologies around network boundaries, geographic locations, available bandwidth, or other operational constraints rather than requiring a single centralized backup destination.

Resumable Transfers

ZSM is designed to handle interrupted replication without requiring an otherwise unnecessary full transfer to be repeated. Persistent replication state, snapshot relationships, bookmarks, and handling of incomplete receives allow interrupted operations to be recovered and continued from an appropriate replication point.

Replication State Management

ZSM maintains the state necessary to coordinate replication independently of the immediate state of an individual transfer. Snapshots, bookmarks, holds, and destination state are treated as related pieces of the replication lifecycle.

This allows replication decisions to persist across individual operations and provides the foundation for retention, cleanup, recovery, and continued incremental synchronization.

Remote Replication

Replication can operate across remote systems rather than being limited to locally attached storage. ZSM coordinates the underlying storage operations with configurable remote execution and transfer mechanisms, allowing the same replication model to be used across local and geographically separated backup infrastructure.

Bookmarks for Recovery and Resynchronization

  • Bookmarks are used to get a remote server back up to date even if its snapshots are lost
  • Because bookmarks are lightweight references (they carry no data on their own), a server that has lost its snapshot history can still be resynchronized from a bookmark held on the backup side
  • This provides a durable, data-efficient mechanism to reconstruct a consistent replication stream without re-sending the full dataset

Controlled Cleanup and Destruction

Backup systems eventually accumulate large numbers of snapshots, and snapshot cleanup can itself become an operational risk. ZSM treats destruction as a controlled operation rather than simply deleting everything older than a particular date.

  • Supports dry-run operation to preview cleanup actions without modifying the storage system
  • Requires explicit authorization for destructive operations rather than allowing deletion to occur implicitly
  • Limits the amount of data that can be destroyed in a single operation
  • Limits the number of snapshots that can be destroyed within a specified range
  • Supports recursive cleanup across ZFS dataset hierarchies
  • Performs cleanup concurrently when appropriate while allowing the amount of concurrency to be controlled
  • Uses ZFS snapshot holds to protect recovery points that are still required by active workflows
  • Verifies expected snapshot holds and removes them when they are no longer required
  • Processes defined snapshot ranges rather than requiring individual snapshots to be handled independently

These controls are intended to make large-scale retention changes more predictable and reduce the blast radius of an incorrect retention decision or operational mistake.

More 'Controlled Cleanup and Destruction' Details

Backup systems eventually accumulate large numbers of snapshots, and snapshot cleanup can itself become an operational risk.

ZSM therefore treats destruction as a controlled operation rather than simply deleting everything older than a particular date.

Cleanup can be performed in bounded operations, limiting both the amount of data removed and the number of snapshots processed at once. Operations can also be previewed without making changes, and snapshot holds are used to prevent data from being removed while it is still required by an active workflow.

These controls are intended to make large-scale retention changes more predictable and reduce the blast radius of an incorrect retention decision or operational mistake.

Dry-Run / No-Op Operation

Dry-run / no-op operation — Cleanup operations can be previewed without making changes, allowing the intended actions to be reviewed before execution.

Operations can be evaluated without making changes to the storage system. This provides a safe way to inspect what ZSM intends to do before allowing destructive or otherwise consequential operations to proceed.

Explicit Destruction

Explicit destruction — Destructive operations require deliberate authorization rather than occurring implicitly as part of normal retention processing.

Destructive operations require explicit authorization rather than being an implicit side effect of normal cleanup or retention processing. This provides an additional safeguard against accidental data removal.

Retention Windows

Snapshot and bookmark retention can be managed independently using configurable retention windows. This allows recovery points to be preserved for an appropriate period while still controlling long-term storage consumption.

Limiting Destruction by Data Volume

Data-volume limits — The amount of data that can be destroyed in a single operation can be bounded to prevent unexpectedly large cleanup operations.

Cleanup operations can be bounded by the amount of data they are allowed to destroy in a single operation. This prevents a single cleanup pass from unexpectedly generating a large destructive workload.

Limiting Destruction by Snapshot Count

Snapshot-count limits — The number of snapshots processed within a range can be limited, providing additional control over the scope of each operation.

The number of snapshots that can be destroyed within a given range can also be bounded. This provides another layer of control over automated cleanup and helps keep individual operations predictable.

Recursive Operation

Recursive operation — Cleanup can be applied across ZFS dataset hierarchies when managing related datasets as a group.

Operations can be applied recursively across ZFS dataset hierarchies. This allows related datasets and their snapshots to be managed as a group rather than requiring each dataset to be processed independently.

Concurrent Cleanup

Concurrent cleanup — Cleanup operations can run concurrently with configurable limits, allowing large retention workloads to be processed efficiently without unrestricted resource consumption.

Cleanup work can be performed concurrently, with separate controls for snapshot and bookmark cleanup. Concurrency can be bounded to balance cleanup performance against the load placed on the storage system.

Protecting Snapshots with Holds

Snapshot holds — ZFS holds can protect snapshots from destruction while they are still required by an active workflow.

ZFS snapshot holds can be used to protect recovery points from deletion while they are still required. ZSM incorporates holds into its retention and cleanup workflow so that protected snapshots are not inadvertently removed.

Verifying and Removing Expected Holds

Hold verification and removal — Expected holds can be checked and removed when they are no longer required, helping prevent stale protection from accumulating.

ZSM can track and manage the holds it expects to exist on snapshots, including removing holds when they are no longer required. This helps prevent stale protection from accumulating indefinitely.

Handling Snapshot Ranges

Snapshot ranges — Cleanup can operate on defined ranges of snapshots, allowing retention policies to process groups of recovery points while still respecting configured limits and protections.

Cleanup can operate on defined ranges of snapshots rather than treating every snapshot independently. This allows retention policies to identify groups of recovery points for evaluation while applying the configured limits and protections to the resulting operation.

Pool and Dataset Reconfiguration

  • Supports changing the zpool configuration at the destination when desired, rather than blindly reproducing the original
  • Applies a new or different layout, including alternate vdev layouts and arrangements
  • Adjusts dataset properties during restore: compression, encryption, and any other user-selectable dataset properties
  • Enables the user to change how a system is provisioned while restoring it to fresh hardware

Optimized Operations with ZFS Channel Programs

  • Uses ZFS channel programs where appropriate to obtain optimal performance per operation
  • Complex, multi-step operations (e.g. evaluating a snapshot chain, computing incremental diffs, or applying retention decisions) are executed atomically inside the kernel
  • Channel programs reduce the number of round trips and the overhead of repeated API calls, keeping every operation as fast and efficient as possible

Metadata Database and Fast Queries

  • Snapshot and bookmark metadata is cached in a database for significantly faster queries and listings
  • Instead of repeatedly walking the ZFS namespace, the framework queries a maintained index of snapshots, bookmarks, holds, sizes, and creation times
  • Enables quick, responsive browsing of history, rapid retention evaluation, and fast operations across many systems and datasets
  • The database is kept consistent with the live ZFS state by the monitoring and detection services

Core Operations

The framework runs a well-defined set of core operations to keep every system protected and recoverable. Each is configurable through the management console and automation layer.

Snapshoting

  • Captures consistent, point-in-time snapshots of each dataset on a schedule
  • Per-system and per-dataset granularity for schedule and scope

Bookmarking

  • Records lightweight bookmark references alongside snapshots
  • Preserves replication continuity independently of full snapshot data
  • Enables resynchronization of a server whose snapshots have been lost

Mirroring (ZFS send/receive)

  • Propagates snapshots and their data between the source, local backup server, and remote off-site server
  • Uses incremental streams to transfer only changes between successive snapshots
  • Optimized for reliability, speed, and minimal bandwidth

Placing Holds

  • Locks critical snapshots with holds to prevent accidental or policy-driven destruction
  • Ensures important recovery points survive retention and purge cycles

Recovery Back to the Source

  • Restores individual files or entire datasets back to the originating computer on demand
  • Supports browsing snapshot history to select the exact point-in-time to recover from

Retention Policy Management and Enforcement

  • Manages and enforces retention policies across every dataset
  • Dictates how many snapshots of each age are retained
  • Applies at the point of capture and cascades through mirroring and restore decisions

Intelligent Purging of Snapshots

  • Removes outdated snapshots from history according to the retention policy
  • Aware of which snapshots are still referenced, held, or required elsewhere
  • Never destroys snapshots that are protected or still needed for replication

Intelligent Purging of Bookmarks

  • Removes bookmarks that are no longer needed once they are no longer required for resynchronization or replication continuity
  • Avoids unbounded bookmark accumulation while preserving those that still serve recovery or mirroring
  • All purge behavior exposed via configuration options to tune aggressiveness and retention

Engineering Focus

Data integrity — snapshots, checksums, holds, and verified restores sit at the core of every decision.

Disaster recovery — the full restore path from network boot to verified, rebooted environment is a first-class, exercised workflow, not an afterthought.

Automation — the common restore path is fully guided; hardware adaptation and validation are handled systematically.

Remote operations — restores are designed to be executed and verified with no physical presence at the destination.

Storage architecture — ZFS pooling, layout migration, dataset placement, and chained replication are deliberate and repeatable.

Operational control — every step is observable, from snapshot history to hardware validation to post-restore verification.