MDEV-14992 BACKUP SERVER

[dr-m]

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory';
BACKUP SERVER TO '/path/to/directory' 1 CONCURRENT;
BACKUP SERVER WITH 'command';
BACKUP SERVER WITH 1 CONCURRENT 'command';

In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to.

For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file.

The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used.

Note: The parameter innodb_log_recovery_start in backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database!

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-39092 Less blocking backup of ENGINE=Aria

The implementation introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_file(): A partial or sparse file-copying service for all systems.

backup_stream_append(): Equivalent to copy_file(), but appending to a stream. On Linux, this uses sendfile(2), which assumes that the source data will not be changed before the data has been consumed from the pipe.

backup_stream_append_async(): A variant of backup_stream_append() where the source file region is guaranteed to be immutable after the call returns. We must not use Linux sendfile(2) for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe.

InnoDB_backup::context: Backup context, attached to backup_sink so that context can continue to exist between the time a BACKUP SERVER releases all locks and another BACKUP SERVER starts executing, with innodb_backup pointing to the new backup, while the old backup is still being finished.

fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup.

fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup.

buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread().

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Thirunarayanan]

I think it is close to completion (Only InnoDB part)

[Thirunarayanan]

Here unlink() is being done while holding log_sys.mutex write lock. It is not right because it could block all commit till unlink happens. May be store the lsn in std::vector and do deleting later?

[dr-m]

std::vector may involve memory allocation and can fail. We do want to delete the log files, no matter what.

Outside Windows, our caller is holding an open file handle, but it will be closed before log_resize_release() will be released. This call should be very rare, only possible if more than one innodb_log_archive=ON file were completed after the end of backup.

Let us think when this code could be invoked. There must be a prior call to InnoDB_backup::commit(), from innodb_backup_start(…, BACKUP_PHASE_NO_COMMIT, …). At that phase we would copy or stream some InnoDB logs, and some non-ACID files, but not any InnoDB data files. Yes, this step could take some time, for example when streaming a lot of log over a slow connection.

We might reduce the probability of invoking this branch by stopping the log archiving as soon as the last log file has been handled, instead of waiting until innodb_backup_end(…, BACKUP_PHASE_FINISH_END, …).

[dr-m]

If we fail, we might want to attempt to remove the incomplete BACKUP SERVER TO target directory. Then again, the directory could be the last resort of a desperate DBA who attempts to rescue what can be rescued from a failing database server. Maybe this is fine as is, and it is up to the user to review the backup directory and the error logs for any clues why the backup failed, or to delete and try again.

[mariadb-SaahilAlam]

Hitting this issue , Please take a look

# 2026-09-15T01:15:12 [3331132] | # 2026-09-15T01:15:03 [3331132] | 2026-09-15  1:15:03 0 [ERROR] InnoDB: ib_0000000000344ff0.log is in unrecognized format
# 2026-09-15T01:15:12 [3331132] | # 2026-09-15T01:15:03 [3331132] | 2026-09-15  1:15:03 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1422] with error Generic error
# 2026-09-15T01:15:12 [3331132] | # 2026-09-15T01:15:03 [3331132] | 2026-09-15  1:15:03 0 [Note] InnoDB: Starting shutdown...
# 2026-09-15T01:15:12 [3331132] | # 2026-09-15T01:15:03 [3331132] | 2026-09-15  1:15:03 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed

saahil@sdp:/data/results/1789459029/000384/1_clone/fbackup/data$ od -Ax -t x1 ib_0000000000344ff0.log|head
000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
003000 fa 00 00 00 00 00 00 00 34 4f f0 01 bb 9a 89 0b
003010 34 01 00 75 87 35 01 44 38 02 86 c3 08 04 ff d3
003020 02 06 0b c3 04 04 ff d3 02 06 0b c3 04 04 ff d3
003030 02 06 0b b5 00 05 d6 69 d2 c4 00 81 80 ff 35 01
003040 00 80 3b aa 34 01 00 3d 3c 12 01 36 37 01 44 72
003050 00 00 00 36 34 01 36 19 06 b2 2b 32 34 01 36 43
003060 44 34 01 36 3f 01 a1 02 b2 13 56 c3 1e 04 ff d3
003070 02 06 0b 34 01 36 49 01 b7 00 00 00 00 36 00 2c

Logs are present on SDP:-
/data/results/1789459029/000384

[dr-m]

Can you present the complete and relevant list of events, including the BACKUP SERVER statement that successfully completed, and which parameters the server was started with? Can you reproduce this with a working trace of the mariadbd process on which BACKUP SERVER was run? The current trace is incomplete:

rr replay /data/results/1789459029/000384/1/rr/mariadbd-1
…
2026-09-15  1:15:08 19 [ERROR] InnoDB: Transaction was aborted due to Deadlock
2026-09-15  1:15:08 18 [ERROR] InnoDB: Transaction was aborted due to Record changed
2026-09-15  1:15:10 0 [Note] InnoDB: Resized log to 150.000MiB; start LSN=3501395
2026-09-15  1:15:12 0 [Note] InnoDB: Resized log to 50.000MiB; start LSN=3502278
2026-09-15  1:15:13 17 [Warning] Could not read packet: fd: 48  state: 1  read_length: 4  errno: 11  vio_errno: 1158  length: 0
2026-09-15  1:15:13 17 [Warning] Aborted connection 17 to db: 'mysql' user: 'root' host: 'localhost' (Got an error reading communication packets)
Remote connection closed
(rr) when
Traceback (most recent call last):
  File "<string>", line 71, in invoke
  File "<string>", line 83, in rr_cmd
gdb.error: packets can only be sent to a remote target
Error occurred in Python: packets can only be sent to a remote target

It looks like the 1_clone/fbackup had been specified as a target directory of a BACKUP SERVER TO statement. That directory appears to contain a valid backup.cnf file. The error is that a log checkpoint header had not been written. Where, I can’t tell, because I do not have a working rr replay trace that covers the BACKUP SERVER TO execution that produced the file.

[mariadb-SaahilAlam]

Please check if this recovery crash is related

# 2026-09-16T13:20:36 [467975] | mariadbd: /data/Server/MDEV-14992_02/storage/innobase/os/os0file.cc:1383: bool os_file_rename_func(const char*, const char*): Assertion !exists' failed.
`
RR trace is on SDP :-

/data/results/1789584127/002401

[dr-m]

The test run includes 8ace36b which is a preliminary version of #5684. It was a mistake to include it before it was stress-tested. I will wait for your test results for that one.

[dr-m]

Is this still reproducible, now that 8ace36b is omitted?