MDEV-41207 Acquire metadata locks for recovered transaction

[Thirunarayanan]

Problem:
=======
A transaction being rolled back during recovery holds LOCK_IX on the
table(not the metadata locks), and the rollback thread holds
a reference on it.

An online ALTER TABLE on that table falls back to acquiring LOCK_S,
which conflicts and fails. prepare_inplace_alter_table_dict()
asserted that the reference count is 1 before checking whether
the table lock was acquired, so the reference still held by the
rollback thread makes the assertion fail.

Solution:
========
trx_recovery_thd: A background connection that owns the metadata
locks of the recovered transactions. One connection is shared by
all of recovering transaction.

trx_lists_init_at_db_start(): Create trx_recovery_thd, before any
metadata lock can be acquired.

trx_resurrect_table_locks(): Acquire a shared metadata lock on each
table that the recovered transaction had modified, and remember the
locks in trx_recovery_mdl. Recovered XA PREPARED transactions are
excluded, because they are completed by a user connection.

trx_recovery_mdl: The metadata locks of the recovered transactions,
by transaction. They are kept outside trx_t. Only a recovered
transaction ever has an entry.

trx_recovery_mdl_exists: Whether trx_recovery_mdl is not empty.
 It is read for every transaction in trx_t::free(),
so that the map will only be consulted while some recovered
transaction still holds metadata locks. It becomes false as
soon as the rollback of the recovered transactions has been
completed, long before trx_recovery_thd is
destroyed.

trx_t::free(): Release the metadata locks of a recovered transaction,
once its rollback has been completed. This is the only place that
releases them, so that a transaction which was rolled back by
trx_rollback_recovered(false) will not keep its locks until shutdown.

trx_recovery_thd_destroy(): Destroy trx_recovery_thd.
It is invoked by innodb_shutdown() only, after trx_sys.close()
has freed any recovered transaction that was left.

row_undo_mod(): Add the debug injection rollback_wait

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[dr-m]

What difference does this move make? The call to lock_sys_tables only impacts data dictionary tables, not user_table.

[dr-m]

If we unconditionally acquired an exclusive table lock here, we would duly wait for the recovered transaction to be rolled back. But we would not want to interfere with normal LOCK=NONE operation.

Can the reported scenario be fixed by acquiring MDL for each affected table in trx_resurrect_table_locks()?