MariaDB ColumnStore崩溃后无法重启，求无数据损失修复方案

From your process status and error logs, we've got two key issues here: a misconfigured mysqld variable triggering warnings, and an InnoDB crash during the purge process (caused by corrupted indexes/data pages) that's keeping mysqld stuck in MAN_OFFLINE. Let's work through this step by step without losing any data:

1. Resolve the mysqld Configuration Warning First #

Your err.log leads with this critical warning:

2020-01-09 14:30:01 0 [Warning] /usr/local/mariadb/columnstore/mysql//bin/mysqld: unknown variable 'loose-server_audit_syslog_info=MyColumnStoreClusterRcc'

This means your MySQL config file (typically /usr/local/mariadb/columnstore/mysql/my.cnf or /etc/my.cnf) includes a variable that either doesn't exist or requires an uninstalled audit plugin.

• Open the config file, locate the line with loose-server_audit_syslog_info, either comment it out (add a # at the start) or verify the audit plugin is installed and correct the variable name.
• Test starting mysqld alone to confirm the warning is resolved:

  /usr/local/mariadb/columnstore/mysql/bin/mysqld_safe --defaults-file=/usr/local/mariadb/columnstore/mysql/my.cnf &
  

  If it starts successfully, shut it down manually with mysqladmin -u root shutdown.

2. Force-Start mysqld with InnoDB Recovery Mode #

The stack trace in your log shows InnoDB crashing during row purge—this is a clear sign of corrupted indexes or data pages. We'll use innodb_force_recovery to get mysqld running, then repair the damaged tables:

• Edit your MySQL config again, add this line under the [mysqld] section:

  innodb_force_recovery = 1
  

  (Stick to the lowest value possible—1 or 2 usually fixes purge-related corruption. If 1 doesn't work, try 2, then 3, etc., up to 6. Higher values impose more restrictions, so use the minimum that allows mysqld to start.)
• Start mysqld with recovery mode enabled:

  /usr/local/mariadb/columnstore/mysql/bin/mysqld_safe --defaults-file=/usr/local/mariadb/columnstore/mysql/my.cnf &
  

• Once mysqld is running, connect to it directly:

  /usr/local/mariadb/columnstore/mysql/bin/mysql -u root
  

• Identify damaged tables: Check the err.log for clues, or run CHECK TABLE on suspect tables (or all tables if you're unsure):

  CHECK TABLE your_database.your_table;
  

• Repair damaged tables: For InnoDB, the safest approach is to dump the table, drop it, then reimport (since REPAIR TABLE has limited support for InnoDB):

  # Dump the damaged table to a backup file
  /usr/local/mariadb/columnstore/mysql/bin/mysqldump -u root your_database your_table > your_table_backup.sql
  # Drop the corrupted table from the database
  mysql -u root your_database -e "DROP TABLE your_table;"
  # Reimport the table from the backup
  mysql -u root your_database < your_table_backup.sql
  

• After fixing all damaged tables, remove the innodb_force_recovery line from your config and restart mysqld to confirm it starts normally without recovery mode.

3. Restore ColumnStore System Processes #

Now that mysqld is working correctly, let's fix the ColumnStore process state:

• Stop all ColumnStore processes completely to clear stale states:

  mcsadmin shutdownSystem y
  

• Clean up any leftover system state files that might be blocking process startup:

  rm -rf /usr/local/mariadb/columnstore/data1/systemFiles/*
  

• Start the ColumnStore system back up:

  mcsadmin startSystem
  

• Wait 2-3 minutes, then check the process status with mcsadmin getProcessStatus—you should see mysqld um1 switch to ACTIVE.

4. Verify Data & Functionality #

Once everything is running, validate the fix:

• Connect using mcsmysql to confirm no connection errors:

  mcsmysql -u root
  

• Query critical tables to ensure data is intact, and test basic DML operations (insert/update/delete) to confirm normal functionality.

Important Notes #

• In innodb_force_recovery mode, MySQL runs in read-only mode—you can only query and dump tables, not modify them. Don't skip the drop/reimport step, as it's the only reliable way to resolve InnoDB corruption here.
• Always back up your data before performing any repair operations, even if you're confident in the steps.
• If you still run into ColumnStore process issues, double-check that all server clocks are synchronized (ColumnStore is sensitive to time drifts) and that there are no filesystem permission issues on ColumnStore directories.

内容的提问来源于stack exchange，提问作者M.ferlay