← Back to team overview

maria-discuss team mailing list archive

Failure upon startup due to changes in /var/lib/mysql/databasename/

 

Hello,

Thanks for the prior suggestions, they ended well.

I found a behavior that I found a slight bit confounding.

While I tested the server, one of the wikis I host was modified by a
friend of mine.

When I broke MariaDB, it was no longer feasible to start it, (to the
degree that it could have taken days of tinkering),  With MariaDB verified
to be not running, I tarred the directory /var/lib/mysql/mywiki/ into a
file mywiki_with_new_data.tar.

I reinstalled MariaDB thoroughly to force the creation of a new
/var/lib/mysql/ and /etc/mysql/ .  I felt it was safe to recover the
databases from the backup SQL file I had created on the currently running
machine.  Everything now works great, but the mywiki database doesn't
reflect the changes anymore.

So I stop the mysql process and then for the sake of recoverability, I tar
up /var/lib/mysql/mywiki into a file called mywiki_without_new_data.tar.
I then rm -rf /var/lib/mysql/mywiki and then untar the
mywiki_with_new_data.tar in it's place.  MariaDB will no longer start.  If
I again rm -rf /var/lib/mysql/mywiki and then untar
mywiki_without_new_data.tar into the location, MariaDB starts without
error.

This isn't a code blue situation, my friend can recreate the change he
made and that is the easiest solution, but others reading this might not
have that luxury.  I am also quite curious about what went wrong and how
to repair the problem.  I didn't think this was possible to fail.

There is an interesting except from the errors described below.  I'm not
certain what exactly this means, or how to correct it.  Given that MariaDB
could have failed uncleanly when the mysql database was imported, recovery
may be infeasible.  (the wiki connected to MariaDB as root, it's bad, I
know)

<excerpt /var/lib/mysql/altdorf.err>
InnoDB: Error: tablespace id is 235 in the data dictionary
InnoDB: but in file ./mywiki/objectcache.ibd it is 254!
2017-06-07 11:45:08 5c7ff420  InnoDB: Assertion failure in thread
1551889440 in file fil0fil.cc line 638
InnoDB: We intentionally generate a memory trap.
</excerpt>



# service mysql start
Job for mysql.service failed. See 'systemctl status mysql.service' and
'journalctl -xn' for details.


# systemctl -l status mysql.service
* mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: failed (Result: exit-code) since Wed 2017-06-07 11:45:40 CEST;
1min 16s ago
  Process: 1692 ExecStop=/etc/init.d/mysql stop (code=exited,
status=0/SUCCESS)
  Process: 1753 ExecStart=/etc/init.d/mysql start (code=exited,
status=1/FAILURE)

Jun 07 11:45:08 altdorf mysql[1753]: /etc/init.d/mysql: WARNING:
/etc/mysql/my.cnf cannot be read. See README.Debian.gz ... (warning).
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: 0 processes alive and
'/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted
in
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: [61B blob data]
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: error: 'Can't connect to
local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No
such file or directory")'
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Check that mysqld is
running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]:
Jun 07 11:45:40 altdorf mysql[1753]: Starting MariaDB database server:
mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
Jun 07 11:45:40 altdorf systemd[1]: mysql.service: control process exited,
code=exited status=1
Jun 07 11:45:40 altdorf systemd[1]: Failed to start LSB: Start and stop
the mysql database server daemon.
Jun 07 11:45:40 altdorf systemd[1]: Unit mysql.service entered failed state.


# journalctl -xn
-- Logs begin at Wed 2017-05-24 10:06:00 CEST, end at Wed 2017-06-07
11:45:40 CEST. --
Jun 07 11:45:08 altdorf /etc/init.d/mysql[1764]: WARNING:
/etc/mysql/my.cnf cannot be read. See
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: 0 processes alive and
'/usr/bin/mysqladmin --de
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: [61B blob data]
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: error: 'Can't connect to
local MySQL server thr
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Check that mysqld is
running and that the socke
Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]:
Jun 07 11:45:40 altdorf mysql[1753]: Starting MariaDB database server:
mysqld . . . . . . . . .
Jun 07 11:45:40 altdorf systemd[1]: mysql.service: control process exited,
code=exited status=1
Jun 07 11:45:40 altdorf systemd[1]: Failed to start LSB: Start and stop
the mysql database serve
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysql.service has failed.
-- 
-- The result is failed.
Jun 07 11:45:40 altdorf systemd[1]: Unit mysql.service entered failed state.


# cat /var/lib/mysql/aldorf.err   (excerpts from file with a database name
changed to protect user privacy)

170607 11:45:08 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql
170607 11:45:08 [Note] /usr/sbin/mysqld (mysqld 10.0.30-MariaDB-0+deb8u2)
starting as process 1851 ...
170607 11:45:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages
170607 11:45:08 [Note] InnoDB: The InnoDB memory heap is disabled
170607 11:45:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
170607 11:45:08 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used
for memory barrier
170607 11:45:08 [Note] InnoDB: Compressed tables use zlib 1.2.8
170607 11:45:08 [Note] InnoDB: Using Linux native AIO
170607 11:45:08 [Note] InnoDB: Not using CPU crc32 instructions
170607 11:45:08 [Note] InnoDB: Initializing buffer pool, size = 128.0M
170607 11:45:08 [Note] InnoDB: Completed initialization of buffer pool
170607 11:45:08 [Note] InnoDB: Highest supported file format is Barracuda.
170607 11:45:08 [Note] InnoDB: 128 rollback segment(s) are active.
170607 11:45:08 [Note] InnoDB: Waiting for purge to start
InnoDB: Error: tablespace id is 235 in the data dictionary
InnoDB: but in file ./mywiki/objectcache.ibd it is 254!
2017-06-07 11:45:08 5c7ff420  InnoDB: Assertion failure in thread
1551889440 in file fil0fil.cc line 638
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170607 11:45:08 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.30-MariaDB-0+deb8u2
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
465934 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170607 11:45:08 mysqld_safe mysqld from pid file
/var/lib/mysql/altdorf.pid ended




Follow ups