maas-devel team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I ended up in a situation where the rabbit on my machine was consuming
150% CPU when there was actually no work to be done. I think I figured
out how we got into this state, and I proposed a patch for it.[1]

However, I still needed to get out of that state. This website was
very valuable:
http://archi-jab-ture.blogspot.com/2012/03/manually-resetting-mnesia-to-bring-up.html

Most of the tools wanted me to list the queues, and then purge them,
but things were so backed up, I couldn't get 'rabbitmqctl list_queues'
to ever return.

I also tried to 'rabbitmqctl reset' but that would fail with 'mnesia
unexpectedly running'. The docs here:
http://www.rabbitmq.com/man/rabbitmqctl.1.man.html

Seem to say that you need to run 'rabbitmqctl stop_app' before doing
'reset'. However doing 'stop_app' would always kill the whole process
(as though I had done 'stop'). And you need the process running in
order to reset it.

One more thing to watch out for from the above 'manually-resetting'
link above

   mnesia:info().

The final '.' is significant. It seems to be the end-of-command
character. (like ; in SQL). The function won't do anything if you
don't type it.
When you run that, you need it to tell you that the directory is being
used, otherwise when you run the 'delete_schema' step, it will say
'ok' but won't actually delete anything, because there was nothing to
delete.

Anyway, celery is happier now, and I can get things to talk again.

John
=:->


[1] Essentially, if you use celery and make a request, it will create
a queue that won't go away until you consume all the responses. But we
never consumed any responses, so we just created an enormous backlog.

https://code.launchpad.net/~jameinel/maas/ignore_results/+merge/128204
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Cygwin)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBuuK0ACgkQJdeBCYSNAAP02gCglrFzM7lhb8uk+dJwWyIueaQL
Q7cAn0d6xqMYTYBFbVzAbMMVbjLoas2I
=mnBX
-----END PGP SIGNATURE-----