yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1737879@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

1)vm shutdown info:
    43 2017-12-13 00:20:15.681+0000: shutting down, reason=crashed

2） nova start vm info:
    2017-12-13 00:24:07.337+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.3 (CentOS BuildSystem <http://bugs.centos.org>, 2017-09-07-11:27:44, c1bm.rdu2.centos.org), qemu versi    on: 1.5.3 (qemu-kvm-1.5.3-141.el7_4.2), hostname: l23-41-5 45 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-0000005c -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-g    uest-core=off -m 16384 -realtime mlock=off -smp 16,sockets=16,cores=16,threads=1 -uuid fc0cc515-5f7b-420c-bcda-d77b66610af2 -smbios 'type=1,manufacturer=Fedora Project,product=OpenStack     Nova,version=13.1.2-1.el7,serial=f7f54ab4-0b4f-499c-8d1d-291e11f50bf2,uuid=fc0cc515-5f7b-420c-bcda-d77b66610af2,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=cha    rmonitor,path=/var/lib/libvirt/qemu/domain-37-instance-0000005c/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-    pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:nova_instances/fc0cc515-5f7b-420c-bcda-d77b66610af2    _disk:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported=cephx\;none:mon_host=10.211.41.4\:6789\;10.211.41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-virtio-disk    0,cache=writeback,throttling.bps-read=52425500,throttling.bps-write=52425500' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -dri    ve 'file=rbd:cinder_volumes/volume-cd2fcd54-0661-4a4d-b13d-419b6585c676:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported=cephx\;none:mon_host=10.211.41.4\:6789\;10.211.    41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-virtio-disk1,serial=cd2fcd54-0661-4a4d-b13d-419b6585c676,cache=writeback' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=    drive-virtio-disk1,id=virtio-disk1 -drive 'file=rbd:nova_instances/fc0cc515-5f7b-420c-bcda-d77b66610af2_disk.config:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported=    cephx\;none:mon_host=10.211.41.4\:6789\;10.211.41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=writeback,throttling.bps-read=52425500,throttling.bps-write=    52425500' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=39 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:6    0:0b:06,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/data/nova/instances/fc0cc515-5f7b-420c-bcda-d77b66610af2/console.log -device isa-serial,chardev=charserial0,id=serial0 -char    dev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:6 -k en-us -vga cirrus -device virtio-balloon-pci,id=b    alloon0,bus=pci.0,addr=0x6 -msg timestamp=on
 46 2017-12-13 00:24:07.337+0000: Domain id=37 is tainted: high-privileges
 47 char device redirected to /dev/pts/7 (label charserial1)

3) Compute nodes system log: / var/log/messages
    Dec 13 08:20:15 l23-41-5 libvirtd: 2017-12-13 00:20:15.461+0000: 22613: error : qemuMonitorIO:697 : internal error: End of file from qemu monitor

4) openstack nova-compute log : /var/log/nova/nova-compute.log

    2017-12-13 08:20:30.751 22579 INFO nova.compute.manager [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Stopped (Lifecycle Event)
    2017-12-13 08:20:30.845 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
    2017-12-13 08:20:30.904 22579 WARNING nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
    2017-12-13 08:20:31.038 22579 INFO nova.extend.network [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william check_result: True
    2017-12-13 08:20:31.039 22579 INFO nova.extend.network [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william,vlan1210 is exists,so is not create
    2017-12-13 08:20:31.039 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william, live migrate vm  bridge_multiple_vlan True
    2017-12-13 08:20:31.054 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance is already powered off in the hypervisor when stop is called.
    2017-12-13 08:20:31.105 22579 INFO nova.virt.libvirt.driver [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance already shutdown.
    2017-12-13 08:20:31.114 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance destroyed successfully.

    2017-12-13 08:24:05.008 22579 INFO nova.compute.resource_tracker [req-493ee54b-e894-471c-93ee-c5c8781f9d36 - - - - -] Compute_service record updated for l23-41-5.lq.cc:l23-41-5
    2017-12-13 08:24:06.429 22579 INFO nova.extend.network [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william check_result: True
    2017-12-13 08:24:06.430 22579 INFO nova.extend.network [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william,vlan1210 is exists,so is not create
    2017-12-13 08:24:06.430 22579 INFO nova.compute.manager [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william, live migrate vm  bridge_multiple_vlan True
    2017-12-13 08:24:06.581 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance destroyed successfully.
    2017-12-13 08:24:06.616 22579 WARNING nova.virt.osinfo [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty)
    2017-12-13 08:24:07.260 22579 WARNING nova.virt.osinfo [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty)
    2017-12-13 08:24:07.281 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william , modify brdige type bridge = vlan1210
    2017-12-13 08:24:07.282 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] wiliam bridge_multiple_vlan True
    2017-12-13 08:24:07.282 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add William: Delete nwfilter rule <Element filterref at 0x7f84eb0>
    2017-12-13 08:24:07.290 22579 INFO nova.virt.libvirt.firewall [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Called setup_basic_filtering in nwfilter
    2017-12-13 08:24:07.290 22579 INFO nova.virt.libvirt.firewall [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Ensuring static filters
    2017-12-13 08:24:08.083 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Resumed (Lifecycle Event)
    2017-12-13 08:24:08.107 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance rebooted successfully.
    2017-12-13 08:24:08.187 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] During sync_power_state the instance has a pending task (powering-on). Skip.
    2017-12-13 08:24:08.187 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Started (Lifecycle Event)
    2017-12-13 08:25:06.395 22579 INFO nova.compute.resource_tracker [req-493ee54b-e894-471c-93ee-c5c8781f9d36 - - - - -] Auditing locally available compute resources for node l23-41-5

5) system info:
    1) libvirt qemu version
        Virsh command line tool of libvirt 3.2.0
        See web site at http://libvirt.org/

        Compiled with support for:
         Hypervisors: QEMU/KVM LXC ESX Test
         Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
         Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Gluster
         Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline Modular
        Note: instance-0000005c.log 8 hours time difference

    2) openstack nova-compute
        openstack-nova-network-13.1.2-1.el7.noarch
        openstack-nova-compute-13.1.2-1.el7.noarch
        python-nova-13.1.2-1.el7.noarch
        openstack-nova-common-13.1.2-1.el7.noarch
        python2-novaclient-3.3.2-1.el7.noarch

    3) ceph version
        ceph-10.2.2-0.el7.x86_64
        ceph-common-10.2.2-0.el7.x86_64
        ceph-base-10.2.2-0.el7.x86_64
        ceph-osd-10.2.2-0.el7.x86_64
        libcephfs1-10.2.2-0.el7.x86_64
        ceph-mon-10.2.2-0.el7.x86_64
        python-cephfs-10.2.2-0.el7.x86_64
        ceph-selinux-10.2.2-0.el7.x86_64
        ceph-mds-10.2.2-0.el7.x86_64
        ceph-deploy-1.5.34-0.noarch

Timing of VM downtime problems every day, don't know where the problem
cause downtime phenomenon, hope the community people can give a solution

Note: / var/log/libvirt/qemu/instance - 0000005 c. The log time is 8
hours

** Affects: nova
     Importance: Undecided
     Assignee: wangqiang.sheng (wangqiangsheng)
         Status: New

-- 
2017-12-13 00:20:15.681+0000: shutting down, reason=crashed
https://bugs.launchpad.net/bugs/1737879
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova).