yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1115153] Re: Instances do not get a (serial) console

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: FlorianOtel <florian.otel@xxxxxxxxx>
Date: Fri, 15 Feb 2013 10:17:21 -0000
Reply-to: Bug 1115153 <1115153@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Incomplete => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1115153

Title:
  Instances do not get a (serial) console

Status in OpenStack Compute (Nova):
  Opinion

Bug description:
  First, a caveat: This might be very well related to a misconfig
  behaving badly. Still, as no error is reported I'll treat this as a
  bug:

  Setup
  --------

  3-node setup -- Controller ("Folsom 1"), Compute ("Folsom 2") and Network ("Folsom 3")
  OS: Ubuntu 12.04
  Code:  OVS and Quamtum directly from trunk; all the rest  as per Ubuntu packages in 12.04.

  OpenStack  Services: 
  (Folsom 1, Controller -- 10 ):   
  Keystone, Swift, Glance on Swift, Quantum server.  For Nova, the _only_ Nova services on the controller are:
          nova-api
          nova-scheduler
          
  (Folsom 2, Compute):  
  Quantum OVS plugin and agent. 
   For Nova, this node is only running "nova-compute" (not:  metadata, novnc, nova-cert, nova-consoleatuth... etc... etc...See below the rationale) 

  (Folsom 3): 
  Quntum OVS agent, Quantum DHCP agent. 

  Please note that I do not run "novnc" "nova-consoleauth" "nova-
  novncproxy"  (nor Horizon for that matter) as my interest is only in
  console/SSH  access to the instances  (no  VNC / GUI / Dashboard )

  Also, as per above, I am not running "nova-api-metadata" either.

  Problem synopsis
  ------------------------

  Instances are spawned successfully --  shown as "ACTIVE" ---  but
  "console.log" are empty (0 bytes) files and with no console listed.
  This is disregarding what image is tested (tried: CirrOS, TTY-linux
  and Ubuntu-12.04 server images, all with same effect):

  On "Compute" (Folsom 2) node:

   root@Folsom-2:~# nova list

  +--------------------------------------+--------------+--------+------------------------+
  | ID                                   | Name         | Status | Networks               |
  +--------------------------------------+--------------+--------+------------------------+
  | c4bd7d22-a859-4603-9ec4-9f91f25f81a4 | Cirros0Sv8   | ACTIVE | SvcIntNet1=169.254.1.3 |
  | 2c4e0e56-9140-46a6-b07d-89a2f3b44730 | TTY-linux-v1 | ACTIVE | SvcIntNet1=169.254.1.4 |
  +--------------------------------------+--------------+--------+------------------------+

  
  ######## The KVM processes: 

  root@server-1355685201-az-2-region-a-geo-1:~# ps axuww | grep -i kvm
  107       3487  0.3  0.1 3607676 48196 ?       Sl   Jan25  48:09 /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -cpu qemu32 -no-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name VM-00000009 -uuid 2c4e0e56-9140-46a6-b07d-89a2f3b44730 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/VM-00000009.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -kernel /var/lib/nova/instances/VM-00000009/kernel -initrd /var/lib/nova/instances/VM-00000009/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/var/lib/nova/instances/VM-00000009/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/var/lib/nova/instances/VM-00000009/disk.local,if=none,id=drive-virtio-disk1,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,ifname=tapcb09e56c-45,script=,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d9:33:cb,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/VM-00000009/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
  107      28392  0.3  0.1 3662384 38420 ?       Sl   Jan25  48:07 /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -cpu qemu32 -no-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name VM-00000008 -uuid c4bd7d22-a859-4603-9ec4-9f91f25f81a4 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/VM-00000008.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/var/lib/nova/instances/VM-00000008/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/VM-00000008/disk.local,if=none,id=drive-virtio-disk1,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,ifname=tap620a3e4d-cb,script=,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d9:d3:89,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/VM-00000008/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

  
  ######## The instance details in Nova: 

  root@Folsom2:~# ll /var/lib/nova/instances/VM-00000008/
  total 740
  drwxr-xr-x 2 root         root   4096 Jan 25 11:34 ./
  drwxr-xr-x 5 nova         nova   4096 Jan 25 11:58 ../
  -rw-rw---- 1 libvirt-qemu kvm       0 Jan 25 11:35 console.log
  -rw-r--r-- 1 libvirt-qemu kvm  720896 Jan 25 11:34 disk
  -rw-r--r-- 1 libvirt-qemu kvm  197120 Jan 25 11:34 disk.local
  -rw-r--r-- 1 root         root   1253 Jan 25 11:34 libvirt.xml

  
  root@Folsom2:~# ll /var/lib/nova/instances/VM-00000009
  total 10816
  drwxr-xr-x 2 root         root    4096 Jan 25 11:58 ./
  drwxr-xr-x 5 nova         nova    4096 Jan 25 11:58 ../
  -rw-rw---- 1 libvirt-qemu kvm        0 Jan 25 11:58 console.log
  -rw-r--r-- 1 libvirt-qemu kvm   720896 Jan 25 11:58 disk
  -rw-r--r-- 1 libvirt-qemu kvm   197120 Jan 25 11:58 disk.local
  -rw-r--r-- 1 libvirt-qemu kvm  4404752 Jan 25 11:58 kernel
  -rw-r--r-- 1 root         root    1412 Jan 25 11:58 libvirt.xml
  -rw-r--r-- 1 libvirt-qemu kvm  5882349 Jan 25 11:58 ramdisk

  ######## The "libvirt.xml" file for "VM-00000008":

  root@server-1355685201-az-2-region-a-geo-1:~# cat /var/lib/nova/instances/VM-00000008/libvirt.xml 
  <domain type="qemu">
    <uuid>c4bd7d22-a859-4603-9ec4-9f91f25f81a4</uuid>
    <name>VM-00000008</name>
    <memory>2097152</memory>
    <vcpu>1</vcpu>
    <os>
      <type>hvm</type>
      <boot dev="hd"/>
    </os>
    <features>
      <acpi/>
    </features>
    <clock offset="utc"/>
    <cpu match="exact">
      <arch>x86_64</arch>
      <model>qemu32</model>
      <vendor>Intel</vendor>
    </cpu>
    <devices>
      <disk type="file" device="disk">
        <driver name="qemu" type="qcow2" cache="none"/>
        <source file="/var/lib/nova/instances/VM-00000008/disk"/>
        <target bus="virtio" dev="vda"/>
      </disk>
      <disk type="file" device="disk">
        <driver name="qemu" type="qcow2" cache="none"/>
        <source file="/var/lib/nova/instances/VM-00000008/disk.local"/>
        <target bus="virtio" dev="vdb"/>
      </disk>
      <interface type="ethernet">
        <mac address="fa:16:3e:d9:d3:89"/>
        <model type="virtio"/>
        <script path=""/>
        <target dev="tap620a3e4d-cb"/>
      </interface>
      <serial type="file">
        <source path="/var/lib/nova/instances/VM-00000008/console.log"/>
      </serial>
      <serial type="pty"/>
      <input type="tablet" bus="usb"/>
      <graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
    </devices>
  </domain>


  ######## Virsh:

  root@server-1355685201-az-2-region-a-geo-1:~# virsh -c qemu:///system list
   Id Name                 State
  ----------------------------------
    7 VM-00000008          running
    8 VM-00000009          running

  
  ########  No serial console for neither of the VMs. 

  root@server-1355685201-az-2-region-a-geo-1:~# virsh console VM-00000008
  Connected to domain VM-00000008
  Escape character is ^]
  error: internal error character device (null) is not using a PTY

  root@server-1355685201-az-2-region-a-geo-1:~# virsh console VM-00000009
  Connected to domain VM-00000009
  Escape character is ^]
  error: internal error character device (null) is not using a PTY

  In other words the instances are reported up and running where in fact
  they are unreachable /  unresponsive.

  Attached is the "nova-compute.log", covering spawning "VM-00000008".
  The "nova.conf" file included below.

  Thanks,

  Florian

  -------

  P.S. The "nova.conf" file. Idenitical on both "Compute" and
  "Controller".   In the config below "10.7.68.179" is the IP address of
  the Controller (Folsom1)

  root@Folsom2:~# cat /etc/nova/nova.conf
  [DEFAULT]

  # LOGS/STATE
  verbose=True
  logdir=/var/log/nova
  state_path=/var/lib/nova
  lock_path=/var/lock/nova

  # AUTHENTICATION
  auth_strategy=keystone

  # DATABASE
  sql_connection=mysql://novadbadmin:novadbadminpasswd@10.7.68.179/nova

  # COMPUTE 
  connection_type=libvirt
  libvirt_type=qemu
  instance_name_template=VM-%08x
  api_paste_config=/etc/nova/api-paste.ini
  allow_resize_to_same_host=True
  libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
  libvirt_use_virtio_for_bridges=True

  
  # APIs -- Only enable compute and metadata (without EC2 compat) 
  enabled_apis = osapi_compute,metadata
  osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions

  # RABBITMQ
  rabbit_host=10.7.68.179
  rabbbit_user=guest
  rabbit_password=guest

  # SCHEDULER
  compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

  # GLANCE
  image_service=nova.image.glance.GlanceImageService
  glance_api_servers=10.7.68.179:9292

  # NETWORK -- Quantum
  network_api_class=nova.network.quantumv2.api.API
  quantum_url=http://10.7.68.179:9696
  quantum_auth_strategy=keystone
  quantum_admin_tenant_name=service
  quantum_admin_username=quantumadmin
  quantum_admin_password=quantumadminpasswd
  quantum_admin_auth_url=http://10.7.68.179:5001/v2.0/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1115153/+subscriptions