yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #00421
[Bug 1115153] Re: Instances do not get a (serial) console
** Changed in: nova
Status: Incomplete => Opinion
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1115153
Title:
Instances do not get a (serial) console
Status in OpenStack Compute (Nova):
Opinion
Bug description:
First, a caveat: This might be very well related to a misconfig
behaving badly. Still, as no error is reported I'll treat this as a
bug:
Setup
--------
3-node setup -- Controller ("Folsom 1"), Compute ("Folsom 2") and Network ("Folsom 3")
OS: Ubuntu 12.04
Code: OVS and Quamtum directly from trunk; all the rest as per Ubuntu packages in 12.04.
OpenStack Services:
(Folsom 1, Controller -- 10 ):
Keystone, Swift, Glance on Swift, Quantum server. For Nova, the _only_ Nova services on the controller are:
nova-api
nova-scheduler
(Folsom 2, Compute):
Quantum OVS plugin and agent.
For Nova, this node is only running "nova-compute" (not: metadata, novnc, nova-cert, nova-consoleatuth... etc... etc...See below the rationale)
(Folsom 3):
Quntum OVS agent, Quantum DHCP agent.
Please note that I do not run "novnc" "nova-consoleauth" "nova-
novncproxy" (nor Horizon for that matter) as my interest is only in
console/SSH access to the instances (no VNC / GUI / Dashboard )
Also, as per above, I am not running "nova-api-metadata" either.
Problem synopsis
------------------------
Instances are spawned successfully -- shown as "ACTIVE" --- but
"console.log" are empty (0 bytes) files and with no console listed.
This is disregarding what image is tested (tried: CirrOS, TTY-linux
and Ubuntu-12.04 server images, all with same effect):
On "Compute" (Folsom 2) node:
root@Folsom-2:~# nova list
+--------------------------------------+--------------+--------+------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------------+--------+------------------------+
| c4bd7d22-a859-4603-9ec4-9f91f25f81a4 | Cirros0Sv8 | ACTIVE | SvcIntNet1=169.254.1.3 |
| 2c4e0e56-9140-46a6-b07d-89a2f3b44730 | TTY-linux-v1 | ACTIVE | SvcIntNet1=169.254.1.4 |
+--------------------------------------+--------------+--------+------------------------+
######## The KVM processes:
root@server-1355685201-az-2-region-a-geo-1:~# ps axuww | grep -i kvm
107 3487 0.3 0.1 3607676 48196 ? Sl Jan25 48:09 /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -cpu qemu32 -no-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name VM-00000009 -uuid 2c4e0e56-9140-46a6-b07d-89a2f3b44730 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/VM-00000009.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -kernel /var/lib/nova/instances/VM-00000009/kernel -initrd /var/lib/nova/instances/VM-00000009/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/var/lib/nova/instances/VM-00000009/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/var/lib/nova/instances/VM-00000009/disk.local,if=none,id=drive-virtio-disk1,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,ifname=tapcb09e56c-45,script=,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d9:33:cb,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/VM-00000009/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
107 28392 0.3 0.1 3662384 38420 ? Sl Jan25 48:07 /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -cpu qemu32 -no-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name VM-00000008 -uuid c4bd7d22-a859-4603-9ec4-9f91f25f81a4 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/VM-00000008.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/var/lib/nova/instances/VM-00000008/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/VM-00000008/disk.local,if=none,id=drive-virtio-disk1,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,ifname=tap620a3e4d-cb,script=,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d9:d3:89,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/VM-00000008/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
######## The instance details in Nova:
root@Folsom2:~# ll /var/lib/nova/instances/VM-00000008/
total 740
drwxr-xr-x 2 root root 4096 Jan 25 11:34 ./
drwxr-xr-x 5 nova nova 4096 Jan 25 11:58 ../
-rw-rw---- 1 libvirt-qemu kvm 0 Jan 25 11:35 console.log
-rw-r--r-- 1 libvirt-qemu kvm 720896 Jan 25 11:34 disk
-rw-r--r-- 1 libvirt-qemu kvm 197120 Jan 25 11:34 disk.local
-rw-r--r-- 1 root root 1253 Jan 25 11:34 libvirt.xml
root@Folsom2:~# ll /var/lib/nova/instances/VM-00000009
total 10816
drwxr-xr-x 2 root root 4096 Jan 25 11:58 ./
drwxr-xr-x 5 nova nova 4096 Jan 25 11:58 ../
-rw-rw---- 1 libvirt-qemu kvm 0 Jan 25 11:58 console.log
-rw-r--r-- 1 libvirt-qemu kvm 720896 Jan 25 11:58 disk
-rw-r--r-- 1 libvirt-qemu kvm 197120 Jan 25 11:58 disk.local
-rw-r--r-- 1 libvirt-qemu kvm 4404752 Jan 25 11:58 kernel
-rw-r--r-- 1 root root 1412 Jan 25 11:58 libvirt.xml
-rw-r--r-- 1 libvirt-qemu kvm 5882349 Jan 25 11:58 ramdisk
######## The "libvirt.xml" file for "VM-00000008":
root@server-1355685201-az-2-region-a-geo-1:~# cat /var/lib/nova/instances/VM-00000008/libvirt.xml
<domain type="qemu">
<uuid>c4bd7d22-a859-4603-9ec4-9f91f25f81a4</uuid>
<name>VM-00000008</name>
<memory>2097152</memory>
<vcpu>1</vcpu>
<os>
<type>hvm</type>
<boot dev="hd"/>
</os>
<features>
<acpi/>
</features>
<clock offset="utc"/>
<cpu match="exact">
<arch>x86_64</arch>
<model>qemu32</model>
<vendor>Intel</vendor>
</cpu>
<devices>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"/>
<source file="/var/lib/nova/instances/VM-00000008/disk"/>
<target bus="virtio" dev="vda"/>
</disk>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"/>
<source file="/var/lib/nova/instances/VM-00000008/disk.local"/>
<target bus="virtio" dev="vdb"/>
</disk>
<interface type="ethernet">
<mac address="fa:16:3e:d9:d3:89"/>
<model type="virtio"/>
<script path=""/>
<target dev="tap620a3e4d-cb"/>
</interface>
<serial type="file">
<source path="/var/lib/nova/instances/VM-00000008/console.log"/>
</serial>
<serial type="pty"/>
<input type="tablet" bus="usb"/>
<graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
</devices>
</domain>
######## Virsh:
root@server-1355685201-az-2-region-a-geo-1:~# virsh -c qemu:///system list
Id Name State
----------------------------------
7 VM-00000008 running
8 VM-00000009 running
######## No serial console for neither of the VMs.
root@server-1355685201-az-2-region-a-geo-1:~# virsh console VM-00000008
Connected to domain VM-00000008
Escape character is ^]
error: internal error character device (null) is not using a PTY
root@server-1355685201-az-2-region-a-geo-1:~# virsh console VM-00000009
Connected to domain VM-00000009
Escape character is ^]
error: internal error character device (null) is not using a PTY
In other words the instances are reported up and running where in fact
they are unreachable / unresponsive.
Attached is the "nova-compute.log", covering spawning "VM-00000008".
The "nova.conf" file included below.
Thanks,
Florian
-------
P.S. The "nova.conf" file. Idenitical on both "Compute" and
"Controller". In the config below "10.7.68.179" is the IP address of
the Controller (Folsom1)
root@Folsom2:~# cat /etc/nova/nova.conf
[DEFAULT]
# LOGS/STATE
verbose=True
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
# AUTHENTICATION
auth_strategy=keystone
# DATABASE
sql_connection=mysql://novadbadmin:novadbadminpasswd@10.7.68.179/nova
# COMPUTE
connection_type=libvirt
libvirt_type=qemu
instance_name_template=VM-%08x
api_paste_config=/etc/nova/api-paste.ini
allow_resize_to_same_host=True
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
libvirt_use_virtio_for_bridges=True
# APIs -- Only enable compute and metadata (without EC2 compat)
enabled_apis = osapi_compute,metadata
osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
# RABBITMQ
rabbit_host=10.7.68.179
rabbbit_user=guest
rabbit_password=guest
# SCHEDULER
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
# GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=10.7.68.179:9292
# NETWORK -- Quantum
network_api_class=nova.network.quantumv2.api.API
quantum_url=http://10.7.68.179:9696
quantum_auth_strategy=keystone
quantum_admin_tenant_name=service
quantum_admin_username=quantumadmin
quantum_admin_password=quantumadminpasswd
quantum_admin_auth_url=http://10.7.68.179:5001/v2.0/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1115153/+subscriptions