openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #23936
wget: can't connect to remote host (169.254.169.254): No route to host, but can ping and ssh
Hello everyone,
I have am having a problem with my metadata service. My VM's log is as
follows. As you may see, it can not find the route to the metadata service
(169.254.169.254). But I can ssh and ping the VM's from the network node.
Although as far as I know, you need to have metadata imported to the
virtual machines, to be able to ping to them and to ssh in to them.
I cant ping or ssh in to the VMs from the controller or compute node.
Anyone any help?
[ 0.018022] Initializing cgroup subsys cpuacct
[ 0.020014] Initializing cgroup subsys memory
[ 0.021322] Initializing cgroup subsys devices
[ 0.022629] Initializing cgroup subsys freezer
[ 0.024007] Initializing cgroup subsys net_cls
[ 0.025310] Initializing cgroup subsys blkio
[ 0.026587] Initializing cgroup subsys perf_event
[ 0.028143] mce: CPU supports 10 MCE banks
[ 0.029686] SMP alternatives: switching to UP code
[ 0.132200] Freeing SMP alternatives: 24k freed
[ 0.133648] ACPI: Core revision 20110413
[ 0.135949] ftrace: allocating 26075 entries in 103 pages
[ 0.144172] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.145806] CPU0: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz stepping 07
[ 0.148008] Performance Events: unsupported p6 CPU model 15 no PMU
driver, software events only.
[ 0.148008] Brought up 1 CPUs
[ 0.148012] Total of 1 processors activated (5320.05 BogoMIPS).
[ 0.150609] devtmpfs: initialized
[ 0.154178] print_constraints: dummy:
[ 0.155400] Time: 13:02:05 Date: 05/24/13
[ 0.156087] NET: Registered protocol family 16
[ 0.157512] ACPI: bus type pci registered
[ 0.158819] PCI: Using configuration type 1 for base access
[ 0.160810] bio: create slab <bio-0> at 0
[ 0.165652] ACPI: Interpreter enabled
[ 0.166763] ACPI: (supports S0 S3 S4 S5)
[ 0.168414] ACPI: Using IOAPIC for interrupt routing
[ 0.173485] ACPI: No dock devices found.
[ 0.174646] HEST: Table not found.
[ 0.175698] PCI: Ignoring host bridge windows from ACPI; if
necessary, use "pci=use_crs" and report a bug
[ 0.176031] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.183300] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by
PIIX4 ACPI
[ 0.184026] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB
[ 0.201264] pci0000:00: Unable to request _OSC control (_OSC
support mask: 0x1e)
[ 0.207189] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.209145] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.211331] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.213126] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.215291] ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
[ 0.217204] vgaarb: device added:
PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.220016] vgaarb: loaded
[ 0.220931] vgaarb: bridge control possible 0000:00:02.0
[ 0.222716] SCSI subsystem initialized
[ 0.224363] usbcore: registered new interface driver usbfs
[ 0.225873] usbcore: registered new interface driver hub
[ 0.227381] usbcore: registered new device driver usb
[ 0.228226] PCI: Using ACPI for IRQ routing
[ 0.229889] NetLabel: Initializing
[ 0.232021] NetLabel: domain hash size = 128
[ 0.233289] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.234683] NetLabel: unlabeled traffic allowed by default
[ 0.236101] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[ 0.237936] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 0.240017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[ 0.248162] Switching to clocksource kvm-clock
[ 0.250575] Switched to NOHz mode on CPU #0
[ 0.260176] AppArmor: AppArmor Filesystem Enabled
[ 0.261587] pnp: PnP ACPI init
[ 0.262592] ACPI: bus type pnp registered
[ 0.264716] pnp: PnP ACPI: found 8 devices
[ 0.265919] ACPI: ACPI bus type pnp unregistered
[ 0.273657] NET: Registered protocol family 2
[ 0.275023] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.277174] TCP established hash table entries: 16384 (order: 6,
262144 bytes)
[ 0.279627] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[ 0.281735] TCP: Hash tables configured (established 16384 bind 16384)
[ 0.283454] TCP reno registered
[ 0.284478] UDP hash table entries: 256 (order: 1, 8192 bytes)
[ 0.286046] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[ 0.287800] NET: Registered protocol family 1
[ 0.289098] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.290692] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.292282] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.294366] audit: initializing netlink socket (disabled)
[ 0.295853] type=2000 audit(1369400527.292:1): initialized
[ 0.315902] Trying to unpack rootfs image as initramfs...
[ 0.332248] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.340281] VFS: Disk quotas dquot_6.5.2
[ 0.341541] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.343868] fuse init (API version 7.16)
[ 0.345164] msgmni has been set to 972
[ 0.360325] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[ 0.368139] io scheduler noop registered
[ 0.369319] io scheduler deadline registered (default)
[ 0.370799] io scheduler cfq registered
[ 0.372965] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 0.374504] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 0.376370] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.378503] ACPI: Power Button [PWRF]
[ 0.388884] ERST: Table is not found!
[ 0.390234] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 0.391810] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI
11 (level, high) -> IRQ 11
[ 0.398358] Freeing initrd memory: 2000k freed
[ 0.400397] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
[ 0.401988] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI
10 (level, high) -> IRQ 10
[ 0.404952] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[ 0.406529] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI
10 (level, high) -> IRQ 10
[ 0.409174] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 0.432969] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.456664] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 0.481202] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.504788] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 0.506483] Linux agpgart interface v0.103
[ 0.508971] brd: module loaded
[ 0.510598] loop: module loaded
[ 0.565931] vda: vda1
[ 0.567917] scsi0 : ata_piix
[ 0.569007] scsi1 : ata_piix
[ 0.569989] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
[ 0.571766] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
[ 0.573818] Fixed MDIO Bus: probed
[ 0.574915] PPP generic driver version 2.4.2
[ 0.576228] tun: Universal TUN/TAP device driver, 1.6
[ 0.577624] tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
[ 0.631792] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.633581] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 0.635274] uhci_hcd: USB Universal Host Controller Interface driver
[ 0.637068] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI
10 (level, high) -> IRQ 10
[ 0.639482] uhci_hcd 0000:00:01.2: UHCI Host Controller
[ 0.641094] uhci_hcd 0000:00:01.2: new USB bus registered, assigned
bus number 1
[ 0.643420] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040
[ 0.645292] hub 1-0:1.0: USB hub found
[ 0.646641] hub 1-0:1.0: 2 ports detected
[ 0.648232] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU]
at 0x60,0x64 irq 1,12
[ 0.651483] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 0.652930] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 0.654454] mousedev: PS/2 mouse device common for all mice
[ 0.656171] rtc_cmos 00:01: RTC can wake from S4
[ 0.657782] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[ 0.659565] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs
[ 0.661383] device-mapper: uevent: version 1.0.3
[ 0.662806] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02)
initialised: dm-devel@xxxxxxxxxx
[ 0.665209] cpuidle: using governor ladder
[ 0.666430] cpuidle: using governor menu
[ 0.667635] EFI Variables Facility v0.08 2004-May-17
[ 0.669285] TCP cubic registered
[ 0.670448] NET: Registered protocol family 10
[ 0.672347] NET: Registered protocol family 17
[ 0.673680] Registering the dns_resolver key type
[ 0.675192] registered taskstats version 1
[ 0.676856] input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input1
[ 0.691184] Magic number: 13:137:29
[ 0.692520] rtc_cmos 00:01: setting system clock to 2013-05-24
13:02:06 UTC (1369400526)
[ 0.694815] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 0.696466] EDD information not available.
[ 0.838045] Freeing unused kernel memory: 900k freed
[ 0.839933] Write protecting the kernel read-only data: 12288k
[ 0.848012] Freeing unused kernel memory: 1968k freed
[ 0.854482] Freeing unused kernel memory: 1368k freed
info: initramfs: up at 0.87
NOCHANGE: partition 1 is size 64260. it cannot be grown
info: initramfs loading root from /dev/vda1
info: /etc/init.d/rc.sysinit: up at 1.14
[ 1.153624] EXT3-fs (vda1): warning: checktime reached, running
e2fsck is recommended
Starting logging: OK
Initializing random number generator... done.
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending select for 50.50.1.3...
Lease of 50.50.1.3 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 50.50.1.2
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 1/30: up 1.52. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 2/30: up 5.55. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 3/30: up 8.55. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 4/30: up 12.56. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 5/30: up 15.57. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 6/30: up 19.58. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 7/30: up 22.58. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 8/30: up 26.59. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 9/30: up 29.60. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 10/30: up 33.61. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 11/30: up 36.61. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 12/30: up 40.62. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 13/30: up 43.63. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 14/30: up 47.64. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 15/30: up 50.65. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 16/30: up 54.66. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 17/30: up 57.66. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 18/30: up 61.67. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 19/30: up 64.68. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 20/30: up 68.69. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 21/30: up 71.69. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 22/30: up 75.71. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 23/30: up 78.71. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 24/30: up 82.72. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 25/30: up 85.73. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 26/30: up 89.74. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 27/30: up 92.74. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 28/30: up 96.75. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 29/30: up 99.76. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 30/30: up 103.77. request failed
cloud-setup: after 30 fails, debugging
cloud-setup: running debug (30 tries reached)
############ debug start ##############
### /etc/rc.d/init.d/sshd start
/etc/rc3.d/S45-cloud-setup: line 66: /etc/rc.d/init.d/sshd: not found
### ifconfig -a
eth0 Link encap:Ethernet HWaddr FA:16:3E:0D:19:2F
inet addr:50.50.1.3 Bcast:50.50.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe0d:192f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:109 errors:0 dropped:0 overruns:0 frame:0
TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6715 (6.5 KiB) TX bytes:5491 (5.3 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:60 errors:0 dropped:0 overruns:0 frame:0
TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5280 (5.1 KiB) TX bytes:5280 (5.1 KiB)
### route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 50.50.1.1 0.0.0.0 UG 0 0 0 eth0
50.50.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
### cat /etc/resolv.conf
search openstacklocal
nameserver 50.50.1.2
### ping -c 5 50.50.1.1
PING 50.50.1.1 (50.50.1.1): 56 data bytes
--- 50.50.1.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
### pinging nameservers
#### ping -c 5 50.50.1.2
PING 50.50.1.2 (50.50.1.2): 56 data bytes
64 bytes from 50.50.1.2: seq=0 ttl=64 time=0.975 ms
64 bytes from 50.50.1.2: seq=1 ttl=64 time=0.472 ms
64 bytes from 50.50.1.2: seq=2 ttl=64 time=0.450 ms
64 bytes from 50.50.1.2: seq=3 ttl=64 time=0.394 ms
64 bytes from 50.50.1.2: seq=4 ttl=64 time=0.403 ms
--- 50.50.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.394/0.538/0.975 ms
### uname -a
Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC
2011 x86_64 GNU/Linux
### lsmod
Module Size Used by Not tainted
vfat 17585 0
fat 61475 1 vfat
isofs 40253 0
ip_tables 27473 0
x_tables 29846 1 ip_tables
pcnet32 42078 0
8139cp 27412 0
ne2k_pci 13691 0
8390 18856 1 ne2k_pci
e1000 108573 0
acpiphp 24080 0
### dmesg | tail
[ 1.275864] acpiphp: Slot [30] registered
[ 1.275877] acpiphp: Slot [31] registered
[ 1.293068] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 1.293072] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 1.308681] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
[ 1.312138] Refined TSC clocksource calibration: 2659.898 MHz.
[ 1.314526] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
[ 1.322363] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
[ 1.331183] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 1.988364] eth0: IPv6 duplicate address fe80::f816:3eff:fe0d:192f detected!
### tail -n 25 /var/log/messages
May 24 07:02:07 cirros kern.info kernel: [ 0.968633] usb 1-1: new
full speed USB device number 2 using uhci_hcd
May 24 07:02:07 cirros kern.info kernel: [ 0.972225] EXT3-fs:
barriers not enabled
May 24 07:02:07 cirros kern.info kernel: [ 0.972552] kjournald
starting. Commit interval 5 seconds
May 24 07:02:07 cirros kern.info kernel: [ 0.972563] EXT3-fs
(vda1): mounted filesystem with ordered data mode
May 24 07:02:07 cirros kern.warn kernel: [ 1.153624] EXT3-fs
(vda1): warning: checktime reached, running e2fsck is recommended
May 24 07:02:07 cirros kern.info kernel: [ 1.156341] EXT3-fs
(vda1): using internal journal
May 24 07:02:07 cirros kern.info kernel: [ 1.275255] acpiphp: ACPI
Hot Plug PCI Controller Driver version: 0.5
May 24 07:02:07 cirros kern.info kernel: [ 1.275421] acpiphp: Slot
[1] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275443] acpiphp: Slot
[2] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275456] acpiphp: Slot
[3] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275468] acpiphp: Slot
[4] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275498] acpiphp: Slot
[5] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275512] acpiphp: Slot
[6] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275536] acpiphp: Slot
[7] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275550] acpiphp: Slot
[8] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275564] acpiphp: Slot
[9] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.275579] acpiphp: Slot
[10] registered
May 24 07:02:07 cirros kern.info kernel: [ 1.293068] e1000:
Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
May 24 07:02:07 cirros kern.info kernel: [ 1.293072] e1000:
Copyright (c) 1999-2006 Intel Corporation.
May 24 07:02:07 cirros kern.info kernel: [ 1.308681]
ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
May 24 07:02:07 cirros kern.info kernel: [ 1.312138] Refined TSC
clocksource calibration: 2659.898 MHz.
May 24 07:02:07 cirros kern.info kernel: [ 1.314526] 8139cp:
8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
May 24 07:02:07 cirros kern.info kernel: [ 1.322363] pcnet32:
pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
May 24 07:02:07 cirros kern.info kernel: [ 1.331183] ip_tables: (C)
2000-2006 Netfilter Core Team
May 24 07:02:07 cirros kern.info kernel: [ 1.988364] eth0: IPv6
duplicate address fe80::f816:3eff:fe0d:192f detected!
############ debug end ##############
cloud-setup: failed to read iid from metadata. tried 30
WARN: /etc/rc3.d/S45-cloud-setup failed
Starting dropbear sshd: generating rsa key... generating dsa key... OK
===== cloud-final: system completely up in 125.14 seconds ====
wget: can't connect to remote host (169.254.169.254): No route to host
wget: can't connect to remote host (169.254.169.254): No route to host
wget: can't connect to remote host (169.254.169.254): No route to host
instance-id:
public-ipv4:
local-ipv4 :
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-userdata: failed to read instance id
WARN: /etc/rc3.d/S99-cloud-userdata failed
____ ____ ____
/ __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/ /_/ \____/___/
http://launchpad.net/cirros
login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
cirros login:
Thanks.
Follow ups
