← Back to team overview

openstack team mailing list archive

wget: can't connect to remote host (169.254.169.254): No route to host, but can ping and ssh

 

Hello everyone,

I have am having a problem with my metadata service. My VM's log is as
follows. As you may see, it can not find the route to the metadata service
(169.254.169.254). But I can ssh and ping the VM's from the network node.
Although as far as I know, you need to have metadata imported to the
virtual machines, to be able to ping to them and to ssh in to them.

I cant ping or ssh in to the VMs from the controller or compute node.

Anyone any help?


[    0.018022] Initializing cgroup subsys cpuacct
[    0.020014] Initializing cgroup subsys memory
[    0.021322] Initializing cgroup subsys devices
[    0.022629] Initializing cgroup subsys freezer
[    0.024007] Initializing cgroup subsys net_cls
[    0.025310] Initializing cgroup subsys blkio
[    0.026587] Initializing cgroup subsys perf_event
[    0.028143] mce: CPU supports 10 MCE banks
[    0.029686] SMP alternatives: switching to UP code
[    0.132200] Freeing SMP alternatives: 24k freed
[    0.133648] ACPI: Core revision 20110413
[    0.135949] ftrace: allocating 26075 entries in 103 pages
[    0.144172] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.145806] CPU0: Intel(R) Xeon(R) CPU           X5355  @ 2.66GHz stepping 07
[    0.148008] Performance Events: unsupported p6 CPU model 15 no PMU
driver, software events only.
[    0.148008] Brought up 1 CPUs
[    0.148012] Total of 1 processors activated (5320.05 BogoMIPS).
[    0.150609] devtmpfs: initialized
[    0.154178] print_constraints: dummy:
[    0.155400] Time: 13:02:05  Date: 05/24/13
[    0.156087] NET: Registered protocol family 16
[    0.157512] ACPI: bus type pci registered
[    0.158819] PCI: Using configuration type 1 for base access
[    0.160810] bio: create slab <bio-0> at 0
[    0.165652] ACPI: Interpreter enabled
[    0.166763] ACPI: (supports S0 S3 S4 S5)
[    0.168414] ACPI: Using IOAPIC for interrupt routing
[    0.173485] ACPI: No dock devices found.
[    0.174646] HEST: Table not found.
[    0.175698] PCI: Ignoring host bridge windows from ACPI; if
necessary, use "pci=use_crs" and report a bug
[    0.176031] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.183300] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by
PIIX4 ACPI
[    0.184026] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    0.201264]  pci0000:00: Unable to request _OSC control (_OSC
support mask: 0x1e)
[    0.207189] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[    0.209145] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.211331] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.213126] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[    0.215291] ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
[    0.217204] vgaarb: device added:
PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.220016] vgaarb: loaded
[    0.220931] vgaarb: bridge control possible 0000:00:02.0
[    0.222716] SCSI subsystem initialized
[    0.224363] usbcore: registered new interface driver usbfs
[    0.225873] usbcore: registered new interface driver hub
[    0.227381] usbcore: registered new device driver usb
[    0.228226] PCI: Using ACPI for IRQ routing
[    0.229889] NetLabel: Initializing
[    0.232021] NetLabel:  domain hash size = 128
[    0.233289] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.234683] NetLabel:  unlabeled traffic allowed by default
[    0.236101] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.237936] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.240017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[    0.248162] Switching to clocksource kvm-clock
[    0.250575] Switched to NOHz mode on CPU #0
[    0.260176] AppArmor: AppArmor Filesystem Enabled
[    0.261587] pnp: PnP ACPI init
[    0.262592] ACPI: bus type pnp registered
[    0.264716] pnp: PnP ACPI: found 8 devices
[    0.265919] ACPI: ACPI bus type pnp unregistered
[    0.273657] NET: Registered protocol family 2
[    0.275023] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.277174] TCP established hash table entries: 16384 (order: 6,
262144 bytes)
[    0.279627] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    0.281735] TCP: Hash tables configured (established 16384 bind 16384)
[    0.283454] TCP reno registered
[    0.284478] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.286046] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.287800] NET: Registered protocol family 1
[    0.289098] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.290692] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.292282] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.294366] audit: initializing netlink socket (disabled)
[    0.295853] type=2000 audit(1369400527.292:1): initialized
[    0.315902] Trying to unpack rootfs image as initramfs...
[    0.332248] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.340281] VFS: Disk quotas dquot_6.5.2
[    0.341541] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.343868] fuse init (API version 7.16)
[    0.345164] msgmni has been set to 972
[    0.360325] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[    0.368139] io scheduler noop registered
[    0.369319] io scheduler deadline registered (default)
[    0.370799] io scheduler cfq registered
[    0.372965] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.374504] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.376370] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.378503] ACPI: Power Button [PWRF]
[    0.388884] ERST: Table is not found!
[    0.390234] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[    0.391810] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI
11 (level, high) -> IRQ 11
[    0.398358] Freeing initrd memory: 2000k freed
[    0.400397] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
[    0.401988] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI
10 (level, high) -> IRQ 10
[    0.404952] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[    0.406529] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI
10 (level, high) -> IRQ 10
[    0.409174] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.432969] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    0.456664] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    0.481202] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    0.504788] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    0.506483] Linux agpgart interface v0.103
[    0.508971] brd: module loaded
[    0.510598] loop: module loaded
[    0.565931]  vda: vda1
[    0.567917] scsi0 : ata_piix
[    0.569007] scsi1 : ata_piix
[    0.569989] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
[    0.571766] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
[    0.573818] Fixed MDIO Bus: probed
[    0.574915] PPP generic driver version 2.4.2
[    0.576228] tun: Universal TUN/TAP device driver, 1.6
[    0.577624] tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
[    0.631792] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.633581] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.635274] uhci_hcd: USB Universal Host Controller Interface driver
[    0.637068] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI
10 (level, high) -> IRQ 10
[    0.639482] uhci_hcd 0000:00:01.2: UHCI Host Controller
[    0.641094] uhci_hcd 0000:00:01.2: new USB bus registered, assigned
bus number 1
[    0.643420] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040
[    0.645292] hub 1-0:1.0: USB hub found
[    0.646641] hub 1-0:1.0: 2 ports detected
[    0.648232] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU]
at 0x60,0x64 irq 1,12
[    0.651483] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.652930] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.654454] mousedev: PS/2 mouse device common for all mice
[    0.656171] rtc_cmos 00:01: RTC can wake from S4
[    0.657782] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[    0.659565] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs
[    0.661383] device-mapper: uevent: version 1.0.3
[    0.662806] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02)
initialised: dm-devel@xxxxxxxxxx
[    0.665209] cpuidle: using governor ladder
[    0.666430] cpuidle: using governor menu
[    0.667635] EFI Variables Facility v0.08 2004-May-17
[    0.669285] TCP cubic registered
[    0.670448] NET: Registered protocol family 10
[    0.672347] NET: Registered protocol family 17
[    0.673680] Registering the dns_resolver key type
[    0.675192] registered taskstats version 1
[    0.676856] input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input1
[    0.691184]   Magic number: 13:137:29
[    0.692520] rtc_cmos 00:01: setting system clock to 2013-05-24
13:02:06 UTC (1369400526)
[    0.694815] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    0.696466] EDD information not available.
[    0.838045] Freeing unused kernel memory: 900k freed
[    0.839933] Write protecting the kernel read-only data: 12288k
[    0.848012] Freeing unused kernel memory: 1968k freed
[    0.854482] Freeing unused kernel memory: 1368k freed

info: initramfs: up at 0.87
NOCHANGE: partition 1 is size 64260. it cannot be grown
info: initramfs loading root from /dev/vda1
info: /etc/init.d/rc.sysinit: up at 1.14
[    1.153624] EXT3-fs (vda1): warning: checktime reached, running
e2fsck is recommended
Starting logging: OK
Initializing random number generator... done.
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending select for 50.50.1.3...
Lease of 50.50.1.3 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 50.50.1.2
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 1/30: up 1.52. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 2/30: up 5.55. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 3/30: up 8.55. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 4/30: up 12.56. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 5/30: up 15.57. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 6/30: up 19.58. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 7/30: up 22.58. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 8/30: up 26.59. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 9/30: up 29.60. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 10/30: up 33.61. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 11/30: up 36.61. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 12/30: up 40.62. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 13/30: up 43.63. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 14/30: up 47.64. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 15/30: up 50.65. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 16/30: up 54.66. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 17/30: up 57.66. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 18/30: up 61.67. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 19/30: up 64.68. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 20/30: up 68.69. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 21/30: up 71.69. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 22/30: up 75.71. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 23/30: up 78.71. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 24/30: up 82.72. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 25/30: up 85.73. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 26/30: up 89.74. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 27/30: up 92.74. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 28/30: up 96.75. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 29/30: up 99.76. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 30/30: up 103.77. request failed
cloud-setup: after 30 fails, debugging
cloud-setup: running debug (30 tries reached)
############ debug start ##############
### /etc/rc.d/init.d/sshd start
/etc/rc3.d/S45-cloud-setup: line 66: /etc/rc.d/init.d/sshd: not found
### ifconfig -a
eth0      Link encap:Ethernet  HWaddr FA:16:3E:0D:19:2F
          inet addr:50.50.1.3  Bcast:50.50.1.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe0d:192f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6715 (6.5 KiB)  TX bytes:5491 (5.3 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:60 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5280 (5.1 KiB)  TX bytes:5280 (5.1 KiB)

### route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         50.50.1.1       0.0.0.0         UG    0      0        0 eth0
50.50.1.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
### cat /etc/resolv.conf
search openstacklocal
nameserver 50.50.1.2
### ping -c 5 50.50.1.1
PING 50.50.1.1 (50.50.1.1): 56 data bytes

--- 50.50.1.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
### pinging nameservers
#### ping -c 5 50.50.1.2
PING 50.50.1.2 (50.50.1.2): 56 data bytes
64 bytes from 50.50.1.2: seq=0 ttl=64 time=0.975 ms
64 bytes from 50.50.1.2: seq=1 ttl=64 time=0.472 ms
64 bytes from 50.50.1.2: seq=2 ttl=64 time=0.450 ms
64 bytes from 50.50.1.2: seq=3 ttl=64 time=0.394 ms
64 bytes from 50.50.1.2: seq=4 ttl=64 time=0.403 ms

--- 50.50.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.394/0.538/0.975 ms
### uname -a
Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC
2011 x86_64 GNU/Linux
### lsmod
Module                  Size  Used by    Not tainted
vfat                   17585  0
fat                    61475  1 vfat
isofs                  40253  0
ip_tables              27473  0
x_tables               29846  1 ip_tables
pcnet32                42078  0
8139cp                 27412  0
ne2k_pci               13691  0
8390                   18856  1 ne2k_pci
e1000                 108573  0
acpiphp                24080  0
### dmesg | tail
[    1.275864] acpiphp: Slot [30] registered
[    1.275877] acpiphp: Slot [31] registered
[    1.293068] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    1.293072] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    1.308681] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
[    1.312138] Refined TSC clocksource calibration: 2659.898 MHz.
[    1.314526] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
[    1.322363] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
[    1.331183] ip_tables: (C) 2000-2006 Netfilter Core Team
[    1.988364] eth0: IPv6 duplicate address fe80::f816:3eff:fe0d:192f detected!
### tail -n 25 /var/log/messages
May 24 07:02:07 cirros kern.info kernel: [    0.968633] usb 1-1: new
full speed USB device number 2 using uhci_hcd
May 24 07:02:07 cirros kern.info kernel: [    0.972225] EXT3-fs:
barriers not enabled
May 24 07:02:07 cirros kern.info kernel: [    0.972552] kjournald
starting.  Commit interval 5 seconds
May 24 07:02:07 cirros kern.info kernel: [    0.972563] EXT3-fs
(vda1): mounted filesystem with ordered data mode
May 24 07:02:07 cirros kern.warn kernel: [    1.153624] EXT3-fs
(vda1): warning: checktime reached, running e2fsck is recommended
May 24 07:02:07 cirros kern.info kernel: [    1.156341] EXT3-fs
(vda1): using internal journal
May 24 07:02:07 cirros kern.info kernel: [    1.275255] acpiphp: ACPI
Hot Plug PCI Controller Driver version: 0.5
May 24 07:02:07 cirros kern.info kernel: [    1.275421] acpiphp: Slot
[1] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275443] acpiphp: Slot
[2] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275456] acpiphp: Slot
[3] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275468] acpiphp: Slot
[4] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275498] acpiphp: Slot
[5] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275512] acpiphp: Slot
[6] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275536] acpiphp: Slot
[7] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275550] acpiphp: Slot
[8] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275564] acpiphp: Slot
[9] registered
May 24 07:02:07 cirros kern.info kernel: [    1.275579] acpiphp: Slot
[10] registered
May 24 07:02:07 cirros kern.info kernel: [    1.293068] e1000:
Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
May 24 07:02:07 cirros kern.info kernel: [    1.293072] e1000:
Copyright (c) 1999-2006 Intel Corporation.
May 24 07:02:07 cirros kern.info kernel: [    1.308681]
ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
May 24 07:02:07 cirros kern.info kernel: [    1.312138] Refined TSC
clocksource calibration: 2659.898 MHz.
May 24 07:02:07 cirros kern.info kernel: [    1.314526] 8139cp:
8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
May 24 07:02:07 cirros kern.info kernel: [    1.322363] pcnet32:
pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
May 24 07:02:07 cirros kern.info kernel: [    1.331183] ip_tables: (C)
2000-2006 Netfilter Core Team
May 24 07:02:07 cirros kern.info kernel: [    1.988364] eth0: IPv6
duplicate address fe80::f816:3eff:fe0d:192f detected!
############ debug end   ##############
cloud-setup: failed to read iid from metadata. tried 30
WARN: /etc/rc3.d/S45-cloud-setup failed
Starting dropbear sshd: generating rsa key... generating dsa key... OK
===== cloud-final: system completely up in 125.14 seconds ====
wget: can't connect to remote host (169.254.169.254): No route to host
wget: can't connect to remote host (169.254.169.254): No route to host
wget: can't connect to remote host (169.254.169.254): No route to host
  instance-id:
  public-ipv4:
  local-ipv4 :
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-userdata: failed to read instance id
WARN: /etc/rc3.d/S99-cloud-userdata failed
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/  /_/   \____/___/
 http://launchpad.net/cirros



login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.

cirros login:


Thanks.

Follow ups