← Back to team overview

openstack team mailing list archive

Re: Cant ping private or floating IP

 

Please ignore my previous email.

The problem still existed after restart. Solution for the problem was to
delete and recreate br-ex and br-int then
restart quantum-plugin-openvswitch-agent, quantum-dhcp-agent, quantum-l3-agent.

Bug: https://bugs.launchpad.net/quantum/+bug/1091605

This solved the issue for me.

On 3 March 2013 16:49, Chathura M. Sarathchandra Magurawalage <
77.chathura@xxxxxxxxx> wrote:

> I have resolved this issue. All it was that I had dhcp enabled on the
> eth0.2 (management network) interface, on the controller and the compute
> node. Although the IP's did not changed as the IPs were reserved
> beforehand.
>
> Thanks everyone for your help.
>
> Chathura
>
> On 28 February 2013 17:58, Chathura M. Sarathchandra Magurawalage <
> 77.chathura@xxxxxxxxx> wrote:
>
>> Hello,
>>
>> Now I am not using virtual aliases (eth0.2) anymore I have got two
>> physical interfaces for management network and the data network.
>>
>> Now when I create an instance I get the following in the instance log
>>
>> no instance data found in start-local
>>
>> cloud-init-nonet waiting 120 seconds for a network device.
>>
>> cloud-init-nonet gave up waiting for a network device.
>>
>> ci-info: lo    : 1 127.0.0.1       255.0.0.0       .
>>
>> ci-info: eth0  : 1 .               .               fa:16:3e:40:60:2f
>>
>> route_info failed
>>
>>
>> Then in /var/log/syslog I get
>>
>> Feb 28 17:42:32 athena dnsmasq-dhcp[19876]: DHCPDISCOVER(tapf71b5b86-5c) fa:16:3e:40:60:2f
>>
>>
>> and I get these errors
>>
>> root@controller:~# less /var/log/quantum/server.log | grep ERROR
>> 2013-02-28 12:19:40    ERROR [quantum.openstack.common.rpc.common] AMQP server on localhost:5672 is unreachable: Socket closed. Trying again in 1 seconds.
>> 2013-02-28 14:12:32    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>> 2013-02-28 14:12:32    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>> 2013-02-28 14:12:35    ERROR [keystone.middleware.auth_token] HTTP connection exception: ''
>> 2013-02-28 14:12:39    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
>> 2013-02-28 15:28:57    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>> 2013-02-28 15:28:57    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>> 2013-02-28 17:35:03    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>> 2013-02-28 17:35:03    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>>
>>
>>
>>
>> any idea?
>>
>>
>> On 22 February 2013 13:55, Chathura M. Sarathchandra Magurawalage <
>> 77.chathura@xxxxxxxxx> wrote:
>>
>>> Yes 10.5.5.5 has ARP entries for 10.5.5.1 and 10.5.5.2 with correct MAC
>>> addresses.
>>> 10.5.5.1 has ARP entries for 10.5.5.5 but most of the time the MAC
>>> address is "(incomplete)"?
>>>
>>> If the eth0.2 VPN interface the problem. Do you know of any solution for
>>> this matter?
>>>
>>> Thanks.
>>>
>>> On 22 February 2013 11:29, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>wrote:
>>>
>>>>  Let me clarify, you have :
>>>>  -ARP cache entry for 10.5.5.1 on 10.5.5.5
>>>>  -ARP cache entry for 10.5.5.5 on 10.5.5.1
>>>> with correct MAC addresses (make sure with 'ip a show qr-XXXX') and
>>>> ICMP request is still not visible on 10.5.5.5 with 'tcpdump -i eth0 -nn
>>>> icmp' ?
>>>>
>>>> If yes, I have to admit I can't help you further. I saw you were using
>>>> virtual alias eth0.2, maybe it's the root cause.
>>>> All my setup is based on physical network cards, I haven't created
>>>> virtual aliases and this basic test works. You have trouble with L2 and I
>>>> don't think Quantum is responsible of that.
>>>>
>>>> -Sylvain
>>>>
>>>> Le 22/02/2013 11:50, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>
>>>> I tried tcpdump on 10.5.5.5 while pinging but I can' t see ICMP
>>>> requests :(
>>>>
>>>>  And I do have right ARP entries on 10.5.5.5
>>>>
>>>>  Thanks.
>>>>
>>>>
>>>> On 22 February 2013 08:21, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>wrote:
>>>>
>>>>>  As per you mention, netnode do have mac address for 10.5.5.5
>>>>> If you tcpdump on 10.5.5.5 while pinging from 10.5.5.1, you should see
>>>>> ICMP request coming from 10.5.5.1 (provided the 'controller' node is the
>>>>> network node)
>>>>> Could you also check ARP cache on 10.5.5.5, especially look at
>>>>> 10.5.5.1 entry ?
>>>>>
>>>>> -Sylvain
>>>>>
>>>>>
>>>>> Le 22/02/2013 01:24, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>>
>>>>>  10.5.5.5 = new fedora vm
>>>>>
>>>>>  arp entries for the vms.
>>>>> root@controller:~# arp -n | grep 10.5.5.3
>>>>> 10.5.5.3                         (incomplete)
>>>>>      tapf71b5b86-5c
>>>>> root@controller:~# arp -n | grep 10.5.5.4
>>>>> 10.5.5.4                         (incomplete)
>>>>>      tapf71b5b86-5c
>>>>> root@controller:~# arp -n | grep 10.5.5.5
>>>>>  10.5.5.5                 ether   fa:16:3e:7b:ef:84   C
>>>>>       qr-4d088f3a-78
>>>>> 10.5.5.5                         (incomplete)
>>>>>      tapf71b5b86-5c
>>>>>
>>>>>  networknode/controller did not have an arp entry for 10.5.5.1.
>>>>>
>>>>>  I tried adding arp entry for the 10.5.5.3 but still it did not work.
>>>>>
>>>>>  any ideas?
>>>>>
>>>>>  thanks.
>>>>>
>>>>>  On 21 February 2013 17:41, Chathura M. Sarathchandra Magurawalage <
>>>>> 77.chathura@xxxxxxxxx> wrote:
>>>>>
>>>>>>  Thanks Sylvain,
>>>>>>
>>>>>>  root@controller:~# ip a
>>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>>>>     inet 127.0.0.1/8 scope host lo
>>>>>>     inet6 ::1/128 scope host
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>>> qlen 1000
>>>>>>     link/ether d4:ae:52:bb:aa:20 brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 10.10.10.1/24 brd 10.10.10.255 scope global eth0
>>>>>>     inet6 fe80::d6ae:52ff:febb:aa20/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
>>>>>> DOWN qlen 1000
>>>>>>     link/ether d4:ae:52:bb:aa:21 brd ff:ff:ff:ff:ff:ff
>>>>>> 4: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UP
>>>>>>     link/ether d4:ae:52:bb:aa:20 brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 192.168.2.225/24 brd 192.168.2.255 scope global eth0.2
>>>>>>     inet6 fe80::d6ae:52ff:febb:aa20/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 5: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>>> state UNKNOWN
>>>>>>      link/ether ba:7a:e9:dc:2b:41 brd ff:ff:ff:ff:ff:ff
>>>>>>     inet6 fe80::b87a:e9ff:fedc:2b41/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>>> state UNKNOWN
>>>>>>     link/ether 9a:41:c8:8a:9e:49 brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 192.168.2.225/24 scope global br-ex
>>>>>> 8: tapf71b5b86-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UNKNOWN
>>>>>>     link/ether 2a:44:a3:d1:7d:f3 brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 10.5.5.2/24 brd 10.5.5.255 scope global tapf71b5b86-5c
>>>>>>     inet6 fe80::2844:a3ff:fed1:7df3/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 9: qr-4d088f3a-78: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UNKNOWN
>>>>>>     link/ether ca:5b:8d:4d:6d:fb brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 10.5.5.1/24 brd 10.5.5.255 scope global qr-4d088f3a-78
>>>>>>     inet6 fe80::c85b:8dff:fe4d:6dfb/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 10: qg-6f8374cb-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UNKNOWN
>>>>>>     link/ether 0e:7f:dd:3a:80:bc brd ff:ff:ff:ff:ff:ff
>>>>>>     inet 192.168.2.151/24 brd 192.168.2.255 scope global
>>>>>> qg-6f8374cb-cb
>>>>>>     inet6 fe80::c7f:ddff:fe3a:80bc/64 scope link
>>>>>>        valid_lft forever preferred_lft forever
>>>>>> 36: eth1.2@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc
>>>>>> noqueue state LOWERLAYERDOWN
>>>>>>     link/ether d4:ae:52:bb:aa:21 brd ff:ff:ff:ff:ff:ff
>>>>>> 39: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>>     link/ether 26:e2:a9:05:3d:4c brd ff:ff:ff:ff:ff:ff
>>>>>>
>>>>>>
>>>>>>  But for some reason the "arp" module is not in the cirros cloud
>>>>>> image. Do you suggest any other image to create instances with?, which I
>>>>>> can log in without having to ssh in first like the ubuntu cloud image.
>>>>>>
>>>>>>  Thanks.
>>>>>>
>>>>>>  On 21 February 2013 16:17, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>>  Could you please :
>>>>>>>  - 'ip a' on network node
>>>>>>>  - 'arp -n' on VM and check '10.5.5.1'
>>>>>>>  - 'arp -n' on netnode and check '10.5.5.3'
>>>>>>>
>>>>>>> Connection in between VM and internal gateway is managed thru ARP
>>>>>>> resolution.
>>>>>>> If 'arp -n' is giving no MAC address for 10.5.5.1 on VM, try :
>>>>>>>  1. get the MAC address for 10.5.5.1 on netnode using 'ip a'
>>>>>>>  2. get the MAC address for 10.5.5.3 on VM using 'ifconfig'
>>>>>>>  2. add manually an ARP entry for 10.5.5.1 by issuing 'arp -s
>>>>>>> 10.5.5.1 <netnode_mac_addr>'
>>>>>>>  3. (if not present on netnode) add manually ARP entry for 10.5.5.3
>>>>>>> using 'arp -s 10.5.5.3 <vm_mac_addr>'
>>>>>>> and check with ping.
>>>>>>>
>>>>>>> If that fixes the bug, you have to investigate why ARP resolution is
>>>>>>> fine for DHCP server but not for gateway. You should see ARP requests on
>>>>>>> physical layer with 'ARP, Request who-has 10.5.5.1 tell 10.5.5.3, length
>>>>>>> 28' and ARP reply as follows 'ARP, Reply 10.5.5.1 is-at <mac_addr>, length
>>>>>>> 28'
>>>>>>>
>>>>>>> -Sylvain
>>>>>>>
>>>>>>>
>>>>>>> Le 21/02/2013 16:29, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>>>>
>>>>>>> So I have got two VMs, they can ping eachother. But they cannot ping
>>>>>>> the gateway.
>>>>>>>
>>>>>>>  Just for the record. From my controller/network-node I can ping
>>>>>>> the provider router.
>>>>>>>
>>>>>>>  root@controller
>>>>>>> :~# quantum router-list
>>>>>>>
>>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>>> | id                                   | name            |
>>>>>>> external_gateway_info                                  |
>>>>>>>
>>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>>> | 1b4b108f-aefa-4548-acc4-d2441038a33f | provider-router |
>>>>>>> {"network_id": "26b88521-5d1e-4f40-a4f5-74f8a8136335"} |
>>>>>>>
>>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>>>
>>>>>>>  [DEFAULT]
>>>>>>>  # Show debugging output in log (sets DEBUG log level output)
>>>>>>>
>>>>>>>
>>>>>>>  # debug = True
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  # L3 requires that an interface driver be set.  Choose the one
>>>>>>> that best
>>>>>>>
>>>>>>> # matches your plugin.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  # OVS
>>>>>>>
>>>>>>>
>>>>>>> interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
>>>>>>> # LinuxBridge
>>>>>>>
>>>>>>>
>>>>>>> #interface_driver =
>>>>>>> quantum.agent.linux.interface.BridgeInterfaceDriver
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  # The Quantum user information for accessing the Quantum API.
>>>>>>>
>>>>>>>
>>>>>>> auth_url = http://localhost:35357/v2.0
>>>>>>> auth_region = RegionOne
>>>>>>> admin_tenant_name = service
>>>>>>> admin_user = quantum
>>>>>>> admin_password = password
>>>>>>>
>>>>>>>  # Use "sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to use
>>>>>>> the real
>>>>>>>
>>>>>>> # root filter facility.
>>>>>>>
>>>>>>>
>>>>>>> # Change to "sudo" to skip the filtering and just run the comand
>>>>>>> directly
>>>>>>>
>>>>>>> root_helper = sudo /usr/bin/quantum-rootwrap
>>>>>>> /etc/quantum/rootwrap.conf
>>>>>>>
>>>>>>>  # Allow overlapping IP (Must have kernel build with
>>>>>>> CONFIG_NET_NS=y and
>>>>>>>
>>>>>>>
>>>>>>> # iproute2 package that supports namespaces).
>>>>>>>
>>>>>>>
>>>>>>> use_namespaces = False
>>>>>>>
>>>>>>>  # If use_namespaces is set as False then the agent can only
>>>>>>> configure one router.
>>>>>>>
>>>>>>>
>>>>>>> # This is done by setting the specific router_id.
>>>>>>>
>>>>>>>
>>>>>>> router_id = 1b4b108f-aefa-4548-acc4-d2441038a33f
>>>>>>>
>>>>>>>  # Each L3 agent can be associated with at most one external
>>>>>>> network.  This
>>>>>>>
>>>>>>>
>>>>>>> # value should be set to the UUID of that external network.  If
>>>>>>> empty,
>>>>>>>
>>>>>>> # the agent will enforce that only a single external networks exists
>>>>>>> and
>>>>>>>
>>>>>>> # use that external network id
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  gateway_external_network_id = 26b88521-5d1e-4f40-a4f5-74f8a8136335
>>>>>>>
>>>>>>>  # Indicates that this L3 agent should also handle routers that do
>>>>>>> not have
>>>>>>>
>>>>>>> # an external network gateway configured.  This option should be
>>>>>>> True only
>>>>>>>
>>>>>>> # for a single agent in a Quantum deployment, and may be False for
>>>>>>> all agents
>>>>>>>
>>>>>>> # if all routers must have an external network gateway
>>>>>>>
>>>>>>>
>>>>>>> # handle_internal_only_routers = True
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  # Name of bridge used for external network traffic. This should be
>>>>>>> set to
>>>>>>>
>>>>>>> # empty value for the linux bridge
>>>>>>>
>>>>>>>
>>>>>>> external_network_bridge = br-ex
>>>>>>>
>>>>>>>  # IP address used by Nova metadata server
>>>>>>>
>>>>>>>
>>>>>>> metadata_ip = 192.168.2.225
>>>>>>>
>>>>>>>  # TCP Port used by Nova metadata server
>>>>>>>  # metadata_port = 8775
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  # The time in seconds between state poll requests
>>>>>>>
>>>>>>>
>>>>>>> # polling_interval = 3
>>>>>>>
>>>>>>>   root@controller:~# sudo tcpdump -i eth0.2 -nn | grep GRE
>>>>>>>  tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>> decode
>>>>>>> listening on eth0.2, link-type EN10MB (Ethernet), capture size 65535
>>>>>>> bytes
>>>>>>> 15:17:58.479140 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6, multicast
>>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length 24
>>>>>>> 15:17:58.627100 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>>> 15:17:59.382410 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:00.378904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:01.378903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:02.385160 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:03.382894 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:03.388210 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:03.388236 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:18:03.388246 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:03.388888 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:08.403093 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:09.402916 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:10.298390 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:10.402924 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:11.294904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:12.294913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:25.312944 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:26.310912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:27.310895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:32.319959 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:33.318897 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:34.318918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:35.322834 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:36.318925 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:36.325485 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:18:36.325511 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:36.325525 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:36.326108 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:18:41.331088 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:42.330900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:43.330903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:19:03.453248 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:04.450896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:05.450899 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:33.483521 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:34.482921 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:35.482911 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:36.390216 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:19:37.386903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:19:38.386905 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:19:48.498690 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:49.494912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:50.494922 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:55.505851 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:56.502912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:57.502911 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:58.508429 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:59.506909 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:59.511616 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:19:59.511641 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:19:59.511652 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:19:59.512382 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:20:03.711078 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6,
>>>>>>> multicast listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length
>>>>>>> 24
>>>>>>> 15:20:03.815067 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>>> 15:20:04.515054 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:20:05.514893 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:20:06.402892 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:06.514919 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:20:07.398910 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:08.398913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:21.404862 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:22.402900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:23.402904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:28.405674 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:29.402894 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:30.402913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:31.405714 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:32.402919 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:32.407948 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:20:32.407974 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:20:32.407986 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:20:32.408728 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:20:37.411052 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:38.410918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:39.410896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:20:59.567749 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:21:00.566914 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:21:01.566893 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:21:20.539385 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>>> 15:21:20.539417 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>>> 15:21:29.583747 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:21:29.587195 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 54: ARP, Reply 10.5.5.2 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>>> 15:21:29.587220 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.2 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>>> 15:21:32.472558 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:21:33.470921 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:21:34.470920 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:02.502741 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:03.498903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:04.498910 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:07.367450 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>>> 15:22:07.367476 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>>> 15:22:08.819063 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6, multicast
>>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length 24
>>>>>>> 15:22:09.335068 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>>> 15:22:17.517928 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:18.514903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:19.514909 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:24.525010 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:25.522896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:26.522895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:27.527868 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:28.526907 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:28.530086 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:22:28.530112 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:22:28.530122 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:22:28.530578 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:22:29.591728 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:22:33.539073 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:34.538891 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:34.595074 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:22:35.538887 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:22:35.594887 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:22:36.594890 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:22:45.371464 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>>> 15:22:45.371493 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>>> icmpicmpicmpicmpicmpicmp15:23:23.375417 IP 192.168.2.225 > 10.0.0.3:
>>>>>>> GREv0, key=0x1, length 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3,
>>>>>>> length 28
>>>>>>> 15:23:23.375446 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>>> uest who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:09.402916 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:10.298390 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:10.402924 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:18:11.294904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:12.294913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:25.312944 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:26.310912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:27.310895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:32.319959 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:33.318897 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:34.318918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:35.322834 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:36.318925 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:36.325485 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:36.325511 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>>  15:18:36.325525 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1,
>>>>>>> length 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:36.326108 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>> 15:18:41.331088 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:42.330900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:18:43.330903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>>> 15:19:03.453248 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> 15:19:04.450896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>>> ......
>>>>>>>
>>>>>>>  Thanks.
>>>>>>>
>>>>>>>  On 21 February 2013 12:45, Sylvain Bauza <
>>>>>>> sylvain.bauza@xxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>>  Thanks for the heads-up. So, the VM is able to reach the DHCP
>>>>>>>> server and get a lease, but unable to contact the provider router IP (on
>>>>>>>> its subnet).
>>>>>>>> Could you please start a second VM using the same way and check
>>>>>>>> network connectivity in between both ?
>>>>>>>>
>>>>>>>> I would say the issue is tied to the L3 agent, as DHCP is working
>>>>>>>> and communication is OK.
>>>>>>>> Please check on the physical layer (probably eth0) on network node
>>>>>>>> that GRE'd ICMP packet is coming in (and hoping coming *back*) when pinging
>>>>>>>> your gateway :
>>>>>>>>
>>>>>>>> sudo tcpdump -i eth0 -nn | grep GRE
>>>>>>>>
>>>>>>>>
>>>>>>>> Also, make sure /etc/quantum/l3_agent.ini is correct with router_id
>>>>>>>> and gateway_external_net_id set correctly.
>>>>>>>>
>>>>>>>> -Sylvain
>>>>>>>>
>>>>>>>> Le 21/02/2013 12:45, Chathura M. Sarathchandra Magurawalage a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>>  Sorry, it comes to the log in prompt but it takes quite long, I
>>>>>>>> just did not realise.
>>>>>>>>
>>>>>>>>  I have got a running Corros VM. This is the log.
>>>>>>>>
>>>>>>>>  [    0.000000] Initializing cgroup subsys cpuset
>>>>>>>> [    0.000000] Initializing cgroup subsys cpu
>>>>>>>> [    0.000000] Linux version 3.0.0-12-virtual (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 (Ubuntu 3.0.0-12.20-virtual 3.0.4)
>>>>>>>> [    0.000000] Command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
>>>>>>>> [    0.000000] KERNEL supported cpus:
>>>>>>>> [    0.000000]   Intel GenuineIntel
>>>>>>>> [    0.000000]   AMD AuthenticAMD
>>>>>>>> [    0.000000]   Centaur CentaurHauls
>>>>>>>> [    0.000000] BIOS-provided physical RAM map:
>>>>>>>> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
>>>>>>>> [    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
>>>>>>>> [    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
>>>>>>>> [    0.000000]  BIOS-e820: 0000000000100000 - 000000001fffd000 (usable)
>>>>>>>> [    0.000000]  BIOS-e820: 000000001fffd000 - 0000000020000000 (reserved)
>>>>>>>> [    0.000000]  BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
>>>>>>>> [    0.000000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
>>>>>>>> [    0.000000] NX (Execute Disable) protection: active
>>>>>>>> [    0.000000] DMI 2.4 present.
>>>>>>>> [    0.000000] No AGP bridge found
>>>>>>>> [    0.000000] last_pfn = 0x1fffd max_arch_pfn = 0x400000000
>>>>>>>> [    0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106
>>>>>>>> [    0.000000] found SMP MP-table at [ffff8800000fdaf0] fdaf0
>>>>>>>> [    0.000000] init_memory_mapping: 0000000000000000-000000001fffd000
>>>>>>>> [    0.000000] RAMDISK: 1fdf9000 - 1ffed000
>>>>>>>> [    0.000000] ACPI: RSDP 00000000000fd990 00014 (v00 BOCHS )
>>>>>>>> [    0.000000] ACPI: RSDT 000000001fffd7b0 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
>>>>>>>> [    0.000000] ACPI: FACP 000000001fffff80 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
>>>>>>>> [    0.000000] ACPI: DSDT 000000001fffd9b0 02589 (v01   BXPC   BXDSDT 00000001 INTL 20100528)
>>>>>>>> [    0.000000] ACPI: FACS 000000001fffff40 00040
>>>>>>>> [    0.000000] ACPI: SSDT 000000001fffd910 0009E (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
>>>>>>>> [    0.000000] ACPI: APIC 000000001fffd830 00072 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
>>>>>>>> [    0.000000] ACPI: HPET 000000001fffd7f0 00038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
>>>>>>>> [    0.000000] No NUMA configuration found
>>>>>>>> [    0.000000] Faking a node at 0000000000000000-000000001fffd000
>>>>>>>> [    0.000000] Initmem setup node 0 0000000000000000-000000001fffd000
>>>>>>>> [    0.000000]   NODE_DATA [000000001fff5000 - 000000001fff9fff]
>>>>>>>> [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
>>>>>>>> [    0.000000] kvm-clock: cpu 0, msr 0:1ce28c1, boot clock
>>>>>>>> [    0.000000] Zone PFN ranges:
>>>>>>>> [    0.000000]   DMA      0x00000010 -> 0x00001000
>>>>>>>> [    0.000000]   DMA32    0x00001000 -> 0x00100000
>>>>>>>> [    0.000000]   Normal   empty
>>>>>>>> [    0.000000] Movable zone start PFN for each node
>>>>>>>> [    0.000000] early_node_map[2] active PFN ranges
>>>>>>>> [    0.000000]     0: 0x00000010 -> 0x0000009d
>>>>>>>> [    0.000000]     0: 0x00000100 -> 0x0001fffd
>>>>>>>> [    0.000000] ACPI: PM-Timer IO Port: 0xb008
>>>>>>>> [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>>>>>>>> [    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
>>>>>>>> [    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
>>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
>>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
>>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
>>>>>>>> [    0.000000] Using ACPI (MADT) for SMP configuration information
>>>>>>>> [    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
>>>>>>>> [    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
>>>>>>>> [    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
>>>>>>>> [    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
>>>>>>>> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
>>>>>>>> [    0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
>>>>>>>> [    0.000000] Allocating PCI resources starting at 20000000 (gap: 20000000:deffc000)
>>>>>>>> [    0.000000] Booting paravirtualized kernel on KVM
>>>>>>>> [    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
>>>>>>>> [    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001fa00000 s79296 r8192 d23104 u2097152
>>>>>>>> [    0.000000] kvm-clock: cpu 0, msr 0:1fa128c1, primary cpu clock
>>>>>>>> [    0.000000] KVM setup async PF for cpu 0
>>>>>>>> [    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 129157
>>>>>>>> [    0.000000] Policy zone: DMA32
>>>>>>>> [    0.000000] Kernel command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
>>>>>>>> [    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
>>>>>>>> [    0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
>>>>>>>> [    0.000000] Checking aperture...
>>>>>>>> [    0.000000] No AGP bridge found
>>>>>>>> [    0.000000] Memory: 497852k/524276k available (6206k kernel code, 460k absent, 25964k reserved, 6907k data, 900k init)
>>>>>>>> [    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
>>>>>>>> [    0.000000] Hierarchical RCU implementation.
>>>>>>>> [    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
>>>>>>>> [    0.000000] NR_IRQS:4352 nr_irqs:256 16
>>>>>>>> [    0.000000] Console: colour VGA+ 80x25
>>>>>>>> [    0.000000] console [tty0] enabled
>>>>>>>> [    0.000000] console [ttyS0] enabled
>>>>>>>> [    0.000000] allocated 4194304 bytes of page_cgroup
>>>>>>>> [    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
>>>>>>>> [    0.000000] Detected 3392.282 MHz processor.
>>>>>>>> [    0.008000] Calibrating delay loop (skipped) preset value.. 6784.56 BogoMIPS (lpj=13569128)
>>>>>>>> [    0.008000] pid_max: default: 32768 minimum: 301
>>>>>>>> [    0.008000] Security Framework initialized
>>>>>>>> [    0.008000] AppArmor: AppArmor initialized
>>>>>>>> [    0.008000] Yama: becoming mindful.
>>>>>>>> [    0.008000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
>>>>>>>> [    0.008292] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
>>>>>>>> [    0.009146] Mount-cache hash table entries: 256
>>>>>>>> [    0.009772] Initializing cgroup subsys cpuacct
>>>>>>>> [    0.010285] Initializing cgroup subsys memory
>>>>>>>> [    0.010791] Initializing cgroup subsys devices
>>>>>>>> [    0.011312] Initializing cgroup subsys freezer
>>>>>>>> [    0.012004] Initializing cgroup subsys net_cls
>>>>>>>> [    0.012523] Initializing cgroup subsys blkio
>>>>>>>> [    0.013027] Initializing cgroup subsys perf_event
>>>>>>>> [    0.013608] mce: CPU supports 10 MCE banks
>>>>>>>> [    0.014264] SMP alternatives: switching to UP code
>>>>>>>> [    0.028279] Freeing SMP alternatives: 24k freed
>>>>>>>> [    0.028805] ACPI: Core revision 20110413
>>>>>>>> [    0.029869] ftrace: allocating 26075 entries in 103 pages
>>>>>>>> [    0.032199] Enabling x2apic
>>>>>>>> [    0.032576] Enabled x2apic
>>>>>>>> [    0.036005] Switched APIC routing to physical x2apic.
>>>>>>>> [    0.037122] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>>>>>>> [    0.037755] CPU0: Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz stepping 09
>>>>>>>> [    0.040002] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
>>>>>>>> [    0.040002] Brought up 1 CPUs
>>>>>>>> [    0.040002] Total of 1 processors activated (6784.56 BogoMIPS).
>>>>>>>> [    0.040002] devtmpfs: initialized
>>>>>>>> [    0.040252] print_constraints: dummy:
>>>>>>>> [    0.040734] Time: 12:52:56  Date: 02/20/13
>>>>>>>> [    0.041230] NET: Registered protocol family 16
>>>>>>>> [    0.041798] ACPI: bus type pci registered
>>>>>>>> [    0.042317] PCI: Using configuration type 1 for base access
>>>>>>>> [    0.043298] bio: create slab <bio-0> at 0
>>>>>>>> [    0.044935] ACPI: Interpreter enabled
>>>>>>>> [    0.045378] ACPI: (supports S0 S3 S4 S5)
>>>>>>>> [    0.046026] ACPI: Using IOAPIC for interrupt routing
>>>>>>>> [    0.048451] ACPI: No dock devices found.
>>>>>>>> [    0.048911] HEST: Table not found.
>>>>>>>> [    0.049321] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
>>>>>>>> [    0.050338] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
>>>>>>>> [    0.052351] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
>>>>>>>> [    0.053173] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
>>>>>>>> [    0.068180]  pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e)
>>>>>>>> [    0.072782] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
>>>>>>>> [    0.073647] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
>>>>>>>> [    0.074501] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
>>>>>>>> [    0.075362] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
>>>>>>>> [    0.076190] ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
>>>>>>>> [    0.077090] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
>>>>>>>> [    0.077982] vgaarb: loaded
>>>>>>>> [    0.078336] vgaarb: bridge control possible 0000:00:02.0
>>>>>>>> [    0.079070] SCSI subsystem initialized
>>>>>>>> [    0.079669] usbcore: registered new interface driver usbfs
>>>>>>>> [    0.080016] usbcore: registered new interface driver hub
>>>>>>>> [    0.080613] usbcore: registered new device driver usb
>>>>>>>> [    0.081265] PCI: Using ACPI for IRQ routing
>>>>>>>> [    0.081952] NetLabel: Initializing
>>>>>>>> [    0.082368] NetLabel:  domain hash size = 128
>>>>>>>> [    0.082869] NetLabel:  protocols = UNLABELED CIPSOv4
>>>>>>>> [    0.083417] NetLabel:  unlabeled traffic allowed by default
>>>>>>>> [    0.084051] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
>>>>>>>> [    0.084774] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
>>>>>>>> [    0.085480] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
>>>>>>>> [    0.092086] Switching to clocksource kvm-clock
>>>>>>>> [    0.095414] Switched to NOHz mode on CPU #0
>>>>>>>> [    0.097611] AppArmor: AppArmor Filesystem Enabled
>>>>>>>> [    0.098148] pnp: PnP ACPI init
>>>>>>>> [    0.098557] ACPI: bus type pnp registered
>>>>>>>> [    0.099474] pnp: PnP ACPI: found 8 devices
>>>>>>>> [    0.099943] ACPI: ACPI bus type pnp unregistered
>>>>>>>> [    0.105880] NET: Registered protocol family 2
>>>>>>>> [    0.106424] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
>>>>>>>> [    0.107293] TCP established hash table entries: 16384 (order: 6, 262144 bytes)
>>>>>>>> [    0.108257] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
>>>>>>>> [    0.109056] TCP: Hash tables configured (established 16384 bind 16384)
>>>>>>>> [    0.109725] TCP reno registered
>>>>>>>> [    0.110113] UDP hash table entries: 256 (order: 1, 8192 bytes)
>>>>>>>> [    0.110736] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
>>>>>>>> [    0.111434] NET: Registered protocol family 1
>>>>>>>> [    0.111938] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
>>>>>>>> [    0.112577] pci 0000:00:01.0: PIIX3: Enabling Passive Release
>>>>>>>> [    0.113189] pci 0000:00:01.0: Activating ISA DMA hang workarounds
>>>>>>>> [    0.114074] Trying to unpack rootfs image as initramfs...
>>>>>>>> [    0.120216] audit: initializing netlink socket (disabled)
>>>>>>>> [    0.120809] type=2000 audit(1361364777.116:1): initialized
>>>>>>>> [    0.150148] HugeTLB registered 2 MB page size, pre-allocated 0 pages
>>>>>>>> [    0.156115] VFS: Disk quotas dquot_6.5.2
>>>>>>>> [    0.156620] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
>>>>>>>> [    0.157660] fuse init (API version 7.16)
>>>>>>>> [    0.158180] msgmni has been set to 972
>>>>>>>> [    0.161560] Freeing initrd memory: 2000k freed
>>>>>>>> [    0.165512] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
>>>>>>>> [    0.166371] io scheduler noop registered
>>>>>>>> [    0.166837] io scheduler deadline registered (default)
>>>>>>>> [    0.167423] io scheduler cfq registered
>>>>>>>> [    0.167937] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
>>>>>>>> [    0.168560] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>>>> [    0.169312] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
>>>>>>>> [    0.170146] ACPI: Power Button [PWRF]
>>>>>>>> [    0.171093] ERST: Table is not found!
>>>>>>>> [    0.171635] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
>>>>>>>> [    0.172268] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11
>>>>>>>> [    0.173342] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
>>>>>>>> [    0.173965] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10
>>>>>>>> [    0.175396] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
>>>>>>>> [    0.176908] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
>>>>>>>> [    0.179328] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
>>>>>>>> [    0.202858] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
>>>>>>>> [    0.223658] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
>>>>>>>> [    0.245496] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
>>>>>>>> [    0.266837] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
>>>>>>>> [    0.267546] Linux agpgart interface v0.103
>>>>>>>> [    0.268626] brd: module loaded
>>>>>>>> [    0.269293] loop: module loaded
>>>>>>>> [    0.283115]  vda: vda1
>>>>>>>> [    0.285258] scsi0 : ata_piix
>>>>>>>> [    0.286262] scsi1 : ata_piix
>>>>>>>> [    0.287206] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
>>>>>>>> [    0.288840] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
>>>>>>>> [    0.290751] Fixed MDIO Bus: probed
>>>>>>>> [    0.291758] PPP generic driver version 2.4.2
>>>>>>>> [    0.292961] tun: Universal TUN/TAP device driver, 1.6
>>>>>>>> [    0.294237] tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
>>>>>>>> [    0.367427] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
>>>>>>>> [    0.369053] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
>>>>>>>> [    0.370623] uhci_hcd: USB Universal Host Controller Interface driver
>>>>>>>> [    0.372247] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10
>>>>>>>> [    0.374444] uhci_hcd 0000:00:01.2: UHCI Host Controller
>>>>>>>> [    0.375886] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
>>>>>>>> [    0.378008] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040
>>>>>>>> [    0.379709] hub 1-0:1.0: USB hub found
>>>>>>>> [    0.380843] hub 1-0:1.0: 2 ports detected
>>>>>>>> [    0.382181] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
>>>>>>>> [    0.383712] serio: i8042 KBD port at 0x60,0x64 irq 1
>>>>>>>> [    0.384268] serio: i8042 AUX port at 0x60,0x64 irq 12
>>>>>>>> [    0.384902] mousedev: PS/2 mouse device common for all mice
>>>>>>>> [    0.385581] rtc_cmos 00:01: RTC can wake from S4
>>>>>>>> [    0.386219] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
>>>>>>>> [    0.386944] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs
>>>>>>>> [    0.387677] device-mapper: uevent: version 1.0.3
>>>>>>>> [    0.388249] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@xxxxxxxxxx
>>>>>>>> [    0.389200] cpuidle: using governor ladder
>>>>>>>> [    0.389694] cpuidle: using governor menu
>>>>>>>> [    0.390148] EFI Variables Facility v0.08 2004-May-17
>>>>>>>> [    0.390826] TCP cubic registered
>>>>>>>> [    0.391311] NET: Registered protocol family 10
>>>>>>>> [    0.392174] NET: Registered protocol family 17
>>>>>>>> [    0.392714] Registering the dns_resolver key type
>>>>>>>> [    0.393321] registered taskstats version 1
>>>>>>>> [    0.393981] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
>>>>>>>> [    0.397108]   Magic number: 1:90:886
>>>>>>>> [    0.397602] rtc_cmos 00:01: setting system clock to 2013-02-20 12:52:56 UTC (1361364776)
>>>>>>>> [    0.398525] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
>>>>>>>> [    0.399188] EDD information not available.
>>>>>>>> [    0.550030] Freeing unused kernel memory: 900k freed
>>>>>>>> [    0.551849] Write protecting the kernel read-only data: 12288k
>>>>>>>> [    0.560335] Freeing unused kernel memory: 1968k freed
>>>>>>>> [    0.564771] Freeing unused kernel memory: 1368k freed
>>>>>>>>
>>>>>>>> info: initramfs: up at 0.56
>>>>>>>> NOCHANGE: partition 1 is size 64260. it cannot be grown
>>>>>>>> info: initramfs loading root from /dev/vda1
>>>>>>>> info: /etc/init.d/rc.sysinit: up at 0.72
>>>>>>>> [    0.722727] EXT3-fs (vda1): warning: checktime reached, running e2fsck is recommended
>>>>>>>> Starting logging: OK
>>>>>>>> Initializing random number generator... done.
>>>>>>>> Starting network...
>>>>>>>> udhcpc (v1.18.5) started
>>>>>>>> Sending discover...
>>>>>>>> Sending select for 10.5.5.3...
>>>>>>>> Lease of 10.5.5.3 obtained, lease time 120
>>>>>>>> deleting routers
>>>>>>>> route: SIOCDELRT: No such process
>>>>>>>> adding dns 8.8.8.8
>>>>>>>> cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 1/30: up 0.84. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 2/30: up 191.17. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 3/30: up 381.64. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 4/30: up 572.10. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 5/30: up 762.57. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 6/30: up 953.03. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 7/30: up 1143.49. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 8/30: up 1333.96. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 9/30: up 1524.42. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 10/30: up 1714.89. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 11/30: up 1905.35. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 12/30: up 2095.81. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 13/30: up 2286.28. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 14/30: up 2476.74. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 15/30: up 2667.21. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 16/30: up 2857.67. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 17/30: up 3048.13. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 18/30: up 3238.60. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 19/30: up 3429.06. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 20/30: up 3619.53. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 21/30: up 3809.99. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 22/30: up 4000.45. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 23/30: up 4190.92. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 24/30: up 4381.38. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 25/30: up 4571.85. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 26/30: up 4762.31. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 27/30: up 4952.77. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 28/30: up 5143.24. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 29/30: up 5333.70. request failed
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-setup: failed 30/30: up 5524.17. request failed
>>>>>>>> cloud-setup: after 30 fails, debugging
>>>>>>>> cloud-setup: running debug (30 tries reached)
>>>>>>>> ############ debug start ##############
>>>>>>>> ### /etc/rc.d/init.d/sshd start
>>>>>>>> /etc/rc3.d/S45-cloud-setup: line 66: /etc/rc.d/init.d/sshd: not found
>>>>>>>> ### ifconfig -a
>>>>>>>> eth0      Link encap:Ethernet  HWaddr FA:16:3E:A2:6E:02
>>>>>>>>           inet addr:10.5.5.3  Bcast:10.5.5.255  Mask:255.255.255.0
>>>>>>>>           inet6 addr: fe80::f816:3eff:fea2:6e02/64 Scope:Link
>>>>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>>>>           RX packets:1025 errors:0 dropped:0 overruns:0 frame:0
>>>>>>>>           TX packets:1190 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>>>           collisions:0 txqueuelen:1000
>>>>>>>>           RX bytes:127682 (124.6 KiB)  TX bytes:125076 (122.1 KiB)
>>>>>>>>
>>>>>>>> lo        Link encap:Local Loopback
>>>>>>>>           inet addr:127.0.0.1  Mask:255.0.0.0
>>>>>>>>           inet6 addr: ::1/128 Scope:Host
>>>>>>>>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>>>>>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>>>>>>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>>>           collisions:0 txqueuelen:0
>>>>>>>>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>>>>>>>>
>>>>>>>> ### route -n
>>>>>>>> Kernel IP routing table
>>>>>>>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>>>>>>>> 0.0.0.0         10.5.5.1        0.0.0.0         UG    0      0        0 eth0
>>>>>>>> 10.5.5.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
>>>>>>>> ### cat /etc/resolv.conf
>>>>>>>> search openstacklocal
>>>>>>>> nameserver 8.8.8.8
>>>>>>>> ### ping -c 5 10.5.5.1
>>>>>>>> PING 10.5.5.1 (10.5.5.1): 56 data bytes
>>>>>>>>
>>>>>>>> --- 10.5.5.1 ping statistics ---
>>>>>>>> 5 packets transmitted, 0 packets received, 100% packet loss
>>>>>>>> ### pinging nameservers
>>>>>>>> #### ping -c 5 8.8.8.8
>>>>>>>> PING 8.8.8.8 (8.8.8.8): 56 data bytes
>>>>>>>>
>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>> 5 packets transmitted, 0 packets received, 100% packet loss
>>>>>>>> ### uname -a
>>>>>>>> Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 x86_64 GNU/Linux
>>>>>>>> ### lsmod
>>>>>>>> Module                  Size  Used by    Not tainted
>>>>>>>> vfat                   17585  0
>>>>>>>> fat                    61475  1 vfat
>>>>>>>> isofs                  40253  0
>>>>>>>> ip_tables              27473  0
>>>>>>>> x_tables               29846  1 ip_tables
>>>>>>>> pcnet32                42078  0
>>>>>>>> 8139cp                 27412  0
>>>>>>>> ne2k_pci               13691  0
>>>>>>>> 8390                   18856  1 ne2k_pci
>>>>>>>> e1000                 108573  0
>>>>>>>> acpiphp                24080  0
>>>>>>>> ### dmesg | tail
>>>>>>>> [    0.763132] acpiphp: Slot [30] registered
>>>>>>>> [    0.763139] acpiphp: Slot [31] registered
>>>>>>>> [    0.767287] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>>>>>> [    0.767291] e1000: Copyright (c) 1999-2006 Intel Corporation.
>>>>>>>> [    0.770137] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
>>>>>>>> [    0.772344] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
>>>>>>>> [    0.781780] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
>>>>>>>> [    0.786775] ip_tables: (C) 2000-2006 Netfilter Core Team
>>>>>>>> [    1.112086] Refined TSC clocksource calibration: 3392.132 MHz.
>>>>>>>> [    1.680348] eth0: IPv6 duplicate address fe80::f816:3eff:fea2:6e02 detected!
>>>>>>>> ### tail -n 25 /var/log/messages
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763011] acpiphp: Slot [15] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763018] acpiphp: Slot [16] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763025] acpiphp: Slot [17] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763033] acpiphp: Slot [18] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763040] acpiphp: Slot [19] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763047] acpiphp: Slot [20] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763054] acpiphp: Slot [21] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763062] acpiphp: Slot [22] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763070] acpiphp: Slot [23] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763077] acpiphp: Slot [24] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763084] acpiphp: Slot [25] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763091] acpiphp: Slot [26] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763099] acpiphp: Slot [27] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763106] acpiphp: Slot [28] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763124] acpiphp: Slot [29] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763132] acpiphp: Slot [30] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763139] acpiphp: Slot [31] registered
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.767287] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.767291] e1000: Copyright (c) 1999-2006 Intel Corporation.
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.770137] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.772344] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.781780] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
>>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.786775] ip_tables: (C) 2000-2006 Netfilter Core Team
>>>>>>>> Feb 20 05:52:57 cirros kern.info kernel: [    1.112086] Refined TSC clocksource calibration: 3392.132 MHz.
>>>>>>>> Feb 20 05:52:57 cirros kern.info kernel: [    1.680348] eth0: IPv6 duplicate address fe80::f816:3eff:fea2:6e02 detected!
>>>>>>>> ############ debug end   ##############
>>>>>>>> cloud-setup: failed to read iid from metadata. tried 30
>>>>>>>> WARN: /etc/rc3.d/S45-cloud-setup failed
>>>>>>>> Starting dropbear sshd: generating rsa key... generating dsa key... OK
>>>>>>>> ===== cloud-final: system completely up in 5743.03 seconds ====
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>>   instance-id:
>>>>>>>>   public-ipv4:
>>>>>>>>   local-ipv4 :
>>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>> cloud-userdata: failed to read instance id
>>>>>>>> WARN: /etc/rc3.d/S99-cloud-userdata failed
>>>>>>>>   ____               ____  ____
>>>>>>>>  / __/ __ ____ ____ / __ \/ __/
>>>>>>>> / /__ / // __// __// /_/ /\ \
>>>>>>>> \___//_//_/  /_/   \____/___/
>>>>>>>>  http://launchpad.net/cirros
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
>>>>>>>>
>>>>>>>> cirros login:
>>>>>>>>
>>>>>>>>
>>>>>>>>  Thanks for the help.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 21 February 2013 08:33, Sylvain Bauza <
>>>>>>>> sylvain.bauza@xxxxxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>>  Le 20/02/2013 23:04, Chathura M. Sarathchandra Magurawalage a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>  Thanks.
>>>>>>>>>
>>>>>>>>>   I would be more concerned about the SIOCDELRT error above. Do
>>>>>>>>>> you try to manually remove a network route at bootup ? Seems like the
>>>>>>>>>> 'route del' is failing because the route is not already existing.
>>>>>>>>>>
>>>>>>>>>>  I am not doing doing anything that I am aware of.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As already said, you absolutely need VNC support for
>>>>>>>>>> investigating. Could you please fix your VNC setup which is incorrect ?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But VNC works fine. Its just that it VM hangs on the boot up it
>>>>>>>>> wont come to the log in prompt, I can't log into it.  :(
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Ahuh. Sorry for bugging you, I haven't understand : is your VM
>>>>>>>>> failing to boot up ?
>>>>>>>>> Which distro is your VM based on ?
>>>>>>>>>
>>>>>>>>> From my POV, your console.log is fine : your VM is booting,
>>>>>>>>> getting DHCP lease, trying to contact metadata server, failing to contact
>>>>>>>>> it, and that's it. You should get a prompt when logging in thru VNC.
>>>>>>>>>
>>>>>>>>> Please clarify : either the VM is failing to boot up properly (and
>>>>>>>>> then try to find a small and cloud-out-of-the-box distro like CirrOS for
>>>>>>>>> testing, or try with runlevel 3 or interactive startup), or your VM is
>>>>>>>>> properly started without network connectivity (and then you have to login
>>>>>>>>> and try more diagnostic tools like ping/tcpdump).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -Sylvain
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Follow ups

References