openstack team mailing list archive

Thread
Date
Re: Cant ping private or floating IP

To: Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>, Anil Vishnoi <vishnoianil@xxxxxxxxx>, Jean-Baptiste RANSY <jean-baptiste.ransy@xxxxxxxxxx>, JuanFra Rodriguez Cardoso <juanfra.rodriguez.cardoso@xxxxxxxxx>
From: "Chathura M. Sarathchandra Magurawalage" <77.chathura@xxxxxxxxx>
Date: Sun, 3 Mar 2013 16:49:59 +0000
Cc: openstack@xxxxxxxxxxxxxxxxxxx
In-reply-to: <CAHw8T_+6v+Vec1J=xfdvq=kiNg+0nhEex1tZfq536UgwXzdm3w@mail.gmail.com>
I have resolved this issue. All it was that I had dhcp enabled on the
eth0.2 (management network) interface, on the controller and the compute
node. Although the IP's did not changed as the IPs were reserved
beforehand.

Thanks everyone for your help.

Chathura

On 28 February 2013 17:58, Chathura M. Sarathchandra Magurawalage <
77.chathura@xxxxxxxxx> wrote:

> Hello,
>
> Now I am not using virtual aliases (eth0.2) anymore I have got two
> physical interfaces for management network and the data network.
>
> Now when I create an instance I get the following in the instance log
>
> no instance data found in start-local
>
> cloud-init-nonet waiting 120 seconds for a network device.
>
> cloud-init-nonet gave up waiting for a network device.
>
> ci-info: lo    : 1 127.0.0.1       255.0.0.0       .
>
> ci-info: eth0  : 1 .               .               fa:16:3e:40:60:2f
>
> route_info failed
>
>
> Then in /var/log/syslog I get
>
> Feb 28 17:42:32 athena dnsmasq-dhcp[19876]: DHCPDISCOVER(tapf71b5b86-5c) fa:16:3e:40:60:2f
>
>
> and I get these errors
>
> root@controller:~# less /var/log/quantum/server.log | grep ERROR
> 2013-02-28 12:19:40    ERROR [quantum.openstack.common.rpc.common] AMQP server on localhost:5672 is unreachable: Socket closed. Trying again in 1 seconds.
> 2013-02-28 14:12:32    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
> 2013-02-28 14:12:32    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
> 2013-02-28 14:12:35    ERROR [keystone.middleware.auth_token] HTTP connection exception: ''
> 2013-02-28 14:12:39    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
> 2013-02-28 15:28:57    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
> 2013-02-28 15:28:57    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
> 2013-02-28 17:35:03    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
> 2013-02-28 17:35:03    ERROR [keystone.middleware.auth_token] Bad response code while validating token: 500
>
>
>
> any idea?
>
>
> On 22 February 2013 13:55, Chathura M. Sarathchandra Magurawalage <
> 77.chathura@xxxxxxxxx> wrote:
>
>> Yes 10.5.5.5 has ARP entries for 10.5.5.1 and 10.5.5.2 with correct MAC
>> addresses.
>> 10.5.5.1 has ARP entries for 10.5.5.5 but most of the time the MAC
>> address is "(incomplete)"?
>>
>> If the eth0.2 VPN interface the problem. Do you know of any solution for
>> this matter?
>>
>> Thanks.
>>
>> On 22 February 2013 11:29, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>wrote:
>>
>>>  Let me clarify, you have :
>>>  -ARP cache entry for 10.5.5.1 on 10.5.5.5
>>>  -ARP cache entry for 10.5.5.5 on 10.5.5.1
>>> with correct MAC addresses (make sure with 'ip a show qr-XXXX') and ICMP
>>> request is still not visible on 10.5.5.5 with 'tcpdump -i eth0 -nn icmp' ?
>>>
>>> If yes, I have to admit I can't help you further. I saw you were using
>>> virtual alias eth0.2, maybe it's the root cause.
>>> All my setup is based on physical network cards, I haven't created
>>> virtual aliases and this basic test works. You have trouble with L2 and I
>>> don't think Quantum is responsible of that.
>>>
>>> -Sylvain
>>>
>>> Le 22/02/2013 11:50, Chathura M. Sarathchandra Magurawalage a écrit :
>>>
>>> I tried tcpdump on 10.5.5.5 while pinging but I can' t see ICMP requests
>>> :(
>>>
>>>  And I do have right ARP entries on 10.5.5.5
>>>
>>>  Thanks.
>>>
>>>
>>> On 22 February 2013 08:21, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>wrote:
>>>
>>>>  As per you mention, netnode do have mac address for 10.5.5.5
>>>> If you tcpdump on 10.5.5.5 while pinging from 10.5.5.1, you should see
>>>> ICMP request coming from 10.5.5.1 (provided the 'controller' node is the
>>>> network node)
>>>> Could you also check ARP cache on 10.5.5.5, especially look at 10.5.5.1
>>>> entry ?
>>>>
>>>> -Sylvain
>>>>
>>>>
>>>> Le 22/02/2013 01:24, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>
>>>>  10.5.5.5 = new fedora vm
>>>>
>>>>  arp entries for the vms.
>>>> root@controller:~# arp -n | grep 10.5.5.3
>>>> 10.5.5.3                         (incomplete)
>>>>    tapf71b5b86-5c
>>>> root@controller:~# arp -n | grep 10.5.5.4
>>>> 10.5.5.4                         (incomplete)
>>>>    tapf71b5b86-5c
>>>> root@controller:~# arp -n | grep 10.5.5.5
>>>>  10.5.5.5                 ether   fa:16:3e:7b:ef:84   C
>>>>       qr-4d088f3a-78
>>>> 10.5.5.5                         (incomplete)
>>>>    tapf71b5b86-5c
>>>>
>>>>  networknode/controller did not have an arp entry for 10.5.5.1.
>>>>
>>>>  I tried adding arp entry for the 10.5.5.3 but still it did not work.
>>>>
>>>>  any ideas?
>>>>
>>>>  thanks.
>>>>
>>>>  On 21 February 2013 17:41, Chathura M. Sarathchandra Magurawalage <
>>>> 77.chathura@xxxxxxxxx> wrote:
>>>>
>>>>>  Thanks Sylvain,
>>>>>
>>>>>  root@controller:~# ip a
>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>>>     inet 127.0.0.1/8 scope host lo
>>>>>     inet6 ::1/128 scope host
>>>>>        valid_lft forever preferred_lft forever
>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>> qlen 1000
>>>>>     link/ether d4:ae:52:bb:aa:20 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 10.10.10.1/24 brd 10.10.10.255 scope global eth0
>>>>>     inet6 fe80::d6ae:52ff:febb:aa20/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
>>>>> DOWN qlen 1000
>>>>>     link/ether d4:ae:52:bb:aa:21 brd ff:ff:ff:ff:ff:ff
>>>>> 4: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UP
>>>>>     link/ether d4:ae:52:bb:aa:20 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 192.168.2.225/24 brd 192.168.2.255 scope global eth0.2
>>>>>     inet6 fe80::d6ae:52ff:febb:aa20/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 5: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>> state UNKNOWN
>>>>>      link/ether ba:7a:e9:dc:2b:41 brd ff:ff:ff:ff:ff:ff
>>>>>     inet6 fe80::b87a:e9ff:fedc:2b41/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>> state UNKNOWN
>>>>>     link/ether 9a:41:c8:8a:9e:49 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 192.168.2.225/24 scope global br-ex
>>>>> 8: tapf71b5b86-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UNKNOWN
>>>>>     link/ether 2a:44:a3:d1:7d:f3 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 10.5.5.2/24 brd 10.5.5.255 scope global tapf71b5b86-5c
>>>>>     inet6 fe80::2844:a3ff:fed1:7df3/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 9: qr-4d088f3a-78: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UNKNOWN
>>>>>     link/ether ca:5b:8d:4d:6d:fb brd ff:ff:ff:ff:ff:ff
>>>>>     inet 10.5.5.1/24 brd 10.5.5.255 scope global qr-4d088f3a-78
>>>>>     inet6 fe80::c85b:8dff:fe4d:6dfb/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 10: qg-6f8374cb-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UNKNOWN
>>>>>     link/ether 0e:7f:dd:3a:80:bc brd ff:ff:ff:ff:ff:ff
>>>>>     inet 192.168.2.151/24 brd 192.168.2.255 scope global
>>>>> qg-6f8374cb-cb
>>>>>     inet6 fe80::c7f:ddff:fe3a:80bc/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 36: eth1.2@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc
>>>>> noqueue state LOWERLAYERDOWN
>>>>>     link/ether d4:ae:52:bb:aa:21 brd ff:ff:ff:ff:ff:ff
>>>>> 39: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>     link/ether 26:e2:a9:05:3d:4c brd ff:ff:ff:ff:ff:ff
>>>>>
>>>>>
>>>>>  But for some reason the "arp" module is not in the cirros cloud
>>>>> image. Do you suggest any other image to create instances with?, which I
>>>>> can log in without having to ssh in first like the ubuntu cloud image.
>>>>>
>>>>>  Thanks.
>>>>>
>>>>>  On 21 February 2013 16:17, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>wrote:
>>>>>
>>>>>>  Could you please :
>>>>>>  - 'ip a' on network node
>>>>>>  - 'arp -n' on VM and check '10.5.5.1'
>>>>>>  - 'arp -n' on netnode and check '10.5.5.3'
>>>>>>
>>>>>> Connection in between VM and internal gateway is managed thru ARP
>>>>>> resolution.
>>>>>> If 'arp -n' is giving no MAC address for 10.5.5.1 on VM, try :
>>>>>>  1. get the MAC address for 10.5.5.1 on netnode using 'ip a'
>>>>>>  2. get the MAC address for 10.5.5.3 on VM using 'ifconfig'
>>>>>>  2. add manually an ARP entry for 10.5.5.1 by issuing 'arp -s
>>>>>> 10.5.5.1 <netnode_mac_addr>'
>>>>>>  3. (if not present on netnode) add manually ARP entry for 10.5.5.3
>>>>>> using 'arp -s 10.5.5.3 <vm_mac_addr>'
>>>>>> and check with ping.
>>>>>>
>>>>>> If that fixes the bug, you have to investigate why ARP resolution is
>>>>>> fine for DHCP server but not for gateway. You should see ARP requests on
>>>>>> physical layer with 'ARP, Request who-has 10.5.5.1 tell 10.5.5.3, length
>>>>>> 28' and ARP reply as follows 'ARP, Reply 10.5.5.1 is-at <mac_addr>, length
>>>>>> 28'
>>>>>>
>>>>>> -Sylvain
>>>>>>
>>>>>>
>>>>>> Le 21/02/2013 16:29, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>>>
>>>>>> So I have got two VMs, they can ping eachother. But they cannot ping
>>>>>> the gateway.
>>>>>>
>>>>>>  Just for the record. From my controller/network-node I can ping the
>>>>>> provider router.
>>>>>>
>>>>>>  root@controller
>>>>>> :~# quantum router-list
>>>>>>
>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>> | id                                   | name            |
>>>>>> external_gateway_info                                  |
>>>>>>
>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>> | 1b4b108f-aefa-4548-acc4-d2441038a33f | provider-router |
>>>>>> {"network_id": "26b88521-5d1e-4f40-a4f5-74f8a8136335"} |
>>>>>>
>>>>>> +--------------------------------------+-----------------+--------------------------------------------------------+
>>>>>>
>>>>>>  [DEFAULT]
>>>>>>  # Show debugging output in log (sets DEBUG log level output)
>>>>>>
>>>>>>
>>>>>>  # debug = True
>>>>>>
>>>>>>
>>>>>>
>>>>>>  # L3 requires that an interface driver be set.  Choose the one that
>>>>>> best
>>>>>>
>>>>>> # matches your plugin.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  # OVS
>>>>>>
>>>>>>
>>>>>> interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
>>>>>> # LinuxBridge
>>>>>>
>>>>>>
>>>>>> #interface_driver =
>>>>>> quantum.agent.linux.interface.BridgeInterfaceDriver
>>>>>>
>>>>>>
>>>>>>
>>>>>>  # The Quantum user information for accessing the Quantum API.
>>>>>>
>>>>>>
>>>>>> auth_url = http://localhost:35357/v2.0
>>>>>> auth_region = RegionOne
>>>>>> admin_tenant_name = service
>>>>>> admin_user = quantum
>>>>>> admin_password = password
>>>>>>
>>>>>>  # Use "sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to use the
>>>>>> real
>>>>>>
>>>>>> # root filter facility.
>>>>>>
>>>>>>
>>>>>> # Change to "sudo" to skip the filtering and just run the comand
>>>>>> directly
>>>>>>
>>>>>> root_helper = sudo /usr/bin/quantum-rootwrap
>>>>>> /etc/quantum/rootwrap.conf
>>>>>>
>>>>>>  # Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y
>>>>>> and
>>>>>>
>>>>>> # iproute2 package that supports namespaces).
>>>>>>
>>>>>>
>>>>>> use_namespaces = False
>>>>>>
>>>>>>  # If use_namespaces is set as False then the agent can only
>>>>>> configure one router.
>>>>>>
>>>>>>
>>>>>> # This is done by setting the specific router_id.
>>>>>>
>>>>>>
>>>>>> router_id = 1b4b108f-aefa-4548-acc4-d2441038a33f
>>>>>>
>>>>>>  # Each L3 agent can be associated with at most one external
>>>>>> network.  This
>>>>>>
>>>>>>
>>>>>> # value should be set to the UUID of that external network.  If
>>>>>> empty,
>>>>>>
>>>>>> # the agent will enforce that only a single external networks exists
>>>>>> and
>>>>>>
>>>>>> # use that external network id
>>>>>>
>>>>>>
>>>>>>
>>>>>>  gateway_external_network_id = 26b88521-5d1e-4f40-a4f5-74f8a8136335
>>>>>>
>>>>>>  # Indicates that this L3 agent should also handle routers that do
>>>>>> not have
>>>>>>
>>>>>> # an external network gateway configured.  This option should be True
>>>>>> only
>>>>>>
>>>>>> # for a single agent in a Quantum deployment, and may be False for
>>>>>> all agents
>>>>>>
>>>>>> # if all routers must have an external network gateway
>>>>>>
>>>>>>
>>>>>> # handle_internal_only_routers = True
>>>>>>
>>>>>>
>>>>>>
>>>>>>  # Name of bridge used for external network traffic. This should be
>>>>>> set to
>>>>>>
>>>>>> # empty value for the linux bridge
>>>>>>
>>>>>>
>>>>>> external_network_bridge = br-ex
>>>>>>
>>>>>>  # IP address used by Nova metadata server
>>>>>>
>>>>>>
>>>>>> metadata_ip = 192.168.2.225
>>>>>>
>>>>>>  # TCP Port used by Nova metadata server
>>>>>>  # metadata_port = 8775
>>>>>>
>>>>>>
>>>>>>
>>>>>>  # The time in seconds between state poll requests
>>>>>>
>>>>>>
>>>>>> # polling_interval = 3
>>>>>>
>>>>>>   root@controller:~# sudo tcpdump -i eth0.2 -nn | grep GRE
>>>>>>  tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>> decode
>>>>>> listening on eth0.2, link-type EN10MB (Ethernet), capture size 65535
>>>>>> bytes
>>>>>> 15:17:58.479140 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length 24
>>>>>> 15:17:58.627100 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>> 15:17:59.382410 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:00.378904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:01.378903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:02.385160 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:03.382894 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:03.388210 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:03.388236 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:18:03.388246 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:03.388888 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:08.403093 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:09.402916 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:10.298390 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:10.402924 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:11.294904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:12.294913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:25.312944 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:26.310912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:27.310895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:32.319959 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:33.318897 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:34.318918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:35.322834 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:36.318925 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:36.325485 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:18:36.325511 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:36.325525 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:36.326108 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:18:41.331088 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:42.330900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:43.330903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:19:03.453248 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:04.450896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:05.450899 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:33.483521 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:34.482921 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:35.482911 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:36.390216 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:19:37.386903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:19:38.386905 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:19:48.498690 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:49.494912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:50.494922 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:55.505851 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:56.502912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:57.502911 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:58.508429 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:59.506909 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:59.511616 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:19:59.511641 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:19:59.511652 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:19:59.512382 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:20:03.711078 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length 24
>>>>>> 15:20:03.815067 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>> 15:20:04.515054 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:20:05.514893 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:20:06.402892 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:06.514919 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:20:07.398910 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:08.398913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:21.404862 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:22.402900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:23.402904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:28.405674 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:29.402894 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:30.402913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:31.405714 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:32.402919 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:32.407948 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:20:32.407974 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:20:32.407986 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:20:32.408728 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:20:37.411052 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:38.410918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:39.410896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:20:59.567749 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:21:00.566914 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:21:01.566893 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:21:20.539385 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>> 15:21:20.539417 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>> 15:21:29.583747 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:21:29.587195 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.2 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>> 15:21:29.587220 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.2 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>> 15:21:32.472558 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:21:33.470921 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:21:34.470920 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:02.502741 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:03.498903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:04.498910 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:07.367450 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>> 15:22:07.367476 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>> 15:22:08.819063 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::c85b:8dff:fe4d:6dfb > ff02::1:ff4d:6dfb: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ff4d:6dfb, length 24
>>>>>> 15:22:09.335068 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 98: IP6 fe80::2844:a3ff:fed1:7df3 > ff02::1:ffd1:7df3: HBH ICMP6, multicast
>>>>>> listener reportmax resp delay: 0 addr: ff02::1:ffd1:7df3, length 24
>>>>>> 15:22:17.517928 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:18.514903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:19.514909 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:24.525010 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:25.522896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:26.522895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:27.527868 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:28.526907 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:28.530086 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:22:28.530112 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:22:28.530122 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:22:28.530578 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:22:29.591728 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.4.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:22:33.539073 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:34.538891 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:34.595074 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:22:35.538887 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:22:35.594887 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:22:36.594890 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:22:45.371464 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3, length 28
>>>>>> 15:22:45.371493 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>> icmpicmpicmpicmpicmpicmp15:23:23.375417 IP 192.168.2.225 > 10.0.0.3:
>>>>>> GREv0, key=0x1, length 54: ARP, Reply 10.5.5.1 is-at 2a:44:a3:d1:7d:f3,
>>>>>> length 28
>>>>>> 15:23:23.375446 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Reply 10.5.5.1 is-at ca:5b:8d:4d:6d:fb, length 28
>>>>>> uest who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:09.402916 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:10.298390 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:10.402924 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:18:11.294904 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:12.294913 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:25.312944 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:26.310912 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:27.310895 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:32.319959 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:33.318897 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:34.318918 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:35.322834 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:36.318925 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:36.325485 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:36.325511 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>>  15:18:36.325525 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:36.326108 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 372: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 318
>>>>>> 15:18:41.331088 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:42.330900 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:18:43.330903 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28
>>>>>> 15:19:03.453248 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> 15:19:04.450896 IP 192.168.2.225 > 10.0.0.3: GREv0, key=0x1, length
>>>>>> 54: ARP, Request who-has 10.5.5.4 tell 10.5.5.2, length 28
>>>>>> ......
>>>>>>
>>>>>>  Thanks.
>>>>>>
>>>>>>  On 21 February 2013 12:45, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>>  Thanks for the heads-up. So, the VM is able to reach the DHCP
>>>>>>> server and get a lease, but unable to contact the provider router IP (on
>>>>>>> its subnet).
>>>>>>> Could you please start a second VM using the same way and check
>>>>>>> network connectivity in between both ?
>>>>>>>
>>>>>>> I would say the issue is tied to the L3 agent, as DHCP is working
>>>>>>> and communication is OK.
>>>>>>> Please check on the physical layer (probably eth0) on network node
>>>>>>> that GRE'd ICMP packet is coming in (and hoping coming *back*) when pinging
>>>>>>> your gateway :
>>>>>>>
>>>>>>> sudo tcpdump -i eth0 -nn | grep GRE
>>>>>>>
>>>>>>>
>>>>>>> Also, make sure /etc/quantum/l3_agent.ini is correct with router_id
>>>>>>> and gateway_external_net_id set correctly.
>>>>>>>
>>>>>>> -Sylvain
>>>>>>>
>>>>>>> Le 21/02/2013 12:45, Chathura M. Sarathchandra Magurawalage a écrit :
>>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>>  Sorry, it comes to the log in prompt but it takes quite long, I
>>>>>>> just did not realise.
>>>>>>>
>>>>>>>  I have got a running Corros VM. This is the log.
>>>>>>>
>>>>>>>  [    0.000000] Initializing cgroup subsys cpuset
>>>>>>> [    0.000000] Initializing cgroup subsys cpu
>>>>>>> [    0.000000] Linux version 3.0.0-12-virtual (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 (Ubuntu 3.0.0-12.20-virtual 3.0.4)
>>>>>>> [    0.000000] Command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
>>>>>>> [    0.000000] KERNEL supported cpus:
>>>>>>> [    0.000000]   Intel GenuineIntel
>>>>>>> [    0.000000]   AMD AuthenticAMD
>>>>>>> [    0.000000]   Centaur CentaurHauls
>>>>>>> [    0.000000] BIOS-provided physical RAM map:
>>>>>>> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
>>>>>>> [    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
>>>>>>> [    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
>>>>>>> [    0.000000]  BIOS-e820: 0000000000100000 - 000000001fffd000 (usable)
>>>>>>> [    0.000000]  BIOS-e820: 000000001fffd000 - 0000000020000000 (reserved)
>>>>>>> [    0.000000]  BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
>>>>>>> [    0.000000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
>>>>>>> [    0.000000] NX (Execute Disable) protection: active
>>>>>>> [    0.000000] DMI 2.4 present.
>>>>>>> [    0.000000] No AGP bridge found
>>>>>>> [    0.000000] last_pfn = 0x1fffd max_arch_pfn = 0x400000000
>>>>>>> [    0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106
>>>>>>> [    0.000000] found SMP MP-table at [ffff8800000fdaf0] fdaf0
>>>>>>> [    0.000000] init_memory_mapping: 0000000000000000-000000001fffd000
>>>>>>> [    0.000000] RAMDISK: 1fdf9000 - 1ffed000
>>>>>>> [    0.000000] ACPI: RSDP 00000000000fd990 00014 (v00 BOCHS )
>>>>>>> [    0.000000] ACPI: RSDT 000000001fffd7b0 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
>>>>>>> [    0.000000] ACPI: FACP 000000001fffff80 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
>>>>>>> [    0.000000] ACPI: DSDT 000000001fffd9b0 02589 (v01   BXPC   BXDSDT 00000001 INTL 20100528)
>>>>>>> [    0.000000] ACPI: FACS 000000001fffff40 00040
>>>>>>> [    0.000000] ACPI: SSDT 000000001fffd910 0009E (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
>>>>>>> [    0.000000] ACPI: APIC 000000001fffd830 00072 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
>>>>>>> [    0.000000] ACPI: HPET 000000001fffd7f0 00038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
>>>>>>> [    0.000000] No NUMA configuration found
>>>>>>> [    0.000000] Faking a node at 0000000000000000-000000001fffd000
>>>>>>> [    0.000000] Initmem setup node 0 0000000000000000-000000001fffd000
>>>>>>> [    0.000000]   NODE_DATA [000000001fff5000 - 000000001fff9fff]
>>>>>>> [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
>>>>>>> [    0.000000] kvm-clock: cpu 0, msr 0:1ce28c1, boot clock
>>>>>>> [    0.000000] Zone PFN ranges:
>>>>>>> [    0.000000]   DMA      0x00000010 -> 0x00001000
>>>>>>> [    0.000000]   DMA32    0x00001000 -> 0x00100000
>>>>>>> [    0.000000]   Normal   empty
>>>>>>> [    0.000000] Movable zone start PFN for each node
>>>>>>> [    0.000000] early_node_map[2] active PFN ranges
>>>>>>> [    0.000000]     0: 0x00000010 -> 0x0000009d
>>>>>>> [    0.000000]     0: 0x00000100 -> 0x0001fffd
>>>>>>> [    0.000000] ACPI: PM-Timer IO Port: 0xb008
>>>>>>> [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>>>>>>> [    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
>>>>>>> [    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
>>>>>>> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
>>>>>>> [    0.000000] Using ACPI (MADT) for SMP configuration information
>>>>>>> [    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
>>>>>>> [    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
>>>>>>> [    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
>>>>>>> [    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
>>>>>>> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
>>>>>>> [    0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
>>>>>>> [    0.000000] Allocating PCI resources starting at 20000000 (gap: 20000000:deffc000)
>>>>>>> [    0.000000] Booting paravirtualized kernel on KVM
>>>>>>> [    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
>>>>>>> [    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001fa00000 s79296 r8192 d23104 u2097152
>>>>>>> [    0.000000] kvm-clock: cpu 0, msr 0:1fa128c1, primary cpu clock
>>>>>>> [    0.000000] KVM setup async PF for cpu 0
>>>>>>> [    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 129157
>>>>>>> [    0.000000] Policy zone: DMA32
>>>>>>> [    0.000000] Kernel command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
>>>>>>> [    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
>>>>>>> [    0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
>>>>>>> [    0.000000] Checking aperture...
>>>>>>> [    0.000000] No AGP bridge found
>>>>>>> [    0.000000] Memory: 497852k/524276k available (6206k kernel code, 460k absent, 25964k reserved, 6907k data, 900k init)
>>>>>>> [    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
>>>>>>> [    0.000000] Hierarchical RCU implementation.
>>>>>>> [    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
>>>>>>> [    0.000000] NR_IRQS:4352 nr_irqs:256 16
>>>>>>> [    0.000000] Console: colour VGA+ 80x25
>>>>>>> [    0.000000] console [tty0] enabled
>>>>>>> [    0.000000] console [ttyS0] enabled
>>>>>>> [    0.000000] allocated 4194304 bytes of page_cgroup
>>>>>>> [    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
>>>>>>> [    0.000000] Detected 3392.282 MHz processor.
>>>>>>> [    0.008000] Calibrating delay loop (skipped) preset value.. 6784.56 BogoMIPS (lpj=13569128)
>>>>>>> [    0.008000] pid_max: default: 32768 minimum: 301
>>>>>>> [    0.008000] Security Framework initialized
>>>>>>> [    0.008000] AppArmor: AppArmor initialized
>>>>>>> [    0.008000] Yama: becoming mindful.
>>>>>>> [    0.008000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
>>>>>>> [    0.008292] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
>>>>>>> [    0.009146] Mount-cache hash table entries: 256
>>>>>>> [    0.009772] Initializing cgroup subsys cpuacct
>>>>>>> [    0.010285] Initializing cgroup subsys memory
>>>>>>> [    0.010791] Initializing cgroup subsys devices
>>>>>>> [    0.011312] Initializing cgroup subsys freezer
>>>>>>> [    0.012004] Initializing cgroup subsys net_cls
>>>>>>> [    0.012523] Initializing cgroup subsys blkio
>>>>>>> [    0.013027] Initializing cgroup subsys perf_event
>>>>>>> [    0.013608] mce: CPU supports 10 MCE banks
>>>>>>> [    0.014264] SMP alternatives: switching to UP code
>>>>>>> [    0.028279] Freeing SMP alternatives: 24k freed
>>>>>>> [    0.028805] ACPI: Core revision 20110413
>>>>>>> [    0.029869] ftrace: allocating 26075 entries in 103 pages
>>>>>>> [    0.032199] Enabling x2apic
>>>>>>> [    0.032576] Enabled x2apic
>>>>>>> [    0.036005] Switched APIC routing to physical x2apic.
>>>>>>> [    0.037122] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>>>>>> [    0.037755] CPU0: Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz stepping 09
>>>>>>> [    0.040002] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
>>>>>>> [    0.040002] Brought up 1 CPUs
>>>>>>> [    0.040002] Total of 1 processors activated (6784.56 BogoMIPS).
>>>>>>> [    0.040002] devtmpfs: initialized
>>>>>>> [    0.040252] print_constraints: dummy:
>>>>>>> [    0.040734] Time: 12:52:56  Date: 02/20/13
>>>>>>> [    0.041230] NET: Registered protocol family 16
>>>>>>> [    0.041798] ACPI: bus type pci registered
>>>>>>> [    0.042317] PCI: Using configuration type 1 for base access
>>>>>>> [    0.043298] bio: create slab <bio-0> at 0
>>>>>>> [    0.044935] ACPI: Interpreter enabled
>>>>>>> [    0.045378] ACPI: (supports S0 S3 S4 S5)
>>>>>>> [    0.046026] ACPI: Using IOAPIC for interrupt routing
>>>>>>> [    0.048451] ACPI: No dock devices found.
>>>>>>> [    0.048911] HEST: Table not found.
>>>>>>> [    0.049321] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
>>>>>>> [    0.050338] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
>>>>>>> [    0.052351] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
>>>>>>> [    0.053173] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
>>>>>>> [    0.068180]  pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e)
>>>>>>> [    0.072782] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
>>>>>>> [    0.073647] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
>>>>>>> [    0.074501] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
>>>>>>> [    0.075362] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
>>>>>>> [    0.076190] ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
>>>>>>> [    0.077090] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
>>>>>>> [    0.077982] vgaarb: loaded
>>>>>>> [    0.078336] vgaarb: bridge control possible 0000:00:02.0
>>>>>>> [    0.079070] SCSI subsystem initialized
>>>>>>> [    0.079669] usbcore: registered new interface driver usbfs
>>>>>>> [    0.080016] usbcore: registered new interface driver hub
>>>>>>> [    0.080613] usbcore: registered new device driver usb
>>>>>>> [    0.081265] PCI: Using ACPI for IRQ routing
>>>>>>> [    0.081952] NetLabel: Initializing
>>>>>>> [    0.082368] NetLabel:  domain hash size = 128
>>>>>>> [    0.082869] NetLabel:  protocols = UNLABELED CIPSOv4
>>>>>>> [    0.083417] NetLabel:  unlabeled traffic allowed by default
>>>>>>> [    0.084051] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
>>>>>>> [    0.084774] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
>>>>>>> [    0.085480] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
>>>>>>> [    0.092086] Switching to clocksource kvm-clock
>>>>>>> [    0.095414] Switched to NOHz mode on CPU #0
>>>>>>> [    0.097611] AppArmor: AppArmor Filesystem Enabled
>>>>>>> [    0.098148] pnp: PnP ACPI init
>>>>>>> [    0.098557] ACPI: bus type pnp registered
>>>>>>> [    0.099474] pnp: PnP ACPI: found 8 devices
>>>>>>> [    0.099943] ACPI: ACPI bus type pnp unregistered
>>>>>>> [    0.105880] NET: Registered protocol family 2
>>>>>>> [    0.106424] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
>>>>>>> [    0.107293] TCP established hash table entries: 16384 (order: 6, 262144 bytes)
>>>>>>> [    0.108257] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
>>>>>>> [    0.109056] TCP: Hash tables configured (established 16384 bind 16384)
>>>>>>> [    0.109725] TCP reno registered
>>>>>>> [    0.110113] UDP hash table entries: 256 (order: 1, 8192 bytes)
>>>>>>> [    0.110736] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
>>>>>>> [    0.111434] NET: Registered protocol family 1
>>>>>>> [    0.111938] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
>>>>>>> [    0.112577] pci 0000:00:01.0: PIIX3: Enabling Passive Release
>>>>>>> [    0.113189] pci 0000:00:01.0: Activating ISA DMA hang workarounds
>>>>>>> [    0.114074] Trying to unpack rootfs image as initramfs...
>>>>>>> [    0.120216] audit: initializing netlink socket (disabled)
>>>>>>> [    0.120809] type=2000 audit(1361364777.116:1): initialized
>>>>>>> [    0.150148] HugeTLB registered 2 MB page size, pre-allocated 0 pages
>>>>>>> [    0.156115] VFS: Disk quotas dquot_6.5.2
>>>>>>> [    0.156620] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
>>>>>>> [    0.157660] fuse init (API version 7.16)
>>>>>>> [    0.158180] msgmni has been set to 972
>>>>>>> [    0.161560] Freeing initrd memory: 2000k freed
>>>>>>> [    0.165512] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
>>>>>>> [    0.166371] io scheduler noop registered
>>>>>>> [    0.166837] io scheduler deadline registered (default)
>>>>>>> [    0.167423] io scheduler cfq registered
>>>>>>> [    0.167937] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
>>>>>>> [    0.168560] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>>> [    0.169312] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
>>>>>>> [    0.170146] ACPI: Power Button [PWRF]
>>>>>>> [    0.171093] ERST: Table is not found!
>>>>>>> [    0.171635] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
>>>>>>> [    0.172268] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11
>>>>>>> [    0.173342] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
>>>>>>> [    0.173965] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10
>>>>>>> [    0.175396] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
>>>>>>> [    0.176908] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
>>>>>>> [    0.179328] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
>>>>>>> [    0.202858] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
>>>>>>> [    0.223658] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
>>>>>>> [    0.245496] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
>>>>>>> [    0.266837] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
>>>>>>> [    0.267546] Linux agpgart interface v0.103
>>>>>>> [    0.268626] brd: module loaded
>>>>>>> [    0.269293] loop: module loaded
>>>>>>> [    0.283115]  vda: vda1
>>>>>>> [    0.285258] scsi0 : ata_piix
>>>>>>> [    0.286262] scsi1 : ata_piix
>>>>>>> [    0.287206] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
>>>>>>> [    0.288840] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
>>>>>>> [    0.290751] Fixed MDIO Bus: probed
>>>>>>> [    0.291758] PPP generic driver version 2.4.2
>>>>>>> [    0.292961] tun: Universal TUN/TAP device driver, 1.6
>>>>>>> [    0.294237] tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
>>>>>>> [    0.367427] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
>>>>>>> [    0.369053] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
>>>>>>> [    0.370623] uhci_hcd: USB Universal Host Controller Interface driver
>>>>>>> [    0.372247] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10
>>>>>>> [    0.374444] uhci_hcd 0000:00:01.2: UHCI Host Controller
>>>>>>> [    0.375886] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
>>>>>>> [    0.378008] uhci_hcd 0000:00:01.2: irq 10, io base 0x0000c040
>>>>>>> [    0.379709] hub 1-0:1.0: USB hub found
>>>>>>> [    0.380843] hub 1-0:1.0: 2 ports detected
>>>>>>> [    0.382181] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
>>>>>>> [    0.383712] serio: i8042 KBD port at 0x60,0x64 irq 1
>>>>>>> [    0.384268] serio: i8042 AUX port at 0x60,0x64 irq 12
>>>>>>> [    0.384902] mousedev: PS/2 mouse device common for all mice
>>>>>>> [    0.385581] rtc_cmos 00:01: RTC can wake from S4
>>>>>>> [    0.386219] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
>>>>>>> [    0.386944] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs
>>>>>>> [    0.387677] device-mapper: uevent: version 1.0.3
>>>>>>> [    0.388249] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@xxxxxxxxxx
>>>>>>> [    0.389200] cpuidle: using governor ladder
>>>>>>> [    0.389694] cpuidle: using governor menu
>>>>>>> [    0.390148] EFI Variables Facility v0.08 2004-May-17
>>>>>>> [    0.390826] TCP cubic registered
>>>>>>> [    0.391311] NET: Registered protocol family 10
>>>>>>> [    0.392174] NET: Registered protocol family 17
>>>>>>> [    0.392714] Registering the dns_resolver key type
>>>>>>> [    0.393321] registered taskstats version 1
>>>>>>> [    0.393981] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
>>>>>>> [    0.397108]   Magic number: 1:90:886
>>>>>>> [    0.397602] rtc_cmos 00:01: setting system clock to 2013-02-20 12:52:56 UTC (1361364776)
>>>>>>> [    0.398525] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
>>>>>>> [    0.399188] EDD information not available.
>>>>>>> [    0.550030] Freeing unused kernel memory: 900k freed
>>>>>>> [    0.551849] Write protecting the kernel read-only data: 12288k
>>>>>>> [    0.560335] Freeing unused kernel memory: 1968k freed
>>>>>>> [    0.564771] Freeing unused kernel memory: 1368k freed
>>>>>>>
>>>>>>> info: initramfs: up at 0.56
>>>>>>> NOCHANGE: partition 1 is size 64260. it cannot be grown
>>>>>>> info: initramfs loading root from /dev/vda1
>>>>>>> info: /etc/init.d/rc.sysinit: up at 0.72
>>>>>>> [    0.722727] EXT3-fs (vda1): warning: checktime reached, running e2fsck is recommended
>>>>>>> Starting logging: OK
>>>>>>> Initializing random number generator... done.
>>>>>>> Starting network...
>>>>>>> udhcpc (v1.18.5) started
>>>>>>> Sending discover...
>>>>>>> Sending select for 10.5.5.3...
>>>>>>> Lease of 10.5.5.3 obtained, lease time 120
>>>>>>> deleting routers
>>>>>>> route: SIOCDELRT: No such process
>>>>>>> adding dns 8.8.8.8
>>>>>>> cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 1/30: up 0.84. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 2/30: up 191.17. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 3/30: up 381.64. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 4/30: up 572.10. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 5/30: up 762.57. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 6/30: up 953.03. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 7/30: up 1143.49. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 8/30: up 1333.96. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 9/30: up 1524.42. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 10/30: up 1714.89. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 11/30: up 1905.35. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 12/30: up 2095.81. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 13/30: up 2286.28. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 14/30: up 2476.74. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 15/30: up 2667.21. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 16/30: up 2857.67. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 17/30: up 3048.13. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 18/30: up 3238.60. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 19/30: up 3429.06. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 20/30: up 3619.53. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 21/30: up 3809.99. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 22/30: up 4000.45. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 23/30: up 4190.92. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 24/30: up 4381.38. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 25/30: up 4571.85. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 26/30: up 4762.31. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 27/30: up 4952.77. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 28/30: up 5143.24. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 29/30: up 5333.70. request failed
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-setup: failed 30/30: up 5524.17. request failed
>>>>>>> cloud-setup: after 30 fails, debugging
>>>>>>> cloud-setup: running debug (30 tries reached)
>>>>>>> ############ debug start ##############
>>>>>>> ### /etc/rc.d/init.d/sshd start
>>>>>>> /etc/rc3.d/S45-cloud-setup: line 66: /etc/rc.d/init.d/sshd: not found
>>>>>>> ### ifconfig -a
>>>>>>> eth0      Link encap:Ethernet  HWaddr FA:16:3E:A2:6E:02
>>>>>>>           inet addr:10.5.5.3  Bcast:10.5.5.255  Mask:255.255.255.0
>>>>>>>           inet6 addr: fe80::f816:3eff:fea2:6e02/64 Scope:Link
>>>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>>>           RX packets:1025 errors:0 dropped:0 overruns:0 frame:0
>>>>>>>           TX packets:1190 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>>           collisions:0 txqueuelen:1000
>>>>>>>           RX bytes:127682 (124.6 KiB)  TX bytes:125076 (122.1 KiB)
>>>>>>>
>>>>>>> lo        Link encap:Local Loopback
>>>>>>>           inet addr:127.0.0.1  Mask:255.0.0.0
>>>>>>>           inet6 addr: ::1/128 Scope:Host
>>>>>>>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>>>>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>>>>>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>>>>>           collisions:0 txqueuelen:0
>>>>>>>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>>>>>>>
>>>>>>> ### route -n
>>>>>>> Kernel IP routing table
>>>>>>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>>>>>>> 0.0.0.0         10.5.5.1        0.0.0.0         UG    0      0        0 eth0
>>>>>>> 10.5.5.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
>>>>>>> ### cat /etc/resolv.conf
>>>>>>> search openstacklocal
>>>>>>> nameserver 8.8.8.8
>>>>>>> ### ping -c 5 10.5.5.1
>>>>>>> PING 10.5.5.1 (10.5.5.1): 56 data bytes
>>>>>>>
>>>>>>> --- 10.5.5.1 ping statistics ---
>>>>>>> 5 packets transmitted, 0 packets received, 100% packet loss
>>>>>>> ### pinging nameservers
>>>>>>> #### ping -c 5 8.8.8.8
>>>>>>> PING 8.8.8.8 (8.8.8.8): 56 data bytes
>>>>>>>
>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>> 5 packets transmitted, 0 packets received, 100% packet loss
>>>>>>> ### uname -a
>>>>>>> Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 x86_64 GNU/Linux
>>>>>>> ### lsmod
>>>>>>> Module                  Size  Used by    Not tainted
>>>>>>> vfat                   17585  0
>>>>>>> fat                    61475  1 vfat
>>>>>>> isofs                  40253  0
>>>>>>> ip_tables              27473  0
>>>>>>> x_tables               29846  1 ip_tables
>>>>>>> pcnet32                42078  0
>>>>>>> 8139cp                 27412  0
>>>>>>> ne2k_pci               13691  0
>>>>>>> 8390                   18856  1 ne2k_pci
>>>>>>> e1000                 108573  0
>>>>>>> acpiphp                24080  0
>>>>>>> ### dmesg | tail
>>>>>>> [    0.763132] acpiphp: Slot [30] registered
>>>>>>> [    0.763139] acpiphp: Slot [31] registered
>>>>>>> [    0.767287] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>>>>> [    0.767291] e1000: Copyright (c) 1999-2006 Intel Corporation.
>>>>>>> [    0.770137] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
>>>>>>> [    0.772344] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
>>>>>>> [    0.781780] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
>>>>>>> [    0.786775] ip_tables: (C) 2000-2006 Netfilter Core Team
>>>>>>> [    1.112086] Refined TSC clocksource calibration: 3392.132 MHz.
>>>>>>> [    1.680348] eth0: IPv6 duplicate address fe80::f816:3eff:fea2:6e02 detected!
>>>>>>> ### tail -n 25 /var/log/messages
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763011] acpiphp: Slot [15] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763018] acpiphp: Slot [16] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763025] acpiphp: Slot [17] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763033] acpiphp: Slot [18] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763040] acpiphp: Slot [19] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763047] acpiphp: Slot [20] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763054] acpiphp: Slot [21] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763062] acpiphp: Slot [22] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763070] acpiphp: Slot [23] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763077] acpiphp: Slot [24] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763084] acpiphp: Slot [25] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763091] acpiphp: Slot [26] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763099] acpiphp: Slot [27] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763106] acpiphp: Slot [28] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763124] acpiphp: Slot [29] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763132] acpiphp: Slot [30] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.763139] acpiphp: Slot [31] registered
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.767287] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.767291] e1000: Copyright (c) 1999-2006 Intel Corporation.
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.770137] ne2k-pci.c:v1.03 9/22/2003 D. Becker/P. Gortmaker
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.772344] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.781780] pcnet32: pcnet32.c:v1.35 21.Apr.2008 tsbogend@xxxxxxxxxxxxxxxx
>>>>>>> Feb 20 05:52:56 cirros kern.info kernel: [    0.786775] ip_tables: (C) 2000-2006 Netfilter Core Team
>>>>>>> Feb 20 05:52:57 cirros kern.info kernel: [    1.112086] Refined TSC clocksource calibration: 3392.132 MHz.
>>>>>>> Feb 20 05:52:57 cirros kern.info kernel: [    1.680348] eth0: IPv6 duplicate address fe80::f816:3eff:fea2:6e02 detected!
>>>>>>> ############ debug end   ##############
>>>>>>> cloud-setup: failed to read iid from metadata. tried 30
>>>>>>> WARN: /etc/rc3.d/S45-cloud-setup failed
>>>>>>> Starting dropbear sshd: generating rsa key... generating dsa key... OK
>>>>>>> ===== cloud-final: system completely up in 5743.03 seconds ====
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>>   instance-id:
>>>>>>>   public-ipv4:
>>>>>>>   local-ipv4 :
>>>>>>> wget: can't connect to remote host (169.254.169.254): Connection timed out
>>>>>>> cloud-userdata: failed to read instance id
>>>>>>> WARN: /etc/rc3.d/S99-cloud-userdata failed
>>>>>>>   ____               ____  ____
>>>>>>>  / __/ __ ____ ____ / __ \/ __/
>>>>>>> / /__ / // __// __// /_/ /\ \
>>>>>>> \___//_//_/  /_/   \____/___/
>>>>>>>  http://launchpad.net/cirros
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
>>>>>>>
>>>>>>> cirros login:
>>>>>>>
>>>>>>>
>>>>>>>  Thanks for the help.
>>>>>>>
>>>>>>>
>>>>>>> On 21 February 2013 08:33, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx
>>>>>>> > wrote:
>>>>>>>
>>>>>>>>  Le 20/02/2013 23:04, Chathura M. Sarathchandra Magurawalage a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>  Thanks.
>>>>>>>>
>>>>>>>>   I would be more concerned about the SIOCDELRT error above. Do
>>>>>>>>> you try to manually remove a network route at bootup ? Seems like the
>>>>>>>>> 'route del' is failing because the route is not already existing.
>>>>>>>>>
>>>>>>>>>  I am not doing doing anything that I am aware of.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> As already said, you absolutely need VNC support for
>>>>>>>>> investigating. Could you please fix your VNC setup which is incorrect ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> But VNC works fine. Its just that it VM hangs on the boot up it
>>>>>>>> wont come to the log in prompt, I can't log into it.  :(
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  Ahuh. Sorry for bugging you, I haven't understand : is your VM
>>>>>>>> failing to boot up ?
>>>>>>>> Which distro is your VM based on ?
>>>>>>>>
>>>>>>>> From my POV, your console.log is fine : your VM is booting, getting
>>>>>>>> DHCP lease, trying to contact metadata server, failing to contact it, and
>>>>>>>> that's it. You should get a prompt when logging in thru VNC.
>>>>>>>>
>>>>>>>> Please clarify : either the VM is failing to boot up properly (and
>>>>>>>> then try to find a small and cloud-out-of-the-box distro like CirrOS for
>>>>>>>> testing, or try with runlevel 3 or interactive startup), or your VM is
>>>>>>>> properly started without network connectivity (and then you have to login
>>>>>>>> and try more diagnostic tools like ping/tcpdump).
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Sylvain
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
Follow ups

Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-03-05
References

Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-14
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-18
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-18
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-18
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-20
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-20
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-20
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-20
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-21
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-21
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-21
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-21
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-21
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-21
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-22
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-22
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-22
Re: Cant ping private or floating IP
From: Sylvain Bauza, 2013-02-22
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-22
Re: Cant ping private or floating IP
From: Chathura M. Sarathchandra Magurawalage, 2013-02-28