openstack team mailing list archive

[Brian Haley <brian.haley@xxxxxx>]

Just some more notes.

It looks like you're running this system as both a network node and compute
node, I think the pdf you found from Redhat assumed the system was a dedicated
network node, i.e. it only had qr- and qg- interfaces, and not ns- as created by
plug() when an instance is booted.

Multiple routes for the same destination, going out two different interfaces not
connected to the same network, are going to cause you trouble.  It's
non-deterministic where a packet will go without ip rules.

I'm going to let you go and debug this some more on your own, as it looks like
it's your iptables config causing it, you just need to get the correct rules in
there.

-Brian

On 07/24/2013 11:34 AM, David Kang wrote:
> 
>  Thanks, Brian.
> My answers are put in your email with "-->".
> 
>  David
> 
> ----- Original Message -----
>> On 07/24/2013 10:42 AM, David Kang wrote:
>>>
>>>   If I remove the following REJECT rules, it works perfectly.
>>> -A INPUT -j REJECT --reject-with icmp-host-prohibited
>>> -A FORWARD -j REJECT --reject-with icmp-host-prohibited
>>>
>>>  With them, it looks like that the packets are dropped at the bridge
>>>  before they can be forwarded.
>>
>> So I'll keep asking - why can't you just remove them? It gets you
>> running and
>> if you're just kicking the tires it's a valid workaround.
>>
> 
> --> My sponsor STRONGLY wants to have the rules for security purpose.
> 
>>> I ran the iptables commands recommended by RedHat.
>>>
>>> When I ping 10.12.182.13 from a VM (192.168.3.3),
>>> I cannot see any packets from qr-32411859-c0,
>>> but I can see packets are dropped at brqf56b3f53-d3.
>>> The outputs of tcpdump is shown below.
>>>
>>> $ brctl show
>>> bridge name bridge id STP enabled interfaces
>>> brq69f480ab-06 8000.001e675ba339 no eth2.82
>>>                                                         tapd8bd73c9-3a
>>> brqf56b3f53-d3 8000.001e675ba338 no eth1.2001
>>>                                                         tap32411859-c0
>>>                                                         tapfa6a1d01-16
>>> $ route -n
>>> Kernel IP routing table
>>> Destination Gateway Genmask Flags Metric Ref Use Iface
>>> 192.168.3.0 0.0.0.0 255.255.255.0 U 0 0 0 ns-fa6a1d01-16
>>> 192.168.3.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-32411859-c0
>>
>> Overlapping IP ranges? That could be a problem.
> 
> --> Those are generated by quantum-linuxbridge-agent.
>   If a quantum network is associated to a quantum l3 router, qr-xxx interface is added.
> 
>>
>>> 10.12.182.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.182
>>> 10.12.82.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-d8bd73c9-3a
>>> 0.0.0.0 10.12.82.1 0.0.0.0 UG 0 0 0 qg-d8bd73c9-3a
>>
>> Why is your default route going out this interface and not eth2.182?
> 
> --> I didn't show it. Another default router of eth2.182 also exist below 10.12.82.1.
>   Quantum automatically made qg-d8bd73c9-3a as a default router.
>   It is the interface to the gateway of the "external network" where
>   floating IP is assigned.
> 
>>
>>> $ tcpdump -i qr-32411859-c0 -nn
>>>    // nothing special
>>
>> What about ns-fa6a1d01-16? That overlapping IP range looks suspicious.
> 
> --> It was made by quantum-linux-bridge.
> 
>>
>>> $ tcpdump -i brqf56b3f53-d3 -nn icmp
>>> tcpdump: WARNING: brqf56b3f53-d3: no IPv4 address assigned
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on brqf56b3f53-d3, link-type EN10MB (Ethernet), capture
>>> size 65535 bytes
>>> 13:48:46.892785 IP 192.168.3.3 > 10.12.182.13: ICMP echo request, id
>>> 46605, seq 1855, length 64
>>> 13:48:46.892825 IP 192.168.3.2 > 192.168.3.3: ICMP host 10.12.182.13
>>> unreachable - admin prohibited, length 92
>>
>> This is the reject iptables rule firing, so those other rules are not
>> matching.
>> You need to look at 'iptables -L -v -n -x' to see if their packet/byte
>> counts
>> are increasing or not. If not, start using things like 'ip route get
>> $dest' to
>> figure out what interfaces the kernel is using for output, which will
>> help you
>> fix those rules to be correct.
> 
> --> Thanks, I will try it.
> 
>>
>> -Brian
>>
>>> ----- Original Message -----
>>>> On 07/23/2013 11:41 PM, David Kang wrote:
>>>>>
>>>>>  A Redhat manual suggests the following rule to enable forwarding
>>>>>  packets
>>>>> among VMs and external network.
>>>>> https://access.redhat.com/site/documentation/en-US/Red_Hat_OpenStack/2/pdf/Release_Notes/Red_Hat_OpenStack-2-Release_Notes-en-US.pdf
>>>>>
>>>>> iptables -t filter -I FORWARD -i qr-+ -o qg-+ -j ACCEPT
>>>>> iptables -t filter -I FORWARD -i qg-+ -o qr-+ -j ACCEPT
>>>>> iptables -t filter -I FORWARD -i qr-+ -o qr-+ -j ACCEPT
>>>>>
>>>>>  But it doesn't work for me.
>>>>
>>>> Can you elaborate on what "it doesn't work" means?
>>>>
>>>> Do any of those rules show increased packet/byte counts, indicating
>>>> they've been
>>>> matched?
>>>>
>>>> Is IP forwarding enabled?
>>>>
>>>> Is there a mis-configuration in your bridge config? Use 'brctl
>>>> show'
>>>> to see
>>>> where all the tap and other devices are attached.
>>>>
>>>> Deleting that one FORWARD rule causing all the trouble is going to
>>>> be
>>>> a much
>>>> quicker solution.
>>>>
>>>> -Brian
>>>>
>>>>> ----- Original Message -----
>>>>>> On 07/23/2013 12:22 PM, David Kang wrote:
>>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>>   We are running OpenStack Folsom on CentOS 6.4.
>>>>>>> Quantum-linuxbridge-agent is used.
>>>>>>> By default, the Quantum node has the following entries in its
>>>>>>> /etc/sysconfig/iptables file.
>>>>>>>
>>>>>>> -A INPUT -j REJECT --reject-with icmp-host-prohibited
>>>>>>> -A FORWARD -j REJECT --reject-with icmp-host-prohibited
>>>>>>>
>>>>>>>  With those two lines, VM cannot get IP address from the DHCP
>>>>>>>  server
>>>>>>>  running on the Quantum node.
>>>>>>> More specifically, the first line prevents a VM from getting IP
>>>>>>> address from DHCP server.
>>>>>>> The second line prevents a VM from talking to other VMs and
>>>>>>> external
>>>>>>> worlds.
>>>>>>> Is there a better way to make the Quantum network work well
>>>>>>> than just commenting them out?
>>>>>>
>>>>>> Since Quantum isn't adding them, and you want the system to act
>>>>>> as
>>>>>> a
>>>>>> DHCP server
>>>>>> and gateway, I think you have two choices:
>>>>>>
>>>>>> 1. Delete them
>>>>>> 2. Craft rules to sit above them (using -I) to allow certain
>>>>>> packets
>>>>>>
>>>>>> #2 gets tricky as you'll need to account for DHCP, metadata, etc.
>>>>>> in
>>>>>> the INPUT
>>>>>> chain, and in the FORWARD chain you could maybe start by allowing
>>>>>> all
>>>>>> traffic
>>>>>> from your bridge. You would need to do some more work there.
>>>>>>
>>>>>> I believe any DHCP iptables rules will be on the compute hosts,
>>>>>> and
>>>>>> will be put
>>>>>> in place for anti-spoofing. Since this is the network node you
>>>>>> won't
>>>>>> see them here.
>>>>>>
>>>>>> -Brian
>>>>>
>>>
>