← Back to team overview

openstack team mailing list archive

Re: Scalability issue in nova-dhcpbridge

 

Hi Vish,

> I believe it is safe to ignore the old leases.  If nova-network has
> been down for a while it could potentially be nice to refresh all of
> the leases that it knows about, but I don't think it will harm
> anything if you remove it.
> 
> Are you running flatdhcp with a single network host on a large
> install?  I would think that multi_host would be a better choice in
> that case.
> 
> There is also a potentially nasty performance issue in linux_net
> where it creates all of the leases.  It is a very expensive operation
> and needs to be reoptimized after the foreign keys were removed from
> the network tables. Currently it is doing 2 database for every active
> instance in the db.

Thanks for making these changes so quickly. Initial testing shows a
huge improvement. If I push things harder we do slow down somewhat
because each call of nova-dhcpbridge takes time just in the import code:


# cat slow
#!/usr/bin/python

from nova.network import linux_net


# time ./slow 
0.295s


So 500 "old" leases will take 150 seconds to process even though we
are just dropping them. dnsmasq has a dbus interface which I was going
to look at, but for now I just wrapped nova-dhcpbridge with a stupid
shell script:


#!/bin/sh

if [ "$1" == "old" ]; then
	exit
fi
exec /opt/stack/nova/bin/nova-dhcpbridge.orig $@


Also "nova list" with 1000 instances is a bit slow (13 seconds on my
setup). It drops to 6.5 seconds if we avoid doing all the queries to
get the hypervisor hostname:


     def _get_hypervisor_hostname(self, context, instance):
+        return "junk"
         compute_node = db.compute_node_get_by_host(context,


I wonder if there is a way we can do this via a join.

Anton


References