← Back to team overview

openstack team mailing list archive

Re: deal with booting lots of instance simultaneously

 

Are you using multi_host setup? If not, as Vish suggested, that will
alleviate much of the problem.

Best,
-jay

On 02/19/2013 04:09 AM, gtt116 wrote:
> Hi Diego
> 
> Thanks for you reply.
> How many hosts do you have? I have 4 hosts. And in this bug,
> https://bugs.launchpad.net/nova/+bug/1094226, The N is 20. In my
> environment N is about 16.
> 
> I found that nova-network is too busy to deal with so many rpc request
> at the same time. The Rabbitmq is strong enough in the scenario.
> 
> 于 2013年02月19日 16:54, Diego Parrilla Santamaría 写道:
>> Hi gtt,
>>
>> what does it mean for you 'lots of instance simultaneously'? 100,
>> 1000, 10000, more?
>>
>> We have launched >100 (but less than <1000) simultaneously without any
>> issue. Rabbit running in a multicore with several gigs of RAM with out
>> of the box configuration.
>>
>> Cheers
>> Diego
>>  -- 
>> Diego Parrilla
>> <http://www.stackops.com/>*CEO*
>> **www.stackops.com*
>> <http://www.stackops.com/> | * diego.parrilla@xxxxxxxxxxxx
>> <mailto:diego.parrilla@xxxxxxxxxxxx>**** | +34 649 94 43 29
>> | skype:diegoparrilla*
>> * <http://www.stackops.com/>
>> *
>>
>> ****
>>
>> *
>>
>>
>>
>> On Tue, Feb 19, 2013 at 9:35 AM, gtt116 <gtt116@xxxxxxx
>> <mailto:gtt116@xxxxxxx>> wrote:
>>
>>     Hi all,
>>
>>     When create lots of instance simultaneously, there will be lots of
>>     instance in ERROR state. And most of them are caused by network
>>     rpc request timeout. This result is not so graceful.
>>
>>     I think it will be better if scheduler keep a queue of creating
>>     request. when he find all the hosts are busy
>>     enough(compute_node.current_workload reach some value), stop cast
>>     the request to host temporarily, until he found some host free
>>     enough. In this way, we can make sure booting lots of instances
>>     simultaneously results in active instances rather than lots of
>>     ERROR instance. but will cause a small weak point, if the top
>>     value of current_workload small enough, create instance processing
>>     will be slow.
>>
>>     Do you have another quick fix?
>>
>>     Thanks,
>>
>>     -- 
>>     best regards,
>>     gtt
>>
>>
>>     _______________________________________________
>>     Mailing list: https://launchpad.net/~openstack
>>     <https://launchpad.net/%7Eopenstack>
>>     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>     Unsubscribe : https://launchpad.net/~openstack
>>     <https://launchpad.net/%7Eopenstack>
>>     More help   : https://help.launchpad.net/ListHelp
>>
>>
> 
> 
> -- 
> best regards,
> gtt
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 


References