openstack team mailing list archive

[Rick Jones <rick.jones2@xxxxxx>]

On 04/30/2013 11:42 AM, Steve Heistand wrote:
> if I launch one vm at a time its doesnt very long to start up the instance. maybe a minute.
> if I launch 4 instances (of the same snapshot as before) it takes 30 minutes.
>
> they are all launching to different compute nodes, the controllers are all multicore,
> I dont see any processes on the compute nodes taking much cpu power, the controller has
> a keystone process mostly sucking up 1 core, loads and loads of beam.smp from rabbitmq but
> none are really taking any cpu time.
>
> the glance image storage is on the controller node, the snapshots are 2-3G in size.
>
> where should I start looking to find out why things are so slow?

I am something of a "networking guy" so that will color my response :)

Is each  instance using the same image or a different one?  If a 
different one, the "workload" to the glance storage will become (I 
suspect) a random read workload accessing the four different images even 
though each individual stream may be sequential.  How much "storage 
oomph" do you have on/serving the glance/controller node?

After that, what do the network statistics look like?  Starting I 
suppose with the glance/controller node.  Take some snapshots over an 
interval in each case and run them through something like beforeafter:

netstat -s > before
...wait a defined/consistent moment...
netstat -s > after
beforeafter before after > delta

and go from there.

rick jones

/*
 * beforeafter
 *
 *   SYNOPSIS
 *
 *       beforeafter before_file after_file
 *
 *   Description
 *
 *       Subtract the numbers in before_file from the numbers in
 *       after_file.
 * 
 *   Example
 *
 *       # netstat -s > netstat.before
 *       # run some test here
 *	 # netstat -s > netstat.after
 *	 # beforeafter netstat.before netstat.after
 *
 *   Note
 *
 *       The "long double" is usually implemented as the IEEE double
 *       extended precision: its mantissa is _at_least_ 64 bits.
 *       Therefore, "long double" should be able to handle 64-bit
 *       integer numbers.
 */

#include <stdio.h>
#include <ctype.h>
#include <unistd.h>
#include <stdlib.h>

char	*USAGE = "before_file after_file";

main(argc, argv)
    int		argc;
    char	*argv[];
    
{
    FILE	*fp1;		/* "before" file */
    FILE	*fp2;		/* "after" file */
    char	*fname1;
    char	*fname2;
    int		i;
    int		c;
    int		c2;
    int		separator  = 1;
    int		separator2 = 1;
    unsigned int	n1 = 0;
    unsigned int	n2 = 0;
    long double		d1 = 0.0;
    long double		d2 = 0.0;
    long double		delta = 0.0;
    long double		p31;
    long double		p32;
    long double		p63;
    long double		p64;
    
    /*
     * Checke # of arguments.
     */
    if (argc != 3) {
	long double	x;

	fprintf(stderr, "Usage: %s %s\n", argv[0], USAGE);

	printf ("\n");
	printf ("     Testing how many decimal digits can be handled ...\n");
	printf ("     #digits #bits  1=pass, 0=fail\n");
	x = 999999999.0L -
	    999999998.0L;
	printf ("%10s %6s   %1.0Lf\n",  "9", "~30", x);
	x = 999999999999999999.0L -
	    999999999999999998.0L;
	printf ("%10s %6s   %1.0Lf\n", "18", "~60", x);
	x = 999999999999999999999999999.0L -
	    999999999999999999999999998.0L;
	printf ("%10s %6s   %1.0Lf\n", "27", "~90", x);
	x = 999999999999999999999999999999.0L -
	    999999999999999999999999999998.0L;
	printf ("%10s %6s   %1.0Lf\n", "30", "~100", x);
	x = 999999999999999999999999999999999.0L -
	    999999999999999999999999999999998.0L;
	printf ("%10s %6s   %1.0Lf\n", "33", "~110", x);
	x = 999999999999999999999999999999999999.0L -
	    999999999999999999999999999999999998.0L;
	printf ("%10s %6s   %1.0Lf\n", "36", "~120", x);

	exit (1);
    }
    /*
     * Open files.
     */
    fname1 = argv[1];
    fname2 = argv[2];
    fp1 = fopen(fname1, "r");
    fp2 = fopen(fname2, "r");
    if (!fp1 || !fp2) {
	fprintf(stderr, "fp1 = %p  fp2 = %p", fp1, fp2);
	perror ("Could not open files");
	exit (2);
    }
    /*
     * Prepare for 32-bit and 64-bit overflow check.
     */
    for (p31=1.0, i=0; i<31; i++) {
	p31 *= 2.0;			/* 2^31 */
    }
    p32 = p31 * 2.0;			/* 2^32 */
    p63 = p31 * p32;			/* 2^63 */
    p64 = p63 * 2.0;			/* 2^64 */

    /*
     * Parse.
     */
    while ((c = getc(fp1)) != EOF) {
	if (c==' ' || c=='\t' || c==':' || c=='(' || c=='\n') {
	    printf("%c", c);
	    separator = 1;
	} else if (!isdigit(c))	{
	    printf("%c", c);
	    separator = 0;
	} else {
	    if (separator == 0) {
		printf("%c", c);	/* this digit is a part of a word */
		continue;
	    }
	    n1 = c - '0';
	    d1 = n1;
	    while ((c = getc(fp1)) != EOF) {
		if (isdigit(c))	{
		    n1 = c - '0';
		    d1 = 10.0 * d1 + n1;
		} else {
		    break;
		}
	    }
	    /*
	     * Find the counterpart in the "after" file.
	     */
	    while ((c2 = getc(fp2)) != EOF) {
		if (c2==' ' || c2=='\t' || c2==':' || c2=='(' || c2=='\n') {
		    separator2 = 1;
		} else if (!isdigit(c2)) {
		    separator2 = 0;
		} else {
		    if (separator2 == 0) {
			continue;
		    }
		    n2 = c2 - '0';
		    d2 = n2;
		    while ((c2 = getc(fp2)) != EOF) {
			if (isdigit(c2)) {
			    n2 = c2 - '0';
			    d2 = 10.0 * d2 + n2;
			} else {
			    break;
			}
		    }
		    break;
		}
	    }
	    delta = d2 - d1;
	    /*
	     * Check a counter overflow.  These checks are not perfect,
	     * but it should work reasonably well when counters are
	     * monotonically increasing.
	     *
	     * Although this program also handles the case where a
	     * counter is decreasing, we don't check underflow of a
	     * decreasing counter because we can not distinguish bitween
	     * a 32-bit counter underflow and a normal 64-bit counter
	     * increase.
	     */
	    if (d2 < p32 && d1 < p32) {
		if (delta < -p31) {
		    /* most likely a 32-bit counter wrapped around */
		    delta += p32;
		}
	    } else if (d2 < p64 && d1 < p64) {
		if (delta < -p63) {
		    /* most likely a 64-bit counter wrapped around */
		    delta += p64;
		}
	    }
	    printf ("%1.0Lf", delta);
	    if (c != EOF) {
		printf("%c", c);
	    }
	}
    }
}