openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On 10/22/2012 12:41 PM, Yaniv Kaul wrote:
>> 1) Tempest executes a series of HTTP calls against public REST endpoints
>> in OpenStack. It has no way of determining what code was run **on the
>> server**. It only has the ability to know what Tempest itself executed,
>> not what percentage of the total API those calls represented
> 
> But you can get number of lines covered (vs. total number of lines in
> the component).

No, you can't... you'd need to run this on the server side, and Tempest
is a client-side thing -- you just point it at an OpenStack environment.
You can't assume that you would be able to profile/SSH as root into the
endpoint running the server to do the above.

>> 2) Specs don't always exist for the APIs -- yes, I know, this isn't
>> good. Especially problematic are some of the Compute API extensions that
>> aren't documented well, or at all.
> 
> which is why you need to look at the code, see what is missed (and
> 'reverse-engineer' it into a test case).

That's what we've been doing up until now.

> But I guess it doesn't have to be either-or and could be both eventually
> - I don't have a lot of faith in a wiki that needs updating, that's all.

Well, like I mentioned in the original post, doing it in the wiki or
automating/scripting some output is fine with me. I agree that a wiki is
not ideal, but unless there is some code-readable description of the
entire API -- via XSchema/JSONSchema, etc -- this isn't very easy.

Best,
-jay

> Y.
> 
>>
>> Best,
>> -jay
>>
>>> Thanks,
>>> Y.
>>>
>>>> * I will start the traceability matrix stuff and publish for people to
>>>> go and update
>>>> * Antoni from HP is going to investigate using things in testtools and
>>>> testrepository for handling module and package-level fixtures and
>>>> removing some of the nose-based cruft
>>>> * I will be working on the YAML file that describes a test environment
>>>> so that different teams that use different deployment frameworks can use
>>>> a commonly-agreed format to describe their envs to the CI system
>>>> * A new member of Gigi's team (really sorry, I've forgotten your name!
>>>> :( ) is going to look at the fuzz testing discussion from earlier and
>>>> see about prototyping something together that would be used for negative
>>>> and security/fuzz testing -- this would enable us to remove the static
>>>> negative tests from Tempest's main test directories. For the record (and
>>>> for the team member whose name I have forgotten, here is the relevant
>>>> link: https://lists.launchpad.net/openstack-qa-team/msg00155.html and
>>>> https://lists.launchpad.net/openstack-qa-team/msg00156.html)
>>>>
>>>> Best,
>>>> -jay
>>>>
>>>> On 10/19/2012 02:47 PM, Sean Gallagher wrote:
>>>>> Daryl,
>>>>> I wasn't able to make the follow-up meeting. :/
>>>>>
>>>>> Can you/ someone send out a recap?
>>>>>
>>>>> David: you had a list of items. Can you post or share that somewhere?
>>>>>
>>>>> We discussed Blueprints for managing some of the planning work.
>>>>>
>>>>> What about higher level planning docs? Useful? Do we have any? Should we?
>>>>>
>>>>> Re Google Hangout next week, I'm interested.
>>>>>
>>>>> -sean
>>>>>
>