dhis2-devs-core team mailing list archive

[Mark Polak <mark@xxxxxxxxxxxxxxx>]

Hey All,

After playing a little bit with it for the last couple hours i must say
that it was a pretty good find by Jason.

The YAML format makes it quite easy to create human readable tests by hand
if you want.
Some hand written tests by Jason
https://github.com/dhis2/dhis2-api-testing/blob/master/tests/crud_operations/dataElementOperations.yaml

But it also lets us generate a bunch of tests using a bit of scripting
Fairly basic  schema integrity tests that we generated.
https://github.com/dhis2/dhis2-api-testing/tree/master/tests/schema_integrity

Which are generated using a simple node script
https://github.com/dhis2/yaml-test-generator/blob/master/index.js

The nice thing about pyresttest seems to be that it is also easily
extensible with extractors (to extract values from the headers or the body)
and/op write your own comparators that compare the requested value vs the
expected value.

A place where we used this currently is to compare the property names on
the schema endpoints on dev vs the properties as extracted from the demo.
As the properties might be the same on both servers the order of the them
in the list might not be the same. The simple equal_items comparer compares
them by sorting the lists before comparing them.

As for where to run them i think we could have two main use cases for this
eventually. The main one currently would be to compare trunk to the latest
release. We could then identify easily where behaviour of the api changed.
Using a bit of extra work we should be able to make a list of which tests
fail so we can check if that is supposed to happen because of changes that
we want or because things broke (regression)

Additionally it might be nice to eventually compare certain versions of
trunk as that could notify us developers if we broke something. (Using
Travis CI that shouldn't have to be too hard to do)

@Morten I think testing using Spring mvc could still be valuable. We pretty
much started this as an initiative to have some kind of integration testing
for the api as a whole and to be able to compare two versions of the api
against each other.

As for other frameworks (that got from a quick google search :)):
https://github.com/jayway/rest-assured is a Java based REST tester which
seems pretty mature. But i'm not sure if that allows you to as easily
configure the tests. The examples seem to just have tests written in Java.
http://dareid.github.io/chakram/ seems to be a javascript version build on
top of Chai. But the tests are also written using code and it seems fairly
new.

Perhaps using either of those could be useful in addition if we run into
really complicated stuff that can't really be done with pyresttest.

Anyhow, figured i'd let you all know :)

Regards,

Mark




On Fri, Aug 28, 2015 at 3:00 PM, Jason Pickering <
jason.p.pickering@xxxxxxxxx> wrote:

> Hi Jim.
>
> No, not really. We just came across this yesterday afternoon, and it
> looked like a good mix of simplicity, but yet quite functional. So, we just
> hacked together some stuff to give it a spin.
>
> We would be happy to consider other tools, but wanted to try this one out
> and see what mileage we might get from it.
>
> So far, I think Mark and I both file the results interesting. The ability
> to integrate with CI is also interesting. Mark spun this up, which shows
> which tests failed and passed
> https://travis-ci.org/dhis2/dhis2-api-testing
>
> In addition to the test results between the /schema endpoints and the
> various individual endpoints, I found another strange thing by accident.
>
>  Trick question. Which of the following will "work" in the sense of
> returning an HTTP 200 ?
>
> curl  "
> https://apps.dhis2.org/demo/api/sharing?type=dataElement&id=fbfJHSPpUQD";
> -X GET -H "Content-Type: application/json" -u admin:district
>
> curl  "
> https://apps.dhis2.org/demo/api/sharing?type=dataElement&id=fbfJHSPpUQD";
> -X GET -H "Content-Type: application/xml" -u admin:district
>
> curl  "
> https://apps.dhis2.org/demo/api/sharing?type=dataElement&id=fbfJHSPpUQD";
> -X GET -H "Content-Type: application/foobar" -u admin:district
>
> Yeah, I know. Content-Type with a GET makes little sense, but give them a
> try. It kind of stumped us.
>
> Anyway, we would be happy to get more input on how to proceed, but feel
> like we need to get somethign up and running pretty soon. Have done enough
> talking about it at the experts meeting in Oslo!
>
> Regards,
> Jason
>
>
>
> On Fri, Aug 28, 2015 at 2:22 PM, Jim Grace <jimgrace@xxxxxxxxx> wrote:
>
>> Hi Jason and Mark,
>>
>> API testing would be quite valuable - great! Thanks! And I agree that we
>> should aim towards developing a test suite for all API functions.
>>
>> I like the simple test description syntax used by pyresttest. Have you
>> compare this tool with other REST testers?
>>
>> Cheers,
>> Jim
>>
>>
>> On Fri, Aug 28, 2015 at 2:58 AM, Jason Pickering <
>> jason.p.pickering@xxxxxxxxx> wrote:
>>
>>> Hi Morten,
>>> I think the idea would be to start with the schema and its various end
>>> points. Reason we were thinking this was
>>>
>>> 1) We could develop many test in a sort of automated way to see how this
>>> framework and approach works and if it is a good fit.
>>>
>>> 2) The schema endpoint in a way defines how we can "trust" the API, as
>>> the first point of contact potentially from external services. If the
>>> schema changes from one version to the next, that itself may be a test, but
>>> then this particular test failure was flagged because even within this
>>> version, the schema and its endpoint were not consistent. It would also
>>> help us potentially to develop generated payloads, if we can trust it. :)
>>>
>>> But, no. I think we would like to develop specific unit tests for
>>> certain operations...creating an orgunit, creating a user, deleting an
>>> orgunit group, etc and provide examples for app developers to create their
>>> own unit tests for their own apps/services to allow people to test against
>>> various versions of the application. I am working on one of these now, so
>>> once it is done, we can see what happens.
>>>
>>> Yes, I think the idea would to be integrate this into the CI server,
>>> which Mark has already done here (
>>> https://travis-ci.org/dhis2/dhis2-api-testing/builds/77504348).
>>>
>>> Regards,
>>> Jason
>>>
>>>
>>> On Fri, Aug 28, 2015 at 8:18 AM, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Jason
>>>>
>>>> That looks interesting, so this would target /api/schemas only? would
>>>> be interesting to see how other parts of the API could be tested also. I
>>>> have also been looking into doing testing using Spring MVC Testing
>>>> framework.
>>>>
>>>> Are you thinking this could be integrated into our CI server also?
>>>>
>>>> --
>>>> Morten
>>>>
>>>> On Fri, Aug 28, 2015 at 12:41 PM, Jason Pickering <
>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Morten,
>>>>> Mark and I have been doing some brainstorming and experimentation
>>>>> around the testing of the API. This came about after some discussions in
>>>>> the experts meeting around consistency of the API across different versions
>>>>> .  Mark and I were thinking what we would like to have would be a suite of
>>>>> external tests of the API, mainly for regression purposes between
>>>>> versions.  The idea being to develop tests from one version based on the
>>>>> /api/schemas (i.e. stable)  end point, and then to test these against
>>>>> another version (i.e. dev) . Mark developed some code  to develop tests
>>>>> against the schemas <https://apps.dhis2.org/demo/api/schemas.json>
>>>>> endpoint  and then we plugged this into something called pyresttest
>>>>> <https://github.com/svanoort/pyresttest>. The repo of the API test
>>>>> for DHIS2 is here <https://github.com/dhis2/dhis2-api-testing>.
>>>>>
>>>>> Interestingly, it has picked up some inconsitencies between what is in
>>>>> the schema and what is in the schema end points.
>>>>>
>>>>> Note carefully here from this link
>>>>> <https://apps.dhis2.org/dev/api/schemas/programRuleAction.json>
>>>>>  (/api/programRuleActions)
>>>>>
>>>>>
>>>>> Note that "trackedEntityAttribute" is present here.
>>>>>
>>>>> However from https://apps.dhis2.org/demo/api/schemas.json, it is
>>>>> missing.
>>>>>
>>>>>
>>>>> This is one type of thing we would like to pick up.
>>>>>
>>>>> We are still thinking through the best way to approach the testing and
>>>>> your thoughts would be most welcome.
>>>>>
>>>>> Regards,
>>>>> Jason
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Jason P. Pickering
>>> email: jason.p.pickering@xxxxxxxxx
>>> tel:+46764147049
>>>
>>> --
>>> Mailing list: https://launchpad.net/~dhis2-devs-core
>>> Post to     : dhis2-devs-core@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs-core
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+46764147049
>
> --
> Mailing list: https://launchpad.net/~dhis2-devs-core
> Post to     : dhis2-devs-core@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs-core
> More help   : https://help.launchpad.net/ListHelp
>
>