arsenal-devel team mailing list archive

[Bryce Harrington <604083@xxxxxxxxxxxxxxxxxx>]

** This bug is no longer flagged as a security vulnerability

-- 
Arsenal Improvements
https://bugs.launchpad.net/bugs/604083
You received this bug notification because you are a member of arsenal-
devel, which is a direct subscriber.

Status in Arsenal: New

Bug description:
(Marked as a bug because Blueprints can't be private.)

0-bryce: I tested send-attachments-upstream.cgi on a few more bugs yesterday.

    There was a small issue which I've patched around in arsenal trunk,
    but you should take a look and think about if there's a better way
    we could solve it.
    [kamran]

0-kamran: I don't know why Edge API was failing on your setup. It's working
    fine with the latest launchpadlib on my machine:

    http://pastebin.com/BTZnnWQB
    http://pastebin.com/1H3YEm0A

    Perhaps you're using an old launchpadlib? I'm not sure if bypassing
    the whole API detection and hardcoding it to be edge is a good idea.
    It would break the script on staging on local Launchpads, making it
    harder to debug.

1. The features and refactoring from the previous email [kamran]

        i. Break out copy_lp_attachments_to_bz() into multiple functions,
           one for the bugzilla login stuff, another for downloading attachments
           from launchpad, and another for uploading to bugzilla.

           The reason to split it up is to add in some flexibility.  E.g., grab
           all the attachments, do some filtering/processing, and then upload a
           subset or superset to bugzilla. [done]

       ii. Add an option to control whether to upload all attachments, or only
           attachments posted by the original bug reporter. [done]
           http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-improvements/annotate/head:/scripts/send-attachments-upstream.py

      iii. Set up a configuration file to store bugzilla username/password and
           other info.  Maybe to a file in ~/.config/arsenal/?

       iv. Add support for "filename exclusions", that allows omitting filenames
           matching a given regex.  This is to prevent uploading of stuff we
           collect(*) but upstream won't care about, like Dependencies.txt, etc.

        v. Add a provision in the logic for de-archiving .zip and .tar files,
           except in the case where the archive contains 6 or more files.

       vi. Add option for setting a threshold maximum filesize; when an
           attachment looks like it will exceed this, skip it, and print
           out the filename at the end of the run.  (Maybe try gzipping it?)

      vii. For filenames ending in .txt or .log, force the content type to text.
           Launchpad gets this wrong sometimes (esp. with Xorg.0.log), and it
           irritates upstream.

2-bryce: send_upstream.cgi needs modified such that instead of setting the
    form action to bugzilla's enter_bug.cgi, it instead points back to
    send_upstream.cgi, and we use bzutils to file the new bug report,
    instead of passing the parameters over to enter_bug.cgi via GET.

    (send_upstream.cgi's form action is specified in
    templates/upstream_form.tt2)

    Sounds you've already started thinking about how to do this, and
    perhaps you'd prefer to extend send-attachments-upstream.py rather
    than build off of send_upstream.cgi.  Either way should be fine, but
    however it's implemented, this should eventually replace the
    functionality currently in send_upstream.cgi.
    [kamran]

2-kamran: Here's what I summarized about Bugzilla communication while writing
    send_attachments_upstream.py:

    There are 4 ways of talking with Bugzillas:
    i) Use standard xmlrpclib:
    Advantages: No dependencies
    Disadvantages: Does not support creating attachments
    ii) A library such as python-bugzilla or bzutils:
    Advantages: Helpful wrappers around xmlrpclib which judge the type
    of Bugzilla being used. For example, python-bugzilla judges whether
    it's a RedHat bugzilla and consequently allows attachment creation
    since RH's Bugzilla supports doing that via xmlrpc.
    Disadvantages: Extra dependency, still does not support creating
    attachments for all Bugzillas since it is effectively dependent
    on XML-RPC.
    iii) Fetching web-pages using httplib2:
    Advantages: No dependencies
    Disadvantages: Sending attachments is a complex job since file data
    has to be form-encoded first, depends on HTML field names on
    web-pages
    iv) Fetching web-pages using pycurl
    Advantages: Takes care of form-encoding attachment data as well
    cookies in a simple, painless manner
    Disadvantages: Dependence on HTML field names

    Currently I'm using method (iv) and check for upload by searching
    for attachment title in the subsequent page. Basically, what it
    does is:

    "Upload abc.txt and then check whether the string abc.txt is
    present in a link which says "Created" in the response body."

    To put it in a nutshell, Bugzilla's XML-RPC API is not mature
    enough yet (unlike Launchpad's REST). It's unsuitable for
    attachment creation. It *can* be used for Bug creation. However
    then we'll have a mixture of XML-RPC and Curl passages to a
    Bugzilla.

3-bryce:  We need a proof-of-concept cgi program that any arbitrary user can
    use, which lets them authenticate against launchpad so that the cgi
    can use their logged in credentials for doing lpl calls.

    Alternatively, send-attachments-upstream.py needs to be modified to
    create LaunchpadService with the 'read-only' config parameter, which
    won't need tokens.  I haven't tested if your script will work
    without login tokens, but if it will, then this should work out
    fine.
3-kamran: I had that in mind. The script should additionally ask to login only if
readonly parameter doesn't work for a private bug. Added to checklist.

4-bryce: After the bug is filed with bugzilla, we need to be able to retrieve
    the bug id of the new bug, and then add a bug watch to the original
    launchpad bug report, along with a message to the reporter that
    their issue has been sent upstream.

    This step is going to require the user be logged in, so if we decide
    to go with the alternate approach for step #3, that'll make this a
    bit more challenging.
    [bryce]

4-kamran: I'll add a "--add-bug-watch" flag in which case user shall be asked to
login.

5.  The html template in send_upstream.cgi needs to be redone using
    genshi. [bryce]

    (I've done a couple example scripts in the web/ directory; looks
    pretty straightforward, just need to spend the few hours to get it
    converted over.)