arsenal-devel team mailing list archive

[Kamran Riaz Khan <krkhan@xxxxxxxxxxxxxx>]

** Description changed:

  (Marked as a bug because Blueprints can't be private.)
  
  0-bryce: I tested send-attachments-upstream.cgi on a few more bugs
  yesterday.
  
      There was a small issue which I've patched around in arsenal trunk,
      but you should take a look and think about if there's a better way
      we could solve it.
      [kamran]
  
  0-kamran: I don't know why Edge API was failing on your setup. It's working
      fine with the latest launchpadlib on my machine:
  
      http://pastebin.com/BTZnnWQB
      http://pastebin.com/1H3YEm0A
  
      Perhaps you're using an old launchpadlib? I'm not sure if bypassing
      the whole API detection and hardcoding it to be edge is a good idea.
      It would break the script on staging on local Launchpads, making it
      harder to debug.
  
  1. The features and refactoring from the previous email [kamran]
  
          i. Break out copy_lp_attachments_to_bz() into multiple functions,
             one for the bugzilla login stuff, another for downloading attachments
             from launchpad, and another for uploading to bugzilla.
  
             The reason to split it up is to add in some flexibility.  E.g., grab
             all the attachments, do some filtering/processing, and then upload a
             subset or superset to bugzilla. [done]
  
         ii. Add an option to control whether to upload all attachments, or only
             attachments posted by the original bug reporter. [done]
             http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-improvements/annotate/head:/scripts/send-attachments-upstream.py
  
        iii. Set up a configuration file to store bugzilla username/password and
             other info.  Maybe to a file in ~/.config/arsenal/? [done]
-            http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-cache-credentials/annotate/head:/scripts/send-attachments-upstream.py
-            http://pastebin.com/xSijK72m
+            http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-cache-credentials/annotate/head:/scripts/send-attachments-upstream.py
+            http://pastebin.com/xSijK72m
  
         iv. Add support for "filename exclusions", that allows omitting filenames
             matching a given regex.  This is to prevent uploading of stuff we
-            collect(*) but upstream won't care about, like Dependencies.txt, etc.
+            collect(*) but upstream won't care about, like Dependencies.txt, etc. [done]
+            http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-exclude-filenames/annotate/head:/scripts/send-attachments-upstream.py
+            http://pastebin.com/V69JgNF6
+            (Unix wildcards are used for filename exclusion instead of regular expressions.
+             A user can input regular expressions which take forever to execute.)
  
          v. Add a provision in the logic for de-archiving .zip and .tar files,
             except in the case where the archive contains 6 or more files.
  
         vi. Add option for setting a threshold maximum filesize; when an
             attachment looks like it will exceed this, skip it, and print
             out the filename at the end of the run.  (Maybe try gzipping it?)
  
        vii. For filenames ending in .txt or .log, force the content type to text.
             Launchpad gets this wrong sometimes (esp. with Xorg.0.log), and it
             irritates upstream.
  
  2-bryce: send_upstream.cgi needs modified such that instead of setting the
      form action to bugzilla's enter_bug.cgi, it instead points back to
      send_upstream.cgi, and we use bzutils to file the new bug report,
      instead of passing the parameters over to enter_bug.cgi via GET.
  
      (send_upstream.cgi's form action is specified in
      templates/upstream_form.tt2)
  
      Sounds you've already started thinking about how to do this, and
      perhaps you'd prefer to extend send-attachments-upstream.py rather
      than build off of send_upstream.cgi.  Either way should be fine, but
      however it's implemented, this should eventually replace the
      functionality currently in send_upstream.cgi.
      [kamran]
  
  2-kamran: Here's what I summarized about Bugzilla communication while writing
      send_attachments_upstream.py:
  
      There are 4 ways of talking with Bugzillas:
      i) Use standard xmlrpclib:
      Advantages: No dependencies
      Disadvantages: Does not support creating attachments
      ii) A library such as python-bugzilla or bzutils:
      Advantages: Helpful wrappers around xmlrpclib which judge the type
      of Bugzilla being used. For example, python-bugzilla judges whether
      it's a RedHat bugzilla and consequently allows attachment creation
      since RH's Bugzilla supports doing that via xmlrpc.
      Disadvantages: Extra dependency, still does not support creating
      attachments for all Bugzillas since it is effectively dependent
      on XML-RPC.
      iii) Fetching web-pages using httplib2:
      Advantages: No dependencies
      Disadvantages: Sending attachments is a complex job since file data
      has to be form-encoded first, depends on HTML field names on
      web-pages
      iv) Fetching web-pages using pycurl
      Advantages: Takes care of form-encoding attachment data as well
      cookies in a simple, painless manner
      Disadvantages: Dependence on HTML field names
  
      Currently I'm using method (iv) and check for upload by searching
      for attachment title in the subsequent page. Basically, what it
      does is:
  
      "Upload abc.txt and then check whether the string abc.txt is
      present in a link which says "Created" in the response body."
  
      To put it in a nutshell, Bugzilla's XML-RPC API is not mature
      enough yet (unlike Launchpad's REST). It's unsuitable for
      attachment creation. It *can* be used for Bug creation. However
      then we'll have a mixture of XML-RPC and Curl passages to a
      Bugzilla.
  
  3-bryce:  We need a proof-of-concept cgi program that any arbitrary user can
      use, which lets them authenticate against launchpad so that the cgi
      can use their logged in credentials for doing lpl calls.
  
      Alternatively, send-attachments-upstream.py needs to be modified to
      create LaunchpadService with the 'read-only' config parameter, which
      won't need tokens.  I haven't tested if your script will work
      without login tokens, but if it will, then this should work out
      fine.
  3-kamran: I had that in mind. The script should additionally ask to login only if
  readonly parameter doesn't work for a private bug. Added to checklist.
  
  4-bryce: After the bug is filed with bugzilla, we need to be able to retrieve
      the bug id of the new bug, and then add a bug watch to the original
      launchpad bug report, along with a message to the reporter that
      their issue has been sent upstream.
  
      This step is going to require the user be logged in, so if we decide
      to go with the alternate approach for step #3, that'll make this a
      bit more challenging.
      [bryce]
  
  4-kamran: I'll add a "--add-bug-watch" flag in which case user shall be asked to
  login.
  
  5.  The html template in send_upstream.cgi needs to be redone using
      genshi. [bryce]
  
      (I've done a couple example scripts in the web/ directory; looks
      pretty straightforward, just need to spend the few hours to get it
      converted over.)

-- 
Arsenal Improvements
https://bugs.launchpad.net/bugs/604083
You received this bug notification because you are a member of arsenal-
devel, which is a direct subscriber.

Status in Arsenal: New

Bug description:
(Marked as a bug because Blueprints can't be private.)

0-bryce: I tested send-attachments-upstream.cgi on a few more bugs yesterday.

    There was a small issue which I've patched around in arsenal trunk,
    but you should take a look and think about if there's a better way
    we could solve it.
    [kamran]

0-kamran: I don't know why Edge API was failing on your setup. It's working
    fine with the latest launchpadlib on my machine:

    http://pastebin.com/BTZnnWQB
    http://pastebin.com/1H3YEm0A

    Perhaps you're using an old launchpadlib? I'm not sure if bypassing
    the whole API detection and hardcoding it to be edge is a good idea.
    It would break the script on staging on local Launchpads, making it
    harder to debug.

1. The features and refactoring from the previous email [kamran]

        i. Break out copy_lp_attachments_to_bz() into multiple functions,
           one for the bugzilla login stuff, another for downloading attachments
           from launchpad, and another for uploading to bugzilla.

           The reason to split it up is to add in some flexibility.  E.g., grab
           all the attachments, do some filtering/processing, and then upload a
           subset or superset to bugzilla. [done]

       ii. Add an option to control whether to upload all attachments, or only
           attachments posted by the original bug reporter. [done]
           http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-improvements/annotate/head:/scripts/send-attachments-upstream.py

      iii. Set up a configuration file to store bugzilla username/password and
           other info.  Maybe to a file in ~/.config/arsenal/? [done]
           http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-cache-credentials/annotate/head:/scripts/send-attachments-upstream.py
           http://pastebin.com/xSijK72m

       iv. Add support for "filename exclusions", that allows omitting filenames
           matching a given regex.  This is to prevent uploading of stuff we
           collect(*) but upstream won't care about, like Dependencies.txt, etc. [done]
           http://bazaar.launchpad.net/~inspirated/arsenal/send-attachments-exclude-filenames/annotate/head:/scripts/send-attachments-upstream.py
           http://pastebin.com/V69JgNF6
           (Unix wildcards are used for filename exclusion instead of regular expressions.
            A user can input regular expressions which take forever to execute.)

        v. Add a provision in the logic for de-archiving .zip and .tar files,
           except in the case where the archive contains 6 or more files.

       vi. Add option for setting a threshold maximum filesize; when an
           attachment looks like it will exceed this, skip it, and print
           out the filename at the end of the run.  (Maybe try gzipping it?)

      vii. For filenames ending in .txt or .log, force the content type to text.
           Launchpad gets this wrong sometimes (esp. with Xorg.0.log), and it
           irritates upstream.

2-bryce: send_upstream.cgi needs modified such that instead of setting the
    form action to bugzilla's enter_bug.cgi, it instead points back to
    send_upstream.cgi, and we use bzutils to file the new bug report,
    instead of passing the parameters over to enter_bug.cgi via GET.

    (send_upstream.cgi's form action is specified in
    templates/upstream_form.tt2)

    Sounds you've already started thinking about how to do this, and
    perhaps you'd prefer to extend send-attachments-upstream.py rather
    than build off of send_upstream.cgi.  Either way should be fine, but
    however it's implemented, this should eventually replace the
    functionality currently in send_upstream.cgi.
    [kamran]

2-kamran: Here's what I summarized about Bugzilla communication while writing
    send_attachments_upstream.py:

    There are 4 ways of talking with Bugzillas:
    i) Use standard xmlrpclib:
    Advantages: No dependencies
    Disadvantages: Does not support creating attachments
    ii) A library such as python-bugzilla or bzutils:
    Advantages: Helpful wrappers around xmlrpclib which judge the type
    of Bugzilla being used. For example, python-bugzilla judges whether
    it's a RedHat bugzilla and consequently allows attachment creation
    since RH's Bugzilla supports doing that via xmlrpc.
    Disadvantages: Extra dependency, still does not support creating
    attachments for all Bugzillas since it is effectively dependent
    on XML-RPC.
    iii) Fetching web-pages using httplib2:
    Advantages: No dependencies
    Disadvantages: Sending attachments is a complex job since file data
    has to be form-encoded first, depends on HTML field names on
    web-pages
    iv) Fetching web-pages using pycurl
    Advantages: Takes care of form-encoding attachment data as well
    cookies in a simple, painless manner
    Disadvantages: Dependence on HTML field names

    Currently I'm using method (iv) and check for upload by searching
    for attachment title in the subsequent page. Basically, what it
    does is:

    "Upload abc.txt and then check whether the string abc.txt is
    present in a link which says "Created" in the response body."

    To put it in a nutshell, Bugzilla's XML-RPC API is not mature
    enough yet (unlike Launchpad's REST). It's unsuitable for
    attachment creation. It *can* be used for Bug creation. However
    then we'll have a mixture of XML-RPC and Curl passages to a
    Bugzilla.

3-bryce:  We need a proof-of-concept cgi program that any arbitrary user can
    use, which lets them authenticate against launchpad so that the cgi
    can use their logged in credentials for doing lpl calls.

    Alternatively, send-attachments-upstream.py needs to be modified to
    create LaunchpadService with the 'read-only' config parameter, which
    won't need tokens.  I haven't tested if your script will work
    without login tokens, but if it will, then this should work out
    fine.
3-kamran: I had that in mind. The script should additionally ask to login only if
readonly parameter doesn't work for a private bug. Added to checklist.

4-bryce: After the bug is filed with bugzilla, we need to be able to retrieve
    the bug id of the new bug, and then add a bug watch to the original
    launchpad bug report, along with a message to the reporter that
    their issue has been sent upstream.

    This step is going to require the user be logged in, so if we decide
    to go with the alternate approach for step #3, that'll make this a
    bit more challenging.
    [bryce]

4-kamran: I'll add a "--add-bug-watch" flag in which case user shall be asked to
login.

5.  The html template in send_upstream.cgi needs to be redone using
    genshi. [bryce]

    (I've done a couple example scripts in the web/ directory; looks
    pretty straightforward, just need to spend the few hours to get it
    converted over.)