yellow team mailing list archive

[Francesco Banconi <francesco.banconi@xxxxxxxxxxxxx>]

Francesco Banconi has proposed merging lp:~frankban/charms/precise/juju-gui/bug-1074412-real-env into lp:~juju-gui/charms/precise/juju-gui/trunk.

Requested reviews:
  Juju GUI Hackers (juju-gui)
Related bugs:
  Bug #1074412 in juju-gui: "Need GUI charm that connects to its environment"
  https://bugs.launchpad.net/juju-gui/+bug/1074412

For more details, see:
https://code.launchpad.net/~frankban/charms/precise/juju-gui/bug-1074412-real-env/+merge/137140

Juju GUI charm connects to its environment

This branch implements the ability to deploy the
Juju GUI charm so that it is actually connected to
the real Juju environment.
This is done by starting the juju API agent pointing
it to the Zookeeper address of the environment.
That part is a translation into Python of what Kapil
already did in the bash charm.

Other changes:

Changed README: now python-charmhelpers needs to be
installed to run functional tests. python-shelltoolbox
is installed as a dependency of python-charmhelpers.

Added a juju-api-port config option. The staging and
real API agents use by default different ports. I
decided to add an option so that the user can select
the port where the websocket server will be listening.
In a future branch we should let users customize the
Juju GUI port too.

Added a staging boolean flag to the config options:
by default is False (real env); users can set it to
True to start the staging server.

Also added a staging-environment config option:
if Juju staging is started, here the users can
select what kind of environment to show up in the GUI.

Fixed start up configuration scripts: now upstart
correctly keeps track of the process ids, and the
services can be correctly stopped.

Added an upstart configuration template for the real
API agent.

Added more functional tests, and implemented a
workaround for bug 872264. I run the test suite in ec2
~10 times without failures, setting the jitsu timeout
to 40 minutes.

Note that rsync raises an error during the test process:
rsync: link_stat "/var/lib/juju/units/*/charm.log" failed:
No such file or directory. This is not an error of the tests,
but a test runner failure while trying to retrieve the logs
from a destroyed unit.

https://codereview.appspot.com/6846132/

-- 
https://code.launchpad.net/~frankban/charms/precise/juju-gui/bug-1074412-real-env/+merge/137140
Your team Juju GUI Hackers is requested to review the proposed merge of lp:~frankban/charms/precise/juju-gui/bug-1074412-real-env into lp:~juju-gui/charms/precise/juju-gui/trunk.

=== modified file 'README.txt'
--- README.txt	2012-11-28 12:01:43 +0000
+++ README.txt	2012-11-30 10:12:50 +0000
@@ -38,7 +38,7 @@
 it you must first install it locally.  The files may be installed globally, or
 into your home directory (as here)::
 
-    sudo apt-get install autoconf libtool python-shelltoolbox
+    sudo apt-get install autoconf libtool python-charmhelpers
     bzr branch lp:~jimbaker/juju-jitsu/unit-test jitsu-unit-test
     cd jitsu-unit-test
     autoreconf
@@ -66,15 +66,16 @@
 Jitsu requires the charm directory be named the same as the charm and it be the
 current working directory when the tests are run::
 
-    JUJU_REPOSITORY=/path/to/charm/repo ~/bin/jitsu test juju-gui --logdir /tmp
+    JUJU_REPOSITORY=/path/to/charm/repo ~/bin/jitsu test juju-gui \
+        --logdir /tmp --timeout 40m
 
 If you are going to run the tests often, you probably want to set up LXC and
 run the tests locally by setting your default environment to a "local" one.
 Among other things you will need to install apt-cacher-ng and LXC to do so.
 
 Unfortunately, currently running tests on a local environment is quite slow
-(with quantal host and precise container at least), so you may want to increase
-the ``jitsu test`` command timeout (e.g. passing ``--timeout 30m``).
+(with quantal host and precise container at least), so you may want to further
+increase the ``jitsu test`` command timeout.
 
 If Jitsu generates errors about not being able bootstrap::
 

=== modified file 'config.yaml'
--- config.yaml	2012-11-20 12:15:10 +0000
+++ config.yaml	2012-11-30 10:12:50 +0000
@@ -4,8 +4,24 @@
       The Juju GUI Bazaar branch containing the source code to be deployed.
     type: string
     default: lp:juju-gui
-  juju-improv-branch:
+  juju-api-branch:
     description: |
-      The Juju Bazaar branch containing the improv script.
+      The Juju API Bazaar branch (containing the improv script).
     type: string
     default: lp:~hazmat/juju/rapi-rollup
+  juju-api-port:
+    description: |
+      The port used by the websocket server.
+    type: int
+    default: 8081
+  staging:
+    description: |
+      Connect the Juju GUI to the staging backend (i.e. the improv script).
+    type: boolean
+    default: False
+  staging-environment:
+    description: |
+      The environment JSON export used by the staging server.
+      Possible values are 'sample' and 'large'.
+    type: string
+    default: sample

=== modified file 'config/config.js.template'
--- config/config.js.template	2012-11-19 17:48:13 +0000
+++ config/config.js.template	2012-11-30 10:12:50 +0000
@@ -6,5 +6,5 @@
   viewContainer: '#main',
   transitions: false,
   charm_store_url: 'http://jujucharms.com/',
-  socket_url: '%(ws_url)s'
+  socket_url: 'ws://%(address)s:%(port)s/ws'
 };

=== added file 'config/juju-api-agent.conf.template'
--- config/juju-api-agent.conf.template	1970-01-01 00:00:00 +0000
+++ config/juju-api-agent.conf.template	2012-11-30 10:12:50 +0000
@@ -0,0 +1,13 @@
+description "Juju API agent"
+author "Canonical"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+env PYTHONPATH=%(juju_dir)s:$PYTHONPATH
+env JUJU_ZOOKEEPER=%(zookeeper)s
+
+# Use --nodaemon so that upstart can correctly retrieve the process ID.
+exec /usr/bin/python -m juju.agents.api --nodaemon --port %(port)s \
+    --logfile /var/log/juju/api-agent.log \
+    --session-file /var/run/juju/api-agent.zksession

=== modified file 'config/juju-api-improv.conf.template'
--- config/juju-api-improv.conf.template	2012-11-27 15:52:05 +0000
+++ config/juju-api-improv.conf.template	2012-11-30 10:12:50 +0000
@@ -1,4 +1,4 @@
-description "Juju GUI"
+description "Juju API improv"
 author "Canonical"
 
 start on runlevel [2345]
@@ -6,4 +6,6 @@
 
 setuid ubuntu
 env PYTHONPATH=%(juju_dir)s:$PYTHONPATH
-exec /usr/bin/python %(juju_dir)s/improv.py -f %(juju_dir)s/sample.json &> /tmp/improv-agent.output
+
+exec /usr/bin/python %(juju_dir)s/improv.py --port %(port)s \
+    -f %(juju_dir)s/%(staging_env)s.json

=== modified file 'config/juju-gui.conf.template'
--- config/juju-gui.conf.template	2012-11-27 15:52:05 +0000
+++ config/juju-gui.conf.template	2012-11-30 10:12:50 +0000
@@ -5,6 +5,6 @@
 stop on runlevel [!2345]
 
 respawn
-
 setuid ubuntu
+
 exec nodejs %(juju_gui_dir)s/server.js

=== modified file 'hooks/install'
--- hooks/install	2012-11-20 12:15:10 +0000
+++ hooks/install	2012-11-30 10:12:50 +0000
@@ -24,16 +24,15 @@
 )
 
 
-def fetch():
+def fetch(juju_gui_branch, juju_api_branch):
     """Install required dependencies and retrieve Juju/Juju-GUI branches."""
     log('Installing dependencies.')
     install_extra_repositories('ppa:chris-lea/node.js')
     apt_get_install('bzr', 'imagemagick', 'make', 'nodejs', 'npm', 'zookeeper')
     log('Retrieving source checkouts.')
-    config = get_config()
     bzr_checkout = command('bzr', 'co', '--lightweight')
-    bzr_checkout(config['juju-gui-branch'], 'juju-gui')
-    bzr_checkout(config['juju-improv-branch'], 'juju')
+    bzr_checkout(juju_gui_branch, 'juju-gui')
+    bzr_checkout(juju_api_branch, 'juju')
 
 
 def build():
@@ -43,10 +42,15 @@
         run('make')
 
 
+def main():
+    config = get_config()
+    fetch(config['juju-gui-branch'], config['juju-api-branch'])
+    build()
+
+
 if __name__ == '__main__':
     log_entry()
     try:
-        fetch()
-        build()
+        main()
     finally:
         log_exit()

=== modified file 'hooks/start'
--- hooks/start	2012-11-27 15:52:05 +0000
+++ hooks/start	2012-11-30 10:12:50 +0000
@@ -1,9 +1,9 @@
 #!/usr/bin/env python2
 
 import os
-import sys
 
 from charmhelpers import (
+    get_config,
     log,
     log_entry,
     log_exit,
@@ -17,7 +17,10 @@
     su,
 )
 
-from utils import render_to_file
+from utils import (
+    get_zookeeper_address,
+    render_to_file,
+)
 
 
 CURRENT_DIR = os.getcwd()
@@ -25,38 +28,83 @@
 JUJU_GUI_DIR = os.path.join(CURRENT_DIR, 'juju-gui')
 
 
-def start_improv():
-    """Start a simulated juju environment using ``improv.py``."""
-    sys.path.insert(0, JUJU_DIR)
-    log('Retrieving websocket URL.')
-    ws_url = 'ws://{0}:8081/ws'.format(unit_get('public-address'))
+def start_gui(juju_api_port):
+    """Set up and start the Juju GUI server."""
     with su('root'):
         run('chown', '-R', 'ubuntu:', JUJU_GUI_DIR)
-    log('Setting up start up scripts.')
-    render_to_file(
-        'juju-api-improv.conf.template', {'juju_dir': JUJU_DIR},
-        '/etc/init/juju-api-improv.conf')
+    log('Setting up Juju GUI start up script.')
     render_to_file(
         'juju-gui.conf.template', {'juju_gui_dir': JUJU_GUI_DIR},
         '/etc/init/juju-gui.conf')
     log('Generating the Juju-GUI configuration file.')
+    context = {'address': unit_get('public-address'), 'port': juju_api_port}
     render_to_file(
-        'config.js.template', {'ws_url': ws_url},
+        'config.js.template', context,
         os.path.join(JUJU_GUI_DIR, 'app', 'config.js'))
     log('Starting Juju-GUI.')
     with su('root'):
         service_control('juju-gui', START)
+
+
+def start_improv(juju_api_port, staging_env):
+    """Start a simulated juju environment using ``improv.py``."""
+    log('Setting up staging start up script.')
+    context = {
+        'juju_dir': JUJU_DIR,
+        'port': juju_api_port,
+        'staging_env': staging_env,
+    }
+    render_to_file(
+        'juju-api-improv.conf.template', context,
+        '/etc/init/juju-api-improv.conf')
+    log('Starting improv.')
+    with su('root'):
         service_control('juju-api-improv', START)
+
+
+def start_agent(juju_api_port):
+    """Start the Juju agent and connect to the current environment."""
+    # Retrieve the Zookeeper address from the start up script.
+    unit_dir = os.path.realpath(os.path.join(CURRENT_DIR, '..'))
+    agent_file = '/etc/init/juju-{0}.conf'.format(os.path.basename(unit_dir))
+    zookeeper = get_zookeeper_address(agent_file)
+    log('Setting up API agent start up script.')
+    context = {
+        'juju_dir': JUJU_DIR,
+        'port': juju_api_port,
+        'zookeeper': zookeeper,
+    }
+    render_to_file(
+        'juju-api-agent.conf.template', context,
+        '/etc/init/juju-api-agent.conf')
+    log('Starting API agent.')
+    with su('root'):
+        service_control('juju-api-agent', START)
+
+
+def open_ports(juju_api_port):
+    """Expose Juju GUI web server and websocket server ports."""
     log('Exposing services.')
     # Open the Juju GUI web server port.
     open_port(8888)
     # Open the Juju websocket server port.
-    open_port(8081)
+    open_port(juju_api_port)
+
+
+def main():
+    config = get_config()
+    juju_api_port = config['juju-api-port']
+    start_gui(juju_api_port)
+    if config.get('staging'):
+        start_improv(juju_api_port, config['staging-environment'])
+    else:
+        start_agent(juju_api_port)
+    open_ports(juju_api_port)
 
 
 if __name__ == '__main__':
     log_entry()
     try:
-        start_improv()
+        main()
     finally:
         log_exit()

=== modified file 'hooks/stop'
--- hooks/stop	2012-11-20 11:58:00 +0000
+++ hooks/stop	2012-11-30 10:12:50 +0000
@@ -1,6 +1,7 @@
 #!/usr/bin/env python2
 
 from charmhelpers import (
+    get_config,
     log,
     log_entry,
     log_exit,
@@ -12,10 +13,16 @@
 
 def stop():
     """Stop the Juju API agent."""
-    log('Stopping Juju API agent.')
+    config = get_config()
     with su('root'):
+        log('Stopping Juju GUI.')
         service_control('juju-gui', STOP)
-        service_control('juju-api-improv', STOP)
+        if config.get('staging'):
+            log('Stopping improv.')
+            service_control('juju-api-improv', STOP)
+        else:
+            log('Stopping API agent.')
+            service_control('juju-api-agent', STOP)
 
 
 if __name__ == '__main__':

=== modified file 'hooks/utils.py'
--- hooks/utils.py	2012-11-20 11:58:00 +0000
+++ hooks/utils.py	2012-11-30 10:12:50 +0000
@@ -2,6 +2,20 @@
 
 import os
 
+from shelltoolbox import search_file
+
+
+def get_zookeeper_address(agent_file_path):
+    """Retrieve the Zookeeper address contained in the given *agent_file_path*.
+
+    The *agent_file_path* is a path to a file containing a line similar to the
+    following::
+
+        env JUJU_ZOOKEEPER="address"
+    """
+    line = search_file('JUJU_ZOOKEEPER', agent_file_path).strip()
+    return line.split('=')[1].strip('"')
+
 
 def render_to_file(template, context, destination):
     """Render the given *template* into *destination* using *context*.

=== modified file 'revision'
--- revision	2012-11-20 13:55:02 +0000
+++ revision	2012-11-30 10:12:50 +0000
@@ -1,1 +1,1 @@
-10
+11

=== modified file 'tests/deploy.test'
--- tests/deploy.test	2012-11-27 15:52:05 +0000
+++ tests/deploy.test	2012-11-30 10:12:50 +0000
@@ -4,11 +4,13 @@
 import unittest
 import urllib2
 
+from charmhelpers import make_charm_config_file
 from shelltoolbox import command
 
 
 jitsu = command('jitsu')
 juju = command('juju')
+ssh = command('ssh')
 
 
 def open_url(url):
@@ -33,20 +35,44 @@
     def setUp(self):
         self.charm = 'juju-gui'
         self.port = '8888'
-        juju('deploy', 'local:{0}'.format(self.charm))
+
+    def tearDown(self):
+        juju('destroy-service', self.charm)
+
+    def stop_services(self, hostname, services):
+        # XXX 2012-11-29 frankban bug=872264:
+            # Just invoking ``juju destroy-service juju-gui`` in tearDown
+            # should execute the ``stop`` hook, stopping all the services
+            # started by the charm in the machine. Right now this does not
+            # work, so the same behavior is accomplished keeping track of
+            # started services and manually stopping them here.
+        target = 'ubuntu@{0}'.format(hostname)
+        for service in services:
+            ssh(target, 'sudo', 'service', service, 'stop')
+
+    def deploy(self, config_path=None):
+        """Deploy and expose the Juju GUI charm. Return the service host name.
+
+        Also wait until the service is started.
+        If *config_path* is provided, it will be used when deploying the charm.
+        """
+        args = ['deploy', 'local:{0}'.format(self.charm)]
+        if config_path is not None:
+            args.extend(['--config', config_path])
+        juju(*args)
         juju('expose', self.charm)
         jitsu(
             'watch', '--failfast', self.charm,
             '--state', 'started', '--open-port', self.port)
-
-    def test_improv(self):
-        # Ensure the Juju GUI and improv services are correctly set up.
         address = jitsu('get-service-info', self.charm, 'public-address')
-        hostname = address.strip().split(':')[1]
+        return address.strip().split(':')[1]
+
+    def check_services(self, hostname, ws_port=8081):
+        """Check the services are listening on their tcp ports."""
         url = 'http://{0}:{1}'.format(hostname, self.port)
         response = open_url(url)
         self.assertEqual(200, response.getcode())
-        ws_url = 'http://{0}:8081/ws'.format(hostname)
+        ws_url = 'http://{0}:{1}/ws'.format(hostname, ws_port)
         # A bad request status code here means the websocket resource is found.
         # It would take an actual websocket client to properly interact with
         # the websocket server.
@@ -54,6 +80,35 @@
             open_url(ws_url)
         self.assertEqual(400, context_manager.exception.getcode())
 
+    def test_api_agent(self):
+        # Ensure the Juju GUI and API agent services are correctly set up.
+        hostname = self.deploy()
+        self.check_services(hostname)
+        # XXX 2012-11-29 frankban bug=872264: see *stop_services* above.
+        self.addCleanup(
+            self.stop_services, hostname, ['juju-api-agent', 'juju-gui'])
+
+    def test_customized_api_port(self):
+        # It is possible to customize the port used by the websocket server.
+        config = {self.charm: {'juju-api-port': 80}}
+        config_file = make_charm_config_file(config)
+        hostname = self.deploy(config_path=config_file.name)
+        self.check_services(hostname, ws_port=80)
+        # XXX 2012-11-29 frankban bug=872264: see *stop_services* above.
+        self.addCleanup(
+            self.stop_services, hostname, ['juju-api-agent', 'juju-gui'])
+
+    def test_staging(self):
+        # Ensure the Juju GUI and improv services are correctly set up.
+        self.services = ('juju-api-improv', 'juju-gui')
+        config = {self.charm: {'staging': 'True'}}
+        config_file = make_charm_config_file(config)
+        hostname = self.deploy(config_path=config_file.name)
+        self.check_services(hostname)
+        # XXX 2012-11-29 frankban bug=872264: see *stop_services* above.
+        self.addCleanup(
+            self.stop_services, hostname, ['juju-api-improv', 'juju-gui'])
+
 
 if __name__ == '__main__':
     unittest.main(verbosity=2)

=== modified file 'tests/test_utils.py'
--- tests/test_utils.py	2012-11-21 11:27:24 +0000
+++ tests/test_utils.py	2012-11-30 10:12:50 +0000
@@ -4,7 +4,26 @@
 import tempfile
 import unittest
 
-from utils import render_to_file
+from utils import (
+    get_zookeeper_address,
+    render_to_file,
+)
+
+
+class GetZookeeperAddressTest(unittest.TestCase):
+
+    def setUp(self):
+        self.zookeeper_address = 'example.com:2000'
+        contents = 'env JUJU_ZOOKEEPER="{0}"\n'.format(self.zookeeper_address)
+        with tempfile.NamedTemporaryFile(delete=False) as agent_file:
+            agent_file.write(contents)
+            self.agent_file_path = agent_file.name
+        self.addCleanup(os.remove, self.agent_file_path)
+
+    def test_get_zookeeper_address(self):
+        # Ensure the Zookeeper address is correctly retreived.
+        address = get_zookeeper_address(self.agent_file_path)
+        self.assertEqual(self.zookeeper_address, address)
 
 
 class RenderToFileTest(unittest.TestCase):