bigdata-dev team mailing list archive
-
bigdata-dev team
-
Mailing list archive
-
Message #00216
[Merge] lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk into lp:charms/trusty/apache-hadoop-plugin
Kevin W Monroe has proposed merging lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk into lp:charms/trusty/apache-hadoop-plugin.
Requested reviews:
Kevin W Monroe (kwmonroe)
For more details, see:
https://code.launchpad.net/~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk/+merge/268670
--
Your team Juju Big Data Development is subscribed to branch lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk.
=== modified file 'DEV-README.md'
--- DEV-README.md 2015-06-29 14:23:33 +0000
+++ DEV-README.md 2015-08-20 23:12:25 +0000
@@ -27,11 +27,19 @@
Additionally, the `JAVA_HOME`, `HADOOP_HOME`, `HADOOP_CONF_DIR`, and other
environment variables will be set via `/etc/environment`. This includes putting
+<<<<<<< TREE
the Hadoop bin and sbin directories on the `PATH`. There are
[helpers](http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/files/head:/common/noarch/)
in `charmhelpers.contrib.bigdata.utils` to assist with using the environment
file. For example, to run the `hdfs` command to create a directory as the
`ubuntu` user:
+=======
+the Hadoop bin and sbin directories on the `PATH`. There are
+[helpers](https://git.launchpad.net/bigdata-data/tree/common/noarch)
+in `charmhelpers.contrib.bigdata.utils` to assist with using the environment
+file. For example, to run the `hdfs` command to create a directory as the
+`ubuntu` user:
+>>>>>>> MERGE-SOURCE
from charmhelpers.contrib.bigdata.utils import run_as
run_as('ubuntu', 'hdfs', 'dfs', '-mkdir', '-p', '/home/ubuntu/foo')
@@ -102,10 +110,17 @@
## Manual Deployment
+<<<<<<< TREE
The easiest way to deploy the core Apache Hadoop platform is to use one of
the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
However, to manually deploy the base Apache Hadoop platform without using one
of the bundles, you can use the following:
+=======
+The easiest way to deploy an Apache Hadoop platform is to use one of
+the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
+However, to manually deploy the base Apache Hadoop platform without using one
+of the bundles, you can use the following:
+>>>>>>> MERGE-SOURCE
juju deploy apache-hadoop-hdfs-master hdfs-master
juju deploy apache-hadoop-hdfs-secondary secondary-namenode
=== modified file 'README.md'
--- README.md 2015-06-18 17:12:11 +0000
+++ README.md 2015-08-20 23:12:25 +0000
@@ -22,7 +22,11 @@
If you wanted to also wanted to be able to analyze your data using Apache Pig,
you could deploy it and attach it to the same plugin:
+<<<<<<< TREE
juju deploy cs:~bigdata-charmers/trusty/apache-pig pig
+=======
+ juju deploy apache-pig pig
+>>>>>>> MERGE-SOURCE
juju add-relation plugin pig
## Benchmarking
@@ -92,17 +96,19 @@
of these resources:
sudo pip install jujuresources
- juju resources fetch --all apache-hadoop-plugin/resources.yaml -d /tmp/resources
- juju resources serve -d /tmp/resources
+ juju-resources fetch --all /path/to/resources.yaml -d /tmp/resources
+ juju-resources serve -d /tmp/resources
This will fetch all of the resources needed by this charm and serve them via a
-simple HTTP server. You can then set the `resources_mirror` config option to
-have the charm use this server for retrieving resources.
+simple HTTP server. The output from `juju-resources serve` will give you a
+URL that you can set as the `resources_mirror` config option for this charm.
+Setting this option will cause all resources required by this charm to be
+downloaded from the configured URL.
You can fetch the resources for all of the Apache Hadoop charms
(`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`,
`apache-hadoop-compute-slave`, `apache-hadoop-plugin`, etc) into a single
-directory and serve them all with a single `juju resources serve` instance.
+directory and serve them all with a single `juju-resources serve` instance.
## Contact Information
=== modified file 'actions/parseTerasort.py'
--- actions/parseTerasort.py 2015-06-08 20:05:37 +0000
+++ actions/parseTerasort.py 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
"""
Simple script to parse cassandra-stress' transaction results
and reformat them as JSON for sending back to juju
=== modified file 'actions/teragen'
--- actions/teragen 2015-06-08 21:32:27 +0000
+++ actions/teragen 2015-08-20 23:12:25 +0000
@@ -3,6 +3,10 @@
SIZE=`action-get size`
IN_DIR=`action-get indir`
+# The plugin uses a venv to avoid conflicting with system bits; ensure our
+# venv is activated.
+. ${CHARM_DIR}/.venv/bin/activate
+
benchmark-start
# NB: Escaped vars in the block below (e.g., \${HADOOP_HOME}) come from
=== modified file 'actions/terasort'
--- actions/terasort 2015-06-08 21:32:27 +0000
+++ actions/terasort 2015-08-20 23:12:25 +0000
@@ -25,6 +25,9 @@
# from this outer scope
su ubuntu << EOF
. /etc/environment
+# The plugin uses a venv to avoid conflicting with system bits; ensure our
+# venv is activated.
+. ${CHARM_DIR}/.venv/bin/activate
mkdir -p /opt/terasort/results/$run
@@ -45,4 +48,4 @@
EOF
PATH=$OLDPATH
-`cat /opt/terasort/results/$run/terasort.log | python $CHARM_DIR/actions/parseTerasort.py`
+`cat /opt/terasort/results/$run/terasort.log | $CHARM_DIR/.venv/bin/python $CHARM_DIR/actions/parseTerasort.py`
=== modified file 'dist.yaml'
--- dist.yaml 2015-04-16 15:45:57 +0000
+++ dist.yaml 2015-08-20 23:12:25 +0000
@@ -68,49 +68,3 @@
owner: 'hdfs'
group: 'hadoop'
perms: 0775
-ports:
- # Ports that need to be exposed, overridden, or manually specified.
- # Only expose ports serving a UI or external API (i.e., namenode and
- # resourcemanager). Communication among units within the cluster does
- # not need ports to be explicitly opened.
- # If adding a port here, you will need to update
- # charmhelpers.contrib.bigdata.handlers.apache or hooks/callbacks.py
- # to ensure that it is supported.
- namenode:
- port: 8020
- exposed_on: 'hdfs-master'
- nn_webapp_http:
- port: 50070
- exposed_on: 'hdfs-master'
- dn_webapp_http:
- port: 50075
- exposed_on: 'compute-slave'
- resourcemanager:
- port: 8032
- exposed_on: 'yarn-master'
- rm_webapp_http:
- port: 8088
- exposed_on: 'yarn-master'
- rm_log:
- port: 19888
- nm_webapp_http:
- port: 8042
- exposed_on: 'compute-slave'
- jobhistory:
- port: 10020
- jh_webapp_http:
- port: 19888
- exposed_on: 'yarn-master'
- # TODO: support SSL
- #nn_webapp_https:
- # port: 50470
- # exposed_on: 'hdfs-master'
- #dn_webapp_https:
- # port: 50475
- # exposed_on: 'compute-slave'
- #rm_webapp_https:
- # port: 8090
- # exposed_on: 'yarn-master'
- #nm_webapp_https:
- # port: 8044
- # exposed_on: 'compute-slave'
=== modified file 'hooks/callbacks.py'
--- hooks/callbacks.py 2015-06-25 15:41:48 +0000
+++ hooks/callbacks.py 2015-08-20 23:12:25 +0000
@@ -24,28 +24,28 @@
def update_blocked_status():
if unitdata.kv().get('charm.active', False):
return
- rels = (
- ('Yarn', 'ResourceManager', ResourceManager()),
+ rels = [
('HDFS', 'NameNode', NameNode()),
- )
+ ]
missing_rel = [rel for rel, res, impl in rels if not impl.connected_units()]
- missing_hosts = [rel for rel, res, impl in rels if not impl.am_i_registered()]
- not_ready = [(rel, res) for rel, res, impl in rels if not impl.is_ready()]
+ rels.append(('Yarn', 'ResourceManager', ResourceManager()))
+ not_ready = [(rel, res) for rel, res, impl in rels if impl.connected_units() and not impl.is_ready()]
+ missing_hosts = [rel for rel, res, impl in rels if impl.connected_units() and not impl.am_i_registered()]
if missing_rel:
hookenv.status_set('blocked', 'Waiting for relation to %s master%s' % (
' and '.join(missing_rel),
's' if len(missing_rel) > 1 else '',
)),
- elif missing_hosts:
- hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % (
- ' and '.join(missing_hosts),
- ))
elif not_ready:
unready_rels, unready_ress = zip(*not_ready)
hookenv.status_set('waiting', 'Waiting for %s to provide %s' % (
' and '.join(unready_rels),
' and '.join(unready_ress),
))
+ elif missing_hosts:
+ hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % (
+ ' and '.join(missing_hosts),
+ ))
def update_working_status():
@@ -56,5 +56,18 @@
def update_active_status():
- unitdata.kv().set('charm.active', True)
- hookenv.status_set('active', 'Ready')
+ hdfs_ready = NameNode().is_ready()
+ yarn_connected = ResourceManager().connected_units()
+ yarn_ready = ResourceManager().is_ready()
+ if hdfs_ready and (not yarn_connected or yarn_ready):
+ unitdata.kv().set('charm.active', True)
+ hookenv.status_set('active', 'Ready%s' % (
+ '' if yarn_ready else ' (HDFS only)'
+ ))
+ else:
+ clear_active_flag()
+ update_blocked_status()
+
+
+def clear_active_flag():
+ unitdata.kv().set('charm.active', False)
=== modified file 'hooks/common.py'
--- hooks/common.py 2015-06-25 15:41:48 +0000
+++ hooks/common.py 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
@@ -14,12 +14,21 @@
Common implementation for all hooks.
"""
+import os
import jujuresources
from charmhelpers.core import hookenv
from charmhelpers.core import unitdata
from charmhelpers.core import charmframework
+# ensure that the venv is used for installing resources
+# (venv is used to ensure library versions needed by plugin
+# don't conflict with the charm the plugin is supporting)
+os.environ['PATH'] = os.pathsep.join([
+ os.path.join(hookenv.charm_dir(), '.venv/bin'),
+ os.environ['PATH']])
+
+
def bootstrap_resources():
"""
Install required resources defined in resources.yaml
@@ -51,7 +60,7 @@
# list of keys required to be in the dist.yaml
client_reqs = ['vendor', 'hadoop_version', 'packages', 'groups', 'users',
- 'dirs', 'ports']
+ 'dirs']
dist_config = jujubigdata.utils.DistConfig(filename='dist.yaml',
required_keys=client_reqs)
hadoop = jujubigdata.handlers.HadoopBase(dist_config)
@@ -71,26 +80,44 @@
],
},
{
- 'name': 'plugin',
+ 'name': 'hdfs',
'provides': [
jujubigdata.relations.HadoopPlugin(),
],
'requires': [
hadoop.is_installed,
+ hdfs_relation,
+ ],
+ 'callbacks': [
+ callbacks.update_working_status,
+ hdfs_relation.register_provided_hosts,
+ jujubigdata.utils.manage_etc_hosts,
+ hdfs.configure_client,
+ callbacks.update_active_status,
+ ],
+ 'cleanup': [
+ callbacks.clear_active_flag,
+ callbacks.update_blocked_status,
+ ],
+ },
+ {
+ 'name': 'yarn',
+ 'provides': [],
+ 'requires': [
+ hadoop.is_installed,
yarn_relation,
- hdfs_relation,
],
'callbacks': [
callbacks.update_working_status,
yarn_relation.register_provided_hosts,
- hdfs_relation.register_provided_hosts,
jujubigdata.utils.manage_etc_hosts,
yarn.install_demo,
yarn.configure_client,
- hdfs.configure_client,
callbacks.update_active_status,
],
- 'cleanup': [],
+ 'cleanup': [
+ callbacks.update_blocked_status,
+ ],
},
])
manager.manage()
=== modified file 'hooks/config-changed'
--- hooks/config-changed 2015-02-09 18:13:28 +0000
+++ hooks/config-changed 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== modified file 'hooks/hadoop-plugin-relation-changed'
--- hooks/hadoop-plugin-relation-changed 2015-04-28 13:38:46 +0000
+++ hooks/hadoop-plugin-relation-changed 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== added file 'hooks/hadoop-plugin-relation-departed'
--- hooks/hadoop-plugin-relation-departed 1970-01-01 00:00:00 +0000
+++ hooks/hadoop-plugin-relation-departed 2015-08-20 23:12:25 +0000
@@ -0,0 +1,15 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+common.manage()
=== modified file 'hooks/namenode-relation-changed'
--- hooks/namenode-relation-changed 2015-05-07 15:27:21 +0000
+++ hooks/namenode-relation-changed 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== added file 'hooks/namenode-relation-departed'
--- hooks/namenode-relation-departed 1970-01-01 00:00:00 +0000
+++ hooks/namenode-relation-departed 2015-08-20 23:12:25 +0000
@@ -0,0 +1,16 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+
+common.manage()
=== modified file 'hooks/resourcemanager-relation-changed'
--- hooks/resourcemanager-relation-changed 2015-05-07 15:11:29 +0000
+++ hooks/resourcemanager-relation-changed 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== added file 'hooks/resourcemanager-relation-departed'
--- hooks/resourcemanager-relation-departed 1970-01-01 00:00:00 +0000
+++ hooks/resourcemanager-relation-departed 2015-08-20 23:12:25 +0000
@@ -0,0 +1,16 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+
+common.manage()
=== modified file 'hooks/setup.py'
--- hooks/setup.py 2015-06-25 15:41:48 +0000
+++ hooks/setup.py 2015-08-20 23:12:25 +0000
@@ -9,6 +9,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+import os
import subprocess
from glob import glob
@@ -17,12 +18,15 @@
"""
Do any setup required before the install hook.
"""
- install_pip()
+ setup_venv()
install_bundled_resources()
-def install_pip():
- subprocess.check_call(['apt-get', 'install', '-yq', 'python-pip', 'bzr'])
+def setup_venv():
+ if not os.path.exists('.venv/bin/python'):
+ subprocess.check_call(['apt-get', 'install', '-yq', 'python-virtualenv'])
+ subprocess.check_call(['virtualenv', '.venv'])
+ execfile('.venv/bin/activate_this.py', {'__file__': '.venv/bin/activate_this.py'})
def install_bundled_resources():
@@ -30,4 +34,4 @@
Install the bundled resources libraries.
"""
archives = glob('resources/python/*')
- subprocess.check_call(['pip', 'install'] + archives)
+ subprocess.check_call(['.venv/bin/pip', 'install'] + archives)
=== modified file 'hooks/start'
--- hooks/start 2015-02-09 18:13:28 +0000
+++ hooks/start 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== modified file 'hooks/stop'
--- hooks/stop 2015-02-09 18:13:28 +0000
+++ hooks/stop 2015-08-20 23:12:25 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
=== modified file 'resources.yaml'
--- resources.yaml 2015-06-25 22:43:49 +0000
+++ resources.yaml 2015-08-20 23:12:25 +0000
@@ -4,7 +4,7 @@
pathlib:
pypi: path.py>=7.0
jujubigdata:
- pypi: jujubigdata>=2.0.2,<3.0.0
+ pypi: jujubigdata>=4.0.0,<5.0.0
charm-benchmark:
pypi: charm-benchmark>=1.0.1,<2.0.0
java-installer:
@@ -12,19 +12,19 @@
# If replaced with an alternate implementation, it must output *only* two
# lines containing the JAVA_HOME path, and the Java version, respectively,
# on stdout. Upon error, it must exit with a non-zero exit code.
- url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150625222410-qfck64q46ubx5i6p/javainstaller.sh-20150311213053-4vq7369jhlvc6qy8-1/java-installer.sh
- hash: 8fdff60270ea4be7bbef1e013e503057fe6efc2f4b5761edebc206a54f303023
+ url: https://git.launchpad.net/bigdata-data/plain/common/noarch/java-installer.sh?id=baa0b74b86587f97b446f255deb96c8420021dd8
+ hash: f7df6937bdb4dcc60de559252b4e6b65c77959f871c7ef2e59af57832d7ddfca
hash_type: sha256
optional_resources:
hadoop-aarch64:
- url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150303192631-swhrf8f7q82si75t/hadoop2.4.1.tar.gz-20150303192554-7gqslr4m8ahkwiax-2/hadoop-2.4.1.tar.gz
+ url: https://git.launchpad.net/bigdata-data/plain/apache/aarch64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
hash: 03ad135835bfe413f85fe176259237a8
hash_type: md5
hadoop-ppc64le:
- url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150130165209-nuz1myezjpdx7eus/hadoop2.4.1ppc64le.t-20150130165148-s8i19s002ht88gio-2/hadoop-2.4.1-ppc64le.tar.gz
+ url: https://git.launchpad.net/bigdata-data/plain/apache/ppc64le/hadoop-2.4.1-ppc64le.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
hash: 09942b168a3db0d183b281477d3dae9deb7b7bc4b5783ba5cda3965b62e71bd5
hash_type: sha256
hadoop-x86_64:
- url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150116154822-x5osw3zfhw6e03b1/hadoop2.4.1.tar.gz-20150116154748-yfa2j12rr5m53xd3-1/hadoop-2.4.1.tar.gz
+ url: https://git.launchpad.net/bigdata-data/plain/apache/x86_64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
hash: a790d39baba3a597bd226042496764e0520c2336eedb28a1a3d5c48572d3b672
hash_type: sha256
=== removed file 'resources/python/jujuresources-0.2.8.tar.gz'
Binary files resources/python/jujuresources-0.2.8.tar.gz 2015-06-25 15:41:48 +0000 and resources/python/jujuresources-0.2.8.tar.gz 1970-01-01 00:00:00 +0000 differ
=== added file 'resources/python/jujuresources-0.2.9.tar.gz'
Binary files resources/python/jujuresources-0.2.9.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/jujuresources-0.2.9.tar.gz 2015-08-20 23:12:25 +0000 differ
Follow ups
