bigdata-dev team mailing list archive
-
bigdata-dev team
-
Mailing list archive
-
Message #00034
[Merge] lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/readme into lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/trunk
Cory Johns has proposed merging lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/readme into lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/trunk.
Requested reviews:
Juju Big Data Development (bigdata-dev)
For more details, see:
https://code.launchpad.net/~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/readme/+merge/252619
New READMEs and minor relation cleanups
--
Your team Juju Big Data Development is requested to review the proposed merge of lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/readme into lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/trunk.
=== added file 'README.dev.md'
--- README.dev.md 1970-01-01 00:00:00 +0000
+++ README.dev.md 2015-03-11 17:16:47 +0000
@@ -0,0 +1,54 @@
+## Overview
+
+This charm provides checkpointing and transaction consolidation for an Apache
+Hadoop deployment, and is intended to be used only as a part of that deployment.
+This document describes how this charm connects to and interacts with the
+other components of the deployment.
+
+
+## Provided Relations
+
+*There are no required relations for this charm.*
+
+## Required Relations
+
+### namenode (dfs)
+
+This relation connects this charm to the apache-hadoop-hdfs-master charm.
+It is a bi-directional interface, with the following keys being exchanged:
+
+* Sent to hdfs-master:
+
+ *There are no keys sent to the hdfs-master*
+
+* Received from hdfs-master:
+
+ * `private-address`: Address of the HDFS master unit, to be used as the NameNode
+ * `ready`: A flag indicating that HDFS is ready to store data
+
+Ports will soon be added to this relation.
+
+
+## Manual Deployment
+
+The easiest way to deploy the core Apache Hadoop platform is to use one of
+the [apache-core-batch-processing-* bundles](https://jujucharms.com/q/bigdata-dev/apache?type=bundle).
+However, to manually deploy the base Apache Hadoop platform without using one of the
+bundles, you can use the following:
+
+ juju deploy apache-hadoop-hdfs-master hdfs-master
+ juju deploy apache-hadoop-hdfs-secondary secondary-namenode
+ juju deploy apache-hadoop-yarn-master yarn-master
+ juju deploy apache-hadoop-compute-slave compute-slave -n3
+ juju deploy apache-hadoop-client client
+ juju add-relation yarn-master hdfs-master
+ juju add-relation secondary-namenode hdfs-master
+ juju add-relation compute-slave yarn-master
+ juju add-relation compute-slave hdfs-master
+ juju add-relation client yarn-master
+ juju add-relation client hdfs-master
+
+This will create a scalable deployment with separate nodes for each master,
+and a three unit compute slave (NodeManager and DataNode) cluster. The master
+charms also support co-locating using the `--to` option to `juju deploy` for
+more dense deployments.
=== modified file 'README.md'
--- README.md 2015-03-04 01:42:08 +0000
+++ README.md 2015-03-11 17:16:47 +0000
@@ -1,117 +1,31 @@
## Overview
-This charm is a component of the Apache Hadoop platform. It is intended
-to be deployed with the other components using the bundle:
-`bundle:~bigdata-charmers/apache-hadoop`
-
-**What is Apache Hadoop?**
-
The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.
-It is designed to scale up from single servers to thousands of machines,
-each offering local computation and storage. Rather than rely on hardware
-to deliver high-avaiability, the library itself is designed to detect
-and handle failures at the application layer, so delivering a
-highly-availabile service on top of a cluster of computers, each of
-which may be prone to failures.
-
-Apache Hadoop 2.4.1 consists of significant improvements over the previous stable
-release (hadoop-1.x).
-
-Here is a short overview of the improvments to both HDFS and MapReduce.
-
- - **HDFS Federation**
- In order to scale the name service horizontally, federation uses multiple
- independent Namenodes/Namespaces. The Namenodes are federated, that is, the
- Namenodes are independent and don't require coordination with each other.
- The datanodes are used as common storage for blocks by all the Namenodes.
- Each datanode registers with all the Namenodes in the cluster. Datanodes
- send periodic heartbeats and block reports and handles commands from the
- Namenodes.
-
- More details are available in the HDFS Federation document:
- <http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/Federation.html>
-
- - **MapReduce NextGen aka YARN aka MRv2**
- The new architecture introduced in hadoop-0.23, divides the two major functions of the
- JobTracker: resource management and job life-cycle management into separate components.
- The new ResourceManager manages the global assignment of compute resources to
- applications and the per-application ApplicationMaster manages the application‚
- scheduling and coordination.
- An application is either a single job in the sense of classic MapReduce jobs or a DAG of
- such jobs.
-
- The ResourceManager and per-machine NodeManager daemon, which manages the user
- processes on that machine, form the computation fabric.
-
- The per-application ApplicationMaster is, in effect, a framework specific
- library and is tasked with negotiating resources from the ResourceManager and
- working with the NodeManager(s) to execute and monitor the tasks.
-
- More details are available in the YARN document:
- <http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/YARN.html>
+This charm deploys a node running the SecondaryNameNode component
+[Apache Hadoop 2.4.1](http://hadoop.apache.org/docs/r2.4.1/), which
+periodically consolidates and checkpoints the work done by the
+NameNode component for efficiency and improved recovery in the case
+of a failure of the NameNode.
## Usage
-This charm manages the HDFS Secondary NameNode.
-It is intended to be used with `apache-hadoop-hdfs-master`.
-
-### Dense Deployment: Single YARN / HDFS master deployment
-
-In this configuration, the YARN and HDFS master components run on the same
-machine. This is useful for lower-resource deployments::
-
- juju deploy apache-hadoop-hdfs-master hdfs-master
- juju deploy apache-hadoop-hdfs-secondary secondary-namenode --to 1
- juju deploy apache-hadoop-yarn-master yarn-master --to 1
- juju deploy apache-hadoop-compute-slave compute-slave
- juju deploy apache-hadoop-client client
- juju add-relation yarn-master hdfs-master
- juju add-relation secondary-namenode hdfs-master
- juju add-relation compute-slave yarn-master
- juju add-relation compute-slave hdfs-master
- juju add-relation client yarn-master
- juju add-relation client hdfs-master
-
-Note that the machine number (`--to 1`) should match the machine number
-for the `hdfs-master` charm. If you previously deployed other services
-in your environment, you may need to adjust the machine number appropriately.
-
-
-### Scale Out Deployment: Separate HDFS, YARN, and compute nodes
-
-In this configuration the HDFS and YARN deployments operate on
-different service units as separate services::
-
- juju deploy apache-hadoop-hdfs-master hdfs-master
- juju deploy apache-hadoop-hdfs-secondary secondary-namenode
- juju deploy apache-hadoop-yarn-master yarn-master
- juju deploy apache-hadoop-compute-slave compute-slave -n 3
- juju deploy apache-hadoop-client client
- juju add-relation yarn-master hdfs-master
- juju add-relation secondary-namenode hdfs-master
- juju add-relation compute-slave yarn-master
- juju add-relation compute-slave hdfs-master
- juju add-relation client yarn-master
- juju add-relation client hdfs-master
-
-The `-n 3` option can be adjusted according to the number of compute nodes
-you need. You can also add additional compute nodes later::
-
- juju add-unit compute-slave -n 2
-
-
-### To deploy a Hadoop service with elasticsearch service::
- # deploy ElasticSearch locally:
- **juju deploy elasticsearch elasticsearch**
- # elasticsearch-hadoop.jar file will be added to LIBJARS path
- # Recommanded to use hadoop -libjars option to included elk jar file
- **juju add-unit -n elasticsearch**
- # deploy hive service by any senarios mentioned above
- # associate Hive with elasticsearch
- **juju add-relation {hadoop master}:elasticsearch elasticsearch:client**
+This charm is intended to be deployed via one of the
+[bundles](https://jujucharms.com/q/bigdata-dev/apache?type=bundle).
+For example:
+
+ juju quickstart u/bigdata-dev/apache-analytics-sql
+
+This will deploy the Apache Hadoop platform with Apache Hive available to
+perform SQL-like queries against your data.
+
+You can also manually load and run map-reduce jobs via the client:
+
+ juju scp my-job.jar client/0:
+ juju ssh client/0
+ hadoop jar my-job.jar
## Deploying in Network-Restricted Environments
@@ -120,12 +34,14 @@
access. To deploy in this environment, you will need a local mirror to serve
the packages and resources required by these charms.
+
### Mirroring Packages
You can setup a local mirror for apt packages using squid-deb-proxy.
For instructions on configuring juju to use this, see the
[Juju Proxy Documentation](https://juju.ubuntu.com/docs/howto-proxies.html).
+
### Mirroring Resources
In addition to apt packages, the Apache Hadoop charms require a few binary
@@ -134,7 +50,7 @@
of these resources:
sudo pip install jujuresources
- juju resources fetch --all apache-hadoop-hdfs-secondary/resources.yaml -d /tmp/resources
+ juju resources fetch --all apache-hadoop-compute-slave/resources.yaml -d /tmp/resources
juju resources serve -d /tmp/resources
This will fetch all of the resources needed by this charm and serve them via a
@@ -143,14 +59,19 @@
You can fetch the resources for all of the Apache Hadoop charms
(`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`,
-`apache-hadoop-compute-slave`, `apache-hadoop-client`, etc) into a single
+`apache-hadoop-hdfs-secondary`, `apache-hadoop-client`, etc) into a single
directory and serve them all with a single `juju resources serve` instance.
## Contact Information
-amir sanjar <amir.sanjar@xxxxxxxxxxxxx>
+
+* Amir Sanjar <amir.sanjar@xxxxxxxxxxxxx>
+* Cory Johns <cory.johns@xxxxxxxxxxxxx>
+* Kevin Monroe <kevin.monroe@xxxxxxxxxxxxx>
+
## Hadoop
+
- [Apache Hadoop](http://hadoop.apache.org/) home page
- [Apache Hadoop bug trackers](http://hadoop.apache.org/issue_tracking.html)
- [Apache Hadoop mailing lists](http://hadoop.apache.org/mailing_lists.html)
=== modified file 'resources.yaml'
--- resources.yaml 2015-03-06 22:41:47 +0000
+++ resources.yaml 2015-03-11 17:16:47 +0000
@@ -8,8 +8,8 @@
six:
pypi: six
charmhelpers:
- pypi: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150306222841-0ibvrfaungfdtkn8/charmhelpers0.2.2.ta-20150304033309-4fa7ewnosqavnwms-1/charmhelpers-0.2.2.tar.gz
- hash: 787fc1cc70fc89e653b08dd192ec702d844709e450ff67577b7c5e99b6bbf39b
+ pypi: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150310214330-f2bk32gk92iinrx8/charmhelpers0.2.2.ta-20150304033309-4fa7ewnosqavnwms-1/charmhelpers-0.2.2.tar.gz
+ hash: a1cafa5e315d3a33db15a8e18f56b4c64d47c2c1c6fcbdba81e42bb00642971c
hash_type: sha256
optional_resources:
hadoop-aarch64:
Follow ups