yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1400477] Re: In Juno Cannot Create Spark Cluster From Horizon

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Lazarev <alazarev@xxxxxxxxxxxx>
Date: Mon, 08 Dec 2014 21:31:05 -0000
Reply-to: Bug 1400477 <1400477@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: sahara
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1400477

Title:
  In Juno Cannot Create Spark Cluster From Horizon

Status in OpenStack Dashboard (Horizon):
  New
Status in OpenStack Data Processing (Sahara, ex. Savanna):
  New

Bug description:
  Trying to instantiate a Spark 1.0.0 cluster, using “Data Processing”
  element under Horizon, and am having the following problems:

  1-      Security Group: Having problem with security groups, when
  either defining "Node Group Templates" or instantiating a cluster. In
  the first case, if I use an existing group, say "default", gui shows
  an error stating "Error Security group '2' not found". Sahara log file
  indicates the same thing:

     2014-12-03 23:41:24.144 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
      2014-12-03 23:41:24.146 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
      2014-12-03 23:41:24.163 30234 DEBUG urllib3.connectionpool [-] "GET /v2/926c31c887f441f6a4e4b8031b8cc528/os-security-groups HTTP/1.1" 200 682 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
      2014-12-03 23:41:24.165 30234 DEBUG sahara.utils.api [-] Validation Error occurred: error_code=400, error_message=Security group '2' not found, error_name=INVALID_REFERENCE bad_request /usr/local/lib/python2.7/dist-packages/sahara/utils/api.py:245
      2014-12-03 23:41:24.165 30234 INFO sahara.cli.sahara_all [-] 10.0.0.86 - - [03/Dec/2014 23:41:24] "POST /v1.1/926c31c887f441f6a4e4b8031b8cc528/node-group-templates HTTP/1.1" 400 221 0.063121
      2014-12-03 23:41:24.257 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
      2014-12-03 23:41:24.258 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

  I can, temporarily, avoid the problem by selecting "Auto Security
  Group" option. This would allow for a node group to be created;
  however, I do not see any new security group, under Compute -> Access
  & Security. At any rate, this also fails during cluster instantiation:

  2014-12-03 23:55:06.285 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
  2014-12-03 23:55:06.286 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
  2014-12-03 23:55:06.409 30234 DEBUG urllib3.connectionpool [-] "POST /v2/926c31c887f441f6a4e4b8031b8cc528/servers HTTP/1.1" 400 116 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
  2014-12-03 23:55:06.451 30234 ERROR sahara.service.ops [-] Error during operating cluster 'Sprk265' (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops Traceback (most recent call last):
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 113, in wrapper
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     f(cluster_id, *args, **kwds)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 198, in _provision_cluster
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     INFRA.create_cluster(cluster)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 51, in create_cluster
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._create_instances(cluster)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 168, in _create_instances
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._run_instance(cluster, node_group, idx, aa_group=aa_group)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 305, in _run_instance
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **nova_kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 883, in create
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **boot_kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 546, in _boot
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return_raw=return_raw, **kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 100, in _create
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     _resp, body = self.api.client.post(url, body=body)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 490, in post
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return self._cs_request(url, 'POST', **kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 465, in _cs_request
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self._time_request(url, method, **kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in _time_request
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self.request(url, method, **kwargs)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 433, in request
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     raise exceptions.from_response(resp, body, url, method)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops BadRequest: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400)
  2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops 
  2014-12-03 23:55:06.611 30234 INFO sahara.service.direct_engine [-] Cluster 'Sprk265' creation rollback (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
  2014-12-03 23:55:06.616 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
  2014-12-03 23:55:06.622 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
  2014-12-03 23:55:07.018 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
  2014-12-03 23:55:07.019 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

  Here is my conf file:

  [DEFAULT]
  use_floating_ips=True
  use_neutron=True
   [keystone_authtoken]
  auth_uri = http://keystone1.maas17:5000/v2.0/
  identity_uri=http://keystone1.maas17:35357/
  admin_user=sahara
  admin_password=sahara
  admin_tenant_name=sahara
  periodic_enable=true
  plugins=vanilla,hdp,idh,spark,cdh
  [database]
  connection=mysql://sahara:sahara@mysql1.maas17/sahara

  Sahara is also registered with Keystone:

  # keystone service-list
  +----------------------------------+----------+-----------------+----------------------------+
  |                id                |   name   |       type      |        description         |
  +----------------------------------+----------+-----------------+----------------------------+
  | 75b2c466c35a44d5bbe7167c1ed38e20 |  cinder  |      volume     |   Cinder Volume Service    |
  | d5b014c2d96e4d619f9d9b8e646f0f5b |   ec2    |       ec2       |  EC2 Compatibility Layer   |
  | 8180061e79a24627be43485910a9e16a |  glance  |      image      |    Glance Image Service    |
  | 4dad7b5145c842a4a8fbbab1f158629c | keystone |     identity    | Keystone Identity Service  |
  | d833594550e14d49be4df4394886c849 |   nova   |     compute     |    Nova Compute Service    |
  | fe42b5328002433989ffd6b18414aacc | quantum  |     network     | Quantum Networking Service |
  | fac3f901841c4528b3902ae1b7265b4e |    s3    |        s3       | S3 Compatible object-store |
  | 12104e23db1441a28cb42cf8c9437139 |  sahara  | data_processing |  Data processing service   |
  +----------------------------------+----------+-----------------+----------------------------+
  See https://ask.openstack.org/en/question/55161/juno-sahara-spark-100-security-group-error/ for workarounds.

  2-      Spark Login: I am not able to login to the recommended Spark
  Image, i.e., http://sahara-files.mirantis.com/saha... . Launching this
  image either by itself for through Sahara/Data Processing, results in
  invalid user ubuntu:

  Generation complete.
   * Stopping Handle applying cloud-config[74G[ OK ]
   * Starting Hadoop namenode: 
  starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-ubuntu.out
  open-vm-tools: not starting as this is not a VMware VM
  landscape-client is not configured, please run landscape-config.
   * Restoring resolver state...       [80G [74G[ OK ]
  chown: invalid user: 'ubuntu:ubuntu'
  chown: invalid user: 'ubuntu:ubuntu'
  rm: cannot remove '/tmp/in_target.d/post-install.d/20-spark': No such file or directory
   * Stopping System V runlevel compatibility[74G[ OK ]
   * Starting execute cloud user/final scripts[74G[ OK ]
  Cloud-init v. 0.7.5 running 'modules:final' at Thu, 04 Dec 2014 04:48:56 +0000. Up 28.44 seconds.
  2014-12-04 04:48:56,208 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
  ec2: 
  ec2: #############################################################
  ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
  ec2: -----END SSH HOST KEY FINGERPRINTS-----
  ec2: #############################################################
  -----BEGIN SSH HOST KEY KEYS-----
  -----END SSH HOST KEY KEYS-----
  Cloud-init v. 0.7.5 finished at Thu, 04 Dec 2014 04:48:56 +0000. Datasource DataSourceNone.  Up 28.55 seconds
  2014-12-04 04:48:56,237 - cc_final_message.py[WARNING]: Used fallback datasource

  Ubuntu 14.04.1 LTS ubuntu ttyS0

  ubuntu login:

  Sahara log file shows repeated failed login attempts:

  2014-12-04 05:01:40.099 30327 DEBUG sahara.service.engine [-] Can't login to node sprk265-worker-002 (10.0.200.59), reason error: [Errno 110] Connection timed out _wait_until_accessible /usr/local/lib/python2.7/dist-packages/sahara/service/engine.py:110
  2014-12-04 05:01:40.116 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-worker-001] _execute_command took 127.5 seconds to complete _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459
  2014-12-04 05:01:40.117 30327 DEBUG sahara.service.engine [-] Can't login to node sprk265-worker-001 (10.0.200.56), reason error: [Errno 110] Connection timed out _wait_until_accessible /usr/local/lib/python2.7/dist-packages/sahara/service/engine.py:110
  2014-12-04 05:01:44.644 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-worker-003] Executing "ls .ssh/authorized_keys" _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459
  2014-12-04 05:01:45.015 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-controller-001] Executing "ls .ssh/authorized_keys" _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459
  2014-12-04 05:01:45.141 30327 DEBUG sahara.openstack.common.periodic_task [-] Running periodic task SaharaPeriodicTasks.update_job_statuses run_periodic_tasks /usr/local/lib/python2.7/dist-packages/sahara/openstack/common/periodic_task.py:193

  Note that I am able to launch other ubuntu images using the same key
  pair

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1400477/+subscriptions
References

[Bug 1400477] [NEW] In Juno Cannot Create Spark Cluster From Horizon
From: Nastooh, 2014-12-08