--- Begin Message ---
-
To:
"Anderson, Marvin" <marvin.anderson@xxxxxx>, "Bouaziz, Khaled" <Khaled.Bouaziz@xxxxxx>
-
From:
"Varnau, Steve (Trafodion)" <steve.varnau@xxxxxx>
-
Date:
Wed, 25 Feb 2015 16:21:28 +0000
-
Accept-language:
en-US
-
Cc:
"Moran, Amanda" <amanda.kay.dierickx@xxxxxx>
-
In-reply-to:
<318555402C42EA458A9CD4B6D9F0B9898A66ED1F@G9W0710.americas.hpqcorp.net>
-
Thread-index:
AdBQZRMJ9VIPstvsQRWuRsVrvd0xOgAAx42gAACu2ZAAA+9DUAAAPvOQAACbm3AAJirZoA==
-
Thread-topic:
Gate job failing
Yeah, hdfs definitely has permission to do anything in the filesystem. The only question is who will be the owner/group of the archive directory. My script assumed that hbase user was the owner of /hbase and was the desired owner of /hbase/archive. So you might want to add a chown command after the mkdir?
-Steve
From: Anderson, Marvin
Sent: Tuesday, February 24, 2015 23:25
To: Varnau, Steve (Trafodion); Bouaziz, Khaled
Cc: Moran, Amanda
Subject: RE: Gate job failing
I get an error using hbase to do the mkdir on our test clusters, it has to be hdfs userid
[andersma@sea-nodepool installer]$ sudo su hbase --command "hdfs dfs -mkdir -p /hbase/archive"
mkdir: Permission denied: user=hbase, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
[andersma@sea-nodepool installer]$ sudo su hdfs --command "hdfs dfs -mkdir -p /hbase/archive"
So, I guess I'll put it in for the hdfs userid.
--Marvin
From: Varnau, Steve (Trafodion)
Sent: Tuesday, February 24, 2015 4:59 PM
To: Anderson, Marvin; Bouaziz, Khaled
Cc: Moran, Amanda
Subject: RE: Gate job failing
Here is what my script does that runs on non-cluster-mgr Cloudera nodes:
sudo -u hbase hdfs dfs -mkdir -p /hbase/archive
sudo -u hdfs hdfs dfs -setfacl -R -m user:jenkins:rwx /hbase/archive
sudo -u hdfs hdfs dfs -setfacl -R -m default:user:jenkins:rwx /hbase/archive
sudo -u hdfs hdfs dfs -setfacl -R -m mask::rwx /hbase/archive
This is sufficient for the tests. I can't say what other implications there might be.
-Steve
From: Anderson, Marvin
Sent: Tuesday, February 24, 2015 13:44
To: Varnau, Steve (Trafodion); Bouaziz, Khaled
Cc: Moran, Amanda
Subject: RE: Gate job failing
Does the installer just do a simple HDFS mkdir command like
Hortonworks
hdfs dfs -mkdir -p /apps/hbase/data/archieve
Cloudera
hdfs dfs -mkdir -p /hbase/archive
Does it need to have permissions set any specific way or other ownership or other settings?
--Marvin
From: Varnau, Steve (Trafodion)
Sent: Tuesday, February 24, 2015 2:58 PM
To: Bouaziz, Khaled; Anderson, Marvin
Cc: Moran, Amanda
Subject: RE: Gate job failing
This may indeed be due to the way we clean up the test nodes between jobs. To protect against any change or test that corrupts hbase data, we completely remove hbase data (in HDFS and in Zookeeper) at the beginning of each job and bring hbase back up to initialize it.
Apparently just bringing up hbase is not creating that directory. So, you could add installer logic to create it if it does not exist, but if you think it should always be there on a normal system, then I can script re-create the archive directory when doing that clean up.
-Steve
From: Bouaziz, Khaled
Sent: Tuesday, February 24, 2015 11:31
To: Anderson, Marvin
Cc: Varnau, Steve (Trafodion); Moran, Amanda
Subject: RE: Gate job failing
Hi Marvin:
I think Steve mentioned something like this before and I think he has a solution already
thanks
From: Anderson, Marvin
Sent: Tuesday, February 24, 2015 1:12 PM
To: Bouaziz, Khaled
Cc: Varnau, Steve (Trafodion); Moran, Amanda
Subject: Gate job failing
Hi Khaled,
I was checking in the changes for snapshot scan support and those changes run fine on several of our test clusters but they are failing on the Jenkins build gate machines for both Cloudera and Hortonworks.
***INFO: Setting HDFS ACLs for snapshot scan support
setfacl: `/hbase/archive': No such file or directory
***ERROR: (hdfs dfs -setfacl -R -m user:trafodion:rwx /hbase/archive) command failed
***ERROR: traf_cloudera_mods98 exited with error.
***INFO: Setting HDFS ACLs for snapshot scan support
setfacl: `/apps/hbase/data/archive': No such file or directory
***ERROR: (hdfs dfs -setfacl -R -m mask::rwx /apps/hbase/data/archive) command failed
***ERROR: traf_hortonworks_mods98 exited with error.
Are these missing directories ones we should be creating or should they already be there. It appears they are not there on the build machines but were there on all our test machines. So, is this a problem with the build machines hadoop env or something we should be creating?
--Marvin
--- End Message ---
