openstack team mailing list archive

[Clay Gerrard <clay.gerrard@xxxxxxxxx>]

Wow, so I glanced quickly at the github project to try and get some
context, but I think I actually start getting *more* confused when I see
"swift" in the same class name as "file-system" ;)

I'd like you (or maybe vaidy, hi vaidy!) to correct me if I'm wrong, but
this hadoop integration will *not* access the filesystem of the object
servers directly?  Everything will happen on a pool of processing boxes
that will talk to swift via HTTP - same as any other client?

In that case, the error message is just a leaky abstraction showing
through.  HDFS probably has permission errors that it tries to helpfully
map back to file system constructs which just don't apply when you're
trying to "simulate" a file system on object storage.  You'll have to get
down the the HTTP to understand what's causing the error.  Presumably a 401
from Swift, so access to swift logs would be helpful.

OTOH, if we're * actually* talking about filesystem permissions; then I'm
totally confused.  But ACL's definitely won't help.  They're just a row
sitting in a sqlite database - probably on a totally different server from
where the one replica of this object is sitting on the filesystem. Nothing
you can set in the api will change the filesystem permissions of the
directory structure or files on the object servers.

Maybe do you have some more overview info on the general approach?  I don't
really have any Hadoop experience, so maybe it'd be better if there's a
hadoop expert out there that also has some experience with swift and can
help get you on the right track...

-Clay



In


On Fri, Apr 26, 2013 at 1:11 AM, Shashank Sahni <shredder12@xxxxxxxxx>wrote:

> Hi everyone,
>
> I've been experimenting with using Swift(1.8) as Hadoop's DFS backend.
>
> https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration
>
> After a few glitches, I'm able to create/access/delete objects/containers
> using hadoop's cli fs tool. But whenever I'm trying to run the job it fails
> with the following error.
>
> ERROR security.UserGroupInformation: PriviledgedActionException
> as:dharmesh cause:java.io.IOException: The ownership/permissions on the
> staging directory
> swift://hadooptest.rackspace/tmp/app/mapred/staging/dharmesh/.staging is
> not as expected. It is owned by and permissions are rwxrwxrwx. The
> directory must be owned by the submitter dharmesh or by dharmesh and
> permissions must be rwx------
>
> Note that, the local system username is "dharmesh" and the openstack
> account and associated tenant is "dharmesh" too.
>
> I tried setting the permissions by creating "tmp" container using "swift
> post -r 'dharmesh:dharmesh'", but unfortunately ended up with the same
> result. Is there an other way to set rwx ACLs in swift?
>
> --
> Shashank Sahni
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>