openstack team mailing list archive

[John Dickinson <me@xxxxxx>]

See http://programmerthoughts.com/programming/nested-folders-in-cloud-files/ for info on how to use a nested directory structure in swift.

--John


On Aug 14, 2011, at 11:07 PM, Thiago Moraes wrote:

> I took a look at some distributed file systems and went a little deeper in Hadoop and his HDFS, for instance. I don't really need full POSIX compliance, but having a nested structure is important, but as far as I know there are way to simulate this on Switf, is that correct?
> 
> The problem I see in using something like hadoop is the single point of failure, not because I need almost 100% availability, but because the people who will access the data does not belong to the same organization. They will be researchers from different institutions that may want to deploy a local server with a subset of the data to improve their productivity, but the data set's size makes impractical to just copy everything.  
> 
> The plan would be that the interface to the system would show which files are stored locally and which are not, so that everyone gets access to everything, almost like a peer to peer system where they download from the closest source and then store for their own use. 
> 
> At first, I though of implementing something by hand, but using an already mature solution makes a lot more sense.
> 
> So, is this plausible or am I trying to use the wrong tools?
> 
> thanks, again
> 
> Thiago Moraes - EnC 07 - UFSCar
> 
> 
> 2011/8/14 Todd Deshane <todd.deshane@xxxxxxx>
> On Sun, Aug 14, 2011 at 4:10 AM, Thiago Moraes
> <thiago.camposmoraes@xxxxxxxxx> wrote:
> > Hey guys,
> >
> > I'm new on the list and I'm currently considering Openstack to solve a data
> > distribution problem. Right now, there's a server which contains very large
> > files (usual files have 30GB or even more). This server is accessed by LAN
> > and over the internet but, of course, it's difficult to do this without
> > local connection.
> >
> > My idea to solve this problem is to deploy new servers on the places which
> > access data more often in an such a way that they get a local copy of the
> > most accessed part of data by then. In my head, I consider that there will
> > be N different clouds, one at my location and the others spread on another
> > networks. Then, these new clouds would download and store parts of the data
> > (entire files) so that they can be accessed through their own LAN.
> >
> 
> It sounds like you are looking for the functionality that Zones (aim
> to?) provide.
> 
> Take a look at:
> 
> http://wiki.openstack.org/MultiClusterZones
> 
> 
> > Is Openstack suitable in this environment? Anyone would recommend another
> > solution?
> >
> 
> Have you also looked at SheepDog, Hadoop or HC2? All of these seem to
> have some OpenStack integration points as well.
> 
> Some links to look into:
> http://wiki.openstack.org/SheepdogSupport
> http://doubleclix.wordpress.com/2011/03/17/hadoop-2-0-openstack-pbj/
> http://www.quora.com/What-features-differentiate-HDFS-and-OpenStack-Object-Storage
> 
> 
> Hope that helps.
> 
> Thanks,
> Todd
> 
> > PS: I know the file size limitations of 5GB. I just need that all parts of a
> > file to be in the same local area network so that a blazingly fast Internet
> > connection is not required all the time.
> >
> > thanks,
> >
> >
> > Thiago Moraes - EnC 07 - UFSCar
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
> 
> 
> 
> --
> Todd Deshane
> http://www.linkedin.com/in/deshantm
> http://www.xen.org/products/cloudxen.html
> http://runningxen.com/
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

Attachment: smime.p7s
Description: S/MIME cryptographic signature