[OmniOS-discuss] Big Data

[Schweiss, Chip]

I've used BeeGFS on Ubuntu 16.04 for about a year now.   I like your idea
of putting it lxzones on OmniOS for scratch space.

I have found it to scale with millions of files very well.   It's running
on a 4 node cluster.  Each node is client, metadata and data nodes.   These
are very big GPU boxes with 9 Tesla GPUs, 40 CPU cores and 256GB ram.

The metadata is mirrored on 2 Samsung Pro SSDs on each node.   It sustains
about 33k metadata ops with never more than one queued.

This is my third iteration of setting it up.  Metadata performance was our
bottleneck each time previously.   What I have found is that latency and
horizontal scaling is king with BeeGFS metadata.   It doesn't take a lot of
CPU, but keep it close as possible on the network to the clients and keep
latency low with fast network and SSDs.

My complaints about BeeGFS is lack of snapshots, so backup is limited to
rsync of a live file system.   For this reason it's only used for this very
high read demand cluster.  I still use ZFS on OmniOS for our PBs of data
where snapshots and replication are priceless.


-Chip



On Sat, Jun 16, 2018 at 12:45 PM, Michael Talbott <mtalbott at lji.org> wrote:

> We've been using OmniOS happily for years now for our storage server
> needs. But we're rapidly increasing our data footprint and growing so much
> (multiple PBs per year) that ideally I'd like to move to a cluster based
> object store based system ontop of OmniOS. I successfully use BeeGFS inside
> lxzones in OmniOS which seems to work nicely for our HPC scratch volume,
> but, it doesn't sound like it scales to hundreds of millions of files very
> well.
>
> I am hoping that someone has some ideas for me. Ideally I'd like something
> that's cluster capable and has erasure coding like Ceph and have cluster
> aware snapshots (not local zfs snaps) and an s3 compatibility/access layer.
>
> Any thoughts on the topic are greatly appreciated.
>
> Thanks,
>
> Michael
> Sent from my iPhone
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20180619/aeae96f5/attachment.html>