From admdwrf at gmail.com Sat Nov 1 00:14:02 2014 From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=) Date: Sat, 1 Nov 2014 01:14:02 +0100 Subject: [OmniOS-discuss] OmniOS doc Message-ID: Hi, Since few years, i'm a big OmniOS fan. I tried to evangelisch it around me. I had a website to share about it, but my ex-business partner delete it and backups too :-( Few days ago, i was at linux hack day at Rennes in France. I talked about OmniOS. Some admin sys were very interesting about it. I restarted a blog at blog.fredalix.com But, in my country, some peoples know SmartOS, and when they go to OmniOS website, they don't like it. I am sad to know sysadmin no want try OmniOS, because the actual wiki is not for newbies. Yesterday, i was at Linux and Docker french event, and i talked about OmniOS and zones. But returns were "OmniOS wiki is good for Solaris admins !" OmniTI's team make a super job, but the OmniOS is not attractive for newbies. it's sad :-( what can we'll do about that ? I am sure with Omnios's community we can publish an OmniOS Survival guide. Yep ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajacoutot at bsdfrog.org Sat Nov 1 00:44:33 2014 From: ajacoutot at bsdfrog.org (Antoine Jacoutot) Date: Sat, 1 Nov 2014 01:44:33 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: Message-ID: <20141101004433.GA12410@janus.obspm.bsdfrog.org> > Few days ago, i was at linux hack day at Rennes in France. I talked > about OmniOS. > > Some admin sys were very interesting about it. I restarted a blog at > [1]blog.fredalix.com > > But, in my country, some peoples know SmartOS, and when they go to > OmniOS website, they don't like it. Yeah but some others *do* like it. > I am sad to know sysadmin no want try OmniOS, because the actual wiki > is not for newbies. > > Yesterday, i was at Linux and Docker french event, and i talked about > OmniOS and zones. > > But returns were ? "OmniOS wiki is good for Solaris admins !" > > OmniTI's team make a super job, but the OmniOS is not attractive for > newbies. > > it's sad :-( > > what can we'll do about that ? > > I am sure with Omnios's community we can publish an OmniOS Survival > guide. > > Yep ? Sure, go ahead... but do we really want to attract "newbies"? I don't know, just asking. -- Antoine From rt at steait.net Sat Nov 1 01:07:33 2014 From: rt at steait.net (Rune Tipsmark) Date: Sat, 1 Nov 2014 01:07:33 +0000 Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% In-Reply-To: References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net> <8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com> Message-ID: So actually started storage vmotions on 3 host, 6 concurrent and am getting about 1GB/sec Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more. I think this is resolved for now.... cpu sitting at 30-40% usage while moving 1GB/sec Iostat -xn 1 pool04 396G 39.5T 9 15.9K 325K 1.01G pool04 396G 39.5T 7 17.0K 270K 1.03G pool04 396G 39.5T 12 17.4K 558K 1.10G pool04 396G 39.5T 10 16.9K 442K 1.03G pool04 397G 39.5T 6 16.9K 332K 1021M pool04 397G 39.5T 1 16.3K 74.9K 1.01G pool04 397G 39.5T 8 17.0K 433K 1.05G pool04 397G 39.5T 20 17.1K 716K 1023M pool04 397G 39.5T 11 18.3K 425K 1.14G pool04 398G 39.5T 0 18.3K 65.9K 1.11G pool04 398G 39.5T 16 17.9K 551K 1.06G pool04 398G 39.5T 0 16.8K 105K 1.03G pool04 398G 39.5T 1 18.2K 124K 1.11G pool04 398G 39.5T 0 17.1K 45.9K 1.05G pool04 399G 39.5T 6 17.3K 454K 1.08G pool04 399G 39.5T 0 17.9K 0 1.06G pool04 399G 39.5T 2 16.9K 116K 1.04G pool04 399G 39.5T 2 18.8K 130K 1.09G pool04 399G 39.5T 0 17.6K 0 1.03G pool04 400G 39.5T 3 17.5K 155K 1.04G pool04 400G 39.5T 0 17.6K 31.5K 1.03G -----Original Message----- From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark Sent: Friday, October 31, 2014 12:38 PM To: Richard Elling; Eric Sproul Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% Ok, makes sense. What other kind of indicators can I look at? I get decent results from DD but still feels a bit slow... Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... Sync=always, block size 1M 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s real 4m56.382s user 0m0.461s sys 3m12.662s Sync=disabled, block size 1M 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s real 1m57.777s user 0m0.237s sys 1m57.466s ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each.... Now looking for aux power cables and will retest... Br, Rune -----Original Message----- From: Richard Elling [mailto:richard.elling at richardelling.com] Sent: Friday, October 31, 2014 9:03 AM To: Eric Sproul Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% On Oct 31, 2014, at 7:14 AM, Eric Sproul wrote: > On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark wrote: > >> Why is this pool showing near 100% busy when the underlying disks are >> doing nothing at all.... > > Simply put, it's just how the accounting works in iostat. It treats > the pool like any other device, so if there is even one outstanding > request to the pool, it counts towards the busy%. Keith W. from > Joyent explained this recently on the illumos-zfs list: > http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pag > e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/ > > The TL;DR is: if your pool has more than one disk in it, the pool-wide > busy% is useless. FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators. -- richard _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From admdwrf at gmail.com Sat Nov 1 01:21:17 2014 From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=) Date: Sat, 1 Nov 2014 02:21:17 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <20141101004433.GA12410@janus.obspm.bsdfrog.org> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: > Yeah but some others *do* like it. yes, like me, but i use Solaris since many years... >Sure, go ahead... but do we really want to attract "newbies"? >I don't know, just asking. Yes of course ! Do you want an IT world with just GNU/TuxTux ???? OmniOS vs Linux/Docker is amazing ! With OmniOS, we can have a real cpu and memory capping ! OpenSolaris is dead, and i think we must show OmniOS is an amazing alternative. We can't stay in our corner. We should show OmniOS Zones and KVM power :) And look how the network stack is manage in a Docker ... beurk !!! Few months ago, one of my client switch from Gnu/Linux to OmniOS and he tell me thank you every day. We should share and display what we can do with OmniOS. 2014-11-01 1:44 GMT+01:00 Antoine Jacoutot : > > Few days ago, i was at linux hack day at Rennes in France. I talked > > about OmniOS. > > > > Some admin sys were very interesting about it. I restarted a blog at > > [1]blog.fredalix.com > > > > But, in my country, some peoples know SmartOS, and when they go to > > OmniOS website, they don't like it. > > Yeah but some others *do* like it. > > > I am sad to know sysadmin no want try OmniOS, because the actual wiki > > is not for newbies. > > > > Yesterday, i was at Linux and Docker french event, and i talked about > > OmniOS and zones. > > > > But returns were ? "OmniOS wiki is good for Solaris admins !" > > > > OmniTI's team make a super job, but the OmniOS is not attractive for > > newbies. > > > > it's sad :-( > > > > what can we'll do about that ? > > > > I am sure with Omnios's community we can publish an OmniOS Survival > > guide. > > > > Yep ? > > Sure, go ahead... but do we really want to attract "newbies"? > I don't know, just asking. > > -- > Antoine > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesus at omniti.com Sat Nov 1 06:17:44 2014 From: jesus at omniti.com (Theo Schlossnagle) Date: Sat, 1 Nov 2014 10:17:44 +0400 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <20141101004433.GA12410@janus.obspm.bsdfrog.org> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: On Sat, Nov 1, 2014 at 3:44 AM, Antoine Jacoutot wrote: > > > I am sure with Omnios's community we can publish an OmniOS Survival > > guide. > > > > Yep ? > > Sure, go ahead... but do we really want to attract "newbies"? > I don't know, just asking. > I think an OmniOS Survival guide would be great. If you have to put something like "newbies" in quotes and talk about them as different than us, you are viewing this community in an unhealthy way. OmniOS is designed to take Illumos and all of its features and package it in a way that makes a minimal and useful general operating system targeted at servers and appliances. Anyone with whom that vision aligns is most welcome in the community and we should encourage efforts to make them feel as an accepted part of the whole. -- Theo Schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Sat Nov 1 21:56:52 2014 From: danmcd at omniti.com (Dan McDonald) Date: Sat, 1 Nov 2014 17:56:52 -0400 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> > On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix wrote: > > >Sure, go ahead... but do we really want to attract "newbies"? > >I don't know, just asking. > Yes of course ! Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. Of course! We don't grow without newbies. We should treat them with kindness, respect, a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly). > We should share and display what we can do with OmniOS. OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be." BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. I won't have it. I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems. If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. Specific examples of what people think will help are appreciated. Dan From mir at miras.org Sat Nov 1 22:44:14 2014 From: mir at miras.org (Michael Rasmussen) Date: Sat, 1 Nov 2014 23:44:14 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> Message-ID: <20141101234414.43f42589@sleipner.datanom.net> On Sat, 1 Nov 2014 17:56:52 -0400 Dan McDonald wrote: > > If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. Specific examples of what people think will help are appreciated. > If we want to be more friendly to newcomers some kind of basic stuff is needed. I can remember my own first journey into Omnios which demanded endless searches and readings from Oracle (Solaris), OpenIndiana, Nexenta, SmartOS, Google, and Youtube etc. And of course reading Omniti and this list. A lot of stuff from Oracle is good but cannot always be used directly since Omnios/Illumos is somewhere in between Solaris 10 and Solaris 11. What would have helped me a great deal would have been able to get all this stuff from one source, and a source with examples directly able to be used. First I think a list of hardware known to work and not known to work would be great. The list should be divided into a section for each stable and LTS version. Something like: 1) CPU 2) Chipset 3) GPU 4) Controller. Onboard, HBA and RAID 5) Nics 6) Motherboards An something more which I am sure I have forgotten;-) I my opinion the best thing would be a web based searchable database. It should be open to everyone registered and everyone should be encourage to add there own findings. Second, I really mis some good tutorials about various subjects. This could be implemented in some kind of wiki. It should be open to everyone registered and everyone should be encourage to add input they think would be of interest to others. There is nothing better than to stand on the shoulders of other wise guys. Third, a section of typical problems users can find them selfs in with guides to either solve the problem or if to solution is found in the section yet and guide to extract all needed information to be send to the list when asking for help. Providing a list of required information to the ones able to help you as part of the first request on the list is more likely to receive a swift answer with, hopefully, the correct solution. As I am very pleased with Omnios I would love to see its popularity rise so I will participate to the best of my knowledge and spare time. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- /usr/games/fortune -es says: It would be illogical to assume that all conditions remain stable. -- Spock, "The Enterprise Incident", stardate 5027.3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From lkateley at kateley.com Sat Nov 1 22:46:03 2014 From: lkateley at kateley.com (Linda Kateley) Date: Sat, 01 Nov 2014 17:46:03 -0500 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: <545562AB.30006@kateley.com> I actually really enjoy doing little how-to videos. I put up a set a couple weeks ago on zfs http://kateleyco.com/?page_id=783 I have taught solaris for most of my career.. so I plan on creating some more.. intro to dtrace.. services.. fma please feel free to link to them or republish or consume. I did them on omni. linda On 10/31/14, 8:21 PM, Fr?d?ric Alix wrote: > > Yeah but some others *do* like it. > yes, like me, but i use Solaris since many years... > > >Sure, go ahead... but do we really want to attract "newbies"? > >I don't know, just asking. > Yes of course ! > Do you want an IT world with just GNU/TuxTux ???? > OmniOS vs Linux/Docker is amazing ! > With OmniOS, we can have a real cpu and memory capping ! > OpenSolaris is dead, and i think we must show OmniOS is an amazing > alternative. > We can't stay in our corner. We should show OmniOS Zones and KVM power :) > And look how the network stack is manage in a Docker ... beurk !!! > Few months ago, one of my client switch from Gnu/Linux to OmniOS and > he tell me thank you every day. > We should share and display what we can do with OmniOS. > > > > > 2014-11-01 1:44 GMT+01:00 Antoine Jacoutot >: > > > Few days ago, i was at linux hack day at Rennes in France. I talked > > about OmniOS. > > > > Some admin sys were very interesting about it. I restarted a > blog at > > [1]blog.fredalix.com > > > > But, in my country, some peoples know SmartOS, and when they > go to > > OmniOS website, they don't like it. > > Yeah but some others *do* like it. > > > I am sad to know sysadmin no want try OmniOS, because the > actual wiki > > is not for newbies. > > > > Yesterday, i was at Linux and Docker french event, and i > talked about > > OmniOS and zones. > > > > But returns were ? "OmniOS wiki is good for Solaris admins !" > > > > OmniTI's team make a super job, but the OmniOS is not > attractive for > > newbies. > > > > it's sad :-( > > > > what can we'll do about that ? > > > > I am sure with Omnios's community we can publish an OmniOS > Survival > > guide. > > > > Yep ? > > Sure, go ahead... but do we really want to attract "newbies"? > I don't know, just asking. > > -- > Antoine > > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From cks at cs.toronto.edu Sat Nov 1 22:48:19 2014 From: cks at cs.toronto.edu (Chris Siebenmann) Date: Sat, 01 Nov 2014 18:48:19 -0400 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: Your message of Sat, 01 Nov 2014 17:56:52 -0400. <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> Message-ID: <20141101224819.2365A5A0974@testapps.cs.toronto.edu> > I do believe, and someone here mentioned this to me offline, that we > might need to promote OmniOS a bit more. I'm not sure how best to > do that, modulo people like the folks here telling their managers, > friends, and customers about how OmniOS, quite simply, Helps You Solve > Problems. One problem with doing such promotion today is that for quite a lot of people, OmniOS does not solve your problems but is instead just a building block for doing so. This is distinctly different from, say, FreeBSD or a random Linux distributions, which *will* solve many people's problems out of the box. Here I'm talking about problems like 'I need a mail server' or 'I need an IMAP server' or 'I need a web server' or 'I need a DB server'. (I am on the mailing list for Exim, and I can assure you that there are many people who show up there wanting an out of the box 'solve my problems' experience with Exim on their system, even for unusual problems. These people have not even seriously read the documentation for Exim.) I understand and accept that OmniOS is never going to explicitly package such things. What I think would help here is a curated list of additional package sources with explicit 'where to get ' for popular s. This would give people who want their problems solved an easily found and explicitly documented two-step process for doing so, in contrast to the current approach where you get to do a bunch of searching (and experimentation and guessing about which is the right package source to use and so on). - cks From rt at steait.net Sun Nov 2 00:45:44 2014 From: rt at steait.net (Rune Tipsmark) Date: Sun, 2 Nov 2014 00:45:44 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Message-ID: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net> Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed... Br, Rune -------------- next part -------------- An HTML attachment was scrubbed... URL: From turbo124 at gmail.com Sun Nov 2 01:01:04 2014 From: turbo124 at gmail.com (David Bomba) Date: Sun, 2 Nov 2014 12:01:04 +1100 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net> References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net> Message-ID: I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. > On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: > > Hi all, > > Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. > > If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed? > > Br, > Rune > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From rt at steait.net Sun Nov 2 06:14:21 2014 From: rt at steait.net (Rune Tipsmark) Date: Sun, 2 Nov 2014 06:14:21 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net> Message-ID: <8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net> Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html Br, Rune From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed? Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From turbo124 at gmail.com Sun Nov 2 06:16:13 2014 From: turbo124 at gmail.com (David Bomba) Date: Sun, 2 Nov 2014 17:16:13 +1100 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: <8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net> References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net> <8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net> Message-ID: <70929479-3ED6-4E12-98F6-7EF9E8ECFD1D@gmail.com> The oracle docs have a step by step for setting up target and initiator. http://docs.oracle.com/cd/E23824_01/html/821-1459/fnnop.html > On 2 Nov 2014, at 5:14 pm, Rune Tipsmark wrote: > > Sounds sensible, how do I do that? > I tried creating a view for a thin lu with my other zfs box, but how do I detect it? > > I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html > > Br, > Rune > > From: David Bomba [mailto:turbo124 at gmail.com] > Sent: Saturday, November 01, 2014 6:01 PM > To: Rune Tipsmark > Cc: omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol > > I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. > > > On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: > > Hi all, > > Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. > > If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed? > > Br, > Rune > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Sun Nov 2 08:26:03 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sun, 2 Nov 2014 09:26:03 +0100 Subject: [OmniOS-discuss] Ang: Re: OmniOS doc In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> References: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>, <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: The answer to this discussion thread would in my opinion be: How do we design, manage and maintain the omnios public documentation? I believe that is a project to take seriously... Johan -----"OmniOS-discuss" skrev: ----- Till: omnios-discuss Fr?n: Dan McDonald S?nt av: "OmniOS-discuss" Datum: 2014-11-01 22:59 ?rende: Re: [OmniOS-discuss] OmniOS doc > On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix wrote: > > >Sure, go ahead... but do we really want to attract "newbies"? > >I don't know, just asking. > Yes of course ! Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. ?Of course! ?We don't grow without newbies. ?We should treat them with kindness, respect, ?a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly). > We should share and display what we can do with OmniOS. OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be." BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. ?I won't have it. I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. ?I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems. If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. ?Specific examples of what people think will help are appreciated. Dan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From johan.kragsterman at capvert.se Sun Nov 2 10:02:13 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sun, 2 Nov 2014 11:02:13 +0100 Subject: [OmniOS-discuss] Ang: Ang: Re: OmniOS doc In-Reply-To: References: , <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>, <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: -----"OmniOS-discuss" skrev: ----- Till: Dan McDonald Fr?n: Johan Kragsterman S?nt av: "OmniOS-discuss" Datum: 2014-11-02 09:27 Kopia: omnios-discuss ?rende: [OmniOS-discuss] Ang: Re: OmniOS doc The answer to this discussion thread would in my opinion be: How do we design, manage and maintain the omnios public documentation? I believe that is a project to take seriously... Johan To reply and add to my own opinon, I'd like to continue with: I'd believe it is OmniTI that has the possibility to start a project around a subject like this. The discussion about it should of coarse be here, on omnios-discuss, but I believe OmniTI should tell us which infrastructure solutions that would be possible to provide for it. ?If OmniTI can provide appropriate infrastructure, I believe the community can provide content. The mantainance of the solution?will of coarse be of great importance, and will of coarse also need man hours, and how this should be done, is a matter of this community to discuss... Johan -----"OmniOS-discuss" skrev: ----- Till: omnios-discuss Fr?n: Dan McDonald S?nt av: "OmniOS-discuss" Datum: 2014-11-01 22:59 ?rende: Re: [OmniOS-discuss] OmniOS doc > On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix wrote: > > >Sure, go ahead... but do we really want to attract "newbies"? > >I don't know, just asking. > Yes of course ! Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. ?Of course! ?We don't grow without newbies. ?We should treat them with kindness, respect, ?a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly). > We should share and display what we can do with OmniOS. OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be." BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. ?I won't have it. I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. ?I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems. If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. ?Specific examples of what people think will help are appreciated. Dan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From tobi at oetiker.ch Sun Nov 2 12:15:20 2014 From: tobi at oetiker.ch (Tobias Oetiker) Date: Sun, 2 Nov 2014 13:15:20 +0100 (CET) Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> Message-ID: Yesterday Dan McDonald wrote: > If we need more docs because it will attract people to OmniOS and > help it grow, I'm all for it. Specific examples of what people > think will help are appreciated. what I notice, when searching for illumos/omnios information I often end up on the oracle website ... an easy way to remedy this in part would be to publish the illumos manual pages in a friendly way on the web, so that google searches have somewhere to go. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From tim at multitalents.net Sun Nov 2 16:21:32 2014 From: tim at multitalents.net (Tim Rice) Date: Sun, 2 Nov 2014 08:21:32 -0800 (PST) Subject: [OmniOS-discuss] Compiling GNUCOBOL : pre-requisites present? In-Reply-To: <20141027164015.GA17098@hp-aio> References: <85f940caa925131c4e59fe2b99c0ae89@kathe.in> <6F3F930C-13BC-4459-B212-7A32EAA757F4@zinascii.com> <20141027164015.GA17098@hp-aio> Message-ID: On Mon, 27 Oct 2014, Mayuresh Kathe wrote: [snip] > > http://omnios.omniti.com/wiki.php/Packaging > > I did what you'd suggested, got BDB installed, but somehow the configure > script is unable to find "gmp.h" at "/usr/include/gmp/". > Is there anything I need to do to make the system know that gmp.h exists > at "/usr/include/gmp/"? Try CPPFLAGS="/usr/include/gmp" ./configure ....... > > Thanks, > > ~Mayuresh > -- Tim Rice Multitalents tim at multitalents.net From johan.kragsterman at capvert.se Sun Nov 2 17:56:12 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sun, 2 Nov 2014 18:56:12 +0100 Subject: [OmniOS-discuss] Ang: Re: zfs send via SRP or other RDMA enabled protocol Message-ID: -----"OmniOS-discuss" skrev: ----- Till: David Bomba Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan ? I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html ? Br, Rune ? From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol ? I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.? ? ? On 2 Nov 2014, at 11:45 am, Rune Tipsmark wrote: ? Hi ?all, ? Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. ? If not, is there a way I can aggregate say 8 or 16? IPoIB partitions and push throughput to a more reasonable speed… ? Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ? _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From rt at steait.net Sun Nov 2 18:10:41 2014 From: rt at steait.net (Rune Tipsmark) Date: Sun, 2 Nov 2014 18:10:41 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: Message-ID: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net> Hi Johan Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine.. What I can't figure out is how to share a LUN with the other zfs box... see pasted info below... ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box root at zfs00:/pool03# stmfadm list-target -v Target: eui.0002C90300095E7C Operational Status: Online Provider Name : srpt Alias : - Protocol : SRP Sessions : 8 Initiator: eui.0002C903000F397C Alias: 8102c90300095e7e:0002c903000f397c Logged in since: Sun Nov 2 02:09:41 2014 Initiator: eui.0002C903000F397B Alias: 8102c90300095e7d:0002c903000f397b Logged in since: Sun Nov 2 02:09:40 2014 Initiator: eui.0002C903000D3D04 Alias: 8102c90300095e7e:0002c903000d3d04 Logged in since: Sat Nov 1 21:14:47 2014 Initiator: eui.0002C90300104F47 Alias: 8102c90300095e7d:0002c90300104f47 Logged in since: Sat Nov 1 21:12:54 2014 Initiator: eui.0002C903000D3D03 Alias: 8102c90300095e7d:0002c903000d3d03 Logged in since: Sat Nov 1 21:12:32 2014 Initiator: eui.0002C90300104F48 Alias: 8102c90300095e7e:0002c90300104f48 Logged in since: Sat Nov 1 21:10:45 2014 Initiator: eui.0002C903000A48FA Alias: 8102c90300095e7e:0002c903000a48fa Logged in since: Sat Nov 1 21:10:40 2014 Initiator: eui.0002C903000D3CA0 Alias: 8102c90300095e7e:0002c903000d3ca0 Logged in since: Sat Nov 1 21:10:39 2014 Target: iqn.2010-09.org.napp-it:1394106801 Operational Status: Online Provider Name : iscsit Alias : 03.06.2014 Protocol : iSCSI Sessions : 0 root at zfs00:/pool03# stmfadm list-lu -v LU Name: 600144F007780B7F00005455EDD50002 Operational Status: Online Provider Name : sbd Alias : /pool03/LU11 View Entry Count : 1 Data File : /pool03/LU11 Meta File : not set Size : 2199023255552 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active root at zfs00:/pool03# stmfadm list-hg -v Host Group: ESX Host Group: Windows Host Group: ESX-iSER Host Group: OmniOS Member: eui.0002C903000923E6 <<--- The other ZFS box Member: iqn.2010-09.org.napp-it:1402013225 Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0 Host group : OmniOS Target group : All LUN : 0 ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00... root at zfs10:/root# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t5000C50055FC9533d0 /scsi_vhci/disk at g5000c50055fc9533 1. c4t5000C50055FE6A63d0 /scsi_vhci/disk at g5000c50055fe6a63 2. c4t5000C500625B7EA7d0 /scsi_vhci/disk at g5000c500625b7ea7 3. c4t5000C500625B86E3d0 /scsi_vhci/disk at g5000c500625b86e3 4. c4t5000C500625B886Fd0 /scsi_vhci/disk at g5000c500625b886f 5. c4t5000C500625B8137d0 /scsi_vhci/disk at g5000c500625b8137 6. c4t5000C500625B8427d0 /scsi_vhci/disk at g5000c500625b8427 7. c4t5000C500625BB773d0 /scsi_vhci/disk at g5000c500625bb773 8. c4t5000C500625BC2C3d0 /scsi_vhci/disk at g5000c500625bc2c3 9. c4t5000C500625BD3EBd0 /scsi_vhci/disk at g5000c500625bd3eb 10. c4t5000C50057085A6Bd0 /scsi_vhci/disk at g5000c50057085a6b 11. c4t5000C50057086B67d0 /scsi_vhci/disk at g5000c50057086b67 12. c4t5000C50062878C0Bd0 /scsi_vhci/disk at g5000c50062878c0b 13. c4t5000C50062878C43d0 /scsi_vhci/disk at g5000c50062878c43 14. c4t5000C500570858EFd0 /scsi_vhci/disk at g5000c500570858ef 15. c4t5000C500570870D3d0 /scsi_vhci/disk at g5000c500570870d3 16. c4t5000C5005708351Bd0 /scsi_vhci/disk at g5000c5005708351b 17. c4t5000C5005708296Fd0 /scsi_vhci/disk at g5000c5005708296f 18. c4t5000C50057089753d0 /scsi_vhci/disk at g5000c50057089753 19. c4t5000C50057086307d0 /scsi_vhci/disk at g5000c50057086307 20. c4t5000C50062879687d0 /scsi_vhci/disk at g5000c50062879687 21. c4t5000C50062879707d0 /scsi_vhci/disk at g5000c50062879707 22. c4t5000C50062879723d0 /scsi_vhci/disk at g5000c50062879723 23. c4t5000C50062879787d0 /scsi_vhci/disk at g5000c50062879787 24. c7t0d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0 25. c7t1d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0 26. c7t4d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0 27. c7t5d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0 28. c10d0 /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0 29. c11d0 /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0 30. c12d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 31. c13d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 32. c14d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 33. c15d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 Specify disk (enter its number): -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 9:56 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----"OmniOS-discuss" skrev: ----- Till: David Bomba Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan ? I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html ? Br, Rune ? From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol ? I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.? ? ? On 2 Nov 2014, at 11:45 am, Rune Tipsmark wrote: ? Hi ?all, ? Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. ? If not, is there a way I can aggregate say 8 or 16? IPoIB partitions and push throughput to a more reasonable speed… ? Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ? _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From johan.kragsterman at capvert.se Sun Nov 2 18:33:19 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Sun, 2 Nov 2014 19:33:19 +0100 Subject: [OmniOS-discuss] Ang: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net> References: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From rt at steait.net Sun Nov 2 19:05:10 2014 From: rt at steait.net (Rune Tipsmark) Date: Sun, 2 Nov 2014 19:05:10 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net> I know, but how do I initiate a session from ZFS10? Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:33 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 19:11 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hi Johan Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine.. What I can't figure out is how to share a LUN with the other zfs box... see pasted info below... ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box root at zfs00:/pool03# stmfadm list-target -v Target: eui.0002C90300095E7C Operational Status: Online Provider Name : srpt Alias : - Protocol : SRP Sessions : 8 Initiator: eui.0002C903000F397C Alias: 8102c90300095e7e:0002c903000f397c Logged in since: Sun Nov 2 02:09:41 2014 Initiator: eui.0002C903000F397B Alias: 8102c90300095e7d:0002c903000f397b Logged in since: Sun Nov 2 02:09:40 2014 Initiator: eui.0002C903000D3D04 Alias: 8102c90300095e7e:0002c903000d3d04 Logged in since: Sat Nov 1 21:14:47 2014 Initiator: eui.0002C90300104F47 Alias: 8102c90300095e7d:0002c90300104f47 Logged in since: Sat Nov 1 21:12:54 2014 Initiator: eui.0002C903000D3D03 Alias: 8102c90300095e7d:0002c903000d3d03 Logged in since: Sat Nov 1 21:12:32 2014 Initiator: eui.0002C90300104F48 Alias: 8102c90300095e7e:0002c90300104f48 Logged in since: Sat Nov 1 21:10:45 2014 Initiator: eui.0002C903000A48FA Alias: 8102c90300095e7e:0002c903000a48fa Logged in since: Sat Nov 1 21:10:40 2014 Initiator: eui.0002C903000D3CA0 Alias: 8102c90300095e7e:0002c903000d3ca0 Logged in since: Sat Nov 1 21:10:39 2014 Target: iqn.2010-09.org.napp-it:1394106801 Operational Status: Online Provider Name : iscsit Alias : 03.06.2014 Protocol : iSCSI Sessions : 0 root at zfs00:/pool03# stmfadm list-lu -v LU Name: 600144F007780B7F00005455EDD50002 Operational Status: Online Provider Name : sbd Alias : /pool03/LU11 View Entry Count : 1 Data File : /pool03/LU11 Meta File : not set Size : 2199023255552 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active root at zfs00:/pool03# stmfadm list-hg -v Host Group: ESX Host Group: Windows Host Group: ESX-iSER Host Group: OmniOS Member: eui.0002C903000923E6 <<--- The other ZFS box Member: iqn.2010-09.org.napp-it:1402013225 Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0 Host group : OmniOS Target group : All LUN : 0 ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00... Hi! You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target. Rgrds Johan root at zfs10:/root# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t5000C50055FC9533d0 /scsi_vhci/disk at g5000c50055fc9533 1. c4t5000C50055FE6A63d0 /scsi_vhci/disk at g5000c50055fe6a63 2. c4t5000C500625B7EA7d0 /scsi_vhci/disk at g5000c500625b7ea7 3. c4t5000C500625B86E3d0 /scsi_vhci/disk at g5000c500625b86e3 4. c4t5000C500625B886Fd0 /scsi_vhci/disk at g5000c500625b886f 5. c4t5000C500625B8137d0 /scsi_vhci/disk at g5000c500625b8137 6. c4t5000C500625B8427d0 /scsi_vhci/disk at g5000c500625b8427 7. c4t5000C500625BB773d0 /scsi_vhci/disk at g5000c500625bb773 8. c4t5000C500625BC2C3d0 /scsi_vhci/disk at g5000c500625bc2c3 9. c4t5000C500625BD3EBd0 /scsi_vhci/disk at g5000c500625bd3eb 10. c4t5000C50057085A6Bd0 /scsi_vhci/disk at g5000c50057085a6b 11. c4t5000C50057086B67d0 /scsi_vhci/disk at g5000c50057086b67 12. c4t5000C50062878C0Bd0 /scsi_vhci/disk at g5000c50062878c0b 13. c4t5000C50062878C43d0 /scsi_vhci/disk at g5000c50062878c43 14. c4t5000C500570858EFd0 /scsi_vhci/disk at g5000c500570858ef 15. c4t5000C500570870D3d0 /scsi_vhci/disk at g5000c500570870d3 16. c4t5000C5005708351Bd0 /scsi_vhci/disk at g5000c5005708351b 17. c4t5000C5005708296Fd0 /scsi_vhci/disk at g5000c5005708296f 18. c4t5000C50057089753d0 /scsi_vhci/disk at g5000c50057089753 19. c4t5000C50057086307d0 /scsi_vhci/disk at g5000c50057086307 20. c4t5000C50062879687d0 /scsi_vhci/disk at g5000c50062879687 21. c4t5000C50062879707d0 /scsi_vhci/disk at g5000c50062879707 22. c4t5000C50062879723d0 /scsi_vhci/disk at g5000c50062879723 23. c4t5000C50062879787d0 /scsi_vhci/disk at g5000c50062879787 24. c7t0d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0 25. c7t1d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0 26. c7t4d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0 27. c7t5d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0 28. c10d0 /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0 29. c11d0 /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0 30. c12d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 31. c13d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 32. c14d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 33. c15d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 Specify disk (enter its number): -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 9:56 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----"OmniOS-discuss" > skrev: ----- Till: David Bomba > Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html Br, Rune From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed... Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoffn at gnaa.net Sun Nov 2 23:34:47 2014 From: geoffn at gnaa.net (Geoff Nordli) Date: Sun, 02 Nov 2014 15:34:47 -0800 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com> Message-ID: <5456BF97.30603@gnaa.net> On 14-11-01 02:56 PM, Dan McDonald wrote: >> On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix wrote: >> >>> Sure, go ahead... but do we really want to attract "newbies"? >>> I don't know, just asking. >> Yes of course ! > Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. Of course! We don't grow without newbies. We should treat them with kindness, respect, a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly). > >> We should share and display what we can do with OmniOS. > OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be." > > BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. I won't have it. > > I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems. > > If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. Specific examples of what people think will help are appreciated. > > How Tos are a good documentation method. One of the issues is knowing if the method you are following from some blog/document is actually the right method for OmniOS or some other solaris derivative. Geoff From richard.elling at richardelling.com Mon Nov 3 02:07:00 2014 From: richard.elling at richardelling.com (Richard Elling) Date: Sun, 2 Nov 2014 18:07:00 -0800 Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% In-Reply-To: References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net> <8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com> Message-ID: <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com> On Oct 31, 2014, at 6:07 PM, Rune Tipsmark wrote: > So actually started storage vmotions on 3 host, 6 concurrent and am getting about 1GB/sec > Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more. > > I think this is resolved for now.... cpu sitting at 30-40% usage while moving 1GB/sec Yes, that seems about right. -- richard > > Iostat -xn 1 > pool04 396G 39.5T 9 15.9K 325K 1.01G > pool04 396G 39.5T 7 17.0K 270K 1.03G > pool04 396G 39.5T 12 17.4K 558K 1.10G > pool04 396G 39.5T 10 16.9K 442K 1.03G > pool04 397G 39.5T 6 16.9K 332K 1021M > pool04 397G 39.5T 1 16.3K 74.9K 1.01G > pool04 397G 39.5T 8 17.0K 433K 1.05G > pool04 397G 39.5T 20 17.1K 716K 1023M > pool04 397G 39.5T 11 18.3K 425K 1.14G > pool04 398G 39.5T 0 18.3K 65.9K 1.11G > pool04 398G 39.5T 16 17.9K 551K 1.06G > pool04 398G 39.5T 0 16.8K 105K 1.03G > pool04 398G 39.5T 1 18.2K 124K 1.11G > pool04 398G 39.5T 0 17.1K 45.9K 1.05G > pool04 399G 39.5T 6 17.3K 454K 1.08G > pool04 399G 39.5T 0 17.9K 0 1.06G > pool04 399G 39.5T 2 16.9K 116K 1.04G > pool04 399G 39.5T 2 18.8K 130K 1.09G > pool04 399G 39.5T 0 17.6K 0 1.03G > pool04 400G 39.5T 3 17.5K 155K 1.04G > pool04 400G 39.5T 0 17.6K 31.5K 1.03G > > -----Original Message----- > From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark > Sent: Friday, October 31, 2014 12:38 PM > To: Richard Elling; Eric Sproul > Cc: omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% > > Ok, makes sense. > What other kind of indicators can I look at? > > I get decent results from DD but still feels a bit slow... > > Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... > > Sync=always, block size 1M > 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s > real 4m56.382s > user 0m0.461s > sys 3m12.662s > > Sync=disabled, block size 1M > 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s > real 1m57.777s > user 0m0.237s > sys 1m57.466s > > ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each.... > > Now looking for aux power cables and will retest... > > Br, > Rune > > -----Original Message----- > From: Richard Elling [mailto:richard.elling at richardelling.com] > Sent: Friday, October 31, 2014 9:03 AM > To: Eric Sproul > Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% > > > On Oct 31, 2014, at 7:14 AM, Eric Sproul wrote: > >> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark wrote: >> >>> Why is this pool showing near 100% busy when the underlying disks are >>> doing nothing at all.... >> >> Simply put, it's just how the accounting works in iostat. It treats >> the pool like any other device, so if there is even one outstanding >> request to the pool, it counts towards the busy%. Keith W. from >> Joyent explained this recently on the illumos-zfs list: >> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pag >> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/ >> >> The TL;DR is: if your pool has more than one disk in it, the pool-wide >> busy% is useless. > > FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators. > -- richard > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From rt at steait.net Mon Nov 3 02:24:29 2014 From: rt at steait.net (Rune Tipsmark) Date: Mon, 3 Nov 2014 02:24:29 +0000 Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% In-Reply-To: <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com> References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net> <8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com> <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com> Message-ID: <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net> Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring? How does compression affect the numbers? Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual? Br, Rune -----Original Message----- From: Richard Elling [mailto:richard.elling at richardelling.com] Sent: Sunday, November 02, 2014 6:07 PM To: Rune Tipsmark Cc: Eric Sproul; omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% On Oct 31, 2014, at 6:07 PM, Rune Tipsmark wrote: > So actually started storage vmotions on 3 host, 6 concurrent and am > getting about 1GB/sec Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more. > > I think this is resolved for now.... cpu sitting at 30-40% usage while > moving 1GB/sec Yes, that seems about right. -- richard > > Iostat -xn 1 > pool04 396G 39.5T 9 15.9K 325K 1.01G > pool04 396G 39.5T 7 17.0K 270K 1.03G > pool04 396G 39.5T 12 17.4K 558K 1.10G > pool04 396G 39.5T 10 16.9K 442K 1.03G > pool04 397G 39.5T 6 16.9K 332K 1021M > pool04 397G 39.5T 1 16.3K 74.9K 1.01G > pool04 397G 39.5T 8 17.0K 433K 1.05G > pool04 397G 39.5T 20 17.1K 716K 1023M > pool04 397G 39.5T 11 18.3K 425K 1.14G > pool04 398G 39.5T 0 18.3K 65.9K 1.11G > pool04 398G 39.5T 16 17.9K 551K 1.06G > pool04 398G 39.5T 0 16.8K 105K 1.03G > pool04 398G 39.5T 1 18.2K 124K 1.11G > pool04 398G 39.5T 0 17.1K 45.9K 1.05G > pool04 399G 39.5T 6 17.3K 454K 1.08G > pool04 399G 39.5T 0 17.9K 0 1.06G > pool04 399G 39.5T 2 16.9K 116K 1.04G > pool04 399G 39.5T 2 18.8K 130K 1.09G > pool04 399G 39.5T 0 17.6K 0 1.03G > pool04 400G 39.5T 3 17.5K 155K 1.04G > pool04 400G 39.5T 0 17.6K 31.5K 1.03G > > -----Original Message----- > From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] > On Behalf Of Rune Tipsmark > Sent: Friday, October 31, 2014 12:38 PM > To: Richard Elling; Eric Sproul > Cc: omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% > > Ok, makes sense. > What other kind of indicators can I look at? > > I get decent results from DD but still feels a bit slow... > > Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... > > Sync=always, block size 1M > 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s > real 4m56.382s > user 0m0.461s > sys 3m12.662s > > Sync=disabled, block size 1M > 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s > real 1m57.777s > user 0m0.237s > sys 1m57.466s > > ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each.... > > Now looking for aux power cables and will retest... > > Br, > Rune > > -----Original Message----- > From: Richard Elling [mailto:richard.elling at richardelling.com] > Sent: Friday, October 31, 2014 9:03 AM > To: Eric Sproul > Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% > > > On Oct 31, 2014, at 7:14 AM, Eric Sproul wrote: > >> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark wrote: >> >>> Why is this pool showing near 100% busy when the underlying disks >>> are doing nothing at all.... >> >> Simply put, it's just how the accounting works in iostat. It treats >> the pool like any other device, so if there is even one outstanding >> request to the pool, it counts towards the busy%. Keith W. from >> Joyent explained this recently on the illumos-zfs list: >> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pa >> g >> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/ >> >> The TL;DR is: if your pool has more than one disk in it, the >> pool-wide busy% is useless. > > FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators. > -- richard > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From johan.kragsterman at capvert.se Mon Nov 3 06:49:08 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 3 Nov 2014 07:49:08 +0100 Subject: [OmniOS-discuss] Ang: RE: RE: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From rt at steait.net Mon Nov 3 07:09:06 2014 From: rt at steait.net (Rune Tipsmark) Date: Mon, 3 Nov 2014 07:09:06 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net> connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out. Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:49 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! I can see I forgot the list in previous mail... Anyway, are you using connectx adapters? Have you checked if the driver is loaded in the host system? Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 00:07 ?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up... Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM. Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to... It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step? Br Rune -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 11:58 AM To: Rune Tipsmark Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 20:05 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I know, but how do I initiate a session from ZFS10? If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager? Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:33 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 19:11 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hi Johan Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine.. What I can't figure out is how to share a LUN with the other zfs box... see pasted info below... ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box root at zfs00:/pool03# stmfadm list-target -v Target: eui.0002C90300095E7C Operational Status: Online Provider Name : srpt Alias : - Protocol : SRP Sessions : 8 Initiator: eui.0002C903000F397C Alias: 8102c90300095e7e:0002c903000f397c Logged in since: Sun Nov 2 02:09:41 2014 Initiator: eui.0002C903000F397B Alias: 8102c90300095e7d:0002c903000f397b Logged in since: Sun Nov 2 02:09:40 2014 Initiator: eui.0002C903000D3D04 Alias: 8102c90300095e7e:0002c903000d3d04 Logged in since: Sat Nov 1 21:14:47 2014 Initiator: eui.0002C90300104F47 Alias: 8102c90300095e7d:0002c90300104f47 Logged in since: Sat Nov 1 21:12:54 2014 Initiator: eui.0002C903000D3D03 Alias: 8102c90300095e7d:0002c903000d3d03 Logged in since: Sat Nov 1 21:12:32 2014 Initiator: eui.0002C90300104F48 Alias: 8102c90300095e7e:0002c90300104f48 Logged in since: Sat Nov 1 21:10:45 2014 Initiator: eui.0002C903000A48FA Alias: 8102c90300095e7e:0002c903000a48fa Logged in since: Sat Nov 1 21:10:40 2014 Initiator: eui.0002C903000D3CA0 Alias: 8102c90300095e7e:0002c903000d3ca0 Logged in since: Sat Nov 1 21:10:39 2014 Target: iqn.2010-09.org.napp-it:1394106801 Operational Status: Online Provider Name : iscsit Alias : 03.06.2014 Protocol : iSCSI Sessions : 0 root at zfs00:/pool03# stmfadm list-lu -v LU Name: 600144F007780B7F00005455EDD50002 Operational Status: Online Provider Name : sbd Alias : /pool03/LU11 View Entry Count : 1 Data File : /pool03/LU11 Meta File : not set Size : 2199023255552 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active root at zfs00:/pool03# stmfadm list-hg -v Host Group: ESX Host Group: Windows Host Group: ESX-iSER Host Group: OmniOS Member: eui.0002C903000923E6 <<--- The other ZFS box Member: iqn.2010-09.org.napp-it:1402013225 Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0 Host group : OmniOS Target group : All LUN : 0 ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00... Hi! You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target. Rgrds Johan root at zfs10:/root# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t5000C50055FC9533d0 /scsi_vhci/disk at g5000c50055fc9533 1. c4t5000C50055FE6A63d0 /scsi_vhci/disk at g5000c50055fe6a63 2. c4t5000C500625B7EA7d0 /scsi_vhci/disk at g5000c500625b7ea7 3. c4t5000C500625B86E3d0 /scsi_vhci/disk at g5000c500625b86e3 4. c4t5000C500625B886Fd0 /scsi_vhci/disk at g5000c500625b886f 5. c4t5000C500625B8137d0 /scsi_vhci/disk at g5000c500625b8137 6. c4t5000C500625B8427d0 /scsi_vhci/disk at g5000c500625b8427 7. c4t5000C500625BB773d0 /scsi_vhci/disk at g5000c500625bb773 8. c4t5000C500625BC2C3d0 /scsi_vhci/disk at g5000c500625bc2c3 9. c4t5000C500625BD3EBd0 /scsi_vhci/disk at g5000c500625bd3eb 10. c4t5000C50057085A6Bd0 /scsi_vhci/disk at g5000c50057085a6b 11. c4t5000C50057086B67d0 /scsi_vhci/disk at g5000c50057086b67 12. c4t5000C50062878C0Bd0 /scsi_vhci/disk at g5000c50062878c0b 13. c4t5000C50062878C43d0 /scsi_vhci/disk at g5000c50062878c43 14. c4t5000C500570858EFd0 /scsi_vhci/disk at g5000c500570858ef 15. c4t5000C500570870D3d0 /scsi_vhci/disk at g5000c500570870d3 16. c4t5000C5005708351Bd0 /scsi_vhci/disk at g5000c5005708351b 17. c4t5000C5005708296Fd0 /scsi_vhci/disk at g5000c5005708296f 18. c4t5000C50057089753d0 /scsi_vhci/disk at g5000c50057089753 19. c4t5000C50057086307d0 /scsi_vhci/disk at g5000c50057086307 20. c4t5000C50062879687d0 /scsi_vhci/disk at g5000c50062879687 21. c4t5000C50062879707d0 /scsi_vhci/disk at g5000c50062879707 22. c4t5000C50062879723d0 /scsi_vhci/disk at g5000c50062879723 23. c4t5000C50062879787d0 /scsi_vhci/disk at g5000c50062879787 24. c7t0d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0 25. c7t1d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0 26. c7t4d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0 27. c7t5d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0 28. c10d0 /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0 29. c11d0 /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0 30. c12d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 31. c13d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 32. c14d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 33. c15d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 Specify disk (enter its number): -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 9:56 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----"OmniOS-discuss" > skrev: ----- Till: David Bomba > Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html Br, Rune From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed... Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Mon Nov 3 07:24:03 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 3 Nov 2014 08:24:03 +0100 Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net> References: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From rt at steait.net Mon Nov 3 07:37:37 2014 From: rt at steait.net (Rune Tipsmark) Date: Mon, 3 Nov 2014 07:37:37 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: Tjenare, Maybe its not possible, would explain why I cannot see the shared lun on the other system. I don't have any spare PCI-E slots on either, all filled with IODrives etc. Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 11:24 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! Hmm, how about the target/initiiator configuration of the HCA's? When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP. Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 08:09 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out. Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:49 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! I can see I forgot the list in previous mail... Anyway, are you using connectx adapters? Have you checked if the driver is loaded in the host system? Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 00:07 ?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up... Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM. Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to... It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step? Br Rune -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 11:58 AM To: Rune Tipsmark Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 20:05 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I know, but how do I initiate a session from ZFS10? If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager? Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:33 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 19:11 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hi Johan Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine.. What I can't figure out is how to share a LUN with the other zfs box... see pasted info below... ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box root at zfs00:/pool03# stmfadm list-target -v Target: eui.0002C90300095E7C Operational Status: Online Provider Name : srpt Alias : - Protocol : SRP Sessions : 8 Initiator: eui.0002C903000F397C Alias: 8102c90300095e7e:0002c903000f397c Logged in since: Sun Nov 2 02:09:41 2014 Initiator: eui.0002C903000F397B Alias: 8102c90300095e7d:0002c903000f397b Logged in since: Sun Nov 2 02:09:40 2014 Initiator: eui.0002C903000D3D04 Alias: 8102c90300095e7e:0002c903000d3d04 Logged in since: Sat Nov 1 21:14:47 2014 Initiator: eui.0002C90300104F47 Alias: 8102c90300095e7d:0002c90300104f47 Logged in since: Sat Nov 1 21:12:54 2014 Initiator: eui.0002C903000D3D03 Alias: 8102c90300095e7d:0002c903000d3d03 Logged in since: Sat Nov 1 21:12:32 2014 Initiator: eui.0002C90300104F48 Alias: 8102c90300095e7e:0002c90300104f48 Logged in since: Sat Nov 1 21:10:45 2014 Initiator: eui.0002C903000A48FA Alias: 8102c90300095e7e:0002c903000a48fa Logged in since: Sat Nov 1 21:10:40 2014 Initiator: eui.0002C903000D3CA0 Alias: 8102c90300095e7e:0002c903000d3ca0 Logged in since: Sat Nov 1 21:10:39 2014 Target: iqn.2010-09.org.napp-it:1394106801 Operational Status: Online Provider Name : iscsit Alias : 03.06.2014 Protocol : iSCSI Sessions : 0 root at zfs00:/pool03# stmfadm list-lu -v LU Name: 600144F007780B7F00005455EDD50002 Operational Status: Online Provider Name : sbd Alias : /pool03/LU11 View Entry Count : 1 Data File : /pool03/LU11 Meta File : not set Size : 2199023255552 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active root at zfs00:/pool03# stmfadm list-hg -v Host Group: ESX Host Group: Windows Host Group: ESX-iSER Host Group: OmniOS Member: eui.0002C903000923E6 <<--- The other ZFS box Member: iqn.2010-09.org.napp-it:1402013225 Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0 Host group : OmniOS Target group : All LUN : 0 ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00... Hi! You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target. Rgrds Johan root at zfs10:/root# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t5000C50055FC9533d0 /scsi_vhci/disk at g5000c50055fc9533 1. c4t5000C50055FE6A63d0 /scsi_vhci/disk at g5000c50055fe6a63 2. c4t5000C500625B7EA7d0 /scsi_vhci/disk at g5000c500625b7ea7 3. c4t5000C500625B86E3d0 /scsi_vhci/disk at g5000c500625b86e3 4. c4t5000C500625B886Fd0 /scsi_vhci/disk at g5000c500625b886f 5. c4t5000C500625B8137d0 /scsi_vhci/disk at g5000c500625b8137 6. c4t5000C500625B8427d0 /scsi_vhci/disk at g5000c500625b8427 7. c4t5000C500625BB773d0 /scsi_vhci/disk at g5000c500625bb773 8. c4t5000C500625BC2C3d0 /scsi_vhci/disk at g5000c500625bc2c3 9. c4t5000C500625BD3EBd0 /scsi_vhci/disk at g5000c500625bd3eb 10. c4t5000C50057085A6Bd0 /scsi_vhci/disk at g5000c50057085a6b 11. c4t5000C50057086B67d0 /scsi_vhci/disk at g5000c50057086b67 12. c4t5000C50062878C0Bd0 /scsi_vhci/disk at g5000c50062878c0b 13. c4t5000C50062878C43d0 /scsi_vhci/disk at g5000c50062878c43 14. c4t5000C500570858EFd0 /scsi_vhci/disk at g5000c500570858ef 15. c4t5000C500570870D3d0 /scsi_vhci/disk at g5000c500570870d3 16. c4t5000C5005708351Bd0 /scsi_vhci/disk at g5000c5005708351b 17. c4t5000C5005708296Fd0 /scsi_vhci/disk at g5000c5005708296f 18. c4t5000C50057089753d0 /scsi_vhci/disk at g5000c50057089753 19. c4t5000C50057086307d0 /scsi_vhci/disk at g5000c50057086307 20. c4t5000C50062879687d0 /scsi_vhci/disk at g5000c50062879687 21. c4t5000C50062879707d0 /scsi_vhci/disk at g5000c50062879707 22. c4t5000C50062879723d0 /scsi_vhci/disk at g5000c50062879723 23. c4t5000C50062879787d0 /scsi_vhci/disk at g5000c50062879787 24. c7t0d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0 25. c7t1d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0 26. c7t4d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0 27. c7t5d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0 28. c10d0 /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0 29. c11d0 /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0 30. c12d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 31. c13d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 32. c14d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 33. c15d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 Specify disk (enter its number): -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 9:56 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----"OmniOS-discuss" > skrev: ----- Till: David Bomba > Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html Br, Rune From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed... Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Mon Nov 3 07:45:39 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 3 Nov 2014 08:45:39 +0100 Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Mon Nov 3 08:40:50 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 3 Nov 2014 09:40:50 +0100 Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From rt at steait.net Mon Nov 3 08:50:56 2014 From: rt at steait.net (Rune Tipsmark) Date: Mon, 3 Nov 2014 08:50:56 +0000 Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: That was a secondary thought, maybe worth testing one day. Primarily I was looking at a way of speeding up zfs send-recv. Guess it's a no go on a single HCA... From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Monday, November 03, 2014 12:41 AM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! If you want to mirror a local volume with a remote LU, I don't think it is a good idea. Asking for trubble... I understand you want to get an HA solution. But I believe HA solutions needs other strategies than that. Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 08:37 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Tjenare, Maybe its not possible, would explain why I cannot see the shared lun on the other system. I don't have any spare PCI-E slots on either, all filled with IODrives etc. Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 11:24 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! Hmm, how about the target/initiiator configuration of the HCA's? When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP. Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 08:09 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out. Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:49 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hej! I can see I forgot the list in previous mail... Anyway, are you using connectx adapters? Have you checked if the driver is loaded in the host system? Best regards from/Med v?nliga h?lsningar fr?n Johan Kragsterman Capvert -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-03 00:07 ?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up... Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM. Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to... It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step? Br Rune -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 11:58 AM To: Rune Tipsmark Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 20:05 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I know, but how do I initiate a session from ZFS10? If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager? Br, Rune From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 10:33 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----Rune Tipsmark > skrev: ----- Till: Johan Kragsterman > Fr?n: Rune Tipsmark > Datum: 2014-11-02 19:11 Kopia: David Bomba >, "omnios-discuss at lists.omniti.com" > ?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Hi Johan Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine.. What I can't figure out is how to share a LUN with the other zfs box... see pasted info below... ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box root at zfs00:/pool03# stmfadm list-target -v Target: eui.0002C90300095E7C Operational Status: Online Provider Name : srpt Alias : - Protocol : SRP Sessions : 8 Initiator: eui.0002C903000F397C Alias: 8102c90300095e7e:0002c903000f397c Logged in since: Sun Nov 2 02:09:41 2014 Initiator: eui.0002C903000F397B Alias: 8102c90300095e7d:0002c903000f397b Logged in since: Sun Nov 2 02:09:40 2014 Initiator: eui.0002C903000D3D04 Alias: 8102c90300095e7e:0002c903000d3d04 Logged in since: Sat Nov 1 21:14:47 2014 Initiator: eui.0002C90300104F47 Alias: 8102c90300095e7d:0002c90300104f47 Logged in since: Sat Nov 1 21:12:54 2014 Initiator: eui.0002C903000D3D03 Alias: 8102c90300095e7d:0002c903000d3d03 Logged in since: Sat Nov 1 21:12:32 2014 Initiator: eui.0002C90300104F48 Alias: 8102c90300095e7e:0002c90300104f48 Logged in since: Sat Nov 1 21:10:45 2014 Initiator: eui.0002C903000A48FA Alias: 8102c90300095e7e:0002c903000a48fa Logged in since: Sat Nov 1 21:10:40 2014 Initiator: eui.0002C903000D3CA0 Alias: 8102c90300095e7e:0002c903000d3ca0 Logged in since: Sat Nov 1 21:10:39 2014 Target: iqn.2010-09.org.napp-it:1394106801 Operational Status: Online Provider Name : iscsit Alias : 03.06.2014 Protocol : iSCSI Sessions : 0 root at zfs00:/pool03# stmfadm list-lu -v LU Name: 600144F007780B7F00005455EDD50002 Operational Status: Online Provider Name : sbd Alias : /pool03/LU11 View Entry Count : 1 Data File : /pool03/LU11 Meta File : not set Size : 2199023255552 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active root at zfs00:/pool03# stmfadm list-hg -v Host Group: ESX Host Group: Windows Host Group: ESX-iSER Host Group: OmniOS Member: eui.0002C903000923E6 <<--- The other ZFS box Member: iqn.2010-09.org.napp-it:1402013225 Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0 Host group : OmniOS Target group : All LUN : 0 ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00... Hi! You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target. Rgrds Johan root at zfs10:/root# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t5000C50055FC9533d0 /scsi_vhci/disk at g5000c50055fc9533 1. c4t5000C50055FE6A63d0 /scsi_vhci/disk at g5000c50055fe6a63 2. c4t5000C500625B7EA7d0 /scsi_vhci/disk at g5000c500625b7ea7 3. c4t5000C500625B86E3d0 /scsi_vhci/disk at g5000c500625b86e3 4. c4t5000C500625B886Fd0 /scsi_vhci/disk at g5000c500625b886f 5. c4t5000C500625B8137d0 /scsi_vhci/disk at g5000c500625b8137 6. c4t5000C500625B8427d0 /scsi_vhci/disk at g5000c500625b8427 7. c4t5000C500625BB773d0 /scsi_vhci/disk at g5000c500625bb773 8. c4t5000C500625BC2C3d0 /scsi_vhci/disk at g5000c500625bc2c3 9. c4t5000C500625BD3EBd0 /scsi_vhci/disk at g5000c500625bd3eb 10. c4t5000C50057085A6Bd0 /scsi_vhci/disk at g5000c50057085a6b 11. c4t5000C50057086B67d0 /scsi_vhci/disk at g5000c50057086b67 12. c4t5000C50062878C0Bd0 /scsi_vhci/disk at g5000c50062878c0b 13. c4t5000C50062878C43d0 /scsi_vhci/disk at g5000c50062878c43 14. c4t5000C500570858EFd0 /scsi_vhci/disk at g5000c500570858ef 15. c4t5000C500570870D3d0 /scsi_vhci/disk at g5000c500570870d3 16. c4t5000C5005708351Bd0 /scsi_vhci/disk at g5000c5005708351b 17. c4t5000C5005708296Fd0 /scsi_vhci/disk at g5000c5005708296f 18. c4t5000C50057089753d0 /scsi_vhci/disk at g5000c50057089753 19. c4t5000C50057086307d0 /scsi_vhci/disk at g5000c50057086307 20. c4t5000C50062879687d0 /scsi_vhci/disk at g5000c50062879687 21. c4t5000C50062879707d0 /scsi_vhci/disk at g5000c50062879707 22. c4t5000C50062879723d0 /scsi_vhci/disk at g5000c50062879723 23. c4t5000C50062879787d0 /scsi_vhci/disk at g5000c50062879787 24. c7t0d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0 25. c7t1d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0 26. c7t4d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0 27. c7t5d0 /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0 28. c10d0 /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0 29. c11d0 /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0 30. c12d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 31. c13d0 /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 32. c14d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0 33. c15d0 /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0 Specify disk (enter its number): -----Original Message----- From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] Sent: Sunday, November 02, 2014 9:56 AM To: Rune Tipsmark Cc: David Bomba; omnios-discuss at lists.omniti.com Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol -----"OmniOS-discuss" > skrev: ----- Till: David Bomba > Fr?n: Rune Tipsmark S?nt av: "OmniOS-discuss" Datum: 2014-11-02 07:55 Kopia: "omnios-discuss at lists.omniti.com" > ?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol Sounds sensible, how do I do that? I tried creating a view for a thin lu with my other zfs box, but how do I detect it? First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application. Did you create SRP target service using COMSTAR? # svcadm enable -r ibsrp/target What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v" Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link? If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA. Rgrds Johan I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html Br, Rune From: David Bomba [mailto:turbo124 at gmail.com] Sent: Saturday, November 01, 2014 6:01 PM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. On 2 Nov 2014, at 11:45 am, Rune Tipsmark > wrote: Hi all, Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy. If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed... Br, Rune _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Mon Nov 3 09:08:35 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 3 Nov 2014 10:08:35 +0100 Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: RE: Re: zfs send via SRP or other RDMA enabled protocol In-Reply-To: References: , , <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, , <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, Message-ID: An HTML attachment was scrubbed... URL: From sk at kram.io Mon Nov 3 10:21:15 2014 From: sk at kram.io (Steffen Kram) Date: Mon, 3 Nov 2014 11:21:15 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> Message-ID: <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io> > I think an OmniOS Survival guide would be great. Theo, I think that would be a really good idea. In my opinion some pages of the current wiki admin guide are already a very good first start (e.g. networking and zones). What?s missing in my point of view are ?Migration Guides? from other operating systems, especially linux and freebsd. Those migration guides should cover all things that look ?strange? for non-solaris folks e.g. partitioning, they should elaborate how to best use the ?specialities? e.g. how to use smf, fmd ..., what are the benefits of crossbow, what?s that automounter thing? Why is my home not in /home? Those are the questions I often get from people not familiar with Illumos/Solaris. I already startet a couple of notes that would cover some of those questions. Helping newbies or linux-sysadmins to get started with omnios. So I would really like to contribute if there would be a real community site. > If you have to put something like "newbies" in quotes and talk about them as different than us, you are viewing this community in an unhealthy way. OmniOS is designed to take Illumos and all of its features and package it in a way that makes a minimal and useful general operating system targeted at servers and appliances. Anyone with whom that vision aligns is most welcome in the community and we should encourage efforts to make them feel as an accepted part of the whole. I think the OmniOS community might be small but I would never call it unhealthy and I must confess I like the familiar atmosphere. One thing why I prefer OmniOS about any other os/distro out there is its focus on minimalism. This is something that totally contradicts with easy accessibility for newbies. However, we as a community could do a better job to pave the way for the more experienced linux/bsd sysadm who wants to dig into OmniOS. This might be to provide a comprehensive guide how to run typical software stacks. This should include how to build from the source, how to package those things and how to run it (smf). Although I provide a public OmniOS package server myself. I would never recommend to use omnios just with 3rd party repositories. I think the provided build scripts are a wealthy source of information to build own packages or to do a quick proof of concept but they should for sure not be viewed as the ?Missing OmniOS Distribution Packages?. Those packages might be highly customized for someone else very own specific needs. What I?m getting at, it?s for sure that it would be nice to be more attractive for a broader user base, but all efforts should be in the spirit of the existing OmniOS. Hey, that?s why the existing user base is here in the first place. So let?s educate people on how to use OmniOS to unleash it?s full potential and don?t try to just provide them with easy to use pre-packaged solutions! Let me close with a pointer to the scala community. They had a similar problem for a long time, and since they come up with http://docs.scala-lang.org the documentation is constantly improving. I really like this approach of a comprehensive website, containing in-depth guides, easy how-tos for primers and handy cheat-cheats for the daily use. I?m ready to contribute to a great community doc and willing to invest time to build it if others also jump in! Cheers, Steffen ? Dr. Steffen Kram Ulm University steffen.kram at uni-ulm.de Institute for Applied www.mathematik.uni-ulm.de/iai Information Processing Helmholtzstra?e 18 Phone: 0731 50-23575 89081 Ulm, Germany Twitter: @stefritz -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2629 bytes Desc: not available URL: From richard at netbsd.org Mon Nov 3 12:07:12 2014 From: richard at netbsd.org (Richard PALO) Date: Mon, 03 Nov 2014 13:07:12 +0100 Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version number Message-ID: Curious, haven't noticed this before: > > richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv > abandon : erreur : _ssl.c:491: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number Some other mercurial sites seem okay, so I wonder if this is primarily on Or*cle's side or omnios (bloody)? found here: https://hg.java.net/hg/solaris-userland~gate this one too: https://hg.java.net/hg/solaris-desktop~spec-files From richard at netbsd.org Mon Nov 3 12:22:30 2014 From: richard at netbsd.org (Richard PALO) Date: Mon, 03 Nov 2014 13:22:30 +0100 Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version number In-Reply-To: References: Message-ID: Le 03/11/14 13:07, Richard PALO a ?crit : > Curious, haven't noticed this before: >> >> richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv >> abandon : erreur : _ssl.c:491: error:1408F10B:SSL >> routines:SSL3_GET_RECORD:wrong version number > > Some other mercurial sites seem okay, so I wonder if this is primarily > on Or*cle's side or omnios (bloody)? > > found here: https://hg.java.net/hg/solaris-userland~gate > this one too: https://hg.java.net/hg/solaris-desktop~spec-files > Found the solution here: http://stackoverflow.com/questions/21477683/mercurial-https-clone-abort-error-wrong-version-number need to add now the following to ~/.hgrc: > tls = false sorry for the noise. From richard at netbsd.org Mon Nov 3 13:57:37 2014 From: richard at netbsd.org (Richard PALO) Date: Mon, 03 Nov 2014 14:57:37 +0100 Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version number In-Reply-To: References: Message-ID: Le 03/11/14 13:22, Richard PALO a ?crit : > Le 03/11/14 13:07, Richard PALO a ?crit : >> Curious, haven't noticed this before: >>> >>> richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv >>> abandon : erreur : _ssl.c:491: error:1408F10B:SSL >>> routines:SSL3_GET_RECORD:wrong version number >> >> Some other mercurial sites seem okay, so I wonder if this is primarily >> on Or*cle's side or omnios (bloody)? >> >> found here: https://hg.java.net/hg/solaris-userland~gate >> this one too: https://hg.java.net/hg/solaris-desktop~spec-files >> > > Found the solution here: > http://stackoverflow.com/questions/21477683/mercurial-https-clone-abort-error-wrong-version-number > > > need to add now the following to ~/.hgrc: > > tls = false > > sorry for the noise. > actually, testing with pkgsrc, upgrading to 3.2 seems to work fine too, so that is probably a preferable solution. From admdwrf at gmail.com Mon Nov 3 17:04:56 2014 From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=) Date: Mon, 3 Nov 2014 18:04:56 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io> References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io> Message-ID: Yep, a survival guide can be a good solution. It should show to Linux/FreeBSD admin how to use ZFS, Zones, Etherstub/Vnic, FSS, Capping memory, tuning the system. In what format can we do this if you are agree ? A page in the actual wiki ? a dedicated website ? cms ? pdf file ? See you, Fred 2014-11-03 11:21 GMT+01:00 Steffen Kram : > > I think an OmniOS Survival guide would be great. > > Theo, I think that would be a really good idea. In my opinion some pages > of the current wiki admin guide are already a very good first start (e.g. > networking and zones). What?s missing in my point of view are ?Migration > Guides? from other operating systems, especially linux and freebsd. Those > migration guides should cover all things that look ?strange? for > non-solaris folks e.g. partitioning, they should elaborate how to best use > the ?specialities? e.g. how to use smf, fmd ..., what are the benefits of > crossbow, what?s that automounter thing? Why is my home not in /home? > > Those are the questions I often get from people not familiar with > Illumos/Solaris. I already startet a couple of notes that would cover some > of those questions. Helping newbies or linux-sysadmins to get started with > omnios. So I would really like to contribute if there would be a real > community site. > > > If you have to put something like "newbies" in quotes and talk about > them as different than us, you are viewing this community in an unhealthy > way. OmniOS is designed to take Illumos and all of its features and > package it in a way that makes a minimal and useful general operating > system targeted at servers and appliances. Anyone with whom that vision > aligns is most welcome in the community and we should encourage efforts to > make them feel as an accepted part of the whole. > > I think the OmniOS community might be small but I would never call it > unhealthy and I must confess I like the familiar atmosphere. > > One thing why I prefer OmniOS about any other os/distro out there is its > focus on minimalism. This is something that totally contradicts with easy > accessibility for newbies. However, we as a community could do a better job > to pave the way for the more experienced linux/bsd sysadm who wants to dig > into OmniOS. This might be to provide a comprehensive guide how to run > typical software stacks. This should include how to build from the source, > how to package those things and how to run it (smf). > > Although I provide a public OmniOS package server myself. I would never > recommend to use omnios just with 3rd party repositories. I think the > provided build scripts are a wealthy source of information to build own > packages or to do a quick proof of concept but they should for sure not be > viewed as the ?Missing OmniOS Distribution Packages?. Those packages might > be highly customized for someone else very own specific needs. > > What I?m getting at, it?s for sure that it would be nice to be more > attractive for a broader user base, but all efforts should be in the spirit > of the existing OmniOS. Hey, that?s why the existing user base is here in > the first place. So let?s educate people on how to use OmniOS to unleash > it?s full potential and don?t try to just provide them with easy to use > pre-packaged solutions! > > Let me close with a pointer to the scala community. They had a similar > problem for a long time, and since they come up with > http://docs.scala-lang.org the documentation is constantly improving. I > really like this approach of a comprehensive website, containing in-depth > guides, easy how-tos for primers and handy cheat-cheats for the daily use. > I?m ready to contribute to a great community doc and willing to invest time > to build it if others also jump in! > > Cheers, Steffen > > ? > Dr. Steffen Kram Ulm University > steffen.kram at uni-ulm.de Institute for Applied > www.mathematik.uni-ulm.de/iai Information Processing > > Helmholtzstra?e 18 Phone: 0731 50-23575 > 89081 Ulm, Germany Twitter: @stefritz > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From estibi at me.com Mon Nov 3 18:37:58 2014 From: estibi at me.com (Piotr Jasiukajtis) Date: Mon, 03 Nov 2014 19:37:58 +0100 Subject: [OmniOS-discuss] Dell R730xd In-Reply-To: References: <8D3E3DD7-EA48-4693-93DE-374C3E67BEFB@ohsu.edu> <3F65C05E-30CA-4916-8304-578B0F927F0D@omniti.com> Message-ID: <47E2A81E-A722-4394-9706-601256F76315@me.com> Hi Randy, I would be interested in your results regarding R730xd, please post them once you have the machine. ? Piotr Jasiukajtis > On 20 Oct 2014, at 23:44, Randy S wrote: > > Thanks for the input guys! > I'll certainly be sure to have it assembled with intel nics and LSI hba's. > If there is any interrest I can post results of preliminary tests here. > It will be some weeks though before I receive the stuff (if all goes well). > > Regards, > > Randy > > > From: danmcd at omniti.com > > Date: Mon, 20 Oct 2014 16:27:47 -0400 > > To: chip at innovates.com > > CC: omnios-discuss at lists.omniti.com > > Subject: Re: [OmniOS-discuss] Dell R730xd > > > > > > > On Oct 20, 2014, at 4:26 PM, Schweiss, Chip wrote: > > > > > > Recently we had a quote from Dell w/ Nexenta. They specify Intel NICs and LSI HBAs when building with Nexenta. > > > > Any HW setup that works with Nexenta should also work with OmniOS. > > > > Dan > > > > _______________________________________________ > > OmniOS-discuss mailing list > > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From richard.elling at richardelling.com Mon Nov 3 19:30:19 2014 From: richard.elling at richardelling.com (Richard Elling) Date: Mon, 3 Nov 2014 11:30:19 -0800 Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% In-Reply-To: <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net> References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net> <8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com> <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com> <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net> Message-ID: <0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com> On Nov 2, 2014, at 6:24 PM, Rune Tipsmark wrote: > Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring? You need to know what you are measuring. iostat and zpool iostat measure the I/O from the pool to the disk. This is often very different than an application to the file system (pool). In practice, it is often so different that it is impractical to correlate the two. > How does compression affect the numbers? > > Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual? Measure both and compare. Usually we measure application side bandwidth using a tool such as fsstat. -- richard > > Br, > Rune > > -----Original Message----- > From: Richard Elling [mailto:richard.elling at richardelling.com] > Sent: Sunday, November 02, 2014 6:07 PM > To: Rune Tipsmark > Cc: Eric Sproul; omnios-discuss at lists.omniti.com > Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% > > > On Oct 31, 2014, at 6:07 PM, Rune Tipsmark wrote: > >> So actually started storage vmotions on 3 host, 6 concurrent and am >> getting about 1GB/sec Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more. >> >> I think this is resolved for now.... cpu sitting at 30-40% usage while >> moving 1GB/sec > > Yes, that seems about right. > -- richard > >> >> Iostat -xn 1 >> pool04 396G 39.5T 9 15.9K 325K 1.01G >> pool04 396G 39.5T 7 17.0K 270K 1.03G >> pool04 396G 39.5T 12 17.4K 558K 1.10G >> pool04 396G 39.5T 10 16.9K 442K 1.03G >> pool04 397G 39.5T 6 16.9K 332K 1021M >> pool04 397G 39.5T 1 16.3K 74.9K 1.01G >> pool04 397G 39.5T 8 17.0K 433K 1.05G >> pool04 397G 39.5T 20 17.1K 716K 1023M >> pool04 397G 39.5T 11 18.3K 425K 1.14G >> pool04 398G 39.5T 0 18.3K 65.9K 1.11G >> pool04 398G 39.5T 16 17.9K 551K 1.06G >> pool04 398G 39.5T 0 16.8K 105K 1.03G >> pool04 398G 39.5T 1 18.2K 124K 1.11G >> pool04 398G 39.5T 0 17.1K 45.9K 1.05G >> pool04 399G 39.5T 6 17.3K 454K 1.08G >> pool04 399G 39.5T 0 17.9K 0 1.06G >> pool04 399G 39.5T 2 16.9K 116K 1.04G >> pool04 399G 39.5T 2 18.8K 130K 1.09G >> pool04 399G 39.5T 0 17.6K 0 1.03G >> pool04 400G 39.5T 3 17.5K 155K 1.04G >> pool04 400G 39.5T 0 17.6K 31.5K 1.03G >> >> -----Original Message----- >> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] >> On Behalf Of Rune Tipsmark >> Sent: Friday, October 31, 2014 12:38 PM >> To: Richard Elling; Eric Sproul >> Cc: omnios-discuss at lists.omniti.com >> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% >> >> Ok, makes sense. >> What other kind of indicators can I look at? >> >> I get decent results from DD but still feels a bit slow... >> >> Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... >> >> Sync=always, block size 1M >> 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s >> real 4m56.382s >> user 0m0.461s >> sys 3m12.662s >> >> Sync=disabled, block size 1M >> 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s >> real 1m57.777s >> user 0m0.237s >> sys 1m57.466s >> >> ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each.... >> >> Now looking for aux power cables and will retest... >> >> Br, >> Rune >> >> -----Original Message----- >> From: Richard Elling [mailto:richard.elling at richardelling.com] >> Sent: Friday, October 31, 2014 9:03 AM >> To: Eric Sproul >> Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com >> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% >> >> >> On Oct 31, 2014, at 7:14 AM, Eric Sproul wrote: >> >>> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark wrote: >>> >>>> Why is this pool showing near 100% busy when the underlying disks >>>> are doing nothing at all.... >>> >>> Simply put, it's just how the accounting works in iostat. It treats >>> the pool like any other device, so if there is even one outstanding >>> request to the pool, it counts towards the busy%. Keith W. from >>> Joyent explained this recently on the illumos-zfs list: >>> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pa >>> g >>> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/ >>> >>> The TL;DR is: if your pool has more than one disk in it, the >>> pool-wide busy% is useless. >> >> FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators. >> -- richard >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > From richard.elling at richardelling.com Mon Nov 3 19:31:49 2014 From: richard.elling at richardelling.com (Richard Elling) Date: Mon, 3 Nov 2014 11:31:49 -0800 Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10% In-Reply-To: <0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com> References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net> <8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com> <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com> <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net> <0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com> Message-ID: [responding to myself, considering the audience] On Nov 3, 2014, at 11:30 AM, Richard Elling wrote: > > On Nov 2, 2014, at 6:24 PM, Rune Tipsmark wrote: > >> Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring? > > You need to know what you are measuring. iostat and zpool iostat measure the I/O from > the pool to the disk. This is often very different than an application to the file system (pool). > In practice, it is often so different that it is impractical to correlate the two. > >> How does compression affect the numbers? >> >> Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual? > > Measure both and compare. Usually we measure application side bandwidth using a tool > such as fsstat. actually, we measure it using some sort of data collector that stores data for long-term analysis and includes handy graphing features (cue Circonus) -- richard From doug at will.to Mon Nov 3 20:33:34 2014 From: doug at will.to (Doug Hughes) Date: Mon, 3 Nov 2014 15:33:34 -0500 Subject: [OmniOS-discuss] strange console message Message-ID: I have a strange console message on a relatively newly installed machine: root at x4275-3-15-39:/root# uname -a SunOS x4275-3-15-39 5.11 omnios-f808c81 i86pc i386 i86pc root at x4275-3-15-39:/root# more /etc/release OmniOS v11 r151012 Copyright 2014 OmniTI Computer Consulting, Inc. All rights reserved. Use is subject to license terms. root at x4275-3-15-39:/root# They go back a ways in time: ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not processed ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not processed ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not processed ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not processed -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Mon Nov 3 20:47:10 2014 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 3 Nov 2014 15:47:10 -0500 Subject: [OmniOS-discuss] strange console message In-Reply-To: References: Message-ID: <9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com> > On Nov 3, 2014, at 3:33 PM, Doug Hughes wrote: > > DispatchInterruptMessage Are you running firefox on this box? I don't see this string in the illumos source at all. Googling it pointed me at bugzilla entries for what appears to be gecko (the firefox rendering engine). Dan From admdwrf at gmail.com Mon Nov 3 21:10:40 2014 From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=) Date: Mon, 3 Nov 2014 22:10:40 +0100 Subject: [OmniOS-discuss] Pb to compile vioif module Message-ID: Since few days, i try to compile the virtio net driver. I downloaded the last vioblk from nexenta git repo, no problem But, when i forward one step on vioif, i eat a new wall after :p Someone already compile it and can help me a bit please ;-) ? Actually, i have this error message: Current working directory /code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif fralix at build0:/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif$ ksh /code/-omnios-151013/illumos-omnios/bldenv.sh -d /code/-omnios-151013/illumos-omnios/illumos.sh 'make' ATLOG=/code/-omnios-151013/illumos-omnios/log HISTFILESIZE=100000 HISTSIZE=200 HOME=/home/fralix HZ=100 LANG=C LC_ALL=C LC_COLLATE=C LC_CTYPE=C LC_MESSAGES=C LC_MONETARY=C LC_NUMERIC=C LC_TIME=C LOGFILE=/code/-omnios-151013/illumos-omnios/log/nightly.log LOGNAME=fralix MACH=i386 MAIL=/var/mail/fralix MAILTO=fralix MULTI_PROTO=no NIGHTLY_OPTIONS=-nDCmpr OLDPWD=/code/-omnios-151013/illumos-omnios/usr/src/uts/intel ON_CLOSED_BINS=/code/-omnios-151013/illumos-omnios/closed ON_CRYPTO_BINS=/code/-omnios-151013/illumos-omnios/on-crypto.i386.tar.bz2 PAGER='/usr/bin/less -ins' PARENT_WS='' PATH=/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin:/usr/ucb:/usr/ccs/bin PWD=/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif REF_PROTO_LIST=/usr/src/proto_list_i386 ROOT=/code/-omnios-151013/illumos-omnios/proto/root_i386 SHELL=/bin/bash SHLVL=2 :ld: fatal: file obj64/vioif.o: open failed: No such file or directory make: Fatal error: Command failed for target `obj64/vioif' Current working directory /code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif make: Fatal error: Command failed for target `def.obj64' Current working directory /code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif SRC=/code/-omnios-151013/illumos-omnios/usr/src STAFFER=fralix TERM=screen _=CLASSPATH Build type is DEBUG RELEASE is VERSION is RELEASE_DATE is nightly The top-level 'setup' target is available to build headers and tools. Using /bin/bash as shell. /usr/ccs/bin/ld -r -dy -N misc/virtio -N misc/mac -o debug64/vioif debug64/vioif.o ld: fatal: file debug64/vioif.o: open failed: No such file or directory *** Error code 1 make: Fatal error: Command failed for target `debug64/vioif' Current working directory /code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif *** Error code 1 The following command caused the error: BUILD_TYPE=DBG64 VERSION='SunOS Development' make def.targ make: Fatal error: Command failed for target `def.debug64' Current working directory /code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif Fred -------------- next part -------------- An HTML attachment was scrubbed... URL: From youzhong at gmail.com Mon Nov 3 21:14:55 2014 From: youzhong at gmail.com (Youzhong Yang) Date: Mon, 3 Nov 2014 16:14:55 -0500 Subject: [OmniOS-discuss] How to release reserved TCP port binding in BOUND state Message-ID: Hello, We are having a very strange issue on one of servers. The issue is that fcntl locking over NFS returns 'no locks available' immediately. dtrace shows that bindresvport() returns error code 125 (EADDRINUSE): # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack(); printf("ret = %d", arg1);}' 9 52692 bindresvport:return rpcmod`connmgr_get+0x560 rpcmod`connmgr_wrapget+0x63 rpcmod`clnt_cots_kcallit+0x198 rpcmod`rpcbind_getaddr+0x245 klmmod`update_host_rpcbinding+0x4f klmmod`nlm_host_get_rpc+0x6d klmmod`nlm_do_lock+0x10d klmmod`nlm4_lock_4_svc+0x2a klmmod`nlm_dispatch+0xe6 klmmod`nlm_prog_4+0x34 rpcmod`svc_getreq+0x1c1 rpcmod`svc_run+0x146 rpcmod`svc_do_run+0x8e nfs`nfssys+0xf1 unix`_sys_sysenter_post_swapgs+0x149 ret = 125 netstat shows that 501 reserved ports are in BOUND state: # netstat -an | grep BOUND *.935 *.* 0 0 1049740 0 BOUND *.801 *.* 0 0 1049740 0 BOUND *.798 *.* 0 0 1049740 0 BOUND *.561 *.* 0 0 1049740 0 BOUND *.613 *.* 0 0 1049740 0 BOUND .... # netstat -an | grep BOUND | wc -l 501 Has anyone seen this similar issue? is it possible to unbind those reserved ports? Rebooting the server is our last resort. Any advice would be very much appreciated. Thanks, -Youzhong -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug at will.to Mon Nov 3 21:55:26 2014 From: doug at will.to (Doug Hughes) Date: Mon, 3 Nov 2014 16:55:26 -0500 Subject: [OmniOS-discuss] strange console message In-Reply-To: <9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com> References: <9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com> Message-ID: Oops, forgot to reply-all. No windowing or windowing tools of any kind. Fresh boot, naked OS. On Mon, Nov 3, 2014 at 3:47 PM, Dan McDonald wrote: > > > On Nov 3, 2014, at 3:33 PM, Doug Hughes wrote: > > > > DispatchInterruptMessage > > > Are you running firefox on this box? I don't see this string in the > illumos source at all. Googling it pointed me at bugzilla entries for what > appears to be gecko (the firefox rendering engine). > > Dan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at novhost.eu Tue Nov 4 02:35:59 2014 From: contact at novhost.eu (NovHost) Date: Tue, 4 Nov 2014 03:35:59 +0100 Subject: [OmniOS-discuss] OmniOS KVM guest with virtio net support Message-ID: I try to run OmniOS under Runabove cloud. To this, i build this OmniOS image for KVM. It's a qcow2 img. I run it with virtio blk and virtio net support on Fedora and Ubuntu with virt-manager, without problem. Actually, i have a problem to run it on Runabove, but it should run in another KVM VPS providers. You can download it at: https://drive.google.com/file/d/0B3424f6EdZMmbFY2VkN0dnZ6TzQ/view?usp=sharing I'll send my notes soon how i put virtio net driver. fred. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.digregorio at gmail.com Tue Nov 4 07:20:28 2014 From: nicolas.digregorio at gmail.com (Nicolas Di Gregorio) Date: Tue, 4 Nov 2014 08:20:28 +0100 Subject: [OmniOS-discuss] Webdav recommandations Message-ID: Hello, What are you recommand as webdav client? I would like to mount a webdav share at boot time is possible? Thanks in advance Kind Regards digrouz -------------- next part -------------- An HTML attachment was scrubbed... URL: From sk at kram.io Tue Nov 4 09:23:06 2014 From: sk at kram.io (Steffen Kram) Date: Tue, 4 Nov 2014 10:23:06 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io> Message-ID: > In what format can we do this if you are agree ? > A page in the actual wiki ? a dedicated website ? cms ? pdf file ? What about putting it on github and just build a nice community doc with jekyll based github pages? That would allow easy contribution and gives us maximum flexibility. Cheers, Steffen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2629 bytes Desc: not available URL: From sk at kram.io Tue Nov 4 12:51:24 2014 From: sk at kram.io (Steffen Kram) Date: Tue, 4 Nov 2014 13:51:24 +0100 Subject: [OmniOS-discuss] OmniOS doc In-Reply-To: References: <20141101004433.GA12410@janus.obspm.bsdfrog.org> <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io> Message-ID: <4469AEB7-B3EA-41D5-BAA3-6C3C1AB0BAC4@kram.io> >> In what format can we do this if you are agree ? >> A page in the actual wiki ? a dedicated website ? cms ? pdf file ? > > What about putting it on github and just build a nice community doc with jekyll based github pages? That would allow easy contribution and gives us maximum flexibility. @omniti: Would it be appreciated if I?m jumping in and just setup such a basic community doc page? I?m not a great artist but I think I?m able to produce something useable. Cheers, Steffen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2629 bytes Desc: not available URL: From lotheac at iki.fi Tue Nov 4 13:43:21 2014 From: lotheac at iki.fi (Lauri Tirkkonen) Date: Tue, 4 Nov 2014 15:43:21 +0200 Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names In-Reply-To: <5458D62B.9040006@scluk.com> References: <20140318155845.GC21841@gutsman.lotheac.fi> <5458D62B.9040006@scluk.com> Message-ID: <20141104134321.GA12810@gutsman.lotheac.fi> On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote: > I have run into the same problem while packaging cmake. Is there a > solution? I have an open pull request for this, but from what I understand OmniOS' pkg isn't currently in a state where they can merge it. The OmniTIers can probably elaborate on that, but if you want to apply the patch yourself, it's at https://github.com/postwait/pkg5/pull/4 -- Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet From admdwrf at gmail.com Tue Nov 4 14:09:11 2014 From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=) Date: Tue, 4 Nov 2014 15:09:11 +0100 Subject: [OmniOS-discuss] OmniOS KVM guest with virtio net support In-Reply-To: References: Message-ID: Hum, very strange, Some KVM host providers use this param to add disk img -drive file=img.qcow2,if=virtio,id=drive-virtiodisk0,cache=none If i use it for OmniOS, the boot process block at the load of vioblk module With NexentaOS, it load without problem. To start OmniOS correctly, i must use this: -drive file=img.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 Since few hours, i am trying to understand why it's ok with nexenta and not with OmniOS. I used the illumos-nexenta source to build virtio, vioblk and blkdev and nothing change. Did you already had this situation ? fred. 2014-11-04 3:35 GMT+01:00 NovHost : > I try to run OmniOS under Runabove cloud. > To this, i build this OmniOS image for KVM. > It's a qcow2 img. I run it with virtio blk and virtio net support on > Fedora and Ubuntu with virt-manager, without problem. > > Actually, i have a problem to run it on Runabove, but it should run in > another KVM VPS providers. > > You can download it at: > > https://drive.google.com/file/d/0B3424f6EdZMmbFY2VkN0dnZ6TzQ/view?usp=sharing > > I'll send my notes soon how i put virtio net driver. > > fred. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From al.slater at scluk.com Tue Nov 4 14:09:16 2014 From: al.slater at scluk.com (Al Slater) Date: Tue, 04 Nov 2014 14:09:16 +0000 Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names In-Reply-To: <20141104134321.GA12810@gutsman.lotheac.fi> References: <20140318155845.GC21841@gutsman.lotheac.fi> <5458D62B.9040006@scluk.com> <20141104134321.GA12810@gutsman.lotheac.fi> Message-ID: <5458DE0C.6040302@scluk.com> On 04/11/14 13:43, Lauri Tirkkonen wrote: > On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote: >> I have run into the same problem while packaging cmake. Is there a >> solution? > > I have an open pull request for this, but from what I understand OmniOS' > pkg isn't currently in a state where they can merge it. The OmniTIers > can probably elaborate on that, but if you want to apply the patch > yourself, it's at https://github.com/postwait/pkg5/pull/4 Thanks for that. -- Al Slater Technical Director SCL Phone : +44 (0)1273 666607 Fax : +44 (0)1273 666601 email : al.slater at scluk.com Stanton Consultancy Ltd Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU Registered in England Company number: 1957652 VAT number: GB 760 2433 55 From youzhong at gmail.com Tue Nov 4 18:32:34 2014 From: youzhong at gmail.com (Youzhong Yang) Date: Tue, 4 Nov 2014 13:32:34 -0500 Subject: [OmniOS-discuss] [developer] How to release reserved TCP port binding in BOUND state In-Reply-To: <20141103215021.GG11456@telcontar.in.telka.sk> References: <20141103215021.GG11456@telcontar.in.telka.sk> Message-ID: Hi Marcel, I really appreciate your input. We will try to find out what is wrong in connmgr_get() which caused port leaking. In the meantime, we want to change clnt_cots_do_bindresvport from 1 to 0 through mdb so that new connection will start using non-reserved port, http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508 Will dynamically changing this variable have any side effect? Thanks, -Youzhong On Mon, Nov 3, 2014 at 4:50 PM, Marcel Telka wrote: > Hi Youzhong, > > On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via > illumos-developer wrote: > > Hello, > > > > We are having a very strange issue on one of servers. The issue is that > > fcntl locking over NFS returns 'no locks available' immediately. > > > > dtrace shows that bindresvport() returns error code 125 (EADDRINUSE): > > > > # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack(); > > printf("ret = %d", arg1);}' > > 9 52692 bindresvport:return > > rpcmod`connmgr_get+0x560 > > rpcmod`connmgr_wrapget+0x63 > > rpcmod`clnt_cots_kcallit+0x198 > > rpcmod`rpcbind_getaddr+0x245 > > klmmod`update_host_rpcbinding+0x4f > > klmmod`nlm_host_get_rpc+0x6d > > klmmod`nlm_do_lock+0x10d > > klmmod`nlm4_lock_4_svc+0x2a > > klmmod`nlm_dispatch+0xe6 > > klmmod`nlm_prog_4+0x34 > > rpcmod`svc_getreq+0x1c1 > > rpcmod`svc_run+0x146 > > rpcmod`svc_do_run+0x8e > > nfs`nfssys+0xf1 > > unix`_sys_sysenter_post_swapgs+0x149 > > ret = 125 > > > > netstat shows that 501 reserved ports are in BOUND state: > > > > # netstat -an | grep BOUND > > *.935 *.* 0 0 1049740 0 > BOUND > > *.801 *.* 0 0 1049740 0 > BOUND > > *.798 *.* 0 0 1049740 0 > BOUND > > *.561 *.* 0 0 1049740 0 > BOUND > > *.613 *.* 0 0 1049740 0 > BOUND > > .... > > # netstat -an | grep BOUND | wc -l > > 501 > > > > Has anyone seen this similar issue? is it possible to unbind those > reserved > > ports? Rebooting the server is our last resort. > > > > Any advice would be very much appreciated. > > I faced similar issue in connmgr_get(). It is filed as #1616 and the > problem > is that the dead connection is not properly closed (there seems to be > missing > connmgr_cancelconn() call somewhere), so the client could properly > reconnect. > Unfortunately, I had no time to finish the analysis of this bug. > > > HTH > > -- > +-------------------------------------------+ > | Marcel Telka e-mail: marcel at telka.sk | > | homepage: http://telka.sk/ | > | jabber: marcel at jabber.sk | > +-------------------------------------------+ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From basil.crow at delphix.com Tue Nov 4 19:07:48 2014 From: basil.crow at delphix.com (Basil Crow) Date: Tue, 4 Nov 2014 11:07:48 -0800 Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names In-Reply-To: <5458DE0C.6040302@scluk.com> References: <20140318155845.GC21841@gutsman.lotheac.fi> <5458D62B.9040006@scluk.com> <20141104134321.GA12810@gutsman.lotheac.fi> <5458DE0C.6040302@scluk.com> Message-ID: On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote: > I have run into the same problem while packaging cmake. Is there a > solution? We ran into three separate issues at Delphix using IPS with filenames that involve spaces. 1.) --- Applying transforms pkgmogrify: File /tmp/build_jenkins/masking.p5m.int line 912: Malformed action at position: 72: file opt/masking/apache-tomcat-7.0.42/webapps/masking/image/Copy of bg.gif group=bin mode=0644 owner=root path="opt/masking/apache-tomcat-7.0.42/webapps/masking/image/Copy of bg.gif" 2.) --- Applying transforms --- Resolving dependencies pkgdepend: Could not parse manifest /tmp/build_delphix/masking.p5m.int.2 because of the following line: file opt/masking/Application/Data Integration 32-bit.app/Contents/Info.plist path="opt/masking/Application/Data Integration 32-bit.app/Contents/Info.plist" owner=root group=bin mode=0755 3.) $ sudo pkg install masking Creating Plan - pkg install: The requested operation cannot be completed due to invalid package metadata. Details follow: Malformed action at position: 56: dir path=opt/masking/Application/Data Integration 32-bit.app/Contents/Resources Each problem above required a different patches: 1.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/707707aba792 (generic.py) 2.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/fe4116bd7737 (pkgfmt.py) 3.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/a543cfbce08f (manifest.py) With these patches applied, we are able to use the full IPS workflow with files that have spaces. On Tue, Nov 4, 2014 at 6:09 AM, Al Slater wrote: > On 04/11/14 13:43, Lauri Tirkkonen wrote: >> On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote: >>> I have run into the same problem while packaging cmake. Is there a >>> solution? >> >> I have an open pull request for this, but from what I understand OmniOS' >> pkg isn't currently in a state where they can merge it. The OmniTIers >> can probably elaborate on that, but if you want to apply the patch >> yourself, it's at https://github.com/postwait/pkg5/pull/4 > > Thanks for that. > > > -- > Al Slater > > Technical Director > SCL > > Phone : +44 (0)1273 666607 > Fax : +44 (0)1273 666601 > email : al.slater at scluk.com > > Stanton Consultancy Ltd > > Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU > > Registered in England Company number: 1957652 VAT number: GB 760 2433 55 > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From marcel at telka.sk Mon Nov 3 21:50:21 2014 From: marcel at telka.sk (Marcel Telka) Date: Mon, 3 Nov 2014 22:50:21 +0100 Subject: [OmniOS-discuss] [developer] How to release reserved TCP port binding in BOUND state In-Reply-To: References: Message-ID: <20141103215021.GG11456@telcontar.in.telka.sk> Hi Youzhong, On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via illumos-developer wrote: > Hello, > > We are having a very strange issue on one of servers. The issue is that > fcntl locking over NFS returns 'no locks available' immediately. > > dtrace shows that bindresvport() returns error code 125 (EADDRINUSE): > > # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack(); > printf("ret = %d", arg1);}' > 9 52692 bindresvport:return > rpcmod`connmgr_get+0x560 > rpcmod`connmgr_wrapget+0x63 > rpcmod`clnt_cots_kcallit+0x198 > rpcmod`rpcbind_getaddr+0x245 > klmmod`update_host_rpcbinding+0x4f > klmmod`nlm_host_get_rpc+0x6d > klmmod`nlm_do_lock+0x10d > klmmod`nlm4_lock_4_svc+0x2a > klmmod`nlm_dispatch+0xe6 > klmmod`nlm_prog_4+0x34 > rpcmod`svc_getreq+0x1c1 > rpcmod`svc_run+0x146 > rpcmod`svc_do_run+0x8e > nfs`nfssys+0xf1 > unix`_sys_sysenter_post_swapgs+0x149 > ret = 125 > > netstat shows that 501 reserved ports are in BOUND state: > > # netstat -an | grep BOUND > *.935 *.* 0 0 1049740 0 BOUND > *.801 *.* 0 0 1049740 0 BOUND > *.798 *.* 0 0 1049740 0 BOUND > *.561 *.* 0 0 1049740 0 BOUND > *.613 *.* 0 0 1049740 0 BOUND > .... > # netstat -an | grep BOUND | wc -l > 501 > > Has anyone seen this similar issue? is it possible to unbind those reserved > ports? Rebooting the server is our last resort. > > Any advice would be very much appreciated. I faced similar issue in connmgr_get(). It is filed as #1616 and the problem is that the dead connection is not properly closed (there seems to be missing connmgr_cancelconn() call somewhere), so the client could properly reconnect. Unfortunately, I had no time to finish the analysis of this bug. HTH -- +-------------------------------------------+ | Marcel Telka e-mail: marcel at telka.sk | | homepage: http://telka.sk/ | | jabber: marcel at jabber.sk | +-------------------------------------------+ From marcel at telka.sk Tue Nov 4 20:02:55 2014 From: marcel at telka.sk (Marcel Telka) Date: Tue, 4 Nov 2014 21:02:55 +0100 Subject: [OmniOS-discuss] [developer] How to release reserved TCP port binding in BOUND state In-Reply-To: References: <20141103215021.GG11456@telcontar.in.telka.sk> Message-ID: <20141104200255.GH11456@telcontar.in.telka.sk> On Tue, Nov 04, 2014 at 01:32:34PM -0500, Youzhong Yang via illumos-developer wrote: > Hi Marcel, > > I really appreciate your input. We will try to find out what is wrong in > connmgr_get() which caused port leaking. > > In the meantime, we want to change clnt_cots_do_bindresvport from 1 to 0 > through mdb so that new connection will start using non-reserved port, > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508 > > Will dynamically changing this variable have any side effect? I do not think it will cause any troubles when changed dynamically. IOW, it might help you :-). -- +-------------------------------------------+ | Marcel Telka e-mail: marcel at telka.sk | | homepage: http://telka.sk/ | | jabber: marcel at jabber.sk | +-------------------------------------------+ From johan.kragsterman at capvert.se Wed Nov 5 20:09:10 2014 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Wed, 5 Nov 2014 21:09:10 +0100 Subject: [OmniOS-discuss] KVM and SMF Message-ID: Hi! I'm on my way to try to get my KVM VM's under SMF control. Have been looking at various exampel xml-files, and there is something I'm not clear about, that I'd like to ask those who knows how to handle this. The service name in this example looks like this: