From rt at steait.net Mon Feb 2 00:21:46 2015 From: rt at steait.net (Rune Tipsmark) Date: Mon, 2 Feb 2015 00:21:46 +0000 Subject: [OmniOS-discuss] Windows crashes my ZFS box Message-ID: <1422836506599.46682@steait.net> hi all, I got some major problems... when using Windows and Fibre Channel I am able to kill my ZFS box totally for at least 15 minutes... it simply drops all connections to all hosts connected via FC. This happens under load, for example doing backups writing to the ZFS, running IO Meter against my ZFS... I have looked at queue depth/length and other stuff, I just cannot seem to find out how this happens... I have tested on 3 different Windows machines and 3 difference ZFS boxes - I have ESXi Servers connected to these ZFS boxes as well and no problem there no matter how much load I put on the LUN's. I tried sync=always, sync=disabled, with and without log devices, dedup=on, dedup=off, volume based lun, thin provisioned lun, zfs based thin provisioned lun... you name it, I tried it... tried all kinds of queue lengths from Windows, default is 65535, tried 16, 32, 64, 256 etc... if I put enough stress on the lun presented to Windows it will cause the ZFS box to drop all FC connections for up to 15 minutes.. a reboot is not possible as it will hang for the same amount of minutes... might as well wait for it to come back... Latest FW on all items, HBA, Switch etc. Monitoring shows a distributed load on the ports as expected using Round Robin and MPIO. One thing that irritates me is that I don't get any more than ~80-120 MB/sec (sync=always) throughput when writing to this LUN in Windows, where I get 6-700 MB/sec (sync=always) when writing from a VM on ESXi... The abysmal performance is a pain, but the fact that I can downright crash or hang my ZFS box just by running IOMeter is disturbing... Any ideas why this might happen? Seems to me like a queue problem but I can't really get any closer than that... maybe Windows is just crappy at handling Fibre Channel... however no problems against HP EVA Storage.... same machine, same tests.... br, Rune -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Mon Feb 2 14:50:42 2015 From: chip at innovates.com (Schweiss, Chip) Date: Mon, 2 Feb 2015 08:50:42 -0600 Subject: [OmniOS-discuss] Windows crashes my ZFS box In-Reply-To: <1422836506599.46682@steait.net> References: <1422836506599.46682@steait.net> Message-ID: On Sun, Feb 1, 2015 at 6:21 PM, Rune Tipsmark wrote: > I got some major problems... when using Windows and Fibre Channel I am > able to kill my ZFS box totally for at least 15 minutes... it simply drops > all connections to all hosts connected via FC. This happens under load, for > example doing backups writing to the ZFS, running IO Meter against my ZFS... > ... > > Latest FW on all items, HBA, Switch etc. Monitoring shows a distributed > load on the ports as expected using Round Robin and MPIO. > > This might be a shot in the dark, but the latest firmware on LSI HBAs is known to have serious problems. It has more to do with data corrupting, so I'm not sure this is your cause. Use P18 or P19, but not P20. -Chip > > > One thing that irritates me is that I don't get any more than ~80-120 > MB/sec (sync=always) throughput when writing to this LUN in Windows, where > I get 6-700 MB/sec (sync=always) when writing from a VM on ESXi... The > abysmal performance is a pain, but the fact that I can downright crash or > hang my ZFS box just by running IOMeter is disturbing... > > > > Any ideas why this might happen? Seems to me like a queue problem but I > can't really get any closer than that... maybe Windows is just crappy at > handling Fibre Channel... however no problems against HP EVA Storage.... > same machine, same tests.... > > > > br, > > Rune > > > > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.kragsterman at capvert.se Mon Feb 2 15:18:18 2015 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Mon, 2 Feb 2015 16:18:18 +0100 Subject: [OmniOS-discuss] Ang: Re: Windows crashes my ZFS box In-Reply-To: References: , <1422836506599.46682@steait.net> Message-ID: Hi! -----"OmniOS-discuss" skrev: ----- Till: Rune Tipsmark Fr?n: "Schweiss, Chip" S?nt av: "OmniOS-discuss" Datum: 2015-02-02 15:52 Kopia: "omnios-discuss at lists.omniti.com" ?rende: Re: [OmniOS-discuss] Windows crashes my ZFS box On Sun, Feb 1, 2015 at 6:21 PM, Rune Tipsmark wrote: I got some major problems... when using Windows and Fibre Channel I am able to kill my ZFS box totally for at least 15 minutes... it simply drops all connections to all hosts connected via FC. This happens under load, for example doing backups writing to the ZFS, running IO Meter against my ZFS... ... Latest FW on all items, HBA, Switch etc. Monitoring shows a distributed load on the ports as expected using Round Robin and MPIO. This might be a shot in the dark, but the latest firmware on LSI HBAs is known to have serious problems.? It has more to do with data corrupting, so I'm not sure this is your cause.? Use P18 or P19, but not P20. -Chip Can't be that, cos' it works fine with other host operating systems. No, when I think about it, the only thing I can imagine is the windows handling of sparse volumes. Are your LU's based on sparse volumes? I had a problem a couple of yrs ago with oVirt nodes, that is the oVirt virtualization management system hypervisor nodes. When I wanted to install these to the SAN, I got problems due to nested qcow volumes. I didn't really dig into it back then, but I know that the problems was the implementation of LVM/qcow they used, on top of ZFS/COMSTAR sparse volumes/LU's. So I thought it MIGHT be something similar on window$, like the software volume handling or so. I am NO expert on win, but it might be worth trying to configure a non sparse volume for a LU to send to window$? Johan ? One thing that irritates me is that I don't get any more than ~80-120 MB/sec (sync=always)?throughput when writing to this LUN in Windows, where I get 6-700 MB/sec (sync=always)?when writing from a VM on ESXi... The abysmal performance is a pain, but the fact that I can downright crash or hang my ZFS box just by running IOMeter is disturbing... ? Any ideas why this might happen? Seems to me like a queue problem but I can't really get any closer than that... maybe Windows is just crappy at handling Fibre Channel... however no problems against HP EVA Storage.... same machine, same tests.... ? br, Rune ? ? _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From danmcd at omniti.com Mon Feb 2 20:39:19 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 2 Feb 2015 15:39:19 -0500 Subject: [OmniOS-discuss] Update to r151006 --> 088 Message-ID: A few customers required some backports, so r151006 got one last update. Here are the release notes: http://omnios.omniti.com/wiki.php/ReleaseNotes/r151006#r151006_088 The bits have been in the repo servers themselves for a couple of weeks, but now there's accompanying release media. Check either your openssl or mpt_sas versions to confirm things. Just typing "openssl version" for the former, and "pkg list -v mpt_sas" should now show: pkg://omnios/driver/storage/mpt_sas at 0.5.11,5.11-0.151006:20150119T182628Z i-- Thanks! Dan From sergey57 at gmail.com Tue Feb 3 18:06:58 2015 From: sergey57 at gmail.com (sergey ivanov) Date: Tue, 3 Feb 2015 13:06:58 -0500 Subject: [OmniOS-discuss] A few packages Message-ID: Hi, I've published today in https://github.com/seriv/omnios-build/tree/cs.umd.edu a branch forked from template with our build scripts for dovecot IMAP server with pigeonhole Managesieve server and with compiled as a plugin ldap authentication (linked against omniti-ms' openldap-client package). Also updated versions of other packages there (mc, fail2ban...). -- Regards, Sergey Ivanov | sergey57 at gmail.com http://www.linkedin.com/pub/sergey-ivanov/8/270/a09 From danmcd at omniti.com Tue Feb 3 18:37:03 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 13:37:03 -0500 Subject: [OmniOS-discuss] A few packages In-Reply-To: References: Message-ID: <31403E7E-F432-4CC4-9518-E404F410B19C@omniti.com> > On Feb 3, 2015, at 1:06 PM, sergey ivanov wrote: > > Hi, > I've published today in > https://github.com/seriv/omnios-build/tree/cs.umd.edu a branch forked > from template with our build scripts for dovecot IMAP server with > pigeonhole Managesieve server and with compiled as a plugin ldap > authentication (linked against omniti-ms' openldap-client package). > Also updated versions of other packages there (mc, fail2ban...). And netperf. Thank you! That's one I may put into master (maybe after 014 ships, maybe even FOR 014...). Dan From danmcd at omniti.com Tue Feb 3 20:35:30 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 15:35:30 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: There is no TL;DR for this. Please read all of this before you upgrade. This is a big one. I've updated install media AND the repo servers. Users who use "pkg update" will be getting a big wad (I'm not taking chances with missed packages or components). I'm sorry it took longer than I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed before I released the next bloody update. This is r151013-20150203: - omnios-build master branch, revision 0c38601 - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own section below. - OpenSSL is now at 1.0.2. - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we can't catch up until VND is upstreamed from SmartOS). - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. (Thanks Richard Yao of ZFS-on-Linux for catching this.) - Curl is now at 7.40.0. - NTP dependency fix, from the community. (Thanks to "takashiary".) - Git is now at 2.2.1. - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri "lotheac" Tirkkonen.) - illumos-omnios master branch, revision aef6850 (last illumos-gate merge 6309835) - Several ZFS enhancements, and sometimes followup fixes for them (one, 5531, was what made this release take longer than it should have). - dis(1) supports cross-target disassembly (Illumos 3317) - NFS improvements in the authentication cache (Illumos 5509) - Several ELF safety improvements from Rich Lowe. - kmem_reap() performance improvements - preadv() and pwritev() NOTE: KVM's poor handling of these (assuming all the world's Linux) is why we have a cherrypick above. - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in omnios-userland, that will be updated closer to 014's release). - Developer prototypes updated to 2015 - Introducing the linked-ipkg ("lipkg") zone brand. See below about how this relates to the new pkg(5) changes. - Reducing RAM used in managing ZFS cache devices - Many new, corrected, and updated man pages. - Longstanding mailx(1) overflow fixes now in place. NEW FOR 2015 -- updated pkg(5) You may have known this from previous bloody releases, but we've been using the same oldish version of pkg(5) since at least r151006 in OmniOS. Bloody releases have had a half-finished pkg(5) that breaks things like "zoneadm -z attach -u". When a release came, we reverted to the old r151006 version of pkg(5). That changes now. If you're interested in source, our new repo for pkg(5) is http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. It's a downstream of OI Hipster's for now. The OpenIndiana Hipster folks have done some work to bring pkg(5) more up to date with its upstream. The new pkg(5) contains many under-the-hood improvements. It also includes two user-visible improvements: - A bit more visible status information during pkg(1) operations. - The ability to link images. Linked images is the big one. Way upstream, linked images are the new default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI Hipster, linked-images are currently disabled. We decided to take a middle ground: THE LINKED IPS BRAND (lipkg) Currently deployed ipkg-branded zones stay the same as they have in previous OmniOS releases. You update them how you're used to updating them: Either the documented OmniOS way, or "Dan's way", where you mount the newly upgraded BE and then update each zone prior to reboot. If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) you may change its brand the the new linked-ipkg (lipkg) brand. The default for a new zone is still "ipkg", so if you want lipkg, you must explicitly "set brand=lipkg" in the zonecfg interaction or script. How to convert an existing "ipkg" zone to an "lipkg" zone: 1.) Install the "lipkg" brand: "pkg install lipkg". 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z detach" 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check regardless: "zoneadm -z attach -u". 5.) Boot the zone: "zoneadm -z boot". The zone 's image will be linked to the global zone's image. Once you are running with lipkg zones, new things happen upon subsequent "pkg update"s. Any packages in the global zone that get updated will also be updated in the lipkg zones at the same time. This includes more than just the "omnios" publisher as well, which is why lipkg zones aren't for everyone. If, like me, you tend to currently use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg update" where all zones are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new BE is created, and all of the zones in the new BE are already updated. Furthermore, if you want to have the semantics of not losing log data during the upgrade, you now merely need to halt the zones, then "pkg update", and reboot. If your zones are autoboot (which they should be) everything's up and running with shiny new bits. Attached as a .txt file is a sample transcript of a linked-image upgrade with a global zone and a non-global zone. Please give this one a try, and we'd really appreciate results from: * lipkg-branded zones. * Kayak images. * The ISO installation experience. * ZFS stressing (there may be some residual issues here from upstream, so be warned). We will really appreciate the feedback. Thanks, Dan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: transcript.txt URL: -------------- next part -------------- From jesus at omniti.com Tue Feb 3 20:51:49 2015 From: jesus at omniti.com (Theo Schlossnagle) Date: Tue, 3 Feb 2015 15:51:49 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: Fantastic work Dan! We're all looking forward to 014! On Tue, Feb 3, 2015 at 3:35 PM, Dan McDonald wrote: > There is no TL;DR for this. Please read all of this before you upgrade. > > This is a big one. I've updated install media AND the repo servers. Users > who use "pkg update" will be getting a big wad (I'm not taking chances with > missed packages or components). I'm sorry it took longer than I expected, > but there was an illumos-gate ZFS bug (5531) I wanted fixed before I > released the next bloody update. > > This is r151013-20150203: > > - omnios-build master branch, revision 0c38601 > > - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own > section below. > > - OpenSSL is now at 1.0.2. > > - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we > can't catch up until VND is upstreamed from SmartOS). > > - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. > (Thanks Richard Yao of ZFS-on-Linux for catching this.) > > - Curl is now at 7.40.0. > > - NTP dependency fix, from the community. (Thanks to "takashiary".) > > - Git is now at 2.2.1. > > - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri > "lotheac" Tirkkonen.) > > - illumos-omnios master branch, revision aef6850 (last illumos-gate merge > 6309835) > > - Several ZFS enhancements, and sometimes followup fixes for them (one, > 5531, was what made this release take longer than it should have). > > - dis(1) supports cross-target disassembly (Illumos 3317) > > - NFS improvements in the authentication cache (Illumos 5509) > > - Several ELF safety improvements from Rich Lowe. > > - kmem_reap() performance improvements > > - preadv() and pwritev() > NOTE: KVM's poor handling of these (assuming all the world's Linux) is > why we have a cherrypick above. > > - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in > omnios-userland, that will be updated closer to 014's release). > > - Developer prototypes updated to 2015 > > - Introducing the linked-ipkg ("lipkg") zone brand. See below about how > this relates to the new pkg(5) changes. > > - Reducing RAM used in managing ZFS cache devices > > - Many new, corrected, and updated man pages. > > - Longstanding mailx(1) overflow fixes now in place. > > > NEW FOR 2015 -- updated pkg(5) > > You may have known this from previous bloody releases, but we've been > using the same oldish version of pkg(5) since at least r151006 in OmniOS. > Bloody releases have had a half-finished pkg(5) that breaks things like > "zoneadm -z attach -u". When a release came, we reverted to the old > r151006 version of pkg(5). That changes now. > > If you're interested in source, our new repo for pkg(5) is > http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. > It's a downstream of OI Hipster's for now. > > The OpenIndiana Hipster folks have done some work to bring pkg(5) more up > to date with its upstream. The new pkg(5) contains many under-the-hood > improvements. It also includes two user-visible improvements: > > - A bit more visible status information during pkg(1) operations. > > - The ability to link images. > > Linked images is the big one. Way upstream, linked images are the new > default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI > Hipster, linked-images are currently disabled. We decided to take a middle > ground: > > > THE LINKED IPS BRAND (lipkg) > > Currently deployed ipkg-branded zones stay the same as they have in > previous OmniOS releases. You update them how you're used to updating > them: Either the documented OmniOS way, or "Dan's way", where you mount > the newly upgraded BE and then update each zone prior to reboot. > > If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) > you may change its brand the the new linked-ipkg (lipkg) brand. The > default for a new zone is still "ipkg", so if you want lipkg, you must > explicitly "set brand=lipkg" in the zonecfg interaction or script. > > How to convert an existing "ipkg" zone to an "lipkg" zone: > > 1.) Install the "lipkg" brand: "pkg install lipkg". > > 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z > detach" > > 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". > > 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check > regardless: "zoneadm -z attach -u". > > 5.) Boot the zone: "zoneadm -z boot". > > The zone 's image will be linked to the global zone's image. > > Once you are running with lipkg zones, new things happen upon subsequent > "pkg update"s. Any packages in the global zone that get updated will also > be updated in the lipkg zones at the same time. This includes more than > just the "omnios" publisher as well, which is why lipkg zones aren't for > everyone. If, like me, you tend to currently use the "Dan's way" of > upgrading, lipkg zones will save you time. "pkg update" where all zones > are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new > BE is created, and all of the zones in the new BE are already updated. > Furthermore, if you want to have the semantics of not losing log data > during the upgrade, you now merely need to halt the zones, then "pkg > update", and reboot. If your zones are autoboot (which they should be) > everything's up and running with shiny new bits. > > Attached as a .txt file is a sample transcript of a linked-image upgrade > with a global zone and a non-global zone. > > Please give this one a try, and we'd really appreciate results from: > > * lipkg-branded zones. > > * Kayak images. > > * The ISO installation experience. > > * ZFS stressing (there may be some residual issues here from > upstream, so be warned). > > We will really appreciate the feedback. > > Thanks, > Dan > > > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Tue Feb 3 20:56:28 2015 From: chip at innovates.com (Schweiss, Chip) Date: Tue, 3 Feb 2015 14:56:28 -0600 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: Excellent! I'll put it on my current project system and get you some feedback. Of particular interest, were there any mpt_sas patches in there? Cheers! -Chip On Tue, Feb 3, 2015 at 2:51 PM, Theo Schlossnagle wrote: > Fantastic work Dan! We're all looking forward to 014! > > On Tue, Feb 3, 2015 at 3:35 PM, Dan McDonald wrote: > >> There is no TL;DR for this. Please read all of this before you upgrade. >> >> This is a big one. I've updated install media AND the repo servers. >> Users who use "pkg update" will be getting a big wad (I'm not taking >> chances with missed packages or components). I'm sorry it took longer than >> I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed >> before I released the next bloody update. >> >> This is r151013-20150203: >> >> - omnios-build master branch, revision 0c38601 >> >> - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own >> section below. >> >> - OpenSSL is now at 1.0.2. >> >> - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because >> we can't catch up until VND is upstreamed from SmartOS). >> >> - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. >> (Thanks Richard Yao of ZFS-on-Linux for catching this.) >> >> - Curl is now at 7.40.0. >> >> - NTP dependency fix, from the community. (Thanks to "takashiary".) >> >> - Git is now at 2.2.1. >> >> - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to >> Lauri "lotheac" Tirkkonen.) >> >> - illumos-omnios master branch, revision aef6850 (last illumos-gate merge >> 6309835) >> >> - Several ZFS enhancements, and sometimes followup fixes for them (one, >> 5531, was what made this release take longer than it should have). >> >> - dis(1) supports cross-target disassembly (Illumos 3317) >> >> - NFS improvements in the authentication cache (Illumos 5509) >> >> - Several ELF safety improvements from Rich Lowe. >> >> - kmem_reap() performance improvements >> >> - preadv() and pwritev() >> NOTE: KVM's poor handling of these (assuming all the world's Linux) is >> why we have a cherrypick above. >> >> - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in >> omnios-userland, that will be updated closer to 014's release). >> >> - Developer prototypes updated to 2015 >> >> - Introducing the linked-ipkg ("lipkg") zone brand. See below about how >> this relates to the new pkg(5) changes. >> >> - Reducing RAM used in managing ZFS cache devices >> >> - Many new, corrected, and updated man pages. >> >> - Longstanding mailx(1) overflow fixes now in place. >> >> >> NEW FOR 2015 -- updated pkg(5) >> >> You may have known this from previous bloody releases, but we've been >> using the same oldish version of pkg(5) since at least r151006 in OmniOS. >> Bloody releases have had a half-finished pkg(5) that breaks things like >> "zoneadm -z attach -u". When a release came, we reverted to the old >> r151006 version of pkg(5). That changes now. >> >> If you're interested in source, our new repo for pkg(5) is >> http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. >> It's a downstream of OI Hipster's for now. >> >> The OpenIndiana Hipster folks have done some work to bring pkg(5) more up >> to date with its upstream. The new pkg(5) contains many under-the-hood >> improvements. It also includes two user-visible improvements: >> >> - A bit more visible status information during pkg(1) operations. >> >> - The ability to link images. >> >> Linked images is the big one. Way upstream, linked images are the new >> default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI >> Hipster, linked-images are currently disabled. We decided to take a middle >> ground: >> >> >> THE LINKED IPS BRAND (lipkg) >> >> Currently deployed ipkg-branded zones stay the same as they have in >> previous OmniOS releases. You update them how you're used to updating >> them: Either the documented OmniOS way, or "Dan's way", where you mount >> the newly upgraded BE and then update each zone prior to reboot. >> >> If you detach a zone: (zoneadm -z halt ; zoneadm -z >> detach) you may change its brand the the new linked-ipkg (lipkg) brand. >> The default for a new zone is still "ipkg", so if you want lipkg, you must >> explicitly "set brand=lipkg" in the zonecfg interaction or script. >> >> How to convert an existing "ipkg" zone to an "lipkg" zone: >> >> 1.) Install the "lipkg" brand: "pkg install lipkg". >> >> 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z >> detach" >> >> 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". >> >> 4.) Attach the zone with the -u flag. pkg(5) needs to run its update >> check regardless: "zoneadm -z attach -u". >> >> 5.) Boot the zone: "zoneadm -z boot". >> >> The zone 's image will be linked to the global zone's image. >> >> Once you are running with lipkg zones, new things happen upon subsequent >> "pkg update"s. Any packages in the global zone that get updated will also >> be updated in the lipkg zones at the same time. This includes more than >> just the "omnios" publisher as well, which is why lipkg zones aren't for >> everyone. If, like me, you tend to currently use the "Dan's way" of >> upgrading, lipkg zones will save you time. "pkg update" where all zones >> are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new >> BE is created, and all of the zones in the new BE are already updated. >> Furthermore, if you want to have the semantics of not losing log data >> during the upgrade, you now merely need to halt the zones, then "pkg >> update", and reboot. If your zones are autoboot (which they should be) >> everything's up and running with shiny new bits. >> >> Attached as a .txt file is a sample transcript of a linked-image upgrade >> with a global zone and a non-global zone. >> >> Please give this one a try, and we'd really appreciate results from: >> >> * lipkg-branded zones. >> >> * Kayak images. >> >> * The ISO installation experience. >> >> * ZFS stressing (there may be some residual issues here from >> upstream, so be warned). >> >> We will really appreciate the feedback. >> >> Thanks, >> Dan >> >> >> >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> > > > -- > > Theo Schlossnagle > > http://omniti.com/is/theo-schlossnagle > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Tue Feb 3 20:59:09 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 15:59:09 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: > On Feb 3, 2015, at 3:56 PM, Schweiss, Chip wrote: > > Excellent! > > I'll put it on my current project system and get you some feedback. > > Of particular interest, were there any mpt_sas patches in there? It has the latest illumos-gate versions of things as of two days ago. Very recently (mid-January) we backported all mpt_sas stuff into 012 and 006. So if you're updated to the latest 012, you won't have much new in mpt_sas. ZFS, OTOH, has newness in it, and alas, not all of it is good (5531 has siblings, apparently). Dan From chip at innovates.com Tue Feb 3 21:07:19 2015 From: chip at innovates.com (Schweiss, Chip) Date: Tue, 3 Feb 2015 15:07:19 -0600 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: On Tue, Feb 3, 2015 at 2:59 PM, Dan McDonald wrote: > > It has the latest illumos-gate versions of things as of two days ago. > Very recently (mid-January) we backported all mpt_sas stuff into 012 and > 006. So if you're updated to the latest 012, you won't have much new in > mpt_sas. ZFS, OTOH, has newness in it, and alas, not all of it is good > (5531 has siblings, apparently). > > Dan > > Good to know. Regardless, I will run bloody until I'm ready to go live on this system. Hopefully I can contribute some valuable information, even if it is in the form of crash dumps. -Chip -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Tue Feb 3 21:08:18 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 16:08:18 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: > On Feb 3, 2015, at 4:07 PM, Schweiss, Chip wrote: > > Good to know. Regardless, I will run bloody until I'm ready to go live on this system. Hopefully I can contribute some valuable information, even if it is in the form of crash dumps. Crash dumps are appreciated. Also, you saw what else (packaging) I'm interested in for this particular one. Thanks! Dan From info at houseofancients.nl Tue Feb 3 22:38:56 2015 From: info at houseofancients.nl (Floris van Essen ..:: House of Ancients Amstafs ::..) Date: Tue, 3 Feb 2015 22:38:56 +0000 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: <356582D1FC91784992ABB4265A16ED48886CF466@vEX01.mindstorm-internet.local> Hi Dan, Immediately after updating, drivers for ntxn ethernet are gone ( they were manually installed, as I had "funky stuff" with the IGB nics), and need reinstalling. Is that normal behavior? Rolling back to the before snapshot, and drivers are there again, and traffic is back to normal Met vriendelijke groet / With kind regards, Floris van Essen Van: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] Namens Dan McDonald Verzonden: dinsdag 3 februari 2015 21:36 Aan: Onderwerp: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) There is no TL;DR for this. Please read all of this before you upgrade. This is a big one. I've updated install media AND the repo servers. Users who use "pkg update" will be getting a big wad (I'm not taking chances with missed packages or components). I'm sorry it took longer than I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed before I released the next bloody update. This is r151013-20150203: - omnios-build master branch, revision 0c38601 - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own section below. - OpenSSL is now at 1.0.2. - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we can't catch up until VND is upstreamed from SmartOS). - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. (Thanks Richard Yao of ZFS-on-Linux for catching this.) - Curl is now at 7.40.0. - NTP dependency fix, from the community. (Thanks to "takashiary".) - Git is now at 2.2.1. - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri "lotheac" Tirkkonen.) - illumos-omnios master branch, revision aef6850 (last illumos-gate merge 6309835) - Several ZFS enhancements, and sometimes followup fixes for them (one, 5531, was what made this release take longer than it should have). - dis(1) supports cross-target disassembly (Illumos 3317) - NFS improvements in the authentication cache (Illumos 5509) - Several ELF safety improvements from Rich Lowe. - kmem_reap() performance improvements - preadv() and pwritev() NOTE: KVM's poor handling of these (assuming all the world's Linux) is why we have a cherrypick above. - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in omnios-userland, that will be updated closer to 014's release). - Developer prototypes updated to 2015 - Introducing the linked-ipkg ("lipkg") zone brand. See below about how this relates to the new pkg(5) changes. - Reducing RAM used in managing ZFS cache devices - Many new, corrected, and updated man pages. - Longstanding mailx(1) overflow fixes now in place. NEW FOR 2015 -- updated pkg(5) You may have known this from previous bloody releases, but we've been using the same oldish version of pkg(5) since at least r151006 in OmniOS. Bloody releases have had a half-finished pkg(5) that breaks things like "zoneadm -z attach -u". When a release came, we reverted to the old r151006 version of pkg(5). That changes now. If you're interested in source, our new repo for pkg(5) is http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. It's a downstream of OI Hipster's for now. The OpenIndiana Hipster folks have done some work to bring pkg(5) more up to date with its upstream. The new pkg(5) contains many under-the-hood improvements. It also includes two user-visible improvements: - A bit more visible status information during pkg(1) operations. - The ability to link images. Linked images is the big one. Way upstream, linked images are the new default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI Hipster, linked-images are currently disabled. We decided to take a middle ground: THE LINKED IPS BRAND (lipkg) Currently deployed ipkg-branded zones stay the same as they have in previous OmniOS releases. You update them how you're used to updating them: Either the documented OmniOS way, or "Dan's way", where you mount the newly upgraded BE and then update each zone prior to reboot. If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) you may change its brand the the new linked-ipkg (lipkg) brand. The default for a new zone is still "ipkg", so if you want lipkg, you must explicitly "set brand=lipkg" in the zonecfg interaction or script. How to convert an existing "ipkg" zone to an "lipkg" zone: 1.) Install the "lipkg" brand: "pkg install lipkg". 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z detach" 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check regardless: "zoneadm -z attach -u". 5.) Boot the zone: "zoneadm -z boot". The zone 's image will be linked to the global zone's image. Once you are running with lipkg zones, new things happen upon subsequent "pkg update"s. Any packages in the global zone that get updated will also be updated in the lipkg zones at the same time. This includes more than just the "omnios" publisher as well, which is why lipkg zones aren't for everyone. If, like me, you tend to currently use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg update" where all zones are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new BE is created, and all of the zones in the new BE are already updated. Furthermore, if you want to have the semantics of not losing log data during the upgrade, you now merely need to halt the zones, then "pkg update", and reboot. If your zones are autoboot (which they should be) everything's up and running with shiny new bits. Attached as a .txt file is a sample transcript of a linked-image upgrade with a global zone and a non-global zone. Please give this one a try, and we'd really appreciate results from: * lipkg-branded zones. * Kayak images. * The ISO installation experience. * ZFS stressing (there may be some residual issues here from upstream, so be warned). We will really appreciate the feedback. Thanks, Dan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ...:: House of Ancients ::... American Staffordshire Terriers +31-628-161-350 +31-614-198-389 Het Perk 48 4903 RB Oosterhout Netherlands www.houseofancients.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobi at oetiker.ch Tue Feb 3 22:46:38 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Tue, 3 Feb 2015 23:46:38 +0100 (CET) Subject: [OmniOS-discuss] in.tftpd not working and how to fix it Message-ID: caveat: due to the kvm performance problem I am still running 010 ... but maybe this is also a problem on more recent versions ... so here goes: after upgrading to 010 the tftp service on our omnios box stopped working properly ... it served a few bytes of the request and then timed out ... we investigated for a bit and then moved our tftp service to linux. Today I ran into the problem again and found the solution: on http://omnios.omniti.com/wiki.php/Installation there is: Activate TFTP server by adding the following line to /etc/inetd.conf and running the "inetconv" command. This will create an SMF service, network/tftp/udp6. Don't worry about the "udp6" bit-- it listens for both IPv4 and IPv6 connections. well as it turnes out, you actually SHOULD worry and use 'udp' instead ... then all will work nicely (on ip4 at least). cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From danmcd at omniti.com Tue Feb 3 22:58:50 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 17:58:50 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: <356582D1FC91784992ABB4265A16ED48886CF466@vEX01.mindstorm-internet.local> References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> <356582D1FC91784992ABB4265A16ED48886CF466@vEX01.mindstorm-internet.local> Message-ID: <1A902494-7F29-48B1-B486-38236BEDA78E@omniti.com> Likely yes. These drivers were installed manually? Yeah, that's likely. Have they survived a prior "pkg update"? Dan Sent from my iPhone (typos, autocorrect, and all) > On Feb 3, 2015, at 5:38 PM, Floris van Essen ..:: House of Ancients Amstafs ::.. wrote: > > Hi Dan, > > Immediately after updating, drivers for ntxn ethernet are gone ( they were manually installed, as I had ?funky stuff? with the IGB nics), and need reinstalling. > Is that normal behavior? > Rolling back to the before snapshot, and drivers are there again, and traffic is back to normal > > Met vriendelijke groet / With kind regards, > > > > Floris van Essen > > Van: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] Namens Dan McDonald > Verzonden: dinsdag 3 februari 2015 21:36 > Aan: > Onderwerp: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) > > There is no TL;DR for this. Please read all of this before you upgrade. > > This is a big one. I've updated install media AND the repo servers. Users who use "pkg update" will be getting a big wad (I'm not taking chances with missed packages or components). I'm sorry it took longer than I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed before I released the next bloody update. > > This is r151013-20150203: > > - omnios-build master branch, revision 0c38601 > > - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own section below. > > - OpenSSL is now at 1.0.2. > > - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we can't catch up until VND is upstreamed from SmartOS). > > - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. (Thanks Richard Yao of ZFS-on-Linux for catching this.) > > - Curl is now at 7.40.0. > > - NTP dependency fix, from the community. (Thanks to "takashiary".) > > - Git is now at 2.2.1. > > - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri "lotheac" Tirkkonen.) > > - illumos-omnios master branch, revision aef6850 (last illumos-gate merge 6309835) > > - Several ZFS enhancements, and sometimes followup fixes for them (one, 5531, was what made this release take longer than it should have). > > - dis(1) supports cross-target disassembly (Illumos 3317) > > - NFS improvements in the authentication cache (Illumos 5509) > > - Several ELF safety improvements from Rich Lowe. > > - kmem_reap() performance improvements > > - preadv() and pwritev() > NOTE: KVM's poor handling of these (assuming all the world's Linux) is why we have a cherrypick above. > > - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in omnios-userland, that will be updated closer to 014's release). > > - Developer prototypes updated to 2015 > > - Introducing the linked-ipkg ("lipkg") zone brand. See below about how this relates to the new pkg(5) changes. > > - Reducing RAM used in managing ZFS cache devices > > - Many new, corrected, and updated man pages. > > - Longstanding mailx(1) overflow fixes now in place. > > > NEW FOR 2015 -- updated pkg(5) > > You may have known this from previous bloody releases, but we've been using the same oldish version of pkg(5) since at least r151006 in OmniOS. Bloody releases have had a half-finished pkg(5) that breaks things like "zoneadm -z attach -u". When a release came, we reverted to the old r151006 version of pkg(5). That changes now. > > If you're interested in source, our new repo for pkg(5) is http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. It's a downstream of OI Hipster's for now. > > The OpenIndiana Hipster folks have done some work to bring pkg(5) more up to date with its upstream. The new pkg(5) contains many under-the-hood improvements. It also includes two user-visible improvements: > > - A bit more visible status information during pkg(1) operations. > > - The ability to link images. > > Linked images is the big one. Way upstream, linked images are the new default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI Hipster, linked-images are currently disabled. We decided to take a middle ground: > > > THE LINKED IPS BRAND (lipkg) > > Currently deployed ipkg-branded zones stay the same as they have in previous OmniOS releases. You update them how you're used to updating them: Either the documented OmniOS way, or "Dan's way", where you mount the newly upgraded BE and then update each zone prior to reboot. > > If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) you may change its brand the the new linked-ipkg (lipkg) brand. The default for a new zone is still "ipkg", so if you want lipkg, you must explicitly "set brand=lipkg" in the zonecfg interaction or script. > > How to convert an existing "ipkg" zone to an "lipkg" zone: > > 1.) Install the "lipkg" brand: "pkg install lipkg". > > 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z detach" > > 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". > > 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check regardless: "zoneadm -z attach -u". > > 5.) Boot the zone: "zoneadm -z boot". > > The zone 's image will be linked to the global zone's image. > > Once you are running with lipkg zones, new things happen upon subsequent "pkg update"s. Any packages in the global zone that get updated will also be updated in the lipkg zones at the same time. This includes more than just the "omnios" publisher as well, which is why lipkg zones aren't for everyone. If, like me, you tend to currently use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg update" where all zones are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new BE is created, and all of the zones in the new BE are already updated. Furthermore, if you want to have the semantics of not losing log data during the upgrade, you now merely need to halt the zones, then "pkg update", and reboot. If your zones are autoboot (which they should be) everything's up and running with shiny new bits. > > Attached as a .txt file is a sample transcript of a linked-image upgrade with a global zone and a non-global zone. > > Please give this one a try, and we'd really appreciate results from: > > * lipkg-branded zones. > > * Kayak images. > > * The ISO installation experience. > > * ZFS stressing (there may be some residual issues here from upstream, so be warned). > > We will really appreciate the feedback. > > Thanks, > Dan > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > ...:: House of Ancients ::... > American Staffordshire Terriers > > +31-628-161-350 > +31-614-198-389 > Het Perk 48 > 4903 RB > Oosterhout > Netherlands > www.houseofancients.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at houseofancients.nl Tue Feb 3 23:05:42 2015 From: info at houseofancients.nl (Floris van Essen ..:: House of Ancients Amstafs ::..) Date: Tue, 3 Feb 2015 23:05:42 +0000 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: <1A902494-7F29-48B1-B486-38236BEDA78E@omniti.com> References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> <356582D1FC91784992ABB4265A16ED48886CF466@vEX01.mindstorm-internet.local> <1A902494-7F29-48B1-B486-38236BEDA78E@omniti.com> Message-ID: <356582D1FC91784992ABB4265A16ED48886D24F3@vEX01.mindstorm-internet.local> Hi Dan, Actually, they have, although those weren?t a major upgrade. Any chance of these drivers making it within the standard OmniOS Drivers packages ? These seem to be in Oracle Solaris : http://docs.oracle.com/cd/E19253-01/816-5177/6mbbc4g8l/index.html and are HP Nic drivers genunix: [ID 469746 kern.info] NOTICE: ntxn0 registered genunix: [ID 805372 kern.info] pcplusmp: pci4040,100 (ntxn) instance 1 irq 0x36 vector 0x63 ioapic 0xff intin 0xff is bound to cpu 2 genunix: [ID 792948 kern.notice] NOTICE: ntxn0: NIC Link is up genunix: [ID 435574 kern.info] NOTICE: ntxn0 link up, 1000 Mbps, full duplex Met vriendelijke groet / With kind regards, Floris van Essen Van: Dan McDonald [mailto:danmcd at omniti.com] Verzonden: dinsdag 3 februari 2015 23:59 Aan: Floris van Essen ..:: House of Ancients Amstafs ::.. CC: Onderwerp: Re: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) Likely yes. These drivers were installed manually? Yeah, that's likely. Have they survived a prior "pkg update"? Dan Sent from my iPhone (typos, autocorrect, and all) On Feb 3, 2015, at 5:38 PM, Floris van Essen ..:: House of Ancients Amstafs ::.. > wrote: Hi Dan, Immediately after updating, drivers for ntxn ethernet are gone ( they were manually installed, as I had ?funky stuff? with the IGB nics), and need reinstalling. Is that normal behavior? Rolling back to the before snapshot, and drivers are there again, and traffic is back to normal Met vriendelijke groet / With kind regards, Floris van Essen Van: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] Namens Dan McDonald Verzonden: dinsdag 3 februari 2015 21:36 Aan: > Onderwerp: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) There is no TL;DR for this. Please read all of this before you upgrade. This is a big one. I've updated install media AND the repo servers. Users who use "pkg update" will be getting a big wad (I'm not taking chances with missed packages or components). I'm sorry it took longer than I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed before I released the next bloody update. This is r151013-20150203: - omnios-build master branch, revision 0c38601 - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own section below. - OpenSSL is now at 1.0.2. - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we can't catch up until VND is upstreamed from SmartOS). - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. (Thanks Richard Yao of ZFS-on-Linux for catching this.) - Curl is now at 7.40.0. - NTP dependency fix, from the community. (Thanks to "takashiary".) - Git is now at 2.2.1. - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri "lotheac" Tirkkonen.) - illumos-omnios master branch, revision aef6850 (last illumos-gate merge 6309835) - Several ZFS enhancements, and sometimes followup fixes for them (one, 5531, was what made this release take longer than it should have). - dis(1) supports cross-target disassembly (Illumos 3317) - NFS improvements in the authentication cache (Illumos 5509) - Several ELF safety improvements from Rich Lowe. - kmem_reap() performance improvements - preadv() and pwritev() NOTE: KVM's poor handling of these (assuming all the world's Linux) is why we have a cherrypick above. - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in omnios-userland, that will be updated closer to 014's release). - Developer prototypes updated to 2015 - Introducing the linked-ipkg ("lipkg") zone brand. See below about how this relates to the new pkg(5) changes. - Reducing RAM used in managing ZFS cache devices - Many new, corrected, and updated man pages. - Longstanding mailx(1) overflow fixes now in place. NEW FOR 2015 -- updated pkg(5) You may have known this from previous bloody releases, but we've been using the same oldish version of pkg(5) since at least r151006 in OmniOS. Bloody releases have had a half-finished pkg(5) that breaks things like "zoneadm -z attach -u". When a release came, we reverted to the old r151006 version of pkg(5). That changes now. If you're interested in source, our new repo for pkg(5) is http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. It's a downstream of OI Hipster's for now. The OpenIndiana Hipster folks have done some work to bring pkg(5) more up to date with its upstream. The new pkg(5) contains many under-the-hood improvements. It also includes two user-visible improvements: - A bit more visible status information during pkg(1) operations. - The ability to link images. Linked images is the big one. Way upstream, linked images are the new default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI Hipster, linked-images are currently disabled. We decided to take a middle ground: THE LINKED IPS BRAND (lipkg) Currently deployed ipkg-branded zones stay the same as they have in previous OmniOS releases. You update them how you're used to updating them: Either the documented OmniOS way, or "Dan's way", where you mount the newly upgraded BE and then update each zone prior to reboot. If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) you may change its brand the the new linked-ipkg (lipkg) brand. The default for a new zone is still "ipkg", so if you want lipkg, you must explicitly "set brand=lipkg" in the zonecfg interaction or script. How to convert an existing "ipkg" zone to an "lipkg" zone: 1.) Install the "lipkg" brand: "pkg install lipkg". 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z detach" 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check regardless: "zoneadm -z attach -u". 5.) Boot the zone: "zoneadm -z boot". The zone 's image will be linked to the global zone's image. Once you are running with lipkg zones, new things happen upon subsequent "pkg update"s. Any packages in the global zone that get updated will also be updated in the lipkg zones at the same time. This includes more than just the "omnios" publisher as well, which is why lipkg zones aren't for everyone. If, like me, you tend to currently use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg update" where all zones are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new BE is created, and all of the zones in the new BE are already updated. Furthermore, if you want to have the semantics of not losing log data during the upgrade, you now merely need to halt the zones, then "pkg update", and reboot. If your zones are autoboot (which they should be) everything's up and running with shiny new bits. Attached as a .txt file is a sample transcript of a linked-image upgrade with a global zone and a non-global zone. Please give this one a try, and we'd really appreciate results from: * lipkg-branded zones. * Kayak images. * The ISO installation experience. * ZFS stressing (there may be some residual issues here from upstream, so be warned). We will really appreciate the feedback. Thanks, Dan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ...:: House of Ancients ::... American Staffordshire Terriers +31-628-161-350 +31-614-198-389 Het Perk 48 4903 RB Oosterhout Netherlands www.houseofancients.nl ...:: House of Ancients ::... American Staffordshire Terriers +31-628-161-350 +31-614-198-389 Het Perk 48 4903 RB Oosterhout Netherlands www.houseofancients.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Tue Feb 3 23:53:35 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 18:53:35 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: <356582D1FC91784992ABB4265A16ED48886D24F3@vEX01.mindstorm-internet.local> References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> <356582D1FC91784992ABB4265A16ED48886CF466@vEX01.mindstorm-internet.local> <1A902494-7F29-48B1-B486-38236BEDA78E@omniti.com> <356582D1FC91784992ABB4265A16ED48886D24F3@vEX01.mindstorm-internet.local> Message-ID: If the aren't open source that's CDDL or BSD, they won't. Dan Sent from my iPhone (typos, autocorrect, and all) > On Feb 3, 2015, at 6:05 PM, Floris van Essen ..:: House of Ancients Amstafs ::.. wrote: > > Hi Dan, > > Actually, they have, although those weren?t a major upgrade. > Any chance of these drivers making it within the standard OmniOS Drivers packages ? > > These seem to be in Oracle Solaris : > > http://docs.oracle.com/cd/E19253-01/816-5177/6mbbc4g8l/index.html > > and are HP Nic drivers > genunix: [ID 469746 kern.info] NOTICE: ntxn0 registered > genunix: [ID 805372 kern.info] pcplusmp: pci4040,100 (ntxn) instance 1 irq 0x36 vector 0x63 ioapic 0xff intin 0xff is bound to cpu 2 > genunix: [ID 792948 kern.notice] NOTICE: ntxn0: NIC Link is up > genunix: [ID 435574 kern.info] NOTICE: ntxn0 link up, 1000 Mbps, full duplex > > > Met vriendelijke groet / With kind regards, > > > > Floris van Essen > > Van: Dan McDonald [mailto:danmcd at omniti.com] > Verzonden: dinsdag 3 februari 2015 23:59 > Aan: Floris van Essen ..:: House of Ancients Amstafs ::.. > CC: > Onderwerp: Re: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) > > Likely yes. These drivers were installed manually? Yeah, that's likely. Have they survived a prior "pkg update"? > > Dan > > Sent from my iPhone (typos, autocorrect, and all) > > On Feb 3, 2015, at 5:38 PM, Floris van Essen ..:: House of Ancients Amstafs ::.. wrote: > > Hi Dan, > > Immediately after updating, drivers for ntxn ethernet are gone ( they were manually installed, as I had ?funky stuff? with the IGB nics), and need reinstalling. > Is that normal behavior? > Rolling back to the before snapshot, and drivers are there again, and traffic is back to normal > > Met vriendelijke groet / With kind regards, > > > > Floris van Essen > > Van: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] Namens Dan McDonald > Verzonden: dinsdag 3 februari 2015 21:36 > Aan: > Onderwerp: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) > > There is no TL;DR for this. Please read all of this before you upgrade. > > This is a big one. I've updated install media AND the repo servers. Users who use "pkg update" will be getting a big wad (I'm not taking chances with missed packages or components). I'm sorry it took longer than I expected, but there was an illumos-gate ZFS bug (5531) I wanted fixed before I released the next bloody update. > > This is r151013-20150203: > > - omnios-build master branch, revision 0c38601 > > - NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own section below. > > - OpenSSL is now at 1.0.2. > > - One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because we can't catch up until VND is upstreamed from SmartOS). > > - Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. (Thanks Richard Yao of ZFS-on-Linux for catching this.) > > - Curl is now at 7.40.0. > > - NTP dependency fix, from the community. (Thanks to "takashiary".) > > - Git is now at 2.2.1. > > - ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to Lauri "lotheac" Tirkkonen.) > > - illumos-omnios master branch, revision aef6850 (last illumos-gate merge 6309835) > > - Several ZFS enhancements, and sometimes followup fixes for them (one, 5531, was what made this release take longer than it should have). > > - dis(1) supports cross-target disassembly (Illumos 3317) > > - NFS improvements in the authentication cache (Illumos 5509) > > - Several ELF safety improvements from Rich Lowe. > > - kmem_reap() performance improvements > > - preadv() and pwritev() > NOTE: KVM's poor handling of these (assuming all the world's Linux) is why we have a cherrypick above. > > - illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy in omnios-userland, that will be updated closer to 014's release). > > - Developer prototypes updated to 2015 > > - Introducing the linked-ipkg ("lipkg") zone brand. See below about how this relates to the new pkg(5) changes. > > - Reducing RAM used in managing ZFS cache devices > > - Many new, corrected, and updated man pages. > > - Longstanding mailx(1) overflow fixes now in place. > > > NEW FOR 2015 -- updated pkg(5) > > You may have known this from previous bloody releases, but we've been using the same oldish version of pkg(5) since at least r151006 in OmniOS. Bloody releases have had a half-finished pkg(5) that breaks things like "zoneadm -z attach -u". When a release came, we reverted to the old r151006 version of pkg(5). That changes now. > > If you're interested in source, our new repo for pkg(5) is http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. It's a downstream of OI Hipster's for now. > > The OpenIndiana Hipster folks have done some work to bring pkg(5) more up to date with its upstream. The new pkg(5) contains many under-the-hood improvements. It also includes two user-visible improvements: > > - A bit more visible status information during pkg(1) operations. > > - The ability to link images. > > Linked images is the big one. Way upstream, linked images are the new default for ipkg-branded zones (i.e. the zones all OmniOS folks use). In OI Hipster, linked-images are currently disabled. We decided to take a middle ground: > > > THE LINKED IPS BRAND (lipkg) > > Currently deployed ipkg-branded zones stay the same as they have in previous OmniOS releases. You update them how you're used to updating them: Either the documented OmniOS way, or "Dan's way", where you mount the newly upgraded BE and then update each zone prior to reboot. > > If you detach a zone: (zoneadm -z halt ; zoneadm -z detach) you may change its brand the the new linked-ipkg (lipkg) brand. The default for a new zone is still "ipkg", so if you want lipkg, you must explicitly "set brand=lipkg" in the zonecfg interaction or script. > > How to convert an existing "ipkg" zone to an "lipkg" zone: > > 1.) Install the "lipkg" brand: "pkg install lipkg". > > 2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z detach" > > 3.) Change the zone's brand: "zonecfg -z set brand=lipkg". > > 4.) Attach the zone with the -u flag. pkg(5) needs to run its update check regardless: "zoneadm -z attach -u". > > 5.) Boot the zone: "zoneadm -z boot". > > The zone 's image will be linked to the global zone's image. > > Once you are running with lipkg zones, new things happen upon subsequent "pkg update"s. Any packages in the global zone that get updated will also be updated in the lipkg zones at the same time. This includes more than just the "omnios" publisher as well, which is why lipkg zones aren't for everyone. If, like me, you tend to currently use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg update" where all zones are lipkg is the equivalent of an automatic "Dan's way" upgrade. The new BE is created, and all of the zones in the new BE are already updated. Furthermore, if you want to have the semantics of not losing log data during the upgrade, you now merely need to halt the zones, then "pkg update", and reboot. If your zones are autoboot (which they should be) everything's up and running with shiny new bits. > > Attached as a .txt file is a sample transcript of a linked-image upgrade with a global zone and a non-global zone. > > Please give this one a try, and we'd really appreciate results from: > > * lipkg-branded zones. > > * Kayak images. > > * The ISO installation experience. > > * ZFS stressing (there may be some residual issues here from upstream, so be warned). > > We will really appreciate the feedback. > > Thanks, > Dan > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > ...:: House of Ancients ::... > American Staffordshire Terriers > > +31-628-161-350 > +31-614-198-389 > Het Perk 48 > 4903 RB > Oosterhout > Netherlands > www.houseofancients.nl > > ...:: House of Ancients ::... > American Staffordshire Terriers > > +31-628-161-350 > +31-614-198-389 > Het Perk 48 > 4903 RB > Oosterhout > Netherlands > www.houseofancients.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Wed Feb 4 01:02:40 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 3 Feb 2015 20:02:40 -0500 Subject: [OmniOS-discuss] in.tftpd not working and how to fix it In-Reply-To: References: Message-ID: <2470DC98-C8FF-423F-905A-2ECEAAAD021A@omniti.com> I'll make sure the wiki gets fixed this week. Thanks, Dan Sent from my iPhone (typos, autocorrect, and all) > On Feb 3, 2015, at 5:46 PM, Tobias Oetiker wrote: > > caveat: due to the kvm performance problem I am still running 010 > ... but maybe this is also a problem on more recent versions ... > > so here goes: > > after upgrading to 010 the tftp service on our omnios box stopped > working properly ... it served a few bytes of the request and then > timed out ... > > we investigated for a bit and then moved our tftp service to linux. > > Today I ran into the problem again and found the solution: > > on http://omnios.omniti.com/wiki.php/Installation there is: > > Activate TFTP server by adding the following line to > /etc/inetd.conf and running the "inetconv" command. This will > create an SMF service, network/tftp/udp6. Don't worry about the > "udp6" bit-- it listens for both IPv4 and IPv6 connections. > > well as it turnes out, you actually SHOULD worry and use 'udp' > instead ... then all will work nicely (on ip4 at least). > > cheers > tobi > > -- > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From chip at innovates.com Wed Feb 4 14:15:00 2015 From: chip at innovates.com (Schweiss, Chip) Date: Wed, 4 Feb 2015 08:15:00 -0600 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: Maybe I haven't had enough coffee this morning or I'm missing something. Where should I set the pkg publisher to point to? I tried: http://pkg.omniti.com/omnios/r151013/ and http://pkg.omniti.com/omnios/r151013-20150203/ Neither works. -Chip On Tue, Feb 3, 2015 at 3:08 PM, Dan McDonald wrote: > > > On Feb 3, 2015, at 4:07 PM, Schweiss, Chip wrote: > > > > Good to know. Regardless, I will run bloody until I'm ready to go live > on this system. Hopefully I can contribute some valuable information, > even if it is in the form of crash dumps. > > Crash dumps are appreciated. Also, you saw what else (packaging) I'm > interested in for this particular one. > > Thanks! > Dan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesus at omniti.com Wed Feb 4 15:00:35 2015 From: jesus at omniti.com (Theo Schlossnagle) Date: Wed, 4 Feb 2015 10:00:35 -0500 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: http://pkg.omniti.com/omnios/bloody/ On Wed, Feb 4, 2015 at 9:15 AM, Schweiss, Chip wrote: > Maybe I haven't had enough coffee this morning or I'm missing something. > > Where should I set the pkg publisher to point to? > > I tried: > http://pkg.omniti.com/omnios/r151013/ > and > http://pkg.omniti.com/omnios/r151013-20150203/ > > Neither works. > > -Chip > > On Tue, Feb 3, 2015 at 3:08 PM, Dan McDonald wrote: > >> >> > On Feb 3, 2015, at 4:07 PM, Schweiss, Chip wrote: >> > >> > Good to know. Regardless, I will run bloody until I'm ready to go live >> on this system. Hopefully I can contribute some valuable information, >> even if it is in the form of crash dumps. >> >> Crash dumps are appreciated. Also, you saw what else (packaging) I'm >> interested in for this particular one. >> >> Thanks! >> Dan >> >> > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Wed Feb 4 16:12:33 2015 From: chip at innovates.com (Schweiss, Chip) Date: Wed, 4 Feb 2015 10:12:33 -0600 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: On Wed, Feb 4, 2015 at 9:00 AM, Theo Schlossnagle wrote: > http://pkg.omniti.com/omnios/bloody/ > That did it. Thanks! -Chip > > On Wed, Feb 4, 2015 at 9:15 AM, Schweiss, Chip wrote: > >> Maybe I haven't had enough coffee this morning or I'm missing something. >> >> Where should I set the pkg publisher to point to? >> >> I tried: >> http://pkg.omniti.com/omnios/r151013/ >> and >> http://pkg.omniti.com/omnios/r151013-20150203/ >> >> Neither works. >> >> -Chip >> >> On Tue, Feb 3, 2015 at 3:08 PM, Dan McDonald wrote: >> >>> >>> > On Feb 3, 2015, at 4:07 PM, Schweiss, Chip wrote: >>> > >>> > Good to know. Regardless, I will run bloody until I'm ready to go >>> live on this system. Hopefully I can contribute some valuable >>> information, even if it is in the form of crash dumps. >>> >>> Crash dumps are appreciated. Also, you saw what else (packaging) I'm >>> interested in for this particular one. >>> >>> Thanks! >>> Dan >>> >>> >> > > > -- > > Theo Schlossnagle > > http://omniti.com/is/theo-schlossnagle > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimklimov at cos.ru Fri Feb 6 13:25:38 2015 From: jimklimov at cos.ru (Jim Klimov) Date: Fri, 06 Feb 2015 14:25:38 +0100 Subject: [OmniOS-discuss] FINALLY - OmniOS bloody is now updated. (long, please read) In-Reply-To: References: <09C29A92-D7F9-48A9-90DE-4C535DE220ED@omniti.com> Message-ID: <481ACD53-C54C-4CBD-8EA8-D30F706026FA@cos.ru> 3 ??????? 2015??. 21:35:30 CET, Dan McDonald ?????: >There is no TL;DR for this. Please read all of this before you >upgrade. > >This is a big one. I've updated install media AND the repo servers. >Users who use "pkg update" will be getting a big wad (I'm not taking >chances with missed packages or components). I'm sorry it took longer >than I expected, but there was an illumos-gate ZFS bug (5531) I wanted >fixed before I released the next bloody update. > >This is r151013-20150203: > >- omnios-build master branch, revision 0c38601 > >- NEW UPDATE to pkg(5). This carries a LOT with it, and gets its own >section below. > >- OpenSSL is now at 1.0.2. > >- One bugfix to illumos-kvm-cmd that needed to be cherrypicked (because >we can't catch up until VND is upstreamed from SmartOS). > >- Longstanding bug with GCC 4.4.4's "cc1" not being properly linked. >(Thanks Richard Yao of ZFS-on-Linux for catching this.) > >- Curl is now at 7.40.0. > >- NTP dependency fix, from the community. (Thanks to "takashiary".) > >- Git is now at 2.2.1. > >- ncurses have their auxiliary libs now in /usr/gnu/lib. (Thanks to >Lauri "lotheac" Tirkkonen.) > >- illumos-omnios master branch, revision aef6850 (last illumos-gate >merge 6309835) > >- Several ZFS enhancements, and sometimes followup fixes for them (one, >5531, was what made this release take longer than it should have). > >- dis(1) supports cross-target disassembly (Illumos 3317) > >- NFS improvements in the authentication cache (Illumos 5509) > >- Several ELF safety improvements from Rich Lowe. > >- kmem_reap() performance improvements > >- preadv() and pwritev() >NOTE: KVM's poor handling of these (assuming all the world's Linux) is >why we have a cherrypick above. > >- illumos-side hwdata (PCI & USB) updates. (NOTE: OmniOS keeps a copy >in omnios-userland, that will be updated closer to 014's release). > >- Developer prototypes updated to 2015 > >- Introducing the linked-ipkg ("lipkg") zone brand. See below about >how this relates to the new pkg(5) changes. > >- Reducing RAM used in managing ZFS cache devices > >- Many new, corrected, and updated man pages. > >- Longstanding mailx(1) overflow fixes now in place. > > >NEW FOR 2015 -- updated pkg(5) > >You may have known this from previous bloody releases, but we've been >using the same oldish version of pkg(5) since at least r151006 in >OmniOS. Bloody releases have had a half-finished pkg(5) that breaks >things like "zoneadm -z attach -u". When a release came, we >reverted to the old r151006 version of pkg(5). That changes now. > >If you're interested in source, our new repo for pkg(5) is >http://github.com/omniti-labs/pkg5 or src.omniti.com:~omnios/core/pkg5. > It's a downstream of OI Hipster's for now. > >The OpenIndiana Hipster folks have done some work to bring pkg(5) more >up to date with its upstream. The new pkg(5) contains many >under-the-hood improvements. It also includes two user-visible >improvements: > > - A bit more visible status information during pkg(1) operations. > > - The ability to link images. > >Linked images is the big one. Way upstream, linked images are the new >default for ipkg-branded zones (i.e. the zones all OmniOS folks use). >In OI Hipster, linked-images are currently disabled. We decided to >take a middle ground: > > >THE LINKED IPS BRAND (lipkg) > >Currently deployed ipkg-branded zones stay the same as they have in >previous OmniOS releases. You update them how you're used to updating >them: Either the documented OmniOS way, or "Dan's way", where you >mount the newly upgraded BE and then update each zone prior to reboot. > >If you detach a zone: (zoneadm -z halt ; zoneadm -z >detach) you may change its brand the the new linked-ipkg (lipkg) brand. >The default for a new zone is still "ipkg", so if you want lipkg, you >must explicitly "set brand=lipkg" in the zonecfg interaction or script. > >How to convert an existing "ipkg" zone to an "lipkg" zone: > >1.) Install the "lipkg" brand: "pkg install lipkg". > >2.) Halt and detach the zone. "zoneadm -z halt ; zoneadm -z > detach" > >3.) Change the zone's brand: "zonecfg -z set brand=lipkg". > >4.) Attach the zone with the -u flag. pkg(5) needs to run its update >check regardless: "zoneadm -z attach -u". > >5.) Boot the zone: "zoneadm -z boot". > >The zone 's image will be linked to the global zone's image. > >Once you are running with lipkg zones, new things happen upon >subsequent "pkg update"s. Any packages in the global zone that get >updated will also be updated in the lipkg zones at the same time. This >includes more than just the "omnios" publisher as well, which is why >lipkg zones aren't for everyone. If, like me, you tend to currently >use the "Dan's way" of upgrading, lipkg zones will save you time. "pkg >update" where all zones are lipkg is the equivalent of an automatic >"Dan's way" upgrade. The new BE is created, and all of the zones in >the new BE are already updated. Furthermore, if you want to have the >semantics of not losing log data during the upgrade, you now merely >need to halt the zones, then "pkg update", and reboot. If your zones >are autoboot (which they should be) everything's up and running with >shiny new bits. > >Attached as a .txt file is a sample transcript of a linked-image >upgrade with a global zone and a non-global zone. > >Please give this one a try, and we'd really appreciate results from: > > * lipkg-branded zones. > > * Kayak images. > > * The ISO installation experience. > > * ZFS stressing (there may be some residual issues here from upstream, >so be warned). > >We will really appreciate the feedback. > >Thanks, >Dan > > > >------------------------------------------------------------------------ > > > > > >------------------------------------------------------------------------ > >_______________________________________________ >OmniOS-discuss mailing list >OmniOS-discuss at lists.omniti.com >http://lists.omniti.com/mailman/listinfo/omnios-discuss Upgrade of my split-root omnios installation with my scripts that manage these more complicated BE's went almost well, the (shared) /var/log dataset needed to be (lofs-)mounted into the cloned BE for pkg to succeed. Relevant updates to beadm-update.sh should now be available in my github at https://github.com/jimklimov/illumos-splitroot-scripts if anyone else does those setups ;) If anyone gets around to port this logic into proper filesystem smf, zfs clone and beadm toolset - i'd be happy ;) Thanks, Jim -- Typos courtesy of K-9 Mail on my Samsung Android From contact at jacobvosmaer.nl Sat Feb 7 23:16:48 2015 From: contact at jacobvosmaer.nl (Jacob Vosmaer) Date: Sun, 8 Feb 2015 00:16:48 +0100 Subject: [OmniOS-discuss] Corrupted ZFS metadata? In-Reply-To: References: <5C57DEC9-2FE9-4047-BB09-8B489A425BC1@marzocchi.net> <95AEF4F6-CFFC-4B24-8AB1-70B32DBDF616@omniti.com> <7E566302-A0D7-4D27-81EB-A22C2A17C00B@marzocchi.net> <4DE2883C-F7DC-4610-A3ED-470786E5C58F@omniti.com> Message-ID: Hi, I use OS X 10.10.x clients to access ZFS filesystems on OmniOS r151012 via NFS. Not quite the same as Olaf but I do sometimes get weird 'corrupted file' errors from OS X. Olaf, have you tried using 'cp' to copy one of the seemingly corrupted files from your OmniOS box to a local OS X volume before opening the file in OS X? Another fun one for me is looking for extended attributes with 'ls -l@' and clearing them with 'xattr -c'. Not sure if it ever fixes anything but it's something one can poke around with. 2015-01-29 22:32 GMT+01:00 Olaf Marzocchi : > > > Il giorno 29/gen/2015, alle ore 22:29, Dan McDonald > ha scritto: > > > > > >> On Jan 29, 2015, at 4:27 PM, Olaf Marzocchi > wrote: > >> > >> Scrubs are performed biweekly by a cron job that should send me the > output in case of errors. I never got anything and indeed I still see no > errors in zpool status. > >> > >> In case it happens again I won?t overwrite the files and I?ll contact > the list to investigate further, I suppose it?s late now to do anything. > > > > One last question: which OmniOS release are you running? > > Latest one, non-bloody: r151012 > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at marzocchi.net Sun Feb 8 15:15:46 2015 From: lists at marzocchi.net (Olaf Marzocchi) Date: Sun, 8 Feb 2015 16:15:46 +0100 Subject: [OmniOS-discuss] Corrupted ZFS metadata? In-Reply-To: References: <5C57DEC9-2FE9-4047-BB09-8B489A425BC1@marzocchi.net> <95AEF4F6-CFFC-4B24-8AB1-70B32DBDF616@omniti.com> <7E566302-A0D7-4D27-81EB-A22C2A17C00B@marzocchi.net> <4DE2883C-F7DC-4610-A3ED-470786E5C58F@omniti.com> Message-ID: <70278C2F-490F-42DD-908F-E103BABA7519@marzocchi.net> Hi, I?ll keep it in mind, but also next time I will try accessing the files via SMB, to verify if the issue is somehow related to netatalk. If you have an idea about how to find those ?falsely corrupted? files let me know, so that I don?t have to wait until I find one by chance. Thanks Olaf > Il giorno 08/feb/2015, alle ore 00:16, Jacob Vosmaer ha scritto: > > Hi, > > I use OS X 10.10.x clients to access ZFS filesystems on OmniOS r151012 via NFS. Not quite the same as Olaf but I do sometimes get weird 'corrupted file' errors from OS X. > > Olaf, have you tried using 'cp' to copy one of the seemingly corrupted files from your OmniOS box to a local OS X volume before opening the file in OS X? Another fun one for me is looking for extended attributes with 'ls -l@' and clearing them with 'xattr -c'. Not sure if it ever fixes anything but it's something one can poke around with. > > 2015-01-29 22:32 GMT+01:00 Olaf Marzocchi >: > > > Il giorno 29/gen/2015, alle ore 22:29, Dan McDonald > ha scritto: > > > > > >> On Jan 29, 2015, at 4:27 PM, Olaf Marzocchi > wrote: > >> > >> Scrubs are performed biweekly by a cron job that should send me the output in case of errors. I never got anything and indeed I still see no errors in zpool status. > >> > >> In case it happens again I won?t overwrite the files and I?ll contact the list to investigate further, I suppose it?s late now to do anything. > > > > One last question: which OmniOS release are you running? > > Latest one, non-bloody: r151012 > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asc1111 at gmail.com Mon Feb 9 17:27:41 2015 From: asc1111 at gmail.com (Aaron Curry) Date: Mon, 9 Feb 2015 10:27:41 -0700 Subject: [OmniOS-discuss] File Server Crash Message-ID: Hi all, We recently set up an OmniOS file server to serve our SMB clients. The server paniced over the weekend. I was hoping someone here would help us interpret the panic info and possibly point us towards a resolution. The stack dump is: panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff007c6e78a0 addr=18 occurred in module "nsmb" due to a NULL pointer dereference panicstack = unix:die+df () | unix:trap+db3 () | unix:cmntrap+e6 () | nsmb:nbssn_peekhdr+75 () | nsmb:nbssn_recv+7f () | nsmb:smb_nbst_recv+a1 () | nsmb:smb_iod_recv1+46 () | nsmb:smb_iod_recvall+65 () | nsmb:smb_iod_vc_work+c6 () | nsmb:smb_usr_iod_work+15c () | nsmb:nsmb_ioctl+d9 () | genunix:cdev_ioctl+39 () | specfs:spec_ioctl+60 () | genunix:fop_ioctl+55 () | genunix:ioctl+9b () | unix:brand_sys_sysenter+1c9 () | I can provide the full crash dump if it will help... once it comes back up. Looks like it crashed again while I was writing this. Thanks, Aaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Brock at 2hoffshore.com Mon Feb 9 17:41:35 2015 From: Robert.Brock at 2hoffshore.com (Robert A. Brock) Date: Mon, 9 Feb 2015 17:41:35 +0000 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: References: Message-ID: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> >From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Aaron Curry >Sent: 09 February 2015 17:28 >To: omnios-discuss at lists.omniti.com >Subject: [OmniOS-discuss] File Server Crash > >Hi all, > >We recently set up an OmniOS file server to serve our SMB clients. The server paniced over the weekend. I was hoping someone here would help us interpret the panic info and possibly point us towards a resolution. The stack dump is: > > ? ? ? ? ? ? ? panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff007c6e78a0 addr=18 occurred in module "nsmb" due to a NULL pointer dereference >? ? ? ? ? ? ? ? panicstack = unix:die+df () | unix:trap+db3 () | unix:cmntrap+e6 () | nsmb:nbssn_peekhdr+75 () | nsmb:nbssn_recv+7f () | nsmb:smb_nbst_recv+a1 () | nsmb:smb_iod_recv1+46 () | nsmb:smb_iod_recvall+65 () | nsmb:smb_iod_vc_work+c6 () | nsmb:smb_usr_iod_work+15c () | nsmb:nsmb_ioctl+d9 () | >genunix:cdev_ioctl+39 () | specfs:spec_ioctl+60 () | genunix:fop_ioctl+55 () | genunix:ioctl+9b () | unix:brand_sys_sysenter+1c9 () | > > >I can provide the full crash dump if it will help... once it comes back up. Looks like it crashed again while I was writing this. > >Thanks, > >Aaron Smells like this: https://github.com/Nexenta/illumos-nexenta/commit/9a60a9d168313d9c350f813abccc6320879ad9b5 Not sure if this fix has made it to OmniOS. From asc1111 at gmail.com Mon Feb 9 18:12:23 2015 From: asc1111 at gmail.com (Aaron Curry) Date: Mon, 9 Feb 2015 11:12:23 -0700 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> Message-ID: Robert, thanks for looking at this. Looks like that fix was committed to OmniOS last September: https://github.com/omniti-labs/illumos-omnios/commit/df1c321ce1e8da6abd16cf562f5cef4ecd9e89c6 Does that mean it made it into r151012 which was released in October? This server is currently running r151010. Can we assume that an update to r151012 will fix the issue? Thanks again, Aaron On Mon, Feb 9, 2015 at 10:41 AM, Robert A. Brock < Robert.Brock at 2hoffshore.com> wrote: > >From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On > Behalf Of Aaron Curry > >Sent: 09 February 2015 17:28 > >To: omnios-discuss at lists.omniti.com > >Subject: [OmniOS-discuss] File Server Crash > > > >Hi all, > > > >We recently set up an OmniOS file server to serve our SMB clients. The > server paniced over the weekend. I was hoping someone here would help us > interpret the panic info and possibly point us towards a resolution. The > stack dump is: > > > > panicstr = BAD TRAP: type=e (#pf Page fault) > rp=ffffff007c6e78a0 addr=18 occurred in module "nsmb" due to a NULL pointer > dereference > > panicstack = unix:die+df () | unix:trap+db3 () | > unix:cmntrap+e6 () | nsmb:nbssn_peekhdr+75 () | nsmb:nbssn_recv+7f () | > nsmb:smb_nbst_recv+a1 () | nsmb:smb_iod_recv1+46 () | > nsmb:smb_iod_recvall+65 () | nsmb:smb_iod_vc_work+c6 () | > nsmb:smb_usr_iod_work+15c () | nsmb:nsmb_ioctl+d9 () | > >genunix:cdev_ioctl+39 () | specfs:spec_ioctl+60 () | genunix:fop_ioctl+55 > () | genunix:ioctl+9b () | unix:brand_sys_sysenter+1c9 () | > > > > > >I can provide the full crash dump if it will help... once it comes back > up. Looks like it crashed again while I was writing this. > > > >Thanks, > > > >Aaron > > Smells like this: > https://github.com/Nexenta/illumos-nexenta/commit/9a60a9d168313d9c350f813abccc6320879ad9b5 > > Not sure if this fix has made it to OmniOS. > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mir at miras.org Mon Feb 9 18:41:49 2015 From: mir at miras.org (Michael Rasmussen) Date: Mon, 9 Feb 2015 19:41:49 +0100 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> Message-ID: <20150209194149.76f210d0@sleipner.datanom.net> On Mon, 9 Feb 2015 11:12:23 -0700 Aaron Curry wrote: > > Does that mean it made it into r151012 which was released in October? This > server is currently running r151010. Can we assume that an update to > r151012 will fix the issue? > Yup. https://github.com/omniti-labs/illumos-omnios/commits/r151012 Scroll down to 'Commits on Sep 8, 2014' -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- /usr/games/fortune -es says: They also surf who only stand on waves. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From danmcd at omniti.com Mon Feb 9 19:07:33 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 9 Feb 2015 14:07:33 -0500 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> Message-ID: > On Feb 9, 2015, at 12:41 PM, Robert A. Brock wrote: > > > Smells like this: https://github.com/Nexenta/illumos-nexenta/commit/9a60a9d168313d9c350f813abccc6320879ad9b5 > > Not sure if this fix has made it to OmniOS. That fix was specially backported to r151012 just before that branch closed for its release. If the original poster is running long-term Support or even r151010, that would explain it. If this is happening on r151012 or bloody, please provide the crash dump, Aaron, as it's NOT the aforementioned bug. Dan From asc1111 at gmail.com Mon Feb 9 19:41:01 2015 From: asc1111 at gmail.com (Aaron Curry) Date: Mon, 9 Feb 2015 12:41:01 -0700 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> Message-ID: Thanks for confirming that. We will be updating tonight. Hopefully we don't see this again! Aaron On Mon, Feb 9, 2015 at 12:07 PM, Dan McDonald wrote: > > > On Feb 9, 2015, at 12:41 PM, Robert A. Brock < > Robert.Brock at 2hoffshore.com> wrote: > > > > > > Smells like this: > https://github.com/Nexenta/illumos-nexenta/commit/9a60a9d168313d9c350f813abccc6320879ad9b5 > > > > Not sure if this fix has made it to OmniOS. > > That fix was specially backported to r151012 just before that branch > closed for its release. If the original poster is running long-term > Support or even r151010, that would explain it. > > If this is happening on r151012 or bloody, please provide the crash dump, > Aaron, as it's NOT the aforementioned bug. > > Dan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Mon Feb 9 19:41:35 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 9 Feb 2015 14:41:35 -0500 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> Message-ID: <8957ABD5-4EC2-4D1F-A1C1-C4445A06C3C3@omniti.com> > On Feb 9, 2015, at 2:41 PM, Aaron Curry wrote: > > Thanks for confirming that. We will be updating tonight. Hopefully we don't see this again! Can you tell me what you *were* running? (For a reference point?) Thanks, Dan From asc1111 at gmail.com Mon Feb 9 20:27:20 2015 From: asc1111 at gmail.com (Aaron Curry) Date: Mon, 9 Feb 2015 13:27:20 -0700 Subject: [OmniOS-discuss] File Server Crash In-Reply-To: <8957ABD5-4EC2-4D1F-A1C1-C4445A06C3C3@omniti.com> References: <2859482C466CCA42AD9B84B9F56212300232BC2A@2H199.2hukwok2.local> <8957ABD5-4EC2-4D1F-A1C1-C4445A06C3C3@omniti.com> Message-ID: This server is currently running r151010. Aaron On Mon, Feb 9, 2015 at 12:41 PM, Dan McDonald wrote: > > > On Feb 9, 2015, at 2:41 PM, Aaron Curry wrote: > > > > Thanks for confirming that. We will be updating tonight. Hopefully we > don't see this again! > > Can you tell me what you *were* running? (For a reference point?) > > Thanks, > Dan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Tue Feb 10 02:22:33 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 9 Feb 2015 21:22:33 -0500 Subject: [OmniOS-discuss] Small update to UNZIP covers CVE 2014-9636 Message-ID: <87BB14EB-DFF5-4FDF-A184-7C7FE6468F18@omniti.com> Thanks to Marissa Murphy for patching this. It's now available for "pkg update" on all supported repos: r151006/LTS, r151010/last-Stable, r151012/Stable, and r151013/bloody. Won't require a reboot, nor service restarts since these are just command replacements. Dan From marc.edwards at siliconcloudinternational.com Tue Feb 10 23:20:34 2015 From: marc.edwards at siliconcloudinternational.com (J Marc Edwards) Date: Tue, 10 Feb 2015 18:20:34 -0500 Subject: [OmniOS-discuss] Installation of OmniOS on Intel NUC... Message-ID: <026f01d04588$2ccbd8d0$86638a70$@siliconcloudinternational.com> I'm working on a desktop-like installation of OmniOS on an Intel NUC (http://www.intel.com/content/www/us/en/nuc/nuc-kit-d54250wykh.html). I have previously installed SmartOS on this NUC with no issues. I built the USB image, but when I boot the NUC, I am dropped into a grub shell with no instructions. Any thoughts on what I should be seeing upon installation startup for OmniOS? I'll get more into the description of my next phase of installation & configuration once I get OmniOS installed. Kind regards, Marc -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Thu Feb 12 01:06:42 2015 From: danmcd at omniti.com (Dan McDonald) Date: Wed, 11 Feb 2015 20:06:42 -0500 Subject: [OmniOS-discuss] Small update to UNZIP covers CVE 2014-9636 In-Reply-To: References: Message-ID: <9FF847D2-0D79-4B12-A2D3-BBBD30E9FC7A@omniti.com> > On Feb 9, 2015, at 9:22 PM, Dan McDonald wrote: > > Thanks to Marissa Murphy for patching this. It's now available for "pkg update" on all supported repos: r151006/LTS, r151010/last-Stable, r151012/Stable, and r151013/bloody. > > Won't require a reboot, nor service restarts since these are just command replacements. You'll have to re-update unzip again, as the upstream (oss-security mailing list) discussion found a problem with the first fix. Sorry about that! Dan From tim at multitalents.net Thu Feb 12 05:32:26 2015 From: tim at multitalents.net (Tim Rice) Date: Wed, 11 Feb 2015 21:32:26 -0800 (PST) Subject: [OmniOS-discuss] Small update to UNZIP covers CVE 2014-9636 In-Reply-To: <9FF847D2-0D79-4B12-A2D3-BBBD30E9FC7A@omniti.com> References: <9FF847D2-0D79-4B12-A2D3-BBBD30E9FC7A@omniti.com> Message-ID: On Wed, 11 Feb 2015, Dan McDonald wrote: | | > On Feb 9, 2015, at 9:22 PM, Dan McDonald wrote: | > | > Thanks to Marissa Murphy for patching this. It's now available for "pkg update" on all supported repos: r151006/LTS, r151010/last-Stable, r151012/Stable, and r151013/bloody. | > | > Won't require a reboot, nor service restarts since these are just command replacements. | | You'll have to re-update unzip again, as the upstream (oss-security mailing list) discussion found a problem with the first fix. And possibly once more. Have a look at CVE-2014-8139 discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1174844 | Sorry about that! These things happen. | Dan | -- Tim Rice Multitalents (707) 456-1146 tim at multitalents.net From Joseph.Cardenuto at gdit.com Thu Feb 12 22:12:46 2015 From: Joseph.Cardenuto at gdit.com (Cardenuto, Joseph P) Date: Thu, 12 Feb 2015 22:12:46 +0000 Subject: [OmniOS-discuss] omnios problem with pc card and sd card Message-ID: <77AE85424D6B1C4FA5F25AB148B55E694F84A479@HQ200-EXMBX03.ad.local> Good Evening I am reaching out to see if anyone can help me with a problem. I am having a problem with OmniOS and I am hoping someone can help me solve the issue. I installed OmniOS on a Getac 300G ruggedized laptop. The Machines contain an "Intel Core i7 vpro" and a Ricoh R5U242 (for the PC and SD card). We are having problems getting the PC and SD cards to be recognized. The problem we are having is twofold. First the SD card is not even recognized so no interrupts are triggered. Second when I insert the pc card which is a 250 mb drive I receive an error from pcmcia_map_reg. The following is a dtrace snippet from inserting the card. From viewing the code the highlighted section is where it appears the problem is. I would appreciate any help you can provide in solving this problem. 3 -> SocketServices [957780418290] pcmcia:SocketServices:entry 3 -> SSSetWindow [957780420042] pcmcia:SSSetWindow:entry 3 -> pcic_set_window [957780421870] pcic:pcic_set_window:entry 3 -> pcmcia_alloc_mem [957780424048] pcmcia:pcmcia_alloc_mem:entry 3 -> pcmcia_ra_alloc [957780425947] pcmcia:pcmcia_ra_alloc:entry 3 -> ddi_prop_free [957780432267] genunix:ddi_prop_free:entry 3 <- ddi_prop_free [957780434818] genunix:ddi_prop_free:return 3 <- pcmcia_ra_alloc [957780436735] pcmcia:pcmcia_ra_alloc:return 3 <- pcmcia_alloc_mem [957780438426] pcmcia:pcmcia_alloc_mem:return 3 -> pcmcia_map_reg [957780440700] pcmcia:pcmcia_map_reg:entry 3 -> pcmcia_cons_regspec [957780442737] pcmcia:pcmcia_cons_regspec:entry 3 <- pcmcia_cons_regspec [957780445271] pcmcia:pcmcia_cons_regspec:return 3 -> ddi_map [957780448254] genunix:ddi_map:entry 3 -> pcieb_bus_map [957780451272] pcieb:pcieb_bus_map:entry 3 -> pci_set_prot_scan [957780457548] npe:pci_set_prot_scan:entry 3 <- pci_set_prot_scan [957780459853] npe:pci_set_prot_scan:return 3 -> npe_bus_map [957780454444] npe:npe_bus_map:entry 3 -> pci_common_get_reg_prop [957780457548] npe:pci_common_get_reg_prop:entry 3 <- pci_common_get_reg_prop [957780459853] npe:pci_common_get_reg_prop:return 3 -> npe_is_mmcfg_supported [957780462061] npe:npe_is_mmcfg_supported:entry 3 -> npe_child_is_pci [957780466185] npe:npe_child_is_pci:entry 3 -> ddi_prop_free [957780468052] genunix:ddi_prop_free:entry 3 <- ddi_prop_free [957780469435] genunix:ddi_prop_free:return 3 -> ddi_prop_free [957780471001] genunix:ddi_prop_free:entry 3 <- ddi_prop_free [957780472417] genunix:ddi_prop_free:return 3 <- npe_child_is_pci [957780474081] npe:npe_child_is_pci:return 3 <- npe_is_mmcfg_supported [957780475909] npe:npe_is_mmcfg_supported:return 3 -> ddi_prop_lookup_int64_array [957780457548] genunix:ddi_prop_lookup_int64_array:entry 3 <- ddi_prop_lookup_int64_array [957780459853] genunix:ddi_prop_lookup_int64_array:return 3 -> ddi_prop_free [957780471001] genunix:ddi_prop_free:entry 3 <- ddi_prop_free [957780472417] genunix:ddi_prop_free:return 3 <- npe_bus_map [957780479691] npe:npe_bus_map:return 3 <- pcieb_bus_map [957780481368] pcieb:pcieb_bus_map:return 3 <- ddi_map [957780483673] genunix:ddi_map:return 3 <- pcmcia_map_reg [957780485827] pcmcia:pcmcia_map_reg:return 3 -> pcmcia_free_mem [957780501512] pcmcia:pcmcia_free_mem:entry 3 -> pcmcia_ra_free [957780513753] pcmcia:pcmcia_ra_free:entry 3 -> ddi_prop_free [957780516005] genunix:ddi_prop_free:entry 3 <- ddi_prop_free [957780517440] genunix:ddi_prop_free:return 3 <- pcmcia_ra_free [957780519082] pcmcia:pcmcia_ra_free:return 3 <- pcmcia_free_mem [957780520734] pcmcia:pcmcia_free_mem:return 3 <- pcic_set_window [957780522347] pcic:pcic_set_window:return 3 <- SSSetWindow [957780524357] pcmcia:SSSetWindow:return 3 <- SocketServices [957780525962] pcmcia:SocketServices:return Thank you in advance for any help you can provide, Joe __________________________________________ Joseph (Joe) P. Cardenuto Principal Engineer -- Software General Dynamics Information Technology 55 Dodge Road, Suite 120 Getzville, NY 14068 Tel: (716) 243-4052 Fax: (716) 243-4011 Email: joseph.cardenuto at gdit.com [cid:image001.gif at 01CE41DA.90630ED0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 2766 bytes Desc: image001.gif URL: From omnios.discuss at virtualunix.eu Fri Feb 13 09:56:23 2015 From: omnios.discuss at virtualunix.eu (Michael) Date: Fri, 13 Feb 2015 09:56:23 +0000 Subject: [OmniOS-discuss] Zone Discrepancy Message-ID: I've run into this issue a couple of times under different circumstances. In my original setup I used a template zone and then created a number of zones that were cloned from it. Somewhere along the line one of the clones would become promoted instead, which messed things up somewhat if I wanted to create a new clone from the template zone. After a hardware upgrade I decided to use full installs for each zone instead of cloning to avoid the above situation, recently I saw the same thing happen again, ie after initial install zones/tmpl0 ~500MB zones/0 ~500MB zones/1 ~500MB and now zones/tmpl0 ~30MB zones/0 ~500MB zones/1 ~1000MB None of the zones had been detached when I noticed this, detaching/attaching/updating didn't change anything. While it's not such a big deal in this scenario because I'm not cloning (at this time) I can't see why this is occurring in the first place. Is this something anyone else is experiencing or any pointers in how to remedy this? Michael From cj.keist at colostate.edu Mon Feb 16 23:44:59 2015 From: cj.keist at colostate.edu (CJ Keist) Date: Mon, 16 Feb 2015 16:44:59 -0700 Subject: [OmniOS-discuss] 3ware LSI 9750-16i4e Message-ID: <54E280FB.3010805@colostate.edu> Hello, Looking for anyone that might have used this raid controller card? LSI web page states the raid controller is supported for OpenSolaris, but I'm not having any luck getting OmniOS 151012 to see any defined drives from the raid card. I've tried to install driver from LSI but get the following error: root at projects1:/root/install/intel/Solaris11# pkgadd -d . The following packages are available: 1 mrsas LSI MegaRAID SAS 2.0 HBA driver (i386) 6.602.03.00 Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]: Processing package instance from LSI MegaRAID SAS 2.0 HBA driver(i386) 6.602.03.00 Copyright (c) 2008-2009, LSI Logic Corporation. All rights reserved. ## Executing checkinstall script. pkgparam: ERROR: unable to locate parameter information for "mrsas" A previous instance of mrsas driver found in the system. Use 'pkgrm mrsas' to remove the previous mrsas package and then do a pkgadd. checkinstall script suspends Installation of was suspended (administration). No changes were made to the system. Try to remove the SUNWmrsas package gets me this: root at projects1:/root# pkgrm SUNWmrsas The following package is currently installed: SUNWmrsas LSI MegaRAID SAS2.0 HBA Driver (i386) 11.11,REV=2009.11.11 Do you want to remove this package? [y,n,?,q] y pkgrm: ERROR: unable to change current working directory to Removal of failed (internal error). No changes were made to the system. Anything I'm missing here? -- C. J. Keist Email: cj.keist at colostate.edu Systems Group Manager Solaris 10 OS (SAI) Engineering Network Services Phone: 970-491-0630 College of Engineering, CSU Fax: 970-491-5569 Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness' From danmcd at omniti.com Tue Feb 17 01:30:30 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 16 Feb 2015 20:30:30 -0500 Subject: [OmniOS-discuss] 3ware LSI 9750-16i4e In-Reply-To: <54E280FB.3010805@colostate.edu> References: <54E280FB.3010805@colostate.edu> Message-ID: <62EC32E5-67D7-47A8-8E18-27E16466F572@omniti.com> > On Feb 16, 2015, at 6:44 PM, CJ Keist wrote: > > Hello, > Looking for anyone that might have used this raid controller card? LSI web page states the raid controller is supported for OpenSolaris, but I'm not having any luck getting OmniOS 151012 to see any defined drives from the raid card. First off... illumos != Solaris. Stop thinking that way. Oracle burned that bridge. Second off, the LSI 2108 chipset SHOULD be supported. I'd be curious for you to share "prtconf | grep 1000," so I can see the PCIe device identifier. 2108 is an older chipset. > Anything I'm missing here? Yes. You shouldn't be getting a hardware RAID card for a ZFS filesystem. This card is based on the LSI 2108 chipset, i.e. HW-RAID. The ONLY reason I can see you wanting such a card is because of the 16i4e (16 + 4). Ideally, you want something based on either the 2008, 2308, or as of r151012, the 3008 chipset. Try something on this page: http://www.lsi.com/products/host-bus-adapters/pages/default.aspx#tab/product-family-tab-1 Or http://www.lsi.com/products/host-bus-adapters/pages/default.aspx#tab/product-family-tab-2 Hope this helps, Dan From jimklimov at cos.ru Tue Feb 17 07:19:01 2015 From: jimklimov at cos.ru (Jim Klimov) Date: Tue, 17 Feb 2015 08:19:01 +0100 Subject: [OmniOS-discuss] 3ware LSI 9750-16i4e In-Reply-To: <54E280FB.3010805@colostate.edu> References: <54E280FB.3010805@colostate.edu> Message-ID: 17 ??????? 2015??. 0:44:59 CET, CJ Keist ?????: >Hello, > Looking for anyone that might have used this raid controller card? >LSI web page states the raid controller is supported for OpenSolaris, >but I'm not having any luck getting OmniOS 151012 to see any defined >drives from the raid card. > >I've tried to install driver from LSI but get the following error: >root at projects1:/root/install/intel/Solaris11# pkgadd -d . > >The following packages are available: > 1 mrsas LSI MegaRAID SAS 2.0 HBA driver > (i386) 6.602.03.00 > >Select package(s) you wish to process (or 'all' to process >all packages). (default: all) [?,??,q]: > >Processing package instance from > > >LSI MegaRAID SAS 2.0 HBA driver(i386) 6.602.03.00 >Copyright (c) 2008-2009, LSI Logic Corporation. >All rights reserved. > >## Executing checkinstall script. >pkgparam: ERROR: unable to locate parameter information for "mrsas" >A previous instance of mrsas driver found in the system. >Use 'pkgrm mrsas' to remove the previous mrsas package and then do a >pkgadd. >checkinstall script suspends > >Installation of was suspended (administration). >No changes were made to the system. > >Try to remove the SUNWmrsas package gets me this: > >root at projects1:/root# pkgrm SUNWmrsas > >The following package is currently installed: > SUNWmrsas LSI MegaRAID SAS2.0 HBA Driver > (i386) 11.11,REV=2009.11.11 > >Do you want to remove this package? [y,n,?,q] y >pkgrm: ERROR: unable to change current working directory to > > >Removal of failed (internal error). >No changes were made to the system. > > >Anything I'm missing here? A couple of things pop out. One is that failures are usually programmatic, i.e. due to tests or other actions in a pre- or post- install/remove scripts. You can inspect those (an SVR4 package is a glorified cpio archive containing the metadata files and ultimate packaged contents - maybe inlaid in a none.(gz,7z,...) archive. More scientifically, you can use pkgtrans to pack\unpack the package delivery formats, such as a single file, a directory, etc. With some handywork (also fixing pkginfo with its cksum-based checksums), you can instrument the pre\post scripts with set -x to trace and repackage, to see what does not work why during an actual installation. It may be that uninstallation fails due to poor legacy dependencies, e.g. if SUNWcs (~kernel) 'depends' on some driver packages like this storage, so you can't remove the old one, and a new driver package may consider itself to be in conflict with the old one. I'm enroute away from a computer now so can't say if this is still the case with illumos metadata. But if this is indeed the roadblock, you might need some pointier surgical tools or a larger hammer to add/remove the driver files and/or the packagjng system's perception of the package. Also look in dmesg - probably some driver modules fail to attach or detach, although generally installation of a dormant driver into an OS image should not depend on actual presence of a device. Oh, also you can clone and mount an alternate BE and use the pkgadd/pkgrm with -R to manipulate this other OS image, maybe then the hardware related checks would be more relaxed. Finally, the 3wares i knew (before they were bought by lsi) were using a chip and firmware of their own. I did not use the later products, but if they are not just a rebrand of mainstream lsi with another known brand - double-check that you need an lsi/megaraid driver or some other after all. HTH, Jim Klimov -- Typos courtesy of K-9 Mail on my Samsung Android From b3niup at gmail.com Tue Feb 17 14:31:38 2015 From: b3niup at gmail.com (=?UTF-8?Q?Benedykt_Przyby=C5=82o?=) Date: Tue, 17 Feb 2015 15:31:38 +0100 Subject: [OmniOS-discuss] L2ARC issue - 16.0E SSD? Message-ID: Hello, I am having problems with L2ARC SSD drives since upgrade to omnios-10b9c79. After filling cache up l2arc size changes to 16.0E and zpool iostat -v shows that cache is still growing (at this moment it is using 2.60T out of 16.0E available space). After removing l2arc drive from pool and reattaching it problem disappears until cache grows up again. Problem affects two machines with similar setup (and my only two machines that runs omnios-10b9c79). For l2arc purpose I am using Intel SSD DC S3500 480GB SATA 2,5". Few system details: # uname -a SunOS nb3 5.11 omnios-10b9c79 i86pc i386 i86pc # zpool status | grep cache -A 1 cache c1t55CD2E404B575172d0 ONLINE 0 0 0 # zpool iostat -v | grep cache -A 1 cache - - - - - - c1t55CD2E404B575172d0 2.60T 16.0E 735 200 1.37M 2.73M # kstat zfs::arcstats:*l2* module: zfs instance: 0 name: arcstats class: misc evict_l2_cached 29975692498432 evict_l2_eligible 10552753603072 evict_l2_ineligible 1821252098048 l2_abort_lowmem 1220 l2_asize 2856707439616 l2_cksum_bad 584529772 l2_compress_failures 0 l2_compress_successes 102235818 l2_compress_zeros 0 l2_evict_lock_retry 409 l2_evict_reading 0 l2_feeds 1300435 l2_free_on_write 11906910 l2_hdr_size 34884171088 l2_hits 765898065 l2_io_error 103979415 l2_misses 1910226514 l2_read_bytes 1495882560512 l2_rw_clash 535147 l2_size 3757422951424 l2_write_bytes 2984588264448 l2_writes_done 916469 l2_writes_error 0 l2_writes_hdr_miss 65671 l2_writes_sent 916469 # fmadm faulty # fmdump -eV TIME CLASS fmdump: warning: /var/fm/fmd/errlog is empty # zpool get version DATA3-BACKUP NAME PROPERTY VALUE SOURCE DATA3-BACKUP version 28 local I found similar issue in freenas: https://bugs.freenas.org/issues/6239 With such patch as solution: https://bugs.freenas.org/projects/freenas/repository/trueos/revisions/6ec48ebf5a1596ec7d2732e891fce3f116105ae5/diff/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c I would like to know if this will be fixed in a stable release soon or if there will be any patch that I will be able to apply to my current environment to fix this? Best regards, Benedykt Przyby?o -------------- next part -------------- An HTML attachment was scrubbed... URL: From cj.keist at colostate.edu Tue Feb 17 16:20:50 2015 From: cj.keist at colostate.edu (CJ Keist) Date: Tue, 17 Feb 2015 09:20:50 -0700 Subject: [OmniOS-discuss] 3ware LSI 9750-16i4e In-Reply-To: <62EC32E5-67D7-47A8-8E18-27E16466F572@omniti.com> References: <54E280FB.3010805@colostate.edu> <62EC32E5-67D7-47A8-8E18-27E16466F572@omniti.com> Message-ID: <54E36A62.8040406@colostate.edu> Thank you. Yes, realize that Illumos is not Solaris. But I know that Illumos is a branch from OpenSolaris after Oracle decided they didn't want to share the same sandbox with everyone else. I thought it safe to purchase this raid controller. This controller does support JBOD in the that you can define each hard disk as a single drive. Just for information. I was able to remove the SUNWmrsas package. 1. mkdir /var/sadm/pkg/SUNWmrsas/install 2. edit /var/sadm/pkg/SUNWmrsas/pkginfo Add "CLASSES=none". Then pkgrm worked. 3. touch /reconfigure and reboot. But the LSI driver would not load, still thought mr_sas driver was present. So modified the checkinstall script to pass. But then got error: pkgadd: ERROR: packaging file is corrupt file size <1391> expected <1071> actual file cksum <47879> expected <23422> actual Edit pkgmap file and changed line: 1 i checkinstall 1391 47879 1384458730 to 1 i checkinstall 1071 23422 1384458730 That got the LSI drive installed. Again "touch /reconfigure" and reboot. But even after all that no luck in seeing the defined drives. modinfo didn't show any sas driver running, So did: modload -p drv/mr_sas modinfo | grep sas 255 fffffffff8125000 43da0 179 1 mr_sas (6.602.03.00) devfsadm Still nothing. Looks like I'll purchase a different raid card. Any recommendations? I have system with Intel based mother board: E99552-510 16 4TB internal drives plus a SAS expansion with another 16 4TB hard drives. On 2/16/15 6:30 PM, Dan McDonald wrote: > >> On Feb 16, 2015, at 6:44 PM, CJ Keist wrote: >> >> Hello, >> Looking for anyone that might have used this raid controller card? LSI web page states the raid controller is supported for OpenSolaris, but I'm not having any luck getting OmniOS 151012 to see any defined drives from the raid card. > > First off... illumos != Solaris. Stop thinking that way. Oracle burned that bridge. > > Second off, the LSI 2108 chipset SHOULD be supported. I'd be curious for you to share "prtconf | grep 1000," so I can see the PCIe device identifier. 2108 is an older chipset. > >> Anything I'm missing here? > > Yes. You shouldn't be getting a hardware RAID card for a ZFS filesystem. This card is based on the LSI 2108 chipset, i.e. HW-RAID. The ONLY reason I can see you wanting such a card is because of the 16i4e (16 + 4). > > Ideally, you want something based on either the 2008, 2308, or as of r151012, the 3008 chipset. Try something on this page: > > http://www.lsi.com/products/host-bus-adapters/pages/default.aspx#tab/product-family-tab-1 > > Or > > http://www.lsi.com/products/host-bus-adapters/pages/default.aspx#tab/product-family-tab-2 > > Hope this helps, > Dan > -- C. J. Keist Email: cj.keist at colostate.edu Systems Group Manager Solaris 10 OS (SAI) Engineering Network Services Phone: 970-491-0630 College of Engineering, CSU Fax: 970-491-5569 Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness' From danmcd at omniti.com Tue Feb 17 16:28:46 2015 From: danmcd at omniti.com (Dan McDonald) Date: Tue, 17 Feb 2015 11:28:46 -0500 Subject: [OmniOS-discuss] 3ware LSI 9750-16i4e In-Reply-To: <54E36A62.8040406@colostate.edu> References: <54E280FB.3010805@colostate.edu> <62EC32E5-67D7-47A8-8E18-27E16466F572@omniti.com> <54E36A62.8040406@colostate.edu> Message-ID: I don't see a PCI ID for your board. It is POSSIBLE the BUILT-IN mr_sas on OmniOS will do what you want, but I *need* to see that PCI ID for your board. Dan From omnios at citrus-it.net Wed Feb 18 13:41:30 2015 From: omnios at citrus-it.net (Andy) Date: Wed, 18 Feb 2015 13:41:30 +0000 (GMT) Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. Message-ID: Hi, We have a number of Dell PE1950 servers running OmniOS which are in need of replacement. Management here have always used Dell and have suggested the R730xd platform (as we want more disk slots in the new servers). I've read some useful posts on this mailing list about the Dell R730xd - specifically that if we ask Dell to configure it for Nexenta then it will come with an LSI HBA and Intel NICs but I haven't seen any further posts saying whether anyone has had any success getting one of these up and running. I got the impression from the posts that it wasn't a great idea and Supermicro is a better option. I'd prefer to run Supermicro but might have some problems convincing those with the purse strings. Is anyone running, or can anyone recommend, a Supermicro server roughly equivalent to the Del R730xd, or give me an idea of what chipsets/HBAs etc. to choose or avoid for OmniOS? Any help appreciated, Thanks Andy -- Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From illumos at cucumber.demon.co.uk Wed Feb 18 14:09:56 2015 From: illumos at cucumber.demon.co.uk (Andrew Gabriel) Date: Wed, 18 Feb 2015 14:09:56 +0000 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: <54E49D34.9090709@cucumber.demon.co.uk> In customers I've dealt with, if there isn't a binding data center policy in place for one or the other, it's usually a trade-off between TCO, and the differing field service offerings with respect to mean time to repair (which can be very location-specific). This can also be influenced by the ability of your own data center staff to diagnose/repair hardware faults and your own spares holding, and of course the availability requirements of your systems. Andy wrote: > Hi, > > We have a number of Dell PE1950 servers running OmniOS which are in need > of replacement. Management here have always used Dell and have suggested > the R730xd platform (as we want more disk slots in the new servers). > > I've read some useful posts on this mailing list about the Dell R730xd - > specifically that if we ask Dell to configure it for Nexenta then it will > come with an LSI HBA and Intel NICs but I haven't seen any further posts > saying whether anyone has had any success getting one of these up and > running. I got the impression from the posts that it wasn't a great idea > and Supermicro is a better option. > > I'd prefer to run Supermicro but might have some problems convincing > those with the purse strings. Is anyone running, or can anyone recommend, > a Supermicro server roughly equivalent to the Del R730xd, or give me an > idea of what chipsets/HBAs etc. to choose or avoid for OmniOS? > > Any help appreciated, > > Thanks > > Andy > > From doug at will.to Wed Feb 18 14:07:36 2015 From: doug at will.to (Doug Hughes) Date: Wed, 18 Feb 2015 09:07:36 -0500 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: <71069fcd-7674-4e3f-8ded-a630df3258c7.maildroid@localhost> The nice thing about Supermicro is that they have much for platform variation than just about anybody else. However, if you need the r730 with the 1.8 disks, SM does not have an option for that. You can, however, easily get 24 2.5" disks in the SC216 (or equivalent) platform, and they have several platforms in 1U and 2U that can accommodate a wide variety of 3.5"disk choices. Which are you looking for? You probably want to give an integrator your specs and let them suggest one. There are many available. Berkeley Communications (my favorite) Microway Colfax many more. You also might look at PSSC labs to see if one of their items fits. Sent from my android device. -----Original Message----- From: Andy To: omnios-discuss at lists.omniti.com Sent: Wed, 18 Feb 2015 8:51 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. Hi, We have a number of Dell PE1950 servers running OmniOS which are in need of replacement. Management here have always used Dell and have suggested the R730xd platform (as we want more disk slots in the new servers). I've read some useful posts on this mailing list about the Dell R730xd - specifically that if we ask Dell to configure it for Nexenta then it will come with an LSI HBA and Intel NICs but I haven't seen any further posts saying whether anyone has had any success getting one of these up and running. I got the impression from the posts that it wasn't a great idea and Supermicro is a better option. I'd prefer to run Supermicro but might have some problems convincing those with the purse strings. Is anyone running, or can anyone recommend, a Supermicro server roughly equivalent to the Del R730xd, or give me an idea of what chipsets/HBAs etc. to choose or avoid for OmniOS? Any help appreciated, Thanks Andy -- Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Wed Feb 18 15:34:18 2015 From: danmcd at omniti.com (Dan McDonald) Date: Wed, 18 Feb 2015 10:34:18 -0500 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: <94F01EF5-4FF3-4737-98CE-097D81D6ED80@omniti.com> > On Feb 18, 2015, at 8:41 AM, Andy wrote: > > I've read some useful posts on this mailing list about the Dell R730xd - > specifically that if we ask Dell to configure it for Nexenta then it will > come with an LSI HBA and Intel NICs but I haven't seen any further posts > saying whether anyone has had any success getting one of these up and > running. I got the impression from the posts that it wasn't a great idea > and Supermicro is a better option. The key is to get an HBA illumos likes. The *default* Dell HBAs aren't all that great, and stick you with HW-RAID. I've not heard success stories with "Nexenta Configured" + OmniOS, but that's because every Nexenta configuration was used to... well... run NexentaStor. :) I know lots of illumos shops run SuperMicro with success. Dan From lkateley at kateley.com Wed Feb 18 15:49:15 2015 From: lkateley at kateley.com (Linda Kateley) Date: Wed, 18 Feb 2015 09:49:15 -0600 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: <94F01EF5-4FF3-4737-98CE-097D81D6ED80@omniti.com> References: <94F01EF5-4FF3-4737-98CE-097D81D6ED80@omniti.com> Message-ID: <54E4B47B.1040205@kateley.com> Just some anecdotal info... sun had a very nice partnership with dell.. back in the day, if you searched for dell in the hardware compat list you would show thousand of entries.. Dell now has a guy who sits on nexenta's board, this fact might have some impact in what they recommend. On the other hand, most if not all of the companies selling openzfs appliances are running supermicro or similar gear...Except for compellent which definitely runs dell :) I agree with dan though, the hba is the most important, as is nics. All this being said, I would personally prefer that dell r730 if i had a choice... lk On 2/18/15 9:34 AM, Dan McDonald wrote: >> On Feb 18, 2015, at 8:41 AM, Andy wrote: >> >> I've read some useful posts on this mailing list about the Dell R730xd - >> specifically that if we ask Dell to configure it for Nexenta then it will >> come with an LSI HBA and Intel NICs but I haven't seen any further posts >> saying whether anyone has had any success getting one of these up and >> running. I got the impression from the posts that it wasn't a great idea >> and Supermicro is a better option. > The key is to get an HBA illumos likes. The *default* Dell HBAs aren't all that great, and stick you with HW-RAID. > > I've not heard success stories with "Nexenta Configured" + OmniOS, but that's because every Nexenta configuration was used to... well... run NexentaStor. :) > > I know lots of illumos shops run SuperMicro with success. > > Dan > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From rt at steait.net Wed Feb 18 20:04:18 2015 From: rt at steait.net (Rune Tipsmark) Date: Wed, 18 Feb 2015 20:04:18 +0000 Subject: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog Message-ID: <1424289858252.20555@steait.net> hi all, I found an entry about zil_slog_limit here: http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWritesAndZILII it basically explains how writes larger than 1MB per default hits the main pool rather than my Slog device - I could not find much further information nor the equivalent setting in OmniOS. I also read http://nex7.blogspot.ca/2013/04/zfs-intent-log.html but it didn't truly help me understand just how I can force every written byte to my ZFS box to go the ZIL regardless of size, I never ever want anything to go directly to my man pool ever. I have sync=always and disabled write back cache on my volume based LU's. Testing with zfs_txg_timeout set to 30 or 60 seconds seems to make no difference if I write large files to my LU's - I don't seem the write speed being consistent with the performance of the Slog devices. It looks as if it goes straight to disk and hence the performance is less than great to say the least. How do I ensure 100% that all writes always goes to my Slog devices - no exceptions. br, Rune -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Wed Feb 18 20:26:55 2015 From: chip at innovates.com (Schweiss, Chip) Date: Wed, 18 Feb 2015 14:26:55 -0600 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: On Wed, Feb 18, 2015 at 7:41 AM, Andy wrote: > > > I'd prefer to run Supermicro but might have some problems convincing > those with the purse strings. Is anyone running, or can anyone recommend, > a Supermicro server roughly equivalent to the Del R730xd, or give me an > idea of what chipsets/HBAs etc. to choose or avoid for OmniOS? > > I've been quite happy with the 6028U-TR4+. It was the first 2U Ultra server Supermicro was shipping. Any of the 2U Ultra's would be a good choice. What I like about these in particular is lots of PCIe slots for HBAs and 10GB NICs. If your running RJ45 10GB they have versions with 10GB built in. If it were available at the time I would have went with the 2028U, since it has all 2 1/2 drives. The only drives I've put in the SATA side has been SSDs for boot and L2ARC. With an internal HBA you could load it up with 24 drives. I think with the on board SATA you are limited to 10 drives. -Chip > Any help appreciated, > > Thanks > > Andy > > -- > Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk > Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ > Registered in England and Wales | Company number 4899123 > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at dojcak.sk Wed Feb 18 22:34:08 2015 From: martin at dojcak.sk (Martin Dojcak) Date: Wed, 18 Feb 2015 23:34:08 +0100 Subject: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog Message-ID: Hi, zil_slog_limit from illumos-gate source code is really 1024*1024 bytes for one write operation. There is nice self explain thread on zfs on linux github (https://github.com/zfsonlinux/zfs/issues/1012) when a write should be logged in indirect mode or in immediate mode. Application rarely use write > 1MB. In the most cases 1MB is enaugh, but it would be nice to have a option to configure zil slog limit (like zfs on linux module). -- Martin Doj??k mail: martin at dojcak.sk jabber: martindojcak at jabbim.sk pgp: http://pgp.dojcak.sk/
-------- P?vodn? spr?va --------
Od: Rune Tipsmark
D?tum:18. 02. 2015 21:04 (GMT+01:00)
Komu: omnios-discuss at lists.omniti.com
Predmet: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog
hi all, I found an entry about zil_slog_limit here: http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWritesAndZILII it basically explains how writes larger than 1MB per default hits the main pool rather than my Slog device - I could not find much further information nor the equivalent setting in OmniOS. I also read http://nex7.blogspot.ca/2013/04/zfs-intent-log.html but it didn't truly help me understand just how I can force every written byte to my ZFS box to go the ZIL regardless of size, I never ever want anything to go directly to my man pool ever. I have sync=always and disabled write back cache on my volume based LU's. Testing with zfs_txg_timeout set to 30 or 60 seconds seems to make no difference if I write large files to my LU's - I don't seem the write speed being consistent with the performance of the Slog devices. It looks as if it goes straight to disk and hence the performance is less than great to say the least. How do I ensure 100% that all writes always goes to my Slog devices - no exceptions. br, Rune -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.elling at richardelling.com Thu Feb 19 00:27:00 2015 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 18 Feb 2015 16:27:00 -0800 Subject: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog In-Reply-To: <1424289858252.20555@steait.net> References: <1424289858252.20555@steait.net> Message-ID: <9B0B4A05-73D1-4124-BEF5-8CEE5E37D0F3@richardelling.com> > On Feb 18, 2015, at 12:04 PM, Rune Tipsmark wrote: > > hi all, > > I found an entry about zil_slog_limit here: http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWritesAndZILII > it basically explains how writes larger than 1MB per default hits the main pool rather than my Slog device - I could not find much further information nor the equivalent setting in OmniOS. I also read http://nex7.blogspot.ca/2013/04/zfs-intent-log.html but it didn't truly help me understand just how I can force every written byte to my ZFS box to go the ZIL regardless of size, I never ever want anything to go directly to my man pool ever. > "never ever want anything to go to main pool" is not feasible. The ZIL is a ZFS Intent Log http://en.wikipedia.org/wiki/Intent_log and, unless you overwrite prior to txg commit, everything ends up in the main pool. > > I have sync=always and disabled write back cache on my volume based LU's. > > Testing with zfs_txg_timeout set to 30 or 60 seconds seems to make no difference if I write large files to my LU's - I don't seem the write speed being consistent with the performance of the Slog devices. It looks as if it goes straight to disk and hence the performance is less than great to say the least. > Ultimately, the pool must be able to sustain the workload, or it will have to throttle. The comment for zil_slog_limit is concise: /* * Use the slog as long as the logbias is 'latency' and the current commit size * doesn't exceed the limit or the total list size doesn't exceed its limit. * Limit checking is disabled by setting zil_slog_limit to UINT64_MAX. */ uint64_t zil_slog_limit = (1024 * 1024); uint64_t zil_slog_list_limit = (1024 * 1024 * 200); and you can change this on the fly using mdb to experiment. > > How do I ensure 100% that all writes always goes to my Slog devices - no exceptions. > The question really isn't "how" the question is "why"? Now that you know what an Intent Log is, and how the performance of the pool is your ultimate limit, perhaps you can explain what you are really trying to accomplish? -- richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From rt at steait.net Thu Feb 19 02:09:36 2015 From: rt at steait.net (Rune Tipsmark) Date: Thu, 19 Feb 2015 02:09:36 +0000 Subject: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog In-Reply-To: <9B0B4A05-73D1-4124-BEF5-8CEE5E37D0F3@richardelling.com> References: <1424289858252.20555@steait.net>, <9B0B4A05-73D1-4124-BEF5-8CEE5E37D0F3@richardelling.com> Message-ID: <1424311776360.64422@steait.net> ________________________________ From: Richard Elling Sent: Thursday, February 19, 2015 1:27 AM To: Rune Tipsmark Cc: omnios-discuss at lists.omniti.com Subject: Re: [OmniOS-discuss] ZFS Slog - force all writes to go to Slog On Feb 18, 2015, at 12:04 PM, Rune Tipsmark > wrote: hi all, I found an entry about zil_slog_limit here: http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWritesAndZILII it basically explains how writes larger than 1MB per default hits the main pool rather than my Slog device - I could not find much further information nor the equivalent setting in OmniOS. I also read http://nex7.blogspot.ca/2013/04/zfs-intent-log.html but it didn't truly help me understand just how I can force every written byte to my ZFS box to go the ZIL regardless of size, I never ever want anything to go directly to my man pool ever. "never ever want anything to go to main pool" is not feasible. The ZIL is a ZFS Intent Log http://en.wikipedia.org/wiki/Intent_log and, unless you overwrite prior to txg commit, everything ends up in the main pool. >> yeah I know, I meant everything needs to go thru the ZIL before hitting the main pool... I have sync=always and disabled write back cache on my volume based LU's. Testing with zfs_txg_timeout set to 30 or 60 seconds seems to make no difference if I write large files to my LU's - I don't seem the write speed being consistent with the performance of the Slog devices. It looks as if it goes straight to disk and hence the performance is less than great to say the least. Ultimately, the pool must be able to sustain the workload, or it will have to throttle. >> it should be OK to take in some hundreds MB/sec (11 SAS mirrors each can do ~150MB/Sec sequential) The comment for zil_slog_limit is concise: /* * Use the slog as long as the logbias is 'latency' and the current commit size * doesn't exceed the limit or the total list size doesn't exceed its limit. * Limit checking is disabled by setting zil_slog_limit to UINT64_MAX. */ uint64_t zil_slog_limit = (1024 * 1024); uint64_t zil_slog_list_limit = (1024 * 1024 * 200); and you can change this on the fly using mdb to experiment. >> how do I do this? I could not find any property matching.... How do I ensure 100% that all writes always goes to my Slog devices - no exceptions. The question really isn't "how" the question is "why"? Now that you know what an Intent Log is, and how the performance of the pool is your ultimate limit, perhaps you can explain what you are really trying to accomplish? >> consistent fast write speeds at all times rather than yoyo write speeds... I get fine benchmarks but rather less fine file copy performance... and I never ever see disks being particular busy during file copy tests.... -- richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Thu Feb 19 02:36:37 2015 From: danmcd at omniti.com (Dan McDonald) Date: Wed, 18 Feb 2015 21:36:37 -0500 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 Message-ID: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. Thanks, Dan From lars.nordin at photonet.se Thu Feb 19 11:04:51 2015 From: lars.nordin at photonet.se (Lars Nordin) Date: Thu, 19 Feb 2015 12:04:51 +0100 Subject: [OmniOS-discuss] no interface found installing kvm Message-ID: <54E5C353.9070004@photonet.se> Hi gurus!in I try to to install pfsence in a kvm virtual machine. First in boot interfaces is found but later it prompts in vnc: No interface found do you want to declare one? And auto detect fails. beginning of vm boot: qemu-system-x86_64: -net vnic,vlan=0,name=net0,ifname=pfwan0,macaddr=2:8:20:93:f5:d9: vnic dhcp disabled qemu-system-x86_64: -net vnic,vlan=1,name=net1,ifname=pflan0,macaddr=2:8:20:41:46:7d: vnic dhcp disabled startscript: #!/usr/bin/bash # on omnios command is /usr/bin/qemu-system-x86_64 # configuration NAME="PfSense2.13" NUM=2 VNIC0=pfwan0 VNIC1=pflan0 #VNIC2=pftlout0 #VNIC3=pftlin0 #VNIC4=print0 #HDD=/dev/zvol/rdsk/mainpool/vm/pfsense/os HDD=/dev/zvol/rdsk/daedaluspool/pfroot/os #CD=/mainpool/nfs/iso/pf213.iso CD=/mnt/pfSense-LiveCD-2.1.5-RELEASE-amd64.iso MEM=2048 # don't change below here! TLN=`expr 7000 + $NUM` mac0=`dladm show-vnic -po macaddress $VNIC0` mac1=`dladm show-vnic -po macaddress $VNIC1` #mac2=`dladm show-vnic -po macaddress $VNIC2` #mac3=`dladm show-vnic -po macaddress $VNIC3` #mac4=`dladm show-vnic -po macaddress $VNIC4` /usr/bin/qemu-system-x86_64 \ -name $NAME \ -boot cd \ -enable-kvm \ -vnc 0.0.0.0:$NUM \ -smp cores=1,threads=2,sockets=1 \ -m $MEM \ -no-hpet \ -localtime \ -drive file=$HDD,if=ide,index=0 \ -drive file=$CD,media=cdrom,if=ide,index=2 \ -net nic,vlan=0,name=net0,model=virtio,macaddr=$mac0 \ -net vnic,vlan=0,name=net0,ifname=$VNIC0,macaddr=$mac0 \ -net nic,vlan=1,name=net1,model=virtio,macaddr=$mac1 \ -net vnic,vlan=1,name=net1,ifname=$VNIC1,macaddr=$mac1 \ #-net nic,vlan=2,name=net2,model=virtio,macaddr=$mac2 \ #-net vnic,vlan=2,name=net2,ifname=$VNIC2,macaddr=$mac2 \ #-net nic,vlan=3,name=net3,model=virtio,macaddr=$mac3 \ #-net vnic,vlan=3,name=net3,ifname=$VNIC3,macaddr=$mac3 \ #-net nic,vlan=4,name=net4,model=virtio,macaddr=$mac4 \ #-net vnic,vlan=4,name=net4,ifname=$VNIC4,macaddr=$mac4 \ -vga std \ -cpu host \ -pidfile /pfroot/pids/$NAME.pid \ -monitor telnet:localhost:$TLN,server,nowait,nodelay \ -daemonize if [ $? -gt 0 ]; then echo "Failed to start VM" exit fi port=`expr 5900 + $NUM` public_nic=$(dladm show-vnic|grep vnic0|awk '{print $2}') public_ip=$(ifconfig $public_nic|grep inet|awk '{print $2}') echo "Started VM: $NAME" echo "VNC available at: host IP ${public_ip} port ${port}" echo "QEMU Monitor, do: # telnet localhost $TLN. Note: use Control ] to exit monitor before quit! " What am I missing? /Lars Nordin From danmcd at omniti.com Thu Feb 19 16:16:02 2015 From: danmcd at omniti.com (Dan McDonald) Date: Thu, 19 Feb 2015 11:16:02 -0500 Subject: [OmniOS-discuss] no interface found installing kvm In-Reply-To: <54E5C353.9070004@photonet.se> References: <54E5C353.9070004@photonet.se> Message-ID: <494DE363-39D3-4060-8838-42B6D5E26635@omniti.com> See what "dladm show-link" and "dladm show-vnic" have to say. It seems like they might not actually be there. Dan From johan.kragsterman at capvert.se Thu Feb 19 18:08:31 2015 From: johan.kragsterman at capvert.se (Johan Kragsterman) Date: Thu, 19 Feb 2015 19:08:31 +0100 Subject: [OmniOS-discuss] Ang: Re: no interface found installing kvm In-Reply-To: <494DE363-39D3-4060-8838-42B6D5E26635@omniti.com> References: <494DE363-39D3-4060-8838-42B6D5E26635@omniti.com>, <54E5C353.9070004@photonet.se> Message-ID: Hi! No need, Dan, this is a PfSense thing...needs to load virtio modules. Johan -----"OmniOS-discuss" skrev: ----- Till: Lars Nordin Fr?n: Dan McDonald S?nt av: "OmniOS-discuss" Datum: 2015-02-19 17:18 Kopia: omnios-discuss at lists.omniti.com ?rende: Re: [OmniOS-discuss] no interface found installing kvm See what "dladm show-link" and "dladm show-vnic" have to say. ?It seems like they might not actually be there. Dan _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss at lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss From groups at tierarzt-mueller.de Fri Feb 20 09:17:49 2015 From: groups at tierarzt-mueller.de (Alexander Lesle) Date: Fri, 20 Feb 2015 10:17:49 +0100 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: References: Message-ID: <521254406.20150220101749@tierarzt-mueller.de> Hello Dan McDonald and List, do your remember my feature request? ,-----[ ]----- | | at Friday, 3. Okt. 2014 17:22 | at mid:A762C059-F687-42DE-A1F7-6CF2A7F5863B at omniti.com wrote: | | > On Oct 3, 2014, at 9:29 AM, Alexander Lesle | > wrote: | | >> Hello Dan McDonald and List, | >> | >> in my mind this was a great proposal what you done at Sep, 02. | >> That's the way to build a great OmniOS near to the users. | >> | >> Here my request for the next release: | >> It would be convenient if open-vm-tools were installed in the new | >> release. http://sourceforge.net/projects/open-vm-tools/files/ | | > Actually, this work has been done. I dropped it on the ground: | | > https://github.com/omniti-labs/omnios-build/pull/39 | | > I will be merging this pull requests now in the master branch, | > and seeing how OOod chews on it. | | | > I know there's other work to be done for OmniOS-as-a-guest as well. | | > Thanks, | > DAn | `------------------- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 wrote in [1]: > There will be only 1-3 more updates prior to the next OmniOS stable > (and for this time, this stable is also Long-Term Support) release. > If you notice anything weird about the bloody bits, please let me > know ASAP. I've heard little/no complaints about the updates to > pkg(5), so either you're very happy, or not using it. :) > This update will only be reaching the repo. > * omnios-build master branch, revision f3d6d48 > * Git to 2.3.0. > * UnZIP fixes. > * OpenJDK7 up to update 76, build 31. > * Microtasking libraries (/lib/libmtsk*) are back as a distinct package > (system/library/mtsk) now that they are not part of the (now open-source) > Math libraries. > * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) > * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 ........ [1] mid:F70C482D-68E8-43A2-BF73-86ADE4B63496 at omniti.com ........ From chip at innovates.com Fri Feb 20 14:20:54 2015 From: chip at innovates.com (Schweiss, Chip) Date: Fri, 20 Feb 2015 08:20:54 -0600 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: <521254406.20150220101749@tierarzt-mueller.de> References: <521254406.20150220101749@tierarzt-mueller.de> Message-ID: I will second that request as my OmniOS experiments always start as VMware VMs. -Chip On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle wrote: > Hello Dan McDonald and List, > > do your remember my feature request? > ,-----[ ]----- > | > | at Friday, 3. Okt. 2014 17:22 > | at mid:A762C059-F687-42DE-A1F7-6CF2A7F5863B at omniti.com wrote: > | > | > On Oct 3, 2014, at 9:29 AM, Alexander Lesle > | > wrote: > | > | >> Hello Dan McDonald and List, > | >> > | >> in my mind this was a great proposal what you done at Sep, 02. > | >> That's the way to build a great OmniOS near to the users. > | >> > | >> Here my request for the next release: > | >> It would be convenient if open-vm-tools were installed in the new > | >> release. http://sourceforge.net/projects/open-vm-tools/files/ > | > | > Actually, this work has been done. I dropped it on the ground: > | > | > https://github.com/omniti-labs/omnios-build/pull/39 > | > | > I will be merging this pull requests now in the master branch, > | > and seeing how OOod chews on it. > | > | > | > I know there's other work to be done for OmniOS-as-a-guest as well. > | > | > Thanks, > | > DAn > | > `------------------- > > Do you find the time to integrate open-vm-tools in the next stable > version? > > Thanks. > > On Februar, 19 2015, 03:36 wrote in [1]: > > > There will be only 1-3 more updates prior to the next OmniOS stable > > (and for this time, this stable is also Long-Term Support) release. > > If you notice anything weird about the bloody bits, please let me > > know ASAP. I've heard little/no complaints about the updates to > > pkg(5), so either you're very happy, or not using it. :) > > > This update will only be reaching the repo. > > > * omnios-build master branch, revision f3d6d48 > > > * Git to 2.3.0. > > > * UnZIP fixes. > > > * OpenJDK7 up to update 76, build 31. > > > * Microtasking libraries (/lib/libmtsk*) are back as a distinct package > > (system/library/mtsk) now that they are not part of the (now > open-source) > > Math libraries. > > > * illumos-omnios master branch, revision cbf73e4 (last illumos-gate > merge 336069c) > > > * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. > > -- > Best Regards > Alexander > Februar, 20 2015 > ........ > [1] mid:F70C482D-68E8-43A2-BF73-86ADE4B63496 at omniti.com > ........ > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Fri Feb 20 15:27:56 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 10:27:56 -0500 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: References: <521254406.20150220101749@tierarzt-mueller.de> Message-ID: Are you two volunteering for testing? If so, I can see what I can do about pushing the package (without it being in an incorporation) into the bloody repo server -- assuming I can build it properly. Dan Sent from my iPhone (typos, autocorrect, and all) > On Feb 20, 2015, at 9:20 AM, Schweiss, Chip wrote: > > I will second that request as my OmniOS experiments always start as VMware VMs. > > -Chip > >> On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle wrote: >> Hello Dan McDonald and List, >> >> do your remember my feature request? >> ,-----[ ]----- >> | >> | at Friday, 3. Okt. 2014 17:22 >> | at mid:A762C059-F687-42DE-A1F7-6CF2A7F5863B at omniti.com wrote: >> | >> | > On Oct 3, 2014, at 9:29 AM, Alexander Lesle >> | > wrote: >> | >> | >> Hello Dan McDonald and List, >> | >> >> | >> in my mind this was a great proposal what you done at Sep, 02. >> | >> That's the way to build a great OmniOS near to the users. >> | >> >> | >> Here my request for the next release: >> | >> It would be convenient if open-vm-tools were installed in the new >> | >> release. http://sourceforge.net/projects/open-vm-tools/files/ >> | >> | > Actually, this work has been done. I dropped it on the ground: >> | >> | > https://github.com/omniti-labs/omnios-build/pull/39 >> | >> | > I will be merging this pull requests now in the master branch, >> | > and seeing how OOod chews on it. >> | >> | >> | > I know there's other work to be done for OmniOS-as-a-guest as well. >> | >> | > Thanks, >> | > DAn >> | >> `------------------- >> >> Do you find the time to integrate open-vm-tools in the next stable >> version? >> >> Thanks. >> >> On Februar, 19 2015, 03:36 wrote in [1]: >> >> > There will be only 1-3 more updates prior to the next OmniOS stable >> > (and for this time, this stable is also Long-Term Support) release. >> > If you notice anything weird about the bloody bits, please let me >> > know ASAP. I've heard little/no complaints about the updates to >> > pkg(5), so either you're very happy, or not using it. :) >> >> > This update will only be reaching the repo. >> >> > * omnios-build master branch, revision f3d6d48 >> >> > * Git to 2.3.0. >> >> > * UnZIP fixes. >> >> > * OpenJDK7 up to update 76, build 31. >> >> > * Microtasking libraries (/lib/libmtsk*) are back as a distinct package >> > (system/library/mtsk) now that they are not part of the (now open-source) >> > Math libraries. >> >> > * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) >> >> > * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. >> >> -- >> Best Regards >> Alexander >> Februar, 20 2015 >> ........ >> [1] mid:F70C482D-68E8-43A2-BF73-86ADE4B63496 at omniti.com >> ........ >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Fri Feb 20 15:50:43 2015 From: chip at innovates.com (Schweiss, Chip) Date: Fri, 20 Feb 2015 09:50:43 -0600 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: References: <521254406.20150220101749@tierarzt-mueller.de> Message-ID: On Fri, Feb 20, 2015 at 9:27 AM, Dan McDonald wrote: > Are you two volunteering for testing? If so, I can see what I can do > about pushing the package (without it being in an incorporation) into the > bloody repo server -- assuming I can build it properly. > Absolutely. I have several OmniOS VMs already. At least 1/2 of them need VMware tools. -Chip > > Dan > > Sent from my iPhone (typos, autocorrect, and all) > > On Feb 20, 2015, at 9:20 AM, Schweiss, Chip wrote: > > I will second that request as my OmniOS experiments always start as VMware > VMs. > > -Chip > > On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle < > groups at tierarzt-mueller.de> wrote: > >> Hello Dan McDonald and List, >> >> do your remember my feature request? >> ,-----[ ]----- >> | >> | at Friday, 3. Okt. 2014 17:22 >> | at mid:A762C059-F687-42DE-A1F7-6CF2A7F5863B at omniti.com wrote: >> | >> | > On Oct 3, 2014, at 9:29 AM, Alexander Lesle >> | > wrote: >> | >> | >> Hello Dan McDonald and List, >> | >> >> | >> in my mind this was a great proposal what you done at Sep, 02. >> | >> That's the way to build a great OmniOS near to the users. >> | >> >> | >> Here my request for the next release: >> | >> It would be convenient if open-vm-tools were installed in the new >> | >> release. http://sourceforge.net/projects/open-vm-tools/files/ >> | >> | > Actually, this work has been done. I dropped it on the ground: >> | >> | > https://github.com/omniti-labs/omnios-build/pull/39 >> | >> | > I will be merging this pull requests now in the master branch, >> | > and seeing how OOod chews on it. >> | >> | >> | > I know there's other work to be done for OmniOS-as-a-guest as well. >> | >> | > Thanks, >> | > DAn >> | >> `------------------- >> >> Do you find the time to integrate open-vm-tools in the next stable >> version? >> >> Thanks. >> >> On Februar, 19 2015, 03:36 wrote in [1]: >> >> > There will be only 1-3 more updates prior to the next OmniOS stable >> > (and for this time, this stable is also Long-Term Support) release. >> > If you notice anything weird about the bloody bits, please let me >> > know ASAP. I've heard little/no complaints about the updates to >> > pkg(5), so either you're very happy, or not using it. :) >> >> > This update will only be reaching the repo. >> >> > * omnios-build master branch, revision f3d6d48 >> >> > * Git to 2.3.0. >> >> > * UnZIP fixes. >> >> > * OpenJDK7 up to update 76, build 31. >> >> > * Microtasking libraries (/lib/libmtsk*) are back as a distinct package >> > (system/library/mtsk) now that they are not part of the (now >> open-source) >> > Math libraries. >> >> > * illumos-omnios master branch, revision cbf73e4 (last illumos-gate >> merge 336069c) >> >> > * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. >> >> -- >> Best Regards >> Alexander >> Februar, 20 2015 >> ........ >> [1] mid:F70C482D-68E8-43A2-BF73-86ADE4B63496 at omniti.com >> ........ >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Fri Feb 20 16:10:13 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 11:10:13 -0500 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: <521254406.20150220101749@tierarzt-mueller.de> References: <521254406.20150220101749@tierarzt-mueller.de> Message-ID: <50173EA8-63C6-4E47-B9AD-DDD922CFA290@omniti.com> > On Feb 20, 2015, at 4:17 AM, Alexander Lesle wrote: > > | > Actually, this work has been done. I dropped it on the ground: > | > | > https://github.com/omniti-labs/omnios-build/pull/39 > | > | > I will be merging this pull requests now in the master branch, > | > and seeing how OOod chews on it. > | Ahhh... turns out, this is *IN* bloody already. If you're running bloody, you can install "open-vm-tools". Dan From danmcd at omniti.com Fri Feb 20 16:16:17 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 11:16:17 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 Message-ID: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Sometime during Q2CY2015 (no earlier than April 1st), OmniOS r151014 will be released. At that time, support for r151010 ("previous stable") will be discontinued. This is in accordance with the OmniOS release cycle: http://omnios.omniti.com/wiki.php/ReleaseCycle If you can, please upgrade to r151012. We *should* be able to allow a jump from r151010 to r151014, as we must support a jump from r151012 to 014 AND from 006 to 014 (because r151014 will also be the next Long-Term Support release). As I mentioned the last time, beadm(1M) is your friend. Creating backup BEs in case of mistakes is easy. Thank you, Dan McDonald -- OmniOS Engineering From jesus at omniti.com Fri Feb 20 16:32:42 2015 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 20 Feb 2015 11:32:42 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Message-ID: You know what would be nice? an omnios.omniti.com/releases web page that has the release name, date and EOL [r151006 : LTS] [birth ] [end-of-life ] [r151010 : stable] [birth ] [end-of-life 2015-04-01] [r151013 : bloody] [birth ] [not supported] [r151014 : LTS] [birth "we're expecting"] [end-of-life ] We could make that nice an pretty and I think it would be very useful. On Fri, Feb 20, 2015 at 11:16 AM, Dan McDonald wrote: > Sometime during Q2CY2015 (no earlier than April 1st), OmniOS r151014 will > be released. At that time, support for r151010 ("previous stable") will be > discontinued. This is in accordance with the OmniOS release cycle: > > http://omnios.omniti.com/wiki.php/ReleaseCycle > > If you can, please upgrade to r151012. We *should* be able to allow a > jump from r151010 to r151014, as we must support a jump from r151012 to 014 > AND from 006 to 014 (because r151014 will also be the next Long-Term > Support release). > > As I mentioned the last time, beadm(1M) is your friend. Creating backup > BEs in case of mistakes is easy. > > Thank you, > Dan McDonald -- OmniOS Engineering > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobi at oetiker.ch Fri Feb 20 19:58:55 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Fri, 20 Feb 2015 20:58:55 +0100 (CET) Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Message-ID: Hi Dan, Today Dan McDonald wrote: > Sometime during Q2CY2015 (no earlier than April 1st), OmniOS > r151014 will be released. At that time, support for r151010 > ("previous stable") will be discontinued. This is in accordance > with the OmniOS release cycle: > > http://omnios.omniti.com/wiki.php/ReleaseCycle > > If you can, please upgrade to r151012. We *should* be able to > allow a jump from r151010 to r151014, as we must support a jump > from r151012 to 014 AND from 006 to 014 (because r151014 will > also be the next Long-Term Support release). > > As I mentioned the last time, beadm(1M) is your friend. Creating > backup BEs in case of mistakes is easy. I do not have a spare machine to test this, but if the dramatic network io performance regression for kvm present in r151012 has not been fixed in r151014 then we will be stranded on r151010. Judging from the echo on the ML, not many people seem to see this problem, or care about it. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From danmcd at omniti.com Fri Feb 20 20:17:16 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 15:17:16 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Message-ID: > On Feb 20, 2015, at 2:58 PM, Tobias Oetiker wrote: > > Hi Dan, > > I do not have a spare machine to test this, but if the dramatic > network io performance regression for kvm present in r151012 has > not been fixed in r151014 then we will be stranded on r151010. And the upstreaming of VND is likely not to happen with this release. That blocks us from properly catching up with illumos-kvm-cmd. There are several commits (Starting with the March 19th one we can't bring in w/o VND) here: https://github.com/joyent/illumos-kvm-cmd/commits/master Currently, we only backport the most recent fix about QEMU using preadv/pwritev recklessly. IF (big if) there are other commits we can backport, you could try it using the patches/ directory of omnios-build. I'm thinking maybe HVM-807's may help you (provided that's indepdent of VND support). Have you seen anything with dtrace or lockstat that could give a clue to what's holding things up? I don't recall seeing anything observed. > Judging from the echo on the ML, not many people seem to see this > problem, or care about it. I'm sorry, but that does appear to be the case. Next week I disappear for other-customer work, but starting in March, I'm doing nothing but r151014. Dan From danmcd at omniti.com Fri Feb 20 20:18:51 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 15:18:51 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Message-ID: <550A9F98-ACDE-4162-B323-F72056045ADA@omniti.com> Unicast... Found this: http://echelog.com/logs/browse/smartos/1408053600 With this to say: So, I can give you a copy of the analysis. Basically the number of iovecs that we can receive is up to the virtio ring buffer size. Which is a lot more than the number of frame io vectors. And while this shouldn't generally happen, Windows has been guilt of it for non-jumbo frames. Why you would break up a <1500 byte packet into 32 iovectors is beyond me. and when it does that QEMU crashes on the assert with the old code, correct? Dan From groups at tierarzt-mueller.de Fri Feb 20 20:43:54 2015 From: groups at tierarzt-mueller.de (Alexander Lesle) Date: Fri, 20 Feb 2015 21:43:54 +0100 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: References: <521254406.20150220101749@tierarzt-mueller.de> Message-ID: <866483450.20150220214354@tierarzt-mueller.de> Hello Dan McDonald and List, thanks for your offer, but I cant do that because I am not a omnios expert enough and my english is not very well. On Februar, 20 2015, 16:27 wrote in [1]: > Are you two volunteering for testing? If so, I can see what I can > do about pushing the package (without it being in an incorporation) > into the bloody repo server -- assuming I can build it properly. > Dan > Sent from my iPhone (typos, autocorrect, and all) > On Feb 20, 2015, at 9:20 AM, Schweiss, Chip wrote: > I will second that request as my OmniOS experiments always start as VMware VMs. > -Chip > On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle > wrote: > Hello Dan McDonald and List, > > do your remember my feature request? > ,-----[ ]----- > | > | at Friday, 3. Okt. 2014 17:22 > | at mid:A762C059-F687-42DE-A1F7-6CF2A7F5863B at omniti.com wrote: > | > | > On Oct 3, 2014, at 9:29 AM, Alexander Lesle > | > wrote: > | > | >> Hello Dan McDonald and List, > | >> > | >> in my mind this was a great proposal what you done at Sep, 02. > | >> That's the way to build a great OmniOS near to the users. > | >> > | >> Here my request for the next release: > | >> It would be convenient if open-vm-tools were installed in the new > | >> release. http://sourceforge.net/projects/open-vm-tools/files/ > | > | > Actually, this work has been done. I dropped it on the ground: > | > | > https://github.com/omniti-labs/omnios-build/pull/39 > | > | > I will be merging this pull requests now in the master branch, > | > and seeing how OOod chews on it. > | > | > | > I know there's other work to be done for OmniOS-as-a-guest as well. > | > | > Thanks, > | > DAn > | > `------------------- > > Do you find the time to integrate open-vm-tools in the next stable > version? > > Thanks. > > On Februar, 19 2015, 03:36 wrote in [1]: > >> There will be only 1-3 more updates prior to the next OmniOS stable >> (and for this time, this stable is also Long-Term Support) release. >> If you notice anything weird about the bloody bits, please let me >> know ASAP. I've heard little/no complaints about the updates to >> pkg(5), so either you're very happy, or not using it. :) > >> This update will only be reaching the repo. > >> * omnios-build master branch, revision f3d6d48 > >> * Git to 2.3.0. > >> * UnZIP fixes. > >> * OpenJDK7 up to update 76, build 31. > >> * Microtasking libraries (/lib/libmtsk*) are back as a distinct package >> (system/library/mtsk) now that they are not part of the (now open-source) >> Math libraries. > >> * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) > >> * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. > > -- > Best Regards > Alexander > Februar, 20 2015 > ........ > [1] mid:F70C482D-68E8-43A2-BF73-86ADE4B63496 at omniti.com > ........ > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Best Regards Alexander Februar, 20 2015 ........ [1] mid:A19863F3-E121-445E-BF4B-DA5C97EF3772 at omniti.com ........ From danmcd at omniti.com Fri Feb 20 20:47:42 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 15:47:42 -0500 Subject: [OmniOS-discuss] OmniOS Bloody update for Feb 18 In-Reply-To: <866483450.20150220214354@tierarzt-mueller.de> References: <521254406.20150220101749@tierarzt-mueller.de> <866483450.20150220214354@tierarzt-mueller.de> Message-ID: > On Feb 20, 2015, at 3:43 PM, Alexander Lesle wrote: > > Hello Dan McDonald and List, > > thanks for your offer, but I cant do that because I am not a omnios > expert enough and my english is not very well. If you are running the bloody version, all you need is this: pkg install open-vm-tools That's it! If you're waiting for something supported, you'll need to wait for r151014 to ship. Dan From hasslerd at gmx.li Fri Feb 20 22:14:48 2015 From: hasslerd at gmx.li (Dominik Hassler) Date: Fri, 20 Feb 2015 23:14:48 +0100 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> Message-ID: <54E7B1D8.3070009@gmx.li> Dan, I had a look at HVM-807 you mentioned. Shouldn't line 251 in 'net/vnic.c' be for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) { instead of for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) { As otherwise the last element of the frameio will not be used at all if iovcnt is greater than the frameio buffer size? Kind regards Dominik On 02/20/2015 09:17 PM, Dan McDonald wrote: > >> On Feb 20, 2015, at 2:58 PM, Tobias Oetiker wrote: >> >> Hi Dan, >> >> I do not have a spare machine to test this, but if the dramatic >> network io performance regression for kvm present in r151012 has >> not been fixed in r151014 then we will be stranded on r151010. > > And the upstreaming of VND is likely not to happen with this release. That blocks us from properly catching up with illumos-kvm-cmd. > > There are several commits (Starting with the March 19th one we can't bring in w/o VND) here: > > https://github.com/joyent/illumos-kvm-cmd/commits/master > > Currently, we only backport the most recent fix about QEMU using preadv/pwritev recklessly. IF (big if) there are other commits we can backport, you could try it using the patches/ directory of omnios-build. I'm thinking maybe HVM-807's may help you (provided that's indepdent of VND support). > > Have you seen anything with dtrace or lockstat that could give a clue to what's holding things up? I don't recall seeing anything observed. > >> Judging from the echo on the ML, not many people seem to see this >> problem, or care about it. > > I'm sorry, but that does appear to be the case. > > Next week I disappear for other-customer work, but starting in March, I'm doing nothing but r151014. > > Dan > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- A non-text attachment was scrubbed... Name: 0xF9ECC6A5.asc Type: application/pgp-keys Size: 2686 bytes Desc: not available URL: From hasslerd at gmx.li Fri Feb 20 22:33:32 2015 From: hasslerd at gmx.li (Dominik Hassler) Date: Fri, 20 Feb 2015 23:33:32 +0100 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: <54E7B1D8.3070009@gmx.li> References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> <54E7B1D8.3070009@gmx.li> Message-ID: <54E7B63C.2040607@gmx.li> never mind, it's getting filled later on. should have had a look at the whole of it before starting to bark :/ sorry for the noise! On 02/20/2015 11:14 PM, Dominik Hassler wrote: > Dan, > > I had a look at HVM-807 you mentioned. > > Shouldn't line 251 in 'net/vnic.c' be > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) { > > instead of > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) { > > As otherwise the last element of the frameio will not be used at all if > iovcnt is greater than the frameio buffer size? > > Kind regards > Dominik > > On 02/20/2015 09:17 PM, Dan McDonald wrote: >> >>> On Feb 20, 2015, at 2:58 PM, Tobias Oetiker wrote: >>> >>> Hi Dan, >>> >>> I do not have a spare machine to test this, but if the dramatic >>> network io performance regression for kvm present in r151012 has >>> not been fixed in r151014 then we will be stranded on r151010. >> >> And the upstreaming of VND is likely not to happen with this release. That blocks us from properly catching up with illumos-kvm-cmd. >> >> There are several commits (Starting with the March 19th one we can't bring in w/o VND) here: >> >> https://github.com/joyent/illumos-kvm-cmd/commits/master >> >> Currently, we only backport the most recent fix about QEMU using preadv/pwritev recklessly. IF (big if) there are other commits we can backport, you could try it using the patches/ directory of omnios-build. I'm thinking maybe HVM-807's may help you (provided that's indepdent of VND support). >> >> Have you seen anything with dtrace or lockstat that could give a clue to what's holding things up? I don't recall seeing anything observed. >> >>> Judging from the echo on the ML, not many people seem to see this >>> problem, or care about it. >> >> I'm sorry, but that does appear to be the case. >> >> Next week I disappear for other-customer work, but starting in March, I'm doing nothing but r151014. >> >> Dan >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss -------------- next part -------------- A non-text attachment was scrubbed... Name: 0xF9ECC6A5.asc Type: application/pgp-keys Size: 2686 bytes Desc: not available URL: From wverb73 at gmail.com Sat Feb 21 01:48:29 2015 From: wverb73 at gmail.com (W Verb) Date: Fri, 20 Feb 2015 17:48:29 -0800 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy Message-ID: Hello all, Each of the things in the subject line are: 1: Horrendously broken 2: Have an extremely poor short-term outlook 3: Will take a huge investment of time by intelligent, dedicated, and insightful people to fix. It's common knowledge that the ixgbe driver in omnios/illumos/opensolaris is broke as hell. The point of this message is not to complain. The point is to pass on a configuration that is working for me, albeit in a screwed-up degraded fashion. I have four ESXi 5.5u2 host servers with 1 Intel PCI-e quad-gigabit NIC installed in each. Three of the gigabit ports on each client are dedicated to carry iSCSI traffic between each host and a single storage server. The storage server is based on a Supermicro X10SLM-F mainboard, which has three PCI-e slots. Two of the slots are used for storage controllers, and a single slot is used for an Intel X520 dual-port fiber 10G NIC. Previously, I had a single storage controller and two quad-gig NICs installed in the storage server, and was able to get close to line-rate on multipath iSCSI with three host clients. But when I added the fourth, I upgraded to 10G. After installation and configuration, I observed all kinds of bad behavior in the network traffic between the hosts and the server. All of this bad behavior is traced to the ixgbe driver on the storage server. Without going into the full troubleshooting process, here are my takeaways: 1: The only tuning factor that appears to have significant effect on the driver is MTU size. This applies to both the MTU of the ixgbe NIC as well as the MTU of the 1-gig NICs used in the hosts. 2: I have seen best performance with the MTU on the ixgbe set to 1000 bytes (yes, 1k). The MTU on the ESXi interfaces is set to 3000 bytes. 3: Setting 9000 byte MTUs on both sides results in about 150MB/s write speeds on a a linux vmware guest running a 10GB dd operation. But read speeds are at 5MB/s. 4: Testing of dd operations on the storage server itself shows that the filesystem is capable of performing 500MB/s+ reads and writes. 5: After setting the MTUs listed in point 2, I am able to get 270-300MB/s writes on the guest OS, and ~200MB/s reads. Not perfect, but I'll take it. 6: No /etc/system or other kernel tunings are in use. 7: Delayed ACK, Nagle, and L2 flow control tests had no effect. 8: pkg -u was performed before all tests, so I should be using the latest kernel code, etc. 9: When capturing traffic on omnios, I used the CSW distribution of tcpdump. It's worth noting that unlike EVERY ... OTHER ... IMPLEMENTATION ... of tcpdump I've ever used (BSD flavors, OSX, various linux distros, various embedded distros), libpcap doesn't appear to get individual frame reports from the omnios kernel, and so aggregates multi-frame TCP segments into a single record. This has the appearance of 20-60kB frames being transmitted by omnios when reading a packet capture with Wireshark. I cannot tell you how irritating this is when troubleshooting network issues. 10: At the wire level, the speed problems are clearly due to pauses in response time by omnios. At 9000 byte frame sizes, I see a good number of duplicate ACKs and fast retransmits during read operations (when omnios is transmitting). But below about a 4100-byte MTU on omnios (which seems to correlate to 4096-byte iSCSI block transfers), the transmission errors fade away and we only see the transmission pause problem. I'm in the process of aggregating the 10G ports and performing some IO testing with the vmware IO performance tool. That should show the performance of the 10G NIC when both physical ports are in use, and hopefully get me some more granularity on the MTU settings. If anyone has a list of kernel tuning parameters to test, I'm happy to try them out and report back. I've found a variety of suggestions online, but between Illumos, solaris, openindiana, Nexenta, opensolaris, etc, the supported variables are, um, inconsistent. -Warren V -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Sat Feb 21 02:10:18 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 21:10:18 -0500 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: Message-ID: You should PLEASE share your original note with the illumos developer's list. Also, please keep in mind that it is VERY possible you're seeing crosscall effects where the interrupt-servicing CPU core is not on the same PCIe bus as where you're card is plugged in. I never got a chance to perform these tests fully when I *had* ixgbe HW handy, but I observed bizarre improvements, or the disappearance of bizarre effects, if I: - disabled the HT-inspired CPUs (psradm -f ) - Disabled one of the two CPUs (again, using psradm). You may wish to try messing around with what OS-reported CPUs are on your Romley (Xeon E5) system. I will also note that it's high time for illumos to pull in the ixgbe updates from upstream. Intel is NOT being very helpful here, partially because of fear-of-Oracle, and partially because there aren't enough illumos customers to make a dent in their HW sales. In the past, illumos developers have found the time to yank in the newest driver updates from upstream. That hasn't happened recently. For the record, OmniTI might be able to contribute here IF AND ONLY IF a sufficiently paying customer motivates us. I suspect the same answer (sufficiently paying customer) applies to engineers from any other illumos shop as well. Dan From danmcd at omniti.com Sat Feb 21 02:12:03 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 20 Feb 2015 21:12:03 -0500 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: Message-ID: <2A2FE1C0-405F-4B95-A2E6-1257FC9BD075@omniti.com> > On Feb 20, 2015, at 9:10 PM, Dan McDonald wrote: > > I never got a chance to perform these tests fully when I *had* ixgbe HW handy, but I observed bizarre improvements, or the disappearance of bizarre effects, if I: > > - disabled the HT-inspired CPUs (psradm -f ) > > - Disabled one of the two CPUs (again, using psradm). > > You may wish to try messing around with what OS-reported CPUs are on your Romley (Xeon E5) system. To see the layouts, the psrinfo(1M) command is your friend, especially "psrinfo -vp". Dan From cks at cs.toronto.edu Sat Feb 21 03:25:59 2015 From: cks at cs.toronto.edu (Chris Siebenmann) Date: Fri, 20 Feb 2015 22:25:59 -0500 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: Your message of Fri, 20 Feb 2015 17:48:29 -0800. Message-ID: <20150221032559.727D07A0792@apps0.cs.toronto.edu> > After installation and configuration, I observed all kinds of bad behavior > in the network traffic between the hosts and the server. All of this bad > behavior is traced to the ixgbe driver on the storage server. Without going > into the full troubleshooting process, here are my takeaways: [...] For what it's worth, we managed to achieve much better line rates on copper 10G ixgbe hardware of various descriptions between OmniOS and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't believe OmniOS could do TCP at full line rate but I think we managed 700+ Mbytes/sec on both transmit and receive and we got basically disk-limited speeds with iSCSI (across multiple disks on multi-disk mirrored pools, OmniOS iSCSI initiator, Linux iSCSI targets). I don't believe we did any specific kernel tuning (and in fact some of our attempts to fiddle ixgbe driver parameters blew up in our face). We did tune iSCSI connection parameters to increase various buffer sizes so that ZFS could do even large single operations in single iSCSI transactions. (More details available if people are interested.) > 10: At the wire level, the speed problems are clearly due to pauses in > response time by omnios. At 9000 byte frame sizes, I see a good number > of duplicate ACKs and fast retransmits during read operations (when > omnios is transmitting). But below about a 4100-byte MTU on omnios > (which seems to correlate to 4096-byte iSCSI block transfers), the > transmission errors fade away and we only see the transmission pause > problem. This is what really attracted my attention. In our OmniOS setup, our specific Intel hardware had ixgbe driver issues that could cause activity stalls during once-a-second link heartbeat checks. This obviously had an effect at the TCP and iSCSI layers. My initial message to illumos-developer sparked a potentially interesting discussion: http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ If you think this is a possibility in your setup, I've put the DTrace script I used to hunt for this up on the web: http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d This isn't the only potential source of driver stalls by any means, it's just the one I found. You may also want to look at lockstat in general, as information it reported is what led us to look specifically at the ixgbe code here. (If you suspect kernel/driver issues, lockstat combined with kernel source is a really excellent resource.) - cks From wverb73 at gmail.com Sat Feb 21 05:46:51 2015 From: wverb73 at gmail.com (W Verb) Date: Fri, 20 Feb 2015 21:46:51 -0800 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: <20150221032559.727D07A0792@apps0.cs.toronto.edu> References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> Message-ID: Hello All, Thank you for your replies. I tried a few things, and found the following: 1: Disabling hyperthreading support in the BIOS drops performance overall by a factor of 4. 2: Disabling VT support also seems to have some effect, although it appears to be minor. But this has the amusing side effect of fixing the hangs I've been experiencing with fast reboot. Probably by disabling kvm. 3: The performance tests are a bit tricky to quantify because of caching effects. In fact, I'm not entirely sure what is happening here. It's just best to describe what I'm seeing: The commands I'm using to test are dd if=/dev/zero of=./test.dd bs=2M count=5000 dd of=/dev/null if=./test.dd bs=2M count=5000 The host vm is running Centos 6.6, and has the latest vmtools installed. There is a host cache on an SSD local to the host that is also in place. Disabling the host cache didn't immediately have an effect as far as I could see. The host MTU set to 3000 on all iSCSI interfaces for all tests. Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test yields an average speed over three tests of 137MB/s. The read test yields an average over three tests of 5MB/s. Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield 140MB/s, and the read tests yield 53MB/s. It's important to note here that if I cut the read test short at only 2-3GB, I get results upwards of 350MB/s, which I assume is local cache-related distortion. Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield about 142MB/s. Test 4: MTU of 1000: Read test at 182MB/s. Test 5: MTU of 900: Read test at 130 MB/s. Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now consistently at about 300MB/s. Test 7: MTU of 1200: Read test at 124MB/s. Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. A few final notes: L1ARC grabs about 10GB of RAM during the tests, so there's definitely some read caching going on. The write operations are easier to observe with iostat, and I'm seeing io rates that closely correlate with the network write speeds. Chris, thanks for your specific details. I'd appreciate it if you could tell me which copper NIC you tried, as well as to pass on the iSCSI tuning parameters. I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of the 82599 in the X520. If I get similar results with my fiber transcievers, I'll see if I can get a hold of copper ones. But I should mention that I did indeed look at PHY/MAC error rates, and they are nil. -Warren V On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann wrote: > > After installation and configuration, I observed all kinds of bad > behavior > > in the network traffic between the hosts and the server. All of this bad > > behavior is traced to the ixgbe driver on the storage server. Without > going > > into the full troubleshooting process, here are my takeaways: > [...] > > For what it's worth, we managed to achieve much better line rates on > copper 10G ixgbe hardware of various descriptions between OmniOS > and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't > believe OmniOS could do TCP at full line rate but I think we managed 700+ > Mbytes/sec on both transmit and receive and we got basically disk-limited > speeds with iSCSI (across multiple disks on multi-disk mirrored pools, > OmniOS iSCSI initiator, Linux iSCSI targets). > > I don't believe we did any specific kernel tuning (and in fact some of > our attempts to fiddle ixgbe driver parameters blew up in our face). > We did tune iSCSI connection parameters to increase various buffer > sizes so that ZFS could do even large single operations in single iSCSI > transactions. (More details available if people are interested.) > > > 10: At the wire level, the speed problems are clearly due to pauses in > > response time by omnios. At 9000 byte frame sizes, I see a good number > > of duplicate ACKs and fast retransmits during read operations (when > > omnios is transmitting). But below about a 4100-byte MTU on omnios > > (which seems to correlate to 4096-byte iSCSI block transfers), the > > transmission errors fade away and we only see the transmission pause > > problem. > > This is what really attracted my attention. In our OmniOS setup, our > specific Intel hardware had ixgbe driver issues that could cause > activity stalls during once-a-second link heartbeat checks. This > obviously had an effect at the TCP and iSCSI layers. My initial message > to illumos-developer sparked a potentially interesting discussion: > > > http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ > > If you think this is a possibility in your setup, I've put the DTrace > script I used to hunt for this up on the web: > > http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d > > This isn't the only potential source of driver stalls by any means, it's > just the one I found. You may also want to look at lockstat in general, > as information it reported is what led us to look specifically at the > ixgbe code here. > > (If you suspect kernel/driver issues, lockstat combined with kernel > source is a really excellent resource.) > > - cks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chip at innovates.com Sat Feb 21 15:59:57 2015 From: chip at innovates.com (Schweiss, Chip) Date: Sat, 21 Feb 2015 09:59:57 -0600 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> Message-ID: I can't say I totally agree with your performance assessment. I run Intel X520 in all my OmniOS boxes. Here is a capture of nfssvrtop I made while running many storage vMotions between two OmniOS boxes hosting NFS datastores. This is a 10 host VMware cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to the in rack Nexus 5010. VMware does 100% sync writes, I use ZeusRAM SSDs for log devices. -Chip 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 KB, awrite: 1875455 KB Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% 4 10.28.17.105 0 0 0 0 0 0 0 0 0 0 0 0 0 4 10.28.17.215 0 0 0 0 0 0 0 0 0 0 0 0 0 4 10.28.17.213 0 0 0 0 0 0 0 0 0 0 0 0 0 4 10.28.16.151 0 0 0 0 0 0 0 0 0 0 0 0 0 4 all 1 0 0 0 0 0 0 0 0 0 0 0 0 3 10.28.16.175 3 0 3 0 0 1 11 0 4806 48 0 0 85 3 10.28.16.183 6 0 6 0 0 3 162 0 549 124 0 0 73 3 10.28.16.180 11 0 10 0 0 3 27 0 776 89 0 0 67 3 10.28.16.176 28 2 26 0 0 10 405 0 2572 198 0 0 100 3 10.28.16.178 4606 4602 4 0 0 294534 3 0 723 49 0 0 99 3 10.28.16.179 4905 4879 26 0 0 312208 311 0 735 271 0 0 99 3 10.28.16.181 5515 5502 13 0 0 352107 77 0 89 87 0 0 99 3 10.28.16.184 12095 12059 10 0 0 763014 39 0 249 147 0 0 99 3 10.28.58.1 15401 6040 116 6354 53 191605 474 202346 192 96 144 83 99 3 all 42574 33086 217 6354 53 *1913488* 1582 202300 348 138 153 105 99 On Fri, Feb 20, 2015 at 11:46 PM, W Verb wrote: > Hello All, > > Thank you for your replies. > I tried a few things, and found the following: > > 1: Disabling hyperthreading support in the BIOS drops performance overall > by a factor of 4. > 2: Disabling VT support also seems to have some effect, although it > appears to be minor. But this has the amusing side effect of fixing the > hangs I've been experiencing with fast reboot. Probably by disabling kvm. > 3: The performance tests are a bit tricky to quantify because of caching > effects. In fact, I'm not entirely sure what is happening here. It's just > best to describe what I'm seeing: > > The commands I'm using to test are > dd if=/dev/zero of=./test.dd bs=2M count=5000 > dd of=/dev/null if=./test.dd bs=2M count=5000 > The host vm is running Centos 6.6, and has the latest vmtools installed. > There is a host cache on an SSD local to the host that is also in place. > Disabling the host cache didn't immediately have an effect as far as I > could see. > > The host MTU set to 3000 on all iSCSI interfaces for all tests. > > Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test > yields an average speed over three tests of 137MB/s. The read test yields > an average over three tests of 5MB/s. > > Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield > 140MB/s, and the read tests yield 53MB/s. It's important to note here that > if I cut the read test short at only 2-3GB, I get results upwards of > 350MB/s, which I assume is local cache-related distortion. > > Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield > about 142MB/s. > Test 4: MTU of 1000: Read test at 182MB/s. > Test 5: MTU of 900: Read test at 130 MB/s. > Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now > consistently at about 300MB/s. > Test 7: MTU of 1200: Read test at 124MB/s. > Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. > > A few final notes: > L1ARC grabs about 10GB of RAM during the tests, so there's definitely some > read caching going on. > The write operations are easier to observe with iostat, and I'm seeing io > rates that closely correlate with the network write speeds. > > > Chris, thanks for your specific details. I'd appreciate it if you could > tell me which copper NIC you tried, as well as to pass on the iSCSI tuning > parameters. > > I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of > the 82599 in the X520. If I get similar results with my fiber transcievers, > I'll see if I can get a hold of copper ones. > > But I should mention that I did indeed look at PHY/MAC error rates, and > they are nil. > > -Warren V > > On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann > wrote: > >> > After installation and configuration, I observed all kinds of bad >> behavior >> > in the network traffic between the hosts and the server. All of this bad >> > behavior is traced to the ixgbe driver on the storage server. Without >> going >> > into the full troubleshooting process, here are my takeaways: >> [...] >> >> For what it's worth, we managed to achieve much better line rates on >> copper 10G ixgbe hardware of various descriptions between OmniOS >> and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't >> believe OmniOS could do TCP at full line rate but I think we managed 700+ >> Mbytes/sec on both transmit and receive and we got basically disk-limited >> speeds with iSCSI (across multiple disks on multi-disk mirrored pools, >> OmniOS iSCSI initiator, Linux iSCSI targets). >> >> I don't believe we did any specific kernel tuning (and in fact some of >> our attempts to fiddle ixgbe driver parameters blew up in our face). >> We did tune iSCSI connection parameters to increase various buffer >> sizes so that ZFS could do even large single operations in single iSCSI >> transactions. (More details available if people are interested.) >> >> > 10: At the wire level, the speed problems are clearly due to pauses in >> > response time by omnios. At 9000 byte frame sizes, I see a good number >> > of duplicate ACKs and fast retransmits during read operations (when >> > omnios is transmitting). But below about a 4100-byte MTU on omnios >> > (which seems to correlate to 4096-byte iSCSI block transfers), the >> > transmission errors fade away and we only see the transmission pause >> > problem. >> >> This is what really attracted my attention. In our OmniOS setup, our >> specific Intel hardware had ixgbe driver issues that could cause >> activity stalls during once-a-second link heartbeat checks. This >> obviously had an effect at the TCP and iSCSI layers. My initial message >> to illumos-developer sparked a potentially interesting discussion: >> >> >> http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ >> >> If you think this is a possibility in your setup, I've put the DTrace >> script I used to hunt for this up on the web: >> >> http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d >> >> This isn't the only potential source of driver stalls by any means, it's >> just the one I found. You may also want to look at lockstat in general, >> as information it reported is what led us to look specifically at the >> ixgbe code here. >> >> (If you suspect kernel/driver issues, lockstat combined with kernel >> source is a really excellent resource.) >> >> - cks >> > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobi at oetiker.ch Sat Feb 21 21:13:54 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Sat, 21 Feb 2015 22:13:54 +0100 (CET) Subject: [OmniOS-discuss] recovering a faulted disk Message-ID: If zfs sees too many errors on a disk, it will 'FAULT' the disk, replace it with a spare disk and start resilvering. We have one system where this keeps happening ... as explained in http://lists.omniti.com/pipermail/omnios-discuss/2014-March/002413.html We have since replaced the firmware on the controllers, and increassed the fan speed, to make sure the contollers are not hot. The disk, in any event seem to be ok, since they do not exhibit any problem when testing afterwards. So my question is, can I somehow convince ZFS that the disk it just FAULTED is actually OK, and it should just fixup the bits that have changed since it abandoned the particular disk ? The reason I am asking this is that resilvering takes quite a lot of time with the stuff we have on these drives, a few days at least ... and just a few hours ago, the scond drive in our RAIDZ2 got faulted ... cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From wverb73 at gmail.com Mon Feb 23 01:54:58 2015 From: wverb73 at gmail.com (W Verb) Date: Sun, 22 Feb 2015 17:54:58 -0800 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> Message-ID: By the way, to those of you who have working setups: please send me your pool/volume settings, interface linkprops, and any kernel tuning parameters you may have set. Thanks, Warren V On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip wrote: > I can't say I totally agree with your performance assessment. I run Intel > X520 in all my OmniOS boxes. > > Here is a capture of nfssvrtop I made while running many storage vMotions > between two OmniOS boxes hosting NFS datastores. This is a 10 host VMware > cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to > the in rack Nexus 5010. > > VMware does 100% sync writes, I use ZeusRAM SSDs for log devices. > > -Chip > > 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 KB, > awrite: 1875455 KB > > Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw > SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% > > 4 10.28.17.105 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.17.215 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.17.213 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.16.151 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > > 4 all 1 0 0 0 0 0 > 0 0 0 0 0 0 0 > > 3 10.28.16.175 3 0 3 0 0 1 > 11 0 4806 48 0 0 85 > > 3 10.28.16.183 6 0 6 0 0 3 > 162 0 549 124 0 0 73 > > 3 10.28.16.180 11 0 10 0 0 3 > 27 0 776 89 0 0 67 > > 3 10.28.16.176 28 2 26 0 0 10 > 405 0 2572 198 0 0 100 > > 3 10.28.16.178 4606 4602 4 0 0 294534 > 3 0 723 49 0 0 99 > > 3 10.28.16.179 4905 4879 26 0 0 312208 > 311 0 735 271 0 0 99 > > 3 10.28.16.181 5515 5502 13 0 0 352107 > 77 0 89 87 0 0 99 > > 3 10.28.16.184 12095 12059 10 0 0 763014 > 39 0 249 147 0 0 99 > > 3 10.28.58.1 15401 6040 116 6354 53 191605 > 474 202346 192 96 144 83 99 > > 3 all 42574 33086 217 6354 53 1913488 > 1582 202300 348 138 153 105 99 > > > > > > On Fri, Feb 20, 2015 at 11:46 PM, W Verb wrote: >> >> Hello All, >> >> Thank you for your replies. >> I tried a few things, and found the following: >> >> 1: Disabling hyperthreading support in the BIOS drops performance overall >> by a factor of 4. >> 2: Disabling VT support also seems to have some effect, although it >> appears to be minor. But this has the amusing side effect of fixing the >> hangs I've been experiencing with fast reboot. Probably by disabling kvm. >> 3: The performance tests are a bit tricky to quantify because of caching >> effects. In fact, I'm not entirely sure what is happening here. It's just >> best to describe what I'm seeing: >> >> The commands I'm using to test are >> dd if=/dev/zero of=./test.dd bs=2M count=5000 >> dd of=/dev/null if=./test.dd bs=2M count=5000 >> The host vm is running Centos 6.6, and has the latest vmtools installed. >> There is a host cache on an SSD local to the host that is also in place. >> Disabling the host cache didn't immediately have an effect as far as I could >> see. >> >> The host MTU set to 3000 on all iSCSI interfaces for all tests. >> >> Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test >> yields an average speed over three tests of 137MB/s. The read test yields an >> average over three tests of 5MB/s. >> >> Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield >> 140MB/s, and the read tests yield 53MB/s. It's important to note here that >> if I cut the read test short at only 2-3GB, I get results upwards of >> 350MB/s, which I assume is local cache-related distortion. >> >> Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield >> about 142MB/s. >> Test 4: MTU of 1000: Read test at 182MB/s. >> Test 5: MTU of 900: Read test at 130 MB/s. >> Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now >> consistently at about 300MB/s. >> Test 7: MTU of 1200: Read test at 124MB/s. >> Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. >> >> A few final notes: >> L1ARC grabs about 10GB of RAM during the tests, so there's definitely some >> read caching going on. >> The write operations are easier to observe with iostat, and I'm seeing io >> rates that closely correlate with the network write speeds. >> >> >> Chris, thanks for your specific details. I'd appreciate it if you could >> tell me which copper NIC you tried, as well as to pass on the iSCSI tuning >> parameters. >> >> I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of >> the 82599 in the X520. If I get similar results with my fiber transcievers, >> I'll see if I can get a hold of copper ones. >> >> But I should mention that I did indeed look at PHY/MAC error rates, and >> they are nil. >> >> -Warren V >> >> On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann >> wrote: >>> >>> > After installation and configuration, I observed all kinds of bad >>> > behavior >>> > in the network traffic between the hosts and the server. All of this >>> > bad >>> > behavior is traced to the ixgbe driver on the storage server. Without >>> > going >>> > into the full troubleshooting process, here are my takeaways: >>> [...] >>> >>> For what it's worth, we managed to achieve much better line rates on >>> copper 10G ixgbe hardware of various descriptions between OmniOS >>> and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't >>> believe OmniOS could do TCP at full line rate but I think we managed 700+ >>> Mbytes/sec on both transmit and receive and we got basically disk-limited >>> speeds with iSCSI (across multiple disks on multi-disk mirrored pools, >>> OmniOS iSCSI initiator, Linux iSCSI targets). >>> >>> I don't believe we did any specific kernel tuning (and in fact some of >>> our attempts to fiddle ixgbe driver parameters blew up in our face). >>> We did tune iSCSI connection parameters to increase various buffer >>> sizes so that ZFS could do even large single operations in single iSCSI >>> transactions. (More details available if people are interested.) >>> >>> > 10: At the wire level, the speed problems are clearly due to pauses in >>> > response time by omnios. At 9000 byte frame sizes, I see a good number >>> > of duplicate ACKs and fast retransmits during read operations (when >>> > omnios is transmitting). But below about a 4100-byte MTU on omnios >>> > (which seems to correlate to 4096-byte iSCSI block transfers), the >>> > transmission errors fade away and we only see the transmission pause >>> > problem. >>> >>> This is what really attracted my attention. In our OmniOS setup, our >>> specific Intel hardware had ixgbe driver issues that could cause >>> activity stalls during once-a-second link heartbeat checks. This >>> obviously had an effect at the TCP and iSCSI layers. My initial message >>> to illumos-developer sparked a potentially interesting discussion: >>> >>> >>> http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ >>> >>> If you think this is a possibility in your setup, I've put the DTrace >>> script I used to hunt for this up on the web: >>> >>> http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d >>> >>> This isn't the only potential source of driver stalls by any means, it's >>> just the one I found. You may also want to look at lockstat in general, >>> as information it reported is what led us to look specifically at the >>> ixgbe code here. >>> >>> (If you suspect kernel/driver issues, lockstat combined with kernel >>> source is a really excellent resource.) >>> >>> - cks >> >> >> >> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > From nagele at wildbit.com Mon Feb 23 04:37:46 2015 From: nagele at wildbit.com (Chris Nagele) Date: Sun, 22 Feb 2015 23:37:46 -0500 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> Message-ID: Is the issue here only related to iSCSI? We've used the X520's for NFS for a couple of years and it has worked really well for us. Not sure if this is an accurate test, but iperf shows the following results for me: Over 1GbE: [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 981 MBytes 823 Mbits/sec Over 10GbE on the same machines: [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 9.42 GBytes 8.09 Gbits/sec I could be going in the wrong direction here, but I was curious as well since we rely on 10G heavily. Chris On Sun, Feb 22, 2015 at 8:54 PM, W Verb wrote: > By the way, to those of you who have working setups: please send me > your pool/volume settings, interface linkprops, and any kernel tuning > parameters you may have set. > > Thanks, > Warren V > > On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip wrote: >> I can't say I totally agree with your performance assessment. I run Intel >> X520 in all my OmniOS boxes. >> >> Here is a capture of nfssvrtop I made while running many storage vMotions >> between two OmniOS boxes hosting NFS datastores. This is a 10 host VMware >> cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to >> the in rack Nexus 5010. >> >> VMware does 100% sync writes, I use ZeusRAM SSDs for log devices. >> >> -Chip >> >> 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 KB, >> awrite: 1875455 KB >> >> Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw >> SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% >> >> 4 10.28.17.105 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.17.215 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.17.213 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.16.151 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 all 1 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 3 10.28.16.175 3 0 3 0 0 1 >> 11 0 4806 48 0 0 85 >> >> 3 10.28.16.183 6 0 6 0 0 3 >> 162 0 549 124 0 0 73 >> >> 3 10.28.16.180 11 0 10 0 0 3 >> 27 0 776 89 0 0 67 >> >> 3 10.28.16.176 28 2 26 0 0 10 >> 405 0 2572 198 0 0 100 >> >> 3 10.28.16.178 4606 4602 4 0 0 294534 >> 3 0 723 49 0 0 99 >> >> 3 10.28.16.179 4905 4879 26 0 0 312208 >> 311 0 735 271 0 0 99 >> >> 3 10.28.16.181 5515 5502 13 0 0 352107 >> 77 0 89 87 0 0 99 >> >> 3 10.28.16.184 12095 12059 10 0 0 763014 >> 39 0 249 147 0 0 99 >> >> 3 10.28.58.1 15401 6040 116 6354 53 191605 >> 474 202346 192 96 144 83 99 >> >> 3 all 42574 33086 217 6354 53 1913488 >> 1582 202300 348 138 153 105 99 >> >> >> >> >> >> On Fri, Feb 20, 2015 at 11:46 PM, W Verb wrote: >>> >>> Hello All, >>> >>> Thank you for your replies. >>> I tried a few things, and found the following: >>> >>> 1: Disabling hyperthreading support in the BIOS drops performance overall >>> by a factor of 4. >>> 2: Disabling VT support also seems to have some effect, although it >>> appears to be minor. But this has the amusing side effect of fixing the >>> hangs I've been experiencing with fast reboot. Probably by disabling kvm. >>> 3: The performance tests are a bit tricky to quantify because of caching >>> effects. In fact, I'm not entirely sure what is happening here. It's just >>> best to describe what I'm seeing: >>> >>> The commands I'm using to test are >>> dd if=/dev/zero of=./test.dd bs=2M count=5000 >>> dd of=/dev/null if=./test.dd bs=2M count=5000 >>> The host vm is running Centos 6.6, and has the latest vmtools installed. >>> There is a host cache on an SSD local to the host that is also in place. >>> Disabling the host cache didn't immediately have an effect as far as I could >>> see. >>> >>> The host MTU set to 3000 on all iSCSI interfaces for all tests. >>> >>> Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test >>> yields an average speed over three tests of 137MB/s. The read test yields an >>> average over three tests of 5MB/s. >>> >>> Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield >>> 140MB/s, and the read tests yield 53MB/s. It's important to note here that >>> if I cut the read test short at only 2-3GB, I get results upwards of >>> 350MB/s, which I assume is local cache-related distortion. >>> >>> Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield >>> about 142MB/s. >>> Test 4: MTU of 1000: Read test at 182MB/s. >>> Test 5: MTU of 900: Read test at 130 MB/s. >>> Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now >>> consistently at about 300MB/s. >>> Test 7: MTU of 1200: Read test at 124MB/s. >>> Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. >>> >>> A few final notes: >>> L1ARC grabs about 10GB of RAM during the tests, so there's definitely some >>> read caching going on. >>> The write operations are easier to observe with iostat, and I'm seeing io >>> rates that closely correlate with the network write speeds. >>> >>> >>> Chris, thanks for your specific details. I'd appreciate it if you could >>> tell me which copper NIC you tried, as well as to pass on the iSCSI tuning >>> parameters. >>> >>> I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of >>> the 82599 in the X520. If I get similar results with my fiber transcievers, >>> I'll see if I can get a hold of copper ones. >>> >>> But I should mention that I did indeed look at PHY/MAC error rates, and >>> they are nil. >>> >>> -Warren V >>> >>> On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann >>> wrote: >>>> >>>> > After installation and configuration, I observed all kinds of bad >>>> > behavior >>>> > in the network traffic between the hosts and the server. All of this >>>> > bad >>>> > behavior is traced to the ixgbe driver on the storage server. Without >>>> > going >>>> > into the full troubleshooting process, here are my takeaways: >>>> [...] >>>> >>>> For what it's worth, we managed to achieve much better line rates on >>>> copper 10G ixgbe hardware of various descriptions between OmniOS >>>> and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't >>>> believe OmniOS could do TCP at full line rate but I think we managed 700+ >>>> Mbytes/sec on both transmit and receive and we got basically disk-limited >>>> speeds with iSCSI (across multiple disks on multi-disk mirrored pools, >>>> OmniOS iSCSI initiator, Linux iSCSI targets). >>>> >>>> I don't believe we did any specific kernel tuning (and in fact some of >>>> our attempts to fiddle ixgbe driver parameters blew up in our face). >>>> We did tune iSCSI connection parameters to increase various buffer >>>> sizes so that ZFS could do even large single operations in single iSCSI >>>> transactions. (More details available if people are interested.) >>>> >>>> > 10: At the wire level, the speed problems are clearly due to pauses in >>>> > response time by omnios. At 9000 byte frame sizes, I see a good number >>>> > of duplicate ACKs and fast retransmits during read operations (when >>>> > omnios is transmitting). But below about a 4100-byte MTU on omnios >>>> > (which seems to correlate to 4096-byte iSCSI block transfers), the >>>> > transmission errors fade away and we only see the transmission pause >>>> > problem. >>>> >>>> This is what really attracted my attention. In our OmniOS setup, our >>>> specific Intel hardware had ixgbe driver issues that could cause >>>> activity stalls during once-a-second link heartbeat checks. This >>>> obviously had an effect at the TCP and iSCSI layers. My initial message >>>> to illumos-developer sparked a potentially interesting discussion: >>>> >>>> >>>> http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ >>>> >>>> If you think this is a possibility in your setup, I've put the DTrace >>>> script I used to hunt for this up on the web: >>>> >>>> http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d >>>> >>>> This isn't the only potential source of driver stalls by any means, it's >>>> just the one I found. You may also want to look at lockstat in general, >>>> as information it reported is what led us to look specifically at the >>>> ixgbe code here. >>>> >>>> (If you suspect kernel/driver issues, lockstat combined with kernel >>>> source is a really excellent resource.) >>>> >>>> - cks >>> >>> >>> >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >> > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From jg at osn.de Mon Feb 23 09:15:20 2015 From: jg at osn.de (Joerg Goltermann) Date: Mon, 23 Feb 2015 10:15:20 +0100 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> Message-ID: <54EAEFA8.4020101@osn.de> Hi, I remember there was a problem with the flow control settings in the ixgbe driver, so I updated it a long time ago for our internal servers to 2.5.8. Last weekend I integrated the latest changes from the FreeBSD driver to bring the illumos ixgbe to 2.5.25 but I had no time to test it, so it's completely untested! If you would like to give the latest driver a try you can fetch the kernel modules from https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9 Clone your boot environment, place the modules in the new environment and update the boot-archive of the new BE. - Joerg On 23.02.2015 02:54, W Verb wrote: > By the way, to those of you who have working setups: please send me > your pool/volume settings, interface linkprops, and any kernel tuning > parameters you may have set. > > Thanks, > Warren V > > On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip wrote: >> I can't say I totally agree with your performance assessment. I run Intel >> X520 in all my OmniOS boxes. >> >> Here is a capture of nfssvrtop I made while running many storage vMotions >> between two OmniOS boxes hosting NFS datastores. This is a 10 host VMware >> cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to >> the in rack Nexus 5010. >> >> VMware does 100% sync writes, I use ZeusRAM SSDs for log devices. >> >> -Chip >> >> 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 KB, >> awrite: 1875455 KB >> >> Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw >> SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% >> >> 4 10.28.17.105 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.17.215 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.17.213 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 10.28.16.151 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 4 all 1 0 0 0 0 0 >> 0 0 0 0 0 0 0 >> >> 3 10.28.16.175 3 0 3 0 0 1 >> 11 0 4806 48 0 0 85 >> >> 3 10.28.16.183 6 0 6 0 0 3 >> 162 0 549 124 0 0 73 >> >> 3 10.28.16.180 11 0 10 0 0 3 >> 27 0 776 89 0 0 67 >> >> 3 10.28.16.176 28 2 26 0 0 10 >> 405 0 2572 198 0 0 100 >> >> 3 10.28.16.178 4606 4602 4 0 0 294534 >> 3 0 723 49 0 0 99 >> >> 3 10.28.16.179 4905 4879 26 0 0 312208 >> 311 0 735 271 0 0 99 >> >> 3 10.28.16.181 5515 5502 13 0 0 352107 >> 77 0 89 87 0 0 99 >> >> 3 10.28.16.184 12095 12059 10 0 0 763014 >> 39 0 249 147 0 0 99 >> >> 3 10.28.58.1 15401 6040 116 6354 53 191605 >> 474 202346 192 96 144 83 99 >> >> 3 all 42574 33086 217 6354 53 1913488 >> 1582 202300 348 138 153 105 99 >> >> >> >> >> >> On Fri, Feb 20, 2015 at 11:46 PM, W Verb wrote: >>> >>> Hello All, >>> >>> Thank you for your replies. >>> I tried a few things, and found the following: >>> >>> 1: Disabling hyperthreading support in the BIOS drops performance overall >>> by a factor of 4. >>> 2: Disabling VT support also seems to have some effect, although it >>> appears to be minor. But this has the amusing side effect of fixing the >>> hangs I've been experiencing with fast reboot. Probably by disabling kvm. >>> 3: The performance tests are a bit tricky to quantify because of caching >>> effects. In fact, I'm not entirely sure what is happening here. It's just >>> best to describe what I'm seeing: >>> >>> The commands I'm using to test are >>> dd if=/dev/zero of=./test.dd bs=2M count=5000 >>> dd of=/dev/null if=./test.dd bs=2M count=5000 >>> The host vm is running Centos 6.6, and has the latest vmtools installed. >>> There is a host cache on an SSD local to the host that is also in place. >>> Disabling the host cache didn't immediately have an effect as far as I could >>> see. >>> >>> The host MTU set to 3000 on all iSCSI interfaces for all tests. >>> >>> Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test >>> yields an average speed over three tests of 137MB/s. The read test yields an >>> average over three tests of 5MB/s. >>> >>> Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield >>> 140MB/s, and the read tests yield 53MB/s. It's important to note here that >>> if I cut the read test short at only 2-3GB, I get results upwards of >>> 350MB/s, which I assume is local cache-related distortion. >>> >>> Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield >>> about 142MB/s. >>> Test 4: MTU of 1000: Read test at 182MB/s. >>> Test 5: MTU of 900: Read test at 130 MB/s. >>> Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now >>> consistently at about 300MB/s. >>> Test 7: MTU of 1200: Read test at 124MB/s. >>> Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. >>> >>> A few final notes: >>> L1ARC grabs about 10GB of RAM during the tests, so there's definitely some >>> read caching going on. >>> The write operations are easier to observe with iostat, and I'm seeing io >>> rates that closely correlate with the network write speeds. >>> >>> >>> Chris, thanks for your specific details. I'd appreciate it if you could >>> tell me which copper NIC you tried, as well as to pass on the iSCSI tuning >>> parameters. >>> >>> I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of >>> the 82599 in the X520. If I get similar results with my fiber transcievers, >>> I'll see if I can get a hold of copper ones. >>> >>> But I should mention that I did indeed look at PHY/MAC error rates, and >>> they are nil. >>> >>> -Warren V >>> >>> On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann >>> wrote: >>>> >>>>> After installation and configuration, I observed all kinds of bad >>>>> behavior >>>>> in the network traffic between the hosts and the server. All of this >>>>> bad >>>>> behavior is traced to the ixgbe driver on the storage server. Without >>>>> going >>>>> into the full troubleshooting process, here are my takeaways: >>>> [...] >>>> >>>> For what it's worth, we managed to achieve much better line rates on >>>> copper 10G ixgbe hardware of various descriptions between OmniOS >>>> and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't >>>> believe OmniOS could do TCP at full line rate but I think we managed 700+ >>>> Mbytes/sec on both transmit and receive and we got basically disk-limited >>>> speeds with iSCSI (across multiple disks on multi-disk mirrored pools, >>>> OmniOS iSCSI initiator, Linux iSCSI targets). >>>> >>>> I don't believe we did any specific kernel tuning (and in fact some of >>>> our attempts to fiddle ixgbe driver parameters blew up in our face). >>>> We did tune iSCSI connection parameters to increase various buffer >>>> sizes so that ZFS could do even large single operations in single iSCSI >>>> transactions. (More details available if people are interested.) >>>> >>>>> 10: At the wire level, the speed problems are clearly due to pauses in >>>>> response time by omnios. At 9000 byte frame sizes, I see a good number >>>>> of duplicate ACKs and fast retransmits during read operations (when >>>>> omnios is transmitting). But below about a 4100-byte MTU on omnios >>>>> (which seems to correlate to 4096-byte iSCSI block transfers), the >>>>> transmission errors fade away and we only see the transmission pause >>>>> problem. >>>> >>>> This is what really attracted my attention. In our OmniOS setup, our >>>> specific Intel hardware had ixgbe driver issues that could cause >>>> activity stalls during once-a-second link heartbeat checks. This >>>> obviously had an effect at the TCP and iSCSI layers. My initial message >>>> to illumos-developer sparked a potentially interesting discussion: >>>> >>>> >>>> http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ >>>> >>>> If you think this is a possibility in your setup, I've put the DTrace >>>> script I used to hunt for this up on the web: >>>> >>>> http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d >>>> >>>> This isn't the only potential source of driver stalls by any means, it's >>>> just the one I found. You may also want to look at lockstat in general, >>>> as information it reported is what led us to look specifically at the >>>> ixgbe code here. >>>> >>>> (If you suspect kernel/driver issues, lockstat combined with kernel >>>> source is a really excellent resource.) >>>> >>>> - cks >>> >>> >>> >>> _______________________________________________ >>> OmniOS-discuss mailing list >>> OmniOS-discuss at lists.omniti.com >>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>> >> > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg Tel: +49 911 39905-0 - Fax: +49 911 39905-55 - http://www.osn.de HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann From wverb73 at gmail.com Mon Feb 23 09:31:55 2015 From: wverb73 at gmail.com (W Verb) Date: Mon, 23 Feb 2015 01:31:55 -0800 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: <54EAEFA8.4020101@osn.de> References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> <54EAEFA8.4020101@osn.de> Message-ID: Thank you Joerg, I've downloaded the package and will try it tomorrow. The only thing I can add at this point is that upon review of my testing, I may have performed my "pkg -u" between the initial quad-gig performance test and installing the 10G NIC. So this may be a new problem introduced in the latest updates. Those of you who are running 10G and have not upgraded to the latest kernel, etc, might want to do some additional testing before running the update. -Warren V On Mon, Feb 23, 2015 at 1:15 AM, Joerg Goltermann wrote: > Hi, > > I remember there was a problem with the flow control settings in the ixgbe > driver, so I updated it a long time ago for our internal servers to 2.5.8. > Last weekend I integrated the latest changes from the FreeBSD driver to > bring > the illumos ixgbe to 2.5.25 but I had no time to test it, so it's > completely > untested! > > > If you would like to give the latest driver a try you can fetch the > kernel modules from https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9 > > Clone your boot environment, place the modules in the new environment > and update the boot-archive of the new BE. > > - Joerg > > > > > > On 23.02.2015 02:54, W Verb wrote: > >> By the way, to those of you who have working setups: please send me >> your pool/volume settings, interface linkprops, and any kernel tuning >> parameters you may have set. >> >> Thanks, >> Warren V >> >> On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip >> wrote: >> >>> I can't say I totally agree with your performance assessment. I run >>> Intel >>> X520 in all my OmniOS boxes. >>> >>> Here is a capture of nfssvrtop I made while running many storage vMotions >>> between two OmniOS boxes hosting NFS datastores. This is a 10 host >>> VMware >>> cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to >>> the in rack Nexus 5010. >>> >>> VMware does 100% sync writes, I use ZeusRAM SSDs for log devices. >>> >>> -Chip >>> >>> 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 >>> KB, >>> awrite: 1875455 KB >>> >>> Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw >>> SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% >>> >>> 4 10.28.17.105 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 >>> >>> 4 10.28.17.215 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 >>> >>> 4 10.28.17.213 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 >>> >>> 4 10.28.16.151 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 >>> >>> 4 all 1 0 0 0 0 0 >>> 0 0 0 0 0 0 0 >>> >>> 3 10.28.16.175 3 0 3 0 0 1 >>> 11 0 4806 48 0 0 85 >>> >>> 3 10.28.16.183 6 0 6 0 0 3 >>> 162 0 549 124 0 0 73 >>> >>> 3 10.28.16.180 11 0 10 0 0 3 >>> 27 0 776 89 0 0 67 >>> >>> 3 10.28.16.176 28 2 26 0 0 10 >>> 405 0 2572 198 0 0 100 >>> >>> 3 10.28.16.178 4606 4602 4 0 0 294534 >>> 3 0 723 49 0 0 99 >>> >>> 3 10.28.16.179 4905 4879 26 0 0 312208 >>> 311 0 735 271 0 0 99 >>> >>> 3 10.28.16.181 5515 5502 13 0 0 352107 >>> 77 0 89 87 0 0 99 >>> >>> 3 10.28.16.184 12095 12059 10 0 0 763014 >>> 39 0 249 147 0 0 99 >>> >>> 3 10.28.58.1 15401 6040 116 6354 53 191605 >>> 474 202346 192 96 144 83 99 >>> >>> 3 all 42574 33086 217 6354 53 1913488 >>> 1582 202300 348 138 153 105 99 >>> >>> >>> >>> >>> >>> On Fri, Feb 20, 2015 at 11:46 PM, W Verb wrote: >>> >>>> >>>> Hello All, >>>> >>>> Thank you for your replies. >>>> I tried a few things, and found the following: >>>> >>>> 1: Disabling hyperthreading support in the BIOS drops performance >>>> overall >>>> by a factor of 4. >>>> 2: Disabling VT support also seems to have some effect, although it >>>> appears to be minor. But this has the amusing side effect of fixing the >>>> hangs I've been experiencing with fast reboot. Probably by disabling >>>> kvm. >>>> 3: The performance tests are a bit tricky to quantify because of caching >>>> effects. In fact, I'm not entirely sure what is happening here. It's >>>> just >>>> best to describe what I'm seeing: >>>> >>>> The commands I'm using to test are >>>> dd if=/dev/zero of=./test.dd bs=2M count=5000 >>>> dd of=/dev/null if=./test.dd bs=2M count=5000 >>>> The host vm is running Centos 6.6, and has the latest vmtools installed. >>>> There is a host cache on an SSD local to the host that is also in place. >>>> Disabling the host cache didn't immediately have an effect as far as I >>>> could >>>> see. >>>> >>>> The host MTU set to 3000 on all iSCSI interfaces for all tests. >>>> >>>> Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test >>>> yields an average speed over three tests of 137MB/s. The read test >>>> yields an >>>> average over three tests of 5MB/s. >>>> >>>> Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield >>>> 140MB/s, and the read tests yield 53MB/s. It's important to note here >>>> that >>>> if I cut the read test short at only 2-3GB, I get results upwards of >>>> 350MB/s, which I assume is local cache-related distortion. >>>> >>>> Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield >>>> about 142MB/s. >>>> Test 4: MTU of 1000: Read test at 182MB/s. >>>> Test 5: MTU of 900: Read test at 130 MB/s. >>>> Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now >>>> consistently at about 300MB/s. >>>> Test 7: MTU of 1200: Read test at 124MB/s. >>>> Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. >>>> >>>> A few final notes: >>>> L1ARC grabs about 10GB of RAM during the tests, so there's definitely >>>> some >>>> read caching going on. >>>> The write operations are easier to observe with iostat, and I'm seeing >>>> io >>>> rates that closely correlate with the network write speeds. >>>> >>>> >>>> Chris, thanks for your specific details. I'd appreciate it if you could >>>> tell me which copper NIC you tried, as well as to pass on the iSCSI >>>> tuning >>>> parameters. >>>> >>>> I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead >>>> of >>>> the 82599 in the X520. If I get similar results with my fiber >>>> transcievers, >>>> I'll see if I can get a hold of copper ones. >>>> >>>> But I should mention that I did indeed look at PHY/MAC error rates, and >>>> they are nil. >>>> >>>> -Warren V >>>> >>>> On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann >>>> wrote: >>>> >>>>> >>>>> After installation and configuration, I observed all kinds of bad >>>>>> behavior >>>>>> in the network traffic between the hosts and the server. All of this >>>>>> bad >>>>>> behavior is traced to the ixgbe driver on the storage server. Without >>>>>> going >>>>>> into the full troubleshooting process, here are my takeaways: >>>>>> >>>>> [...] >>>>> >>>>> For what it's worth, we managed to achieve much better line rates on >>>>> copper 10G ixgbe hardware of various descriptions between OmniOS >>>>> and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't >>>>> believe OmniOS could do TCP at full line rate but I think we managed >>>>> 700+ >>>>> Mbytes/sec on both transmit and receive and we got basically >>>>> disk-limited >>>>> speeds with iSCSI (across multiple disks on multi-disk mirrored pools, >>>>> OmniOS iSCSI initiator, Linux iSCSI targets). >>>>> >>>>> I don't believe we did any specific kernel tuning (and in fact some >>>>> of >>>>> our attempts to fiddle ixgbe driver parameters blew up in our face). >>>>> We did tune iSCSI connection parameters to increase various buffer >>>>> sizes so that ZFS could do even large single operations in single iSCSI >>>>> transactions. (More details available if people are interested.) >>>>> >>>>> 10: At the wire level, the speed problems are clearly due to pauses in >>>>>> response time by omnios. At 9000 byte frame sizes, I see a good number >>>>>> of duplicate ACKs and fast retransmits during read operations (when >>>>>> omnios is transmitting). But below about a 4100-byte MTU on omnios >>>>>> (which seems to correlate to 4096-byte iSCSI block transfers), the >>>>>> transmission errors fade away and we only see the transmission pause >>>>>> problem. >>>>>> >>>>> >>>>> This is what really attracted my attention. In our OmniOS setup, our >>>>> specific Intel hardware had ixgbe driver issues that could cause >>>>> activity stalls during once-a-second link heartbeat checks. This >>>>> obviously had an effect at the TCP and iSCSI layers. My initial message >>>>> to illumos-developer sparked a potentially interesting discussion: >>>>> >>>>> >>>>> http://www.listbox.com/member/archive/182179/2014/10/sort/ >>>>> time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D- >>>>> 11E4-A39C-D534381BA44D/ >>>>> >>>>> If you think this is a possibility in your setup, I've put the DTrace >>>>> script I used to hunt for this up on the web: >>>>> >>>>> http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d >>>>> >>>>> This isn't the only potential source of driver stalls by any means, >>>>> it's >>>>> just the one I found. You may also want to look at lockstat in general, >>>>> as information it reported is what led us to look specifically at the >>>>> ixgbe code here. >>>>> >>>>> (If you suspect kernel/driver issues, lockstat combined with kernel >>>>> source is a really excellent resource.) >>>>> >>>>> - cks >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> OmniOS-discuss mailing list >>>> OmniOS-discuss at lists.omniti.com >>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss >>>> >>>> >>> _______________________________________________ >> OmniOS-discuss mailing list >> OmniOS-discuss at lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> >> > -- > OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg > Tel: +49 911 39905-0 - Fax: +49 911 39905-55 - http://www.osn.de > HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danmcd at omniti.com Mon Feb 23 16:06:15 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 23 Feb 2015 11:06:15 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: <54E7B1D8.3070009@gmx.li> References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> <54E7B1D8.3070009@gmx.li> Message-ID: <07CAF7F6-D55E-47AB-AF40-96B717B03106@omniti.com> > On Feb 20, 2015, at 5:14 PM, Dominik Hassler wrote: > > Dan, > > I had a look at HVM-807 you mentioned. > > Shouldn't line 251 in 'net/vnic.c' be > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) { > > instead of > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) { > > As otherwise the last element of the frameio will not be used at all if > iovcnt is greater than the frameio buffer size? I've no idea, honestly. I count on upstream (Joyent) to look at all of that. If you've found a genuine problem in illumos-kvm or illumos-kvm-cmd, perhaps pinging smartos-discuss, or some other appropriate list may be your best bet. Dan From jg at osn.de Mon Feb 23 16:21:38 2015 From: jg at osn.de (Joerg Goltermann) Date: Mon, 23 Feb 2015 17:21:38 +0100 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: References: <20150221032559.727D07A0792@apps0.cs.toronto.edu> <54EAEFA8.4020101@osn.de> Message-ID: <54EB5392.6030900@osn.de> Hi, I think your problem is caused by your link properties or your switch settings. In general the standard ixgbe seems to perform well. I had trouble after changing the default flow control settings to "bi" and this was my motivation to update the ixgbe driver a long time ago. After I have updated our systems to ixgbe 2.5.8 I never had any problems .... Make sure your switch has support for jumbo frames and you use the same mtu on all ports, otherwise the smallest will be used. What switch do you use? I can tell you nice horror stories about different vendors.... - Joerg On 23.02.2015 10:31, W Verb wrote: > Thank you Joerg, > > I've downloaded the package and will try it tomorrow. > > The only thing I can add at this point is that upon review of my > testing, I may have performed my "pkg -u" between the initial quad-gig > performance test and installing the 10G NIC. So this may be a new > problem introduced in the latest updates. > > Those of you who are running 10G and have not upgraded to the latest > kernel, etc, might want to do some additional testing before running the > update. > > -Warren V > > On Mon, Feb 23, 2015 at 1:15 AM, Joerg Goltermann > wrote: > > Hi, > > I remember there was a problem with the flow control settings in the > ixgbe > driver, so I updated it a long time ago for our internal servers to > 2.5.8. > Last weekend I integrated the latest changes from the FreeBSD driver > to bring > the illumos ixgbe to 2.5.25 but I had no time to test it, so it's > completely > untested! > > > If you would like to give the latest driver a try you can fetch the > kernel modules from > https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9 > > > Clone your boot environment, place the modules in the new environment > and update the boot-archive of the new BE. > > - Joerg > > > > > > On 23.02.2015 02:54, W Verb wrote: > > By the way, to those of you who have working setups: please send me > your pool/volume settings, interface linkprops, and any kernel > tuning > parameters you may have set. > > Thanks, > Warren V > > On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip > > wrote: > > I can't say I totally agree with your performance > assessment. I run Intel > X520 in all my OmniOS boxes. > > Here is a capture of nfssvrtop I made while running many > storage vMotions > between two OmniOS boxes hosting NFS datastores. This is a > 10 host VMware > cluster. Both OmniOS boxes are dual 10G connected with > copper twin-ax to > the in rack Nexus 5010. > > VMware does 100% sync writes, I use ZeusRAM SSDs for log > devices. > > -Chip > > 2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, > swrite: 15985 KB, > awrite: 1875455 KB > > Ver Client NFSOPS Reads SWrites AWrites > Commits Rd_bw > SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align% > > 4 10.28.17.105 0 0 0 0 > 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.17.215 0 0 0 0 > 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.17.213 0 0 0 0 > 0 0 > 0 0 0 0 0 0 0 > > 4 10.28.16.151 0 0 0 0 > 0 0 > 0 0 0 0 0 0 0 > > 4 all 1 0 0 0 > 0 0 > 0 0 0 0 0 0 0 > > 3 10.28.16.175 3 0 3 0 > 0 1 > 11 0 4806 48 0 0 85 > > 3 10.28.16.183 6 0 6 0 > 0 3 > 162 0 549 124 0 0 73 > > 3 10.28.16.180 11 0 10 0 > 0 3 > 27 0 776 89 0 0 67 > > 3 10.28.16.176 28 2 26 0 > 0 10 > 405 0 2572 198 0 0 100 > > 3 10.28.16.178 4606 4602 4 0 > 0 294534 > 3 0 723 49 0 0 99 > > 3 10.28.16.179 4905 4879 26 0 > 0 312208 > 311 0 735 271 0 0 99 > > 3 10.28.16.181 5515 5502 13 0 > 0 352107 > 77 0 89 87 0 0 99 > > 3 10.28.16.184 12095 12059 10 0 > 0 763014 > 39 0 249 147 0 0 99 > > 3 10.28.58.1 15401 6040 116 6354 > 53 191605 > 474 202346 192 96 144 83 99 > > 3 all 42574 33086 217 > 6354 53 1913488 > 1582 202300 348 138 153 105 99 > > > > > > On Fri, Feb 20, 2015 at 11:46 PM, W Verb > wrote: > > > Hello All, > > Thank you for your replies. > I tried a few things, and found the following: > > 1: Disabling hyperthreading support in the BIOS drops > performance overall > by a factor of 4. > 2: Disabling VT support also seems to have some effect, > although it > appears to be minor. But this has the amusing side > effect of fixing the > hangs I've been experiencing with fast reboot. Probably > by disabling kvm. > 3: The performance tests are a bit tricky to quantify > because of caching > effects. In fact, I'm not entirely sure what is > happening here. It's just > best to describe what I'm seeing: > > The commands I'm using to test are > dd if=/dev/zero of=./test.dd bs=2M count=5000 > dd of=/dev/null if=./test.dd bs=2M count=5000 > The host vm is running Centos 6.6, and has the latest > vmtools installed. > There is a host cache on an SSD local to the host that > is also in place. > Disabling the host cache didn't immediately have an > effect as far as I could > see. > > The host MTU set to 3000 on all iSCSI interfaces for all > tests. > > Test 1: Right after reboot, with an ixgbe MTU of 9000, > the write test > yields an average speed over three tests of 137MB/s. The > read test yields an > average over three tests of 5MB/s. > > Test 2: After setting "ifconfig ixgbe0 mtu 3000", the > write tests yield > 140MB/s, and the read tests yield 53MB/s. It's important > to note here that > if I cut the read test short at only 2-3GB, I get > results upwards of > 350MB/s, which I assume is local cache-related distortion. > > Test 3: MTU of 1500. Read tests are up to 156 MB/s. > Write tests yield > about 142MB/s. > Test 4: MTU of 1000: Read test at 182MB/s. > Test 5: MTU of 900: Read test at 130 MB/s. > Test 6: MTU of 1000: Read test at 160MB/s. Write tests > are now > consistently at about 300MB/s. > Test 7: MTU of 1200: Read test at 124MB/s. > Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. > > A few final notes: > L1ARC grabs about 10GB of RAM during the tests, so > there's definitely some > read caching going on. > The write operations are easier to observe with iostat, > and I'm seeing io > rates that closely correlate with the network write speeds. > > > Chris, thanks for your specific details. I'd appreciate > it if you could > tell me which copper NIC you tried, as well as to pass > on the iSCSI tuning > parameters. > > I've ordered an Intel EXPX9502AFXSR, which uses the > 82598 chip instead of > the 82599 in the X520. If I get similar results with my > fiber transcievers, > I'll see if I can get a hold of copper ones. > > But I should mention that I did indeed look at PHY/MAC > error rates, and > they are nil. > > -Warren V > > On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann > > > wrote: > > > After installation and configuration, I observed > all kinds of bad > behavior > in the network traffic between the hosts and the > server. All of this > bad > behavior is traced to the ixgbe driver on the > storage server. Without > going > into the full troubleshooting process, here are > my takeaways: > > [...] > > For what it's worth, we managed to achieve much > better line rates on > copper 10G ixgbe hardware of various descriptions > between OmniOS > and CentOS 7 (I don't think we ever tested OmniOS to > OmniOS). I don't > believe OmniOS could do TCP at full line rate but I > think we managed 700+ > Mbytes/sec on both transmit and receive and we got > basically disk-limited > speeds with iSCSI (across multiple disks on > multi-disk mirrored pools, > OmniOS iSCSI initiator, Linux iSCSI targets). > > I don't believe we did any specific kernel tuning > (and in fact some of > our attempts to fiddle ixgbe driver parameters blew > up in our face). > We did tune iSCSI connection parameters to increase > various buffer > sizes so that ZFS could do even large single > operations in single iSCSI > transactions. (More details available if people are > interested.) > > 10: At the wire level, the speed problems are > clearly due to pauses in > response time by omnios. At 9000 byte frame > sizes, I see a good number > of duplicate ACKs and fast retransmits during > read operations (when > omnios is transmitting). But below about a > 4100-byte MTU on omnios > (which seems to correlate to 4096-byte iSCSI > block transfers), the > transmission errors fade away and we only see > the transmission pause > problem. > > > This is what really attracted my attention. In > our OmniOS setup, our > specific Intel hardware had ixgbe driver issues that > could cause > activity stalls during once-a-second link heartbeat > checks. This > obviously had an effect at the TCP and iSCSI layers. > My initial message > to illumos-developer sparked a potentially > interesting discussion: > > > http://www.listbox.com/member/__archive/182179/2014/10/sort/__time_rev/page/16/entry/6:405/__20141003125035:6357079A-4B1D-__11E4-A39C-D534381BA44D/ > > > If you think this is a possibility in your setup, > I've put the DTrace > script I used to hunt for this up on the web: > > http://www.cs.toronto.edu/~__cks/src/omnios-ixgbe/ixgbe___delay.d > > > This isn't the only potential source of driver > stalls by any means, it's > just the one I found. You may also want to look at > lockstat in general, > as information it reported is what led us to look > specifically at the > ixgbe code here. > > (If you suspect kernel/driver issues, lockstat > combined with kernel > source is a really excellent resource.) > > - cks > > > > > _________________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.__com > > http://lists.omniti.com/__mailman/listinfo/omnios-__discuss > > > > _________________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.__com > > http://lists.omniti.com/__mailman/listinfo/omnios-__discuss > > > > -- > OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg > Tel: +49 911 39905-0 - Fax: +49 911 > 39905-55 - http://www.osn.de > HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann > > -- OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg Tel: +49 911 39905-0 - Fax: +49 911 39905-55 - http://www.osn.de HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann From cks at cs.toronto.edu Mon Feb 23 16:29:48 2015 From: cks at cs.toronto.edu (Chris Siebenmann) Date: Mon, 23 Feb 2015 11:29:48 -0500 Subject: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy In-Reply-To: Your message of Fri, 20 Feb 2015 21:46:51 -0800. Message-ID: <20150223162948.F3F777A082C@apps0.cs.toronto.edu> > Chris, thanks for your specific details. I'd appreciate it if you > could tell me which copper NIC you tried, as well as to pass on the > iSCSI tuning parameters. Our copper NIC experience is with onboard X540-AT2 ports on SuperMicro hardware (which have the guaranteed 10-20 msec lock hold) and dual-port 82599EB TN cards (which have some sort of driver/hardware failure under load that eventually leads to 2-second lock holds). I can't recommend either with the current driver; we had to revert to 1G networking in order to get stable servers. The iSCSI parameter modifications we do, across both initiators and targets, are: initialr2t no firstburstlength 128k maxrecvdataseglen 128k [only on Linux backends] maxxmitdataseglen 128k [only on Linux backends] The OmniOS initiator doesn't need tuning for more than the first two parameters; on the Linux backends we tune up all four. My extended thoughts on these tuning parameters and why we touch them can be found here: http://utcc.utoronto.ca/~cks/space/blog/tech/UnderstandingiSCSIProtocol http://utcc.utoronto.ca/~cks/space/blog/tech/LikelyISCSITuning The short version is that these parameters probably only make a small difference but their overall goal is to do 128KB ZFS reads and writes in single iSCSI operations (although they will be fragmented at the TCP layer) and to do iSCSI writes without a back-and-forth delay between initiator and target (that's 'initialr2t no'). I think basically everyone should use InitialR2T set to no and in fact that it should be the software default. These days only unusually limited iSCSI targets should need it to be otherwise and they can change their setting for it (initiator and target must both agree to it being 'yes', so either can veto it). - cks From danmcd at omniti.com Mon Feb 23 16:06:15 2015 From: danmcd at omniti.com (Dan McDonald) Date: Mon, 23 Feb 2015 11:06:15 -0500 Subject: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010 In-Reply-To: <54E7B1D8.3070009@gmx.li> References: <75A4C8F1-0452-4B40-94A9-210D85516725@omniti.com> <54E7B1D8.3070009@gmx.li> Message-ID: <07CAF7F6-D55E-47AB-AF40-96B717B03106@omniti.com> > On Feb 20, 2015, at 5:14 PM, Dominik Hassler wrote: > > Dan, > > I had a look at HVM-807 you mentioned. > > Shouldn't line 251 in 'net/vnic.c' be > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) { > > instead of > for (i = 0; i < MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) { > > As otherwise the last element of the frameio will not be used at all if > iovcnt is greater than the frameio buffer size? I've no idea, honestly. I count on upstream (Joyent) to look at all of that. If you've found a genuine problem in illumos-kvm or illumos-kvm-cmd, perhaps pinging smartos-discuss, or some other appropriate list may be your best bet. Dan From steve at linuxsuite.org Tue Feb 24 16:55:44 2015 From: steve at linuxsuite.org (steve at linuxsuite.org) Date: Tue, 24 Feb 2015 11:55:44 -0500 Subject: [OmniOS-discuss] Same hardware show different cfgadm? Message-ID: Howdy! I have 2 identical zfs storage configurations consisting of OmniOS v11 r151004 on Dell R710 server LSI - SAS9201-16e: attached to 16 bay Rackable JBOD LSI - SAS9200-8e : attached to 45 bay Supermicro JBOD Things seem to work fine, but there are very different outputs for cfgadm. There are 53 JBOD disks total with 10 raidz zpools + 3 hot spares. There are 5 system disks on the main Dell 710 All pools function ok. Disks are all 3TB SAS cfgadm shows very different information.. root at zfs-1:~# cfgadm Ap_Id Type Receptacle Occupant Condition c1 scsi-bus connected configured unknown c3 scsi-sas connected unconfigured unknown c4 scsi-sas connected configured unknown c6 scsi-sas connected configured unknown c7 scsi-sas connected configured unknown c8 scsi-sas connected unconfigured unknown [snip] root at zfs-2:~# cfgadm Ap_Id Type Receptacle Occupant Condition c0 scsi-sas connected configured unknown c1 scsi-bus connected configured unknown c2 scsi-sas connected unconfigured unknown c3 scsi-sas connected unconfigured unknown c4 scsi-sas connected configured unknown c5 scsi-sas connected configured unknown [snip] A more complete listing is even stranger.. ( see below for complete output ) root at zfs-1:~# cfgadm -al |grep dsk |wc -l 32 root at zfs-2:~# cfgadm -al|grep dsk|wc -l 53 Why do some disks show up as c5::w5000c50055c9a011,0 disk-path connected configured unknown and others as c5::dsk/c5t5000CCA03E202671d0 disk connected configured unknown However, format command sees all disks and can access them normally... Ideas? Does this matter? Why does cfgadm show such different configs? thanx - steve root at zfs-1:~# cfgadm -al Ap_Id Type Receptacle Occupant Conditionc5::w5000c50055c9a011,0 disk-path connected configured unknown c1 scsi-bus connected configured unknown c1::dsk/c1t0d0 disk connected configured unknown c1::dsk/c1t1d0 disk connected configured unknown c1::dsk/c1t2d0 disk connected configured unknown c1::dsk/c1t4d0 disk connected configured unknown c1::dsk/c1t5d0 disk connected configured unknown c3 scsi-sas connected unconfigured unknown c4 scsi-sas connected configured unknown c4::dsk/c4t5000CCA01A7C4FE5d0 disk connected configured unknown c4::dsk/c4t5000CCA01A7CD8A1d0 disk connected configured unknown c4::dsk/c4t5000CCA01A7CDFB5d0 disk connected configured unknown c4::dsk/c4t5000CCA01A72D899d0 disk connected configured unknown c4::dsk/c4t5000CCA01A72DDDDd0 disk connected configured unknown c4::es/ses0 ESI connected configured unknown c4::smp/expd0 smp connected configured unknown c4::w5000c50041a6ab01,0 disk-path connected configured unknown c4::w5000c50041aafa81,0 disk-path connected configured unknown c4::w5000c50041ac4e3d,0 disk-path connected configured unknown c4::w5000c50041e9e081,0 disk-path connected configured unknown c4::w5000c50041f2f09d,0 disk-path connected configured unknown c4::w5000c50055a24fc1,0 disk-path connected configured unknown c4::w5000c50055a25a01,0 disk-path connected configured unknown c4::w5000c50055a25e99,0 disk-path connected configured unknown c4::w5000c50055a259b9,0 disk-path connected configured unknown c4::w5000c50055c9bed5,0 disk-path connected unconfigured unknown c4::w5000c50055c66abd,0 disk-path connected configured unknown c4::w5000c50055ca227d,0 disk-path connected configured unknown c4::w5000c50055ca363d,0 disk-path connected unconfigured unknown c4::w5000c50055e66c29,0 disk-path connected configured unknown c4::w5000c50055e75ad1,0 disk-path connected unconfigured unknown c4::w5000c50055e7546d,0 disk-path connected configured unknown c4::w5000c500423b5e9d,0 disk-path connected unconfigured unknown c4::w5000c500423da1bd,0 disk-path connected unconfigured unknown c4::w5000c500423e977d,0 disk-path connected configured unknown c4::w5000c500424c5e29,0 disk-path connected configured unknown c4::w5000c5004198a2c5,0 disk-path connected configured unknown c6 scsi-sas connected configured unknown c6::dsk/c6t5000CCA03E1E5FF5d0 disk connected configured unknown c6::dsk/c6t5000CCA03E1F33D9d0 disk connected configured unknown c6::dsk/c6t5000CCA03E09BFB1d0 disk connected configured unknown c6::dsk/c6t5000CCA03E014A2Dd0 disk connected configured unknown c6::dsk/c6t5000CCA03E15B6F1d0 disk connected configured unknown c6::dsk/c6t5000CCA03E20AF2Dd0 disk connected configured unknown c6::dsk/c6t5000CCA03E20AF3Dd0 disk connected configured unknown c6::dsk/c6t5000CCA03E167AFDd0 disk connected configured unknown c6::dsk/c6t5000CCA03E167B7Dd0 disk connected configured unknown c6::dsk/c6t5000CCA03E167B29d0 disk connected configured unknown c6::dsk/c6t5000CCA03E172A05d0 disk connected configured unknown c6::dsk/c6t5000CCA03E210B6Dd0 disk connected configured unknown c6::dsk/c6t5000CCA03E0626F1d0 disk connected configured unknown c6::dsk/c6t5000CCA03E012635d0 disk connected configured unknown c6::dsk/c6t5000CCA03E113475d0 disk connected configured unknown c6::dsk/c6t5000CCA03E126771d0 disk connected configured unknown c6::dsk/c6t5000CCA03E167559d0 disk connected configured unknown c6::dsk/c6t5000CCA03E167649d0 disk connected configured unknown c6::dsk/c6t5000CCA03E168005d0 disk connected configured unknown c6::dsk/c6t5000CCA03E168141d0 disk connected configured unknown c6::dsk/c6t5000CCA03E212455d0 disk connected configured unknown c6::es/ses1 ESI connected configured unknown c6::smp/expd1 smp connected configured unknown c7 scsi-sas connected configured unknown c7::dsk/c7t5000CCA03E2126E1d0 disk connected configured unknown c7::es/ses2 ESI connected configured unknown c7::smp/expd2 smp connected configured unknown c7::w5000c50055c9bed5,0 disk-path connected configured unknown c7::w5000c50055ca39e9,0 disk-path connected configured unknown c7::w5000c50055ca363d,0 disk-path connected configured unknown c7::w5000c50055e7b4d5,0 disk-path connected unconfigured unknown c7::w5000c50055e75ad1,0 disk-path connected configured unknown c7::w5000c50055e77a25,0 disk-path connected configured unknown c7::w5000c50055e75671,0 disk-path connected configured unknown c7::w5000c50055e75679,0 disk-path connected configured unknown c7::w5000c500423b5e9d,0 disk-path connected configured unknown c7::w5000c500423da1bd,0 disk-path connected configured unknown c7::w5000c500423dd3d9,0 disk-path connected configured unknown c8 scsi-sas connected unconfigured unknown usb8/1 unknown empty unconfigured ok usb8/2 unknown empty unconfigured ok usb9/1 unknown empty unconfigured ok usb9/2 unknown empty unconfigured ok usb10/1 unknown empty unconfigured ok usb10/2 unknown empty unconfigured ok usb10/3 usb-hub connected configured ok usb10/3.1 unknown empty unconfigured ok usb10/3.2 unknown empty unconfigured ok usb10/3.3 unknown empty unconfigured ok usb10/4 unknown empty unconfigured ok usb11/1 unknown empty unconfigured ok usb11/2 usb-device connected configured ok usb12/1 unknown empty unconfigured ok usb12/2 unknown empty unconfigured ok usb13/1 unknown empty unconfigured ok usb13/2 unknown empty unconfigured ok usb13/3 unknown empty unconfigured ok usb13/4 unknown empty unconfigured ok root at zfs-2:~# cfgadm -al Ap_Id Type Receptacle Occupant Condition c0 scsi-sas connected configured unknown c0::dsk/c0t5000CCA01A7E89F5d0 disk connected configured unknown c0::dsk/c0t5000CCA01A80A9CDd0 disk connected configured unknown c0::dsk/c0t5000CCA01A80AB65d0 disk connected configured unknown c0::dsk/c0t5000CCA01A886801d0 disk connected configured unknown c0::dsk/c0t5000CCA01ABC7339d0 disk connected configured unknown c0::dsk/c0t5000CCA01ABC9445d0 disk connected configured unknown c0::dsk/c0t5000CCA01ABC9535d0 disk connected configured unknown c0::dsk/c0t5000CCA03E1AA175d0 disk connected configured unknown c0::dsk/c0t5000CCA03E11FABDd0 disk connected configured unknown c0::dsk/c0t5000CCA03E210E5Dd0 disk connected configured unknown c0::dsk/c0t5000CCA03E0622C5d0 disk connected configured unknown c0::es/ses2 ESI connected configured unknown c0::smp/expd2 smp connected configured unknown c1 scsi-bus connected configured unknown c1::dsk/c1t0d0 disk connected configured unknown c1::dsk/c1t1d0 disk connected configured unknown c1::dsk/c1t3d0 disk connected configured unknown c1::dsk/c1t4d0 disk connected configured unknown c1::dsk/c1t5d0 disk connected configured unknown c2 scsi-sas connected unconfigured unknown c3 scsi-sas connected unconfigured unknown c4 scsi-sas connected configured unknown c4::dsk/c4t5000CCA01A58FD39d0 disk connected configured unknown c4::dsk/c4t5000CCA03E0B00B5d0 disk connected configured unknown c4::dsk/c4t5000CCA03E1C4401d0 disk connected configured unknown c4::dsk/c4t5000CCA03E1E4A51d0 disk connected configured unknown c4::dsk/c4t5000CCA03E1EFECDd0 disk connected configured unknown c4::dsk/c4t5000CCA03E0065B5d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167AF9d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167B45d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167BA9d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167C0Dd0 disk connected configured unknown c4::dsk/c4t5000CCA03E167C49d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167CA1d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167D71d0 disk connected configured unknown c4::dsk/c4t5000CCA03E215DCDd0 disk connected configured unknown c4::dsk/c4t5000CCA03E1676E9d0 disk connected configured unknown c4::dsk/c4t5000CCA03E16708Dd0 disk connected configured unknown c4::dsk/c4t5000CCA03E16767Dd0 disk connected configured unknown c4::dsk/c4t5000CCA03E16807Dd0 disk connected configured unknown c4::dsk/c4t5000CCA03E21594Dd0 disk connected configured unknown c4::dsk/c4t5000CCA03E167269d0 disk connected configured unknown c4::dsk/c4t5000CCA03E167801d0 disk connected configured unknown c4::es/ses0 ESI connected configured unknown c4::smp/expd0 smp connected configured unknown c5 scsi-sas connected configured unknown c5::dsk/c5t5000CCA01A7D4D85d0 disk connected configured unknown c5::dsk/c5t5000CCA01A7E0499d0 disk connected configured unknown c5::dsk/c5t5000CCA01A7F5F69d0 disk connected configured unknown c5::dsk/c5t5000CCA01A80E98Dd0 disk connected configured unknown c5::dsk/c5t5000CCA01A807E75d0 disk connected configured unknown c5::dsk/c5t5000CCA01AD9C2F9d0 disk connected configured unknown c5::dsk/c5t5000CCA01AD331EDd0 disk connected configured unknown c5::dsk/c5t5000CCA01ADAF64Dd0 disk connected configured unknown c5::dsk/c5t5000CCA01ADDCBF9d0 disk connected configured unknown c5::dsk/c5t5000CCA03E1FD499d0 disk connected configured unknown c5::dsk/c5t5000CCA03E19F571d0 disk connected configured unknown c5::dsk/c5t5000CCA03E00745Dd0 disk connected configured unknown c5::dsk/c5t5000CCA03E202671d0 disk connected configured unknown c5::dsk/c5t5000CCA03E216269d0 disk connected configured unknown c5::dsk/c5t5000CCA03E227781d0 disk connected configured unknown c5::dsk/c5t5000CCA03E726699d0 disk connected configured unknown c5::es/ses1 ESI connected configured unknown c5::smp/expd1 smp connected configured unknown c5::w5000c50055c9a011,0 disk-path connected configured unknown c5::w5000c50055c9ffb9,0 disk-path connected configured unknown c5::w5000c50055e6a5c9,0 disk-path connected configured unknown c5::w5000c50055e6664d,0 disk-path connected configured unknown c5::w5000c5005596543d,0 disk-path connected configured unknown usb8/1 unknown empty unconfigured ok usb8/2 unknown empty unconfigured ok usb9/1 unknown empty unconfigured ok usb9/2 unknown empty unconfigured ok usb10/1 unknown empty unconfigured ok usb10/2 unknown empty unconfigured ok usb10/3 usb-hub connected configured ok usb10/3.1 unknown empty unconfigured ok usb10/3.2 unknown empty unconfigured ok usb10/3.3 unknown empty unconfigured ok usb10/4 unknown empty unconfigured ok usb11/1 unknown empty unconfigured ok usb11/2 usb-device connected configured ok usb12/1 unknown empty unconfigured ok usb12/2 unknown empty unconfigured ok usb13/1 unknown empty unconfigured ok usb13/2 unknown empty unconfigured ok usb13/3 unknown empty unconfigured ok usb13/4 unknown empty unconfigured ok From mark0x01 at gmail.com Wed Feb 25 07:23:58 2015 From: mark0x01 at gmail.com (Mark) Date: Wed, 25 Feb 2015 20:23:58 +1300 Subject: [OmniOS-discuss] Same hardware show different cfgadm? In-Reply-To: References: Message-ID: <54ED788E.8040907@gmail.com> Do you have single or dual path enclosures? Are the disks SAS or Sata? It looks like it is seeing some of the disks on two paths, but not all. c4::w5000c50055ca363d,0 disk-path c7::w5000c50055ca363d,0 disk-path Mark. On 25/02/2015 5:55 a.m., steve at linuxsuite.org wrote: > > Howdy! > > I have 2 identical zfs storage configurations consisting of > OmniOS v11 r151004 > on > > Dell R710 server > LSI - SAS9201-16e: attached to 16 bay Rackable JBOD > LSI - SAS9200-8e : attached to 45 bay Supermicro JBOD > > Things seem to work fine, but there are very different outputs > for cfgadm. There are 53 JBOD disks total with 10 raidz zpools + 3 hot > spares. > There are 5 system disks on the main Dell 710 > > All pools function ok. Disks are all 3TB SAS > > cfgadm shows very different information.. > > root at zfs-1:~# cfgadm > Ap_Id Type Receptacle Occupant > Condition > c1 scsi-bus connected configured unknown > c3 scsi-sas connected unconfigured unknown > c4 scsi-sas connected configured unknown > c6 scsi-sas connected configured unknown > c7 scsi-sas connected configured unknown > c8 scsi-sas connected unconfigured unknown > [snip] > > root at zfs-2:~# cfgadm > Ap_Id Type Receptacle Occupant > Condition > c0 scsi-sas connected configured unknown > c1 scsi-bus connected configured unknown > c2 scsi-sas connected unconfigured unknown > c3 scsi-sas connected unconfigured unknown > c4 scsi-sas connected configured unknown > c5 scsi-sas connected configured unknown > [snip] > > > A more complete listing is even stranger.. ( see below for > complete output ) > > root at zfs-1:~# cfgadm -al |grep dsk |wc -l > 32 > > root at zfs-2:~# cfgadm -al|grep dsk|wc -l > 53 > > Why do some disks show up as > > c5::w5000c50055c9a011,0 disk-path connected configured unknown > > and others as > > c5::dsk/c5t5000CCA03E202671d0 disk connected configured unknown > > > However, format command sees all disks and can access them normally... > > Ideas? Does this matter? Why does cfgadm show such different configs? > > thanx - steve > > root at zfs-1:~# cfgadm -al > Ap_Id Type Receptacle Occupant > Conditionc5::w5000c50055c9a011,0 disk-path connected > configured unknown > > c1 scsi-bus connected configured unknown > c1::dsk/c1t0d0 disk connected configured unknown > c1::dsk/c1t1d0 disk connected configured unknown > c1::dsk/c1t2d0 disk connected configured unknown > c1::dsk/c1t4d0 disk connected configured unknown > c1::dsk/c1t5d0 disk connected configured unknown > c3 scsi-sas connected unconfigured unknown > c4 scsi-sas connected configured unknown > c4::dsk/c4t5000CCA01A7C4FE5d0 disk connected configured unknown > c4::dsk/c4t5000CCA01A7CD8A1d0 disk connected configured unknown > c4::dsk/c4t5000CCA01A7CDFB5d0 disk connected configured unknown > c4::dsk/c4t5000CCA01A72D899d0 disk connected configured unknown > c4::dsk/c4t5000CCA01A72DDDDd0 disk connected configured unknown > c4::es/ses0 ESI connected configured unknown > c4::smp/expd0 smp connected configured unknown > c4::w5000c50041a6ab01,0 disk-path connected configured unknown > c4::w5000c50041aafa81,0 disk-path connected configured unknown > c4::w5000c50041ac4e3d,0 disk-path connected configured unknown > c4::w5000c50041e9e081,0 disk-path connected configured unknown > c4::w5000c50041f2f09d,0 disk-path connected configured unknown > c4::w5000c50055a24fc1,0 disk-path connected configured unknown > c4::w5000c50055a25a01,0 disk-path connected configured unknown > c4::w5000c50055a25e99,0 disk-path connected configured unknown > c4::w5000c50055a259b9,0 disk-path connected configured unknown > c4::w5000c50055c9bed5,0 disk-path connected unconfigured unknown > c4::w5000c50055c66abd,0 disk-path connected configured unknown > c4::w5000c50055ca227d,0 disk-path connected configured unknown > c4::w5000c50055ca363d,0 disk-path connected unconfigured unknown > c4::w5000c50055e66c29,0 disk-path connected configured unknown > c4::w5000c50055e75ad1,0 disk-path connected unconfigured unknown > c4::w5000c50055e7546d,0 disk-path connected configured unknown > c4::w5000c500423b5e9d,0 disk-path connected unconfigured unknown > c4::w5000c500423da1bd,0 disk-path connected unconfigured unknown > c4::w5000c500423e977d,0 disk-path connected configured unknown > c4::w5000c500424c5e29,0 disk-path connected configured unknown > c4::w5000c5004198a2c5,0 disk-path connected configured unknown > c6 scsi-sas connected configured unknown > c6::dsk/c6t5000CCA03E1E5FF5d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E1F33D9d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E09BFB1d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E014A2Dd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E15B6F1d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E20AF2Dd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E20AF3Dd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E167AFDd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E167B7Dd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E167B29d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E172A05d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E210B6Dd0 disk connected configured unknown > c6::dsk/c6t5000CCA03E0626F1d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E012635d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E113475d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E126771d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E167559d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E167649d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E168005d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E168141d0 disk connected configured unknown > c6::dsk/c6t5000CCA03E212455d0 disk connected configured unknown > c6::es/ses1 ESI connected configured unknown > c6::smp/expd1 smp connected configured unknown > c7 scsi-sas connected configured unknown > c7::dsk/c7t5000CCA03E2126E1d0 disk connected configured unknown > c7::es/ses2 ESI connected configured unknown > c7::smp/expd2 smp connected configured unknown > c7::w5000c50055c9bed5,0 disk-path connected configured unknown > c7::w5000c50055ca39e9,0 disk-path connected configured unknown > c7::w5000c50055ca363d,0 disk-path connected configured unknown > c7::w5000c50055e7b4d5,0 disk-path connected unconfigured unknown > c7::w5000c50055e75ad1,0 disk-path connected configured unknown > c7::w5000c50055e77a25,0 disk-path connected configured unknown > c7::w5000c50055e75671,0 disk-path connected configured unknown > c7::w5000c50055e75679,0 disk-path connected configured unknown > c7::w5000c500423b5e9d,0 disk-path connected configured unknown > c7::w5000c500423da1bd,0 disk-path connected configured unknown > c7::w5000c500423dd3d9,0 disk-path connected configured unknown > c8 scsi-sas connected unconfigured unknown > usb8/1 unknown empty unconfigured ok > usb8/2 unknown empty unconfigured ok > usb9/1 unknown empty unconfigured ok > usb9/2 unknown empty unconfigured ok > usb10/1 unknown empty unconfigured ok > usb10/2 unknown empty unconfigured ok > usb10/3 usb-hub connected configured ok > usb10/3.1 unknown empty unconfigured ok > usb10/3.2 unknown empty unconfigured ok > usb10/3.3 unknown empty unconfigured ok > usb10/4 unknown empty unconfigured ok > usb11/1 unknown empty unconfigured ok > usb11/2 usb-device connected configured ok > usb12/1 unknown empty unconfigured ok > usb12/2 unknown empty unconfigured ok > usb13/1 unknown empty unconfigured ok > usb13/2 unknown empty unconfigured ok > usb13/3 unknown empty unconfigured ok > usb13/4 unknown empty unconfigured ok > > > root at zfs-2:~# cfgadm -al > Ap_Id Type Receptacle Occupant > Condition > c0 scsi-sas connected configured unknown > c0::dsk/c0t5000CCA01A7E89F5d0 disk connected configured unknown > c0::dsk/c0t5000CCA01A80A9CDd0 disk connected configured unknown > c0::dsk/c0t5000CCA01A80AB65d0 disk connected configured unknown > c0::dsk/c0t5000CCA01A886801d0 disk connected configured unknown > c0::dsk/c0t5000CCA01ABC7339d0 disk connected configured unknown > c0::dsk/c0t5000CCA01ABC9445d0 disk connected configured unknown > c0::dsk/c0t5000CCA01ABC9535d0 disk connected configured unknown > c0::dsk/c0t5000CCA03E1AA175d0 disk connected configured unknown > c0::dsk/c0t5000CCA03E11FABDd0 disk connected configured unknown > c0::dsk/c0t5000CCA03E210E5Dd0 disk connected configured unknown > c0::dsk/c0t5000CCA03E0622C5d0 disk connected configured unknown > c0::es/ses2 ESI connected configured unknown > c0::smp/expd2 smp connected configured unknown > c1 scsi-bus connected configured unknown > c1::dsk/c1t0d0 disk connected configured unknown > c1::dsk/c1t1d0 disk connected configured unknown > c1::dsk/c1t3d0 disk connected configured unknown > c1::dsk/c1t4d0 disk connected configured unknown > c1::dsk/c1t5d0 disk connected configured unknown > c2 scsi-sas connected unconfigured unknown > c3 scsi-sas connected unconfigured unknown > c4 scsi-sas connected configured unknown > c4::dsk/c4t5000CCA01A58FD39d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E0B00B5d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E1C4401d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E1E4A51d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E1EFECDd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E0065B5d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167AF9d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167B45d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167BA9d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167C0Dd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167C49d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167CA1d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167D71d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E215DCDd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E1676E9d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E16708Dd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E16767Dd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E16807Dd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E21594Dd0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167269d0 disk connected configured unknown > c4::dsk/c4t5000CCA03E167801d0 disk connected configured unknown > c4::es/ses0 ESI connected configured unknown > c4::smp/expd0 smp connected configured unknown > c5 scsi-sas connected configured unknown > c5::dsk/c5t5000CCA01A7D4D85d0 disk connected configured unknown > c5::dsk/c5t5000CCA01A7E0499d0 disk connected configured unknown > c5::dsk/c5t5000CCA01A7F5F69d0 disk connected configured unknown > c5::dsk/c5t5000CCA01A80E98Dd0 disk connected configured unknown > c5::dsk/c5t5000CCA01A807E75d0 disk connected configured unknown > c5::dsk/c5t5000CCA01AD9C2F9d0 disk connected configured unknown > c5::dsk/c5t5000CCA01AD331EDd0 disk connected configured unknown > c5::dsk/c5t5000CCA01ADAF64Dd0 disk connected configured unknown > c5::dsk/c5t5000CCA01ADDCBF9d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E1FD499d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E19F571d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E00745Dd0 disk connected configured unknown > c5::dsk/c5t5000CCA03E202671d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E216269d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E227781d0 disk connected configured unknown > c5::dsk/c5t5000CCA03E726699d0 disk connected configured unknown > c5::es/ses1 ESI connected configured unknown > c5::smp/expd1 smp connected configured unknown > c5::w5000c50055c9a011,0 disk-path connected configured unknown > c5::w5000c50055c9ffb9,0 disk-path connected configured unknown > c5::w5000c50055e6a5c9,0 disk-path connected configured unknown > c5::w5000c50055e6664d,0 disk-path connected configured unknown > c5::w5000c5005596543d,0 disk-path connected configured unknown > usb8/1 unknown empty unconfigured ok > usb8/2 unknown empty unconfigured ok > usb9/1 unknown empty unconfigured ok > usb9/2 unknown empty unconfigured ok > usb10/1 unknown empty unconfigured ok > usb10/2 unknown empty unconfigured ok > usb10/3 usb-hub connected configured ok > usb10/3.1 unknown empty unconfigured ok > usb10/3.2 unknown empty unconfigured ok > usb10/3.3 unknown empty unconfigured ok > usb10/4 unknown empty unconfigured ok > usb11/1 unknown empty unconfigured ok > usb11/2 usb-device connected configured ok > usb12/1 unknown empty unconfigured ok > usb12/2 unknown empty unconfigured ok > usb13/1 unknown empty unconfigured ok > usb13/2 unknown empty unconfigured ok > usb13/3 unknown empty unconfigured ok > usb13/4 unknown empty unconfigured ok > > > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > From tobi at oetiker.ch Wed Feb 25 23:17:35 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Thu, 26 Feb 2015 00:17:35 +0100 (CET) Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool Message-ID: experts! If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for 512n like in the olden days, or use the new 4Kn. I know ZFS can deal with both ... So what would be your choice, and WHY? cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From mir at miras.org Wed Feb 25 23:35:57 2015 From: mir at miras.org (Michael Rasmussen) Date: Thu, 26 Feb 2015 00:35:57 +0100 Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool In-Reply-To: References: Message-ID: <20150226003557.2f534c44@sleipner.datanom.net> On Thu, 26 Feb 2015 00:17:35 +0100 (CET) Tobias Oetiker wrote: > experts! > > If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for > 512n like in the olden days, or use the new 4Kn. > > I know ZFS can deal with both ... > > So what would be your choice, and WHY? > I would try to analyze the expected workload. Writing in small chunks points at 512n while large chunks points towards 4Kn. Small chunks on 4Kn will waste a lot of space. If all else is equal the performance should be the same. IMHO. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael rasmussen cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir datanom net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir miras org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- /usr/games/fortune -es says: To have died once is enough. -- Publius Vergilius Maro (Virgil) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From richard.elling at richardelling.com Wed Feb 25 23:39:19 2015 From: richard.elling at richardelling.com (Richard Elling) Date: Wed, 25 Feb 2015 15:39:19 -0800 Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool In-Reply-To: References: Message-ID: <40D943AB-D3F6-4FF0-AD32-E4F6A7706FF4@richardelling.com> > On Feb 25, 2015, at 3:17 PM, Tobias Oetiker wrote: > > experts! > > If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for > 512n like in the olden days, or use the new 4Kn. > > I know ZFS can deal with both ... > > So what would be your choice, and WHY? Better yet, what would be your requirements, then your choice, then why? -- richard > > cheers > tobi > > -- > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From protonwrangler at gmail.com Thu Feb 26 00:09:21 2015 From: protonwrangler at gmail.com (Warren Marts) Date: Wed, 25 Feb 2015 17:09:21 -0700 Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool In-Reply-To: <40D943AB-D3F6-4FF0-AD32-E4F6A7706FF4@richardelling.com> References: <40D943AB-D3F6-4FF0-AD32-E4F6A7706FF4@richardelling.com> Message-ID: At some point sourcing true 512 B/sector disks will get more difficult and expensive - unless you knew this filesystem is likely to see lots of small writes and performance or space efficiency are critical I'd lean to the 4KB sector disks. On Wed, Feb 25, 2015 at 4:39 PM, Richard Elling < richard.elling at richardelling.com> wrote: > > > On Feb 25, 2015, at 3:17 PM, Tobias Oetiker wrote: > > > > experts! > > > > If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for > > 512n like in the olden days, or use the new 4Kn. > > > > I know ZFS can deal with both ... > > > > So what would be your choice, and WHY? > > Better yet, what would be your requirements, then your choice, then why? > -- richard > > > > > cheers > > tobi > > > > -- > > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > > www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 > > > > _______________________________________________ > > OmniOS-discuss mailing list > > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobi at oetiker.ch Thu Feb 26 07:34:54 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Thu, 26 Feb 2015 08:34:54 +0100 (CET) Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool In-Reply-To: References: <40D943AB-D3F6-4FF0-AD32-E4F6A7706FF4@richardelling.com> Message-ID: Hi Marts, Yesterday Warren Marts wrote: > At some point sourcing true 512 B/sector disks will get more difficult and > expensive - unless you knew this filesystem is likely to see lots of small > writes and performance or space efficiency are critical I'd lean to the 4KB > sector disks. Amazingly, the new He6 drives from HGST only seem to exist in an 512n variant ... while the also new He8 drives only come in 4Kn and 512e types ... > On Wed, Feb 25, 2015 at 4:39 PM, Richard Elling < > richard.elling at richardelling.com> wrote: > > > > > > On Feb 25, 2015, at 3:17 PM, Tobias Oetiker wrote: > > > > > > experts! > > > > > > If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for > > > 512n like in the olden days, or use the new 4Kn. > > > > > > I know ZFS can deal with both ... > > > > > > So what would be your choice, and WHY? > > > > Better yet, what would be your requirements, then your choice, then why? > > -- richard > > > > > > > > cheers > > > tobi > > > > > > -- > > > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > > > www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 > > > > > > _______________________________________________ > > > OmniOS-discuss mailing list > > > OmniOS-discuss at lists.omniti.com > > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > _______________________________________________ > > OmniOS-discuss mailing list > > OmniOS-discuss at lists.omniti.com > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From tobi at oetiker.ch Thu Feb 26 08:37:34 2015 From: tobi at oetiker.ch (Tobias Oetiker) Date: Thu, 26 Feb 2015 09:37:34 +0100 (CET) Subject: [OmniOS-discuss] HGST 7K6000 for a RAIDZ2 pool In-Reply-To: References: <40D943AB-D3F6-4FF0-AD32-E4F6A7706FF4@richardelling.com> Message-ID: Today Tobias Oetiker wrote: > Hi Marts, > > Yesterday Warren Marts wrote: > > > At some point sourcing true 512 B/sector disks will get more difficult and > > expensive - unless you knew this filesystem is likely to see lots of small > > writes and performance or space efficiency are critical I'd lean to the 4KB > > sector disks. > > Amazingly, the new He6 drives from HGST only seem to exist in an > 512n variant ... while the also new He8 drives only come in 4Kn and > 512e types ... also while on the subject, there is https://github.com/zfsonlinux/zfs/issues/548 https://github.com/zfsonlinux/zfs/issues/1807 which would indicate that using 512n is probably a better choice in an environment where we are using zvol a lot cheers tobi > > > > On Wed, Feb 25, 2015 at 4:39 PM, Richard Elling < > > richard.elling at richardelling.com> wrote: > > > > > > > > > On Feb 25, 2015, at 3:17 PM, Tobias Oetiker wrote: > > > > > > > > experts! > > > > > > > > If you were to buy 6TB disks for a RAIDZ2 Pool, would you go for > > > > 512n like in the olden days, or use the new 4Kn. > > > > > > > > I know ZFS can deal with both ... > > > > > > > > So what would be your choice, and WHY? > > > > > > Better yet, what would be your requirements, then your choice, then why? > > > -- richard > > > > > > > > > > > cheers > > > > tobi > > > > > > > > -- > > > > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > > > > www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 > > > > > > > > _______________________________________________ > > > > OmniOS-discuss mailing list > > > > OmniOS-discuss at lists.omniti.com > > > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > > > _______________________________________________ > > > OmniOS-discuss mailing list > > > OmniOS-discuss at lists.omniti.com > > > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > > > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi at oetiker.ch +41 62 775 9902 From omnios at citrus-it.net Fri Feb 27 10:05:26 2015 From: omnios at citrus-it.net (Andy) Date: Fri, 27 Feb 2015 10:05:26 +0000 (GMT) Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: On Wed, 18 Feb 2015, Andy wrote: ; Is anyone running, or can anyone recommend, ; a Supermicro server roughly ; equivalent to the Del R730xd, or give me an idea of what chipsets/HBAs ; etc. to choose or avoid for OmniOS? ; ; Any help appreciated, Thanks for all of the responses. We've been researching and mulling it over and the latest plan is to go for Dell R730 servers with a 16 x 2.5" SAS backplane, Intel Ethernet I350 networking and the standard Dell PERC H730 SAS controller. Dell now officially support IT mode with the latest firmware for the PERC H730 and I've read some encouraging reports. If we experience problems then there are options to flash standard LSI firmware or to buy different cards but I'm hopeful that it will just work. If we go ahead, I'll let you all know how it goes! Thanks, Andy -- Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From danmcd at omniti.com Fri Feb 27 15:54:36 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 10:54:36 -0500 Subject: [OmniOS-discuss] Dell vs. Supermicro and any recommendations.. In-Reply-To: References: Message-ID: <81F2A38D-5298-4FB8-B0BB-24E4506D6040@omniti.com> > On Feb 27, 2015, at 5:05 AM, Andy wrote: > > If we go ahead, I'll let you all know how it goes! Please do that. If you can zap a Dell Standard HBA out of HW-RAID and into a raw-disk controller, that'd be a HUGE WIN for illumos distros everywhere. Dan From danmcd at omniti.com Fri Feb 27 16:20:50 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 11:20:50 -0500 Subject: [OmniOS-discuss] (From illumos) Request for e1000g and igb testing for latest parts References: <54F09887.10003@joyent.com> Message-ID: <3ED021A3-FCDB-4D15-8F35-6C4CB1605A6E@omniti.com> Hello OmniOS fans! Robert Mustacchi from Joyent has updated e1000g & igb with more modern code from Intel. If you are an OmniOS person looking to try this out, and need help updating either your e1000g or igb, please let me know via the OmniOS discussion list, or via unicast mail. The binaries he supplies below should just *DROP IN* on any OmniOS revision. I'll be trying this out on 64-bit e1000g (for 82579LM or 85274L) in bloody and maybe 64-bit igb in r151012 (for I210) respectively. NOTE: Use the non-debug bits unless you've built an entire BE around a DEBUG illumos-omnios build. My advice for those who want to try is: beadm create new-intel beadm mount new-intel /mnt beadm umount new-intel beadm activate new-intel reboot You can and should re-activate the existing BE when you're done testing. Happy Intel Ethernetting! Dan > Begin forwarded message: > > Date: February 27, 2015 at 11:17:11 AM EST > From: "Robert Mustacchi via illumos-developer" > To: illumos Developer , "smartos-discuss at lists.smartos.org" > Subject: Re: [developer] Request for e1000g and igb testing for latest parts > Reply-To: "Robert Mustacchi" > > On 2/27/15 7:51 , Robert Mustacchi via illumos-developer wrote: >> Hi, >> >> Various Broadwell and related hardware is using newer e1000g parts, I >> took the opportunity to sync up our common code to get support for them >> and some newer igb parts, being more I218 and I210 branded parts. As >> such, I have a newer igb and e1000g driver and would appreciate some >> broader community testing on both the older devices and if you happen to >> have one of the previously unsupported devices, some of the newer ones. >> >> If you test it, can you please let me know what part you're using. You >> can get the information from prtconf -d. >> >> Here's all the pieces: >> >> e1000g i386: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv/e1000g >> e1000g amd64: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv/amd64/e1000g >> igb i386: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv/igb >> igb amd64: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv/amd64/igb >> >> smartos platform: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/smartos/platform-20150223T225455Z.tgz >> smartos iso: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/smartos/platform-20150223T225455Z.iso >> smartos usb: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/smartos/platform-20150223T225455Z.usb.bz2 >> >> sdc platform: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/sdc/platform-20150223T225545Z.tgz >> >> webrev: >> https://us-east.manta.joyent.com/rmustacc/public/webrevs/e1000g/index.html >> >> patch: >> https://us-east.manta.joyent.com/rmustacc/public/preview/i218/e1000g.patch > > It was kindly pointed out that I had provided debug-build drivers. Here > are links to the non-debug variants: > > e1000g i386: > https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv-nd/e1000g > e1000g amd64: > https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv-nd/amd64/e1000g > igb i386: > https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv-nd/igb > igb amd64: > https://us-east.manta.joyent.com/rmustacc/public/preview/i218/drv-nd/amd64/igb > > Thanks again, > Robert > > > > ------------------------------------------- > illumos-developer > Archives: https://www.listbox.com/member/archive/182179/=now > RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175029-813097db > Modify Your Subscription: https://www.listbox.com/member/?member_id=21175029&id_secret=21175029-471fe0d4 > Powered by Listbox: http://www.listbox.com From danmcd at omniti.com Fri Feb 27 21:27:18 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 16:27:18 -0500 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? Message-ID: This is a serious question. A recent illumos push mildly affects people who have, but in OmniOS's case, it's a mild change (not a horrible one). I'd appreciate some on-list feedback about who does and doesn't change their /etc/profile or /etc/.login vs. what we ship. The next bloody update (and r151014) will alter the default file, and because of IPS, it won't be replaced IF the file was altered from the stock configuration. Fortunately for OmniOS, it's mostly a don't-care. I appreciate your feedback! Thanks, Dan McD -- OmniOS Engineering From peter.tribble at gmail.com Fri Feb 27 21:49:19 2015 From: peter.tribble at gmail.com (Peter Tribble) Date: Fri, 27 Feb 2015 21:49:19 +0000 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: On Fri, Feb 27, 2015 at 9:27 PM, Dan McDonald wrote: > This is a serious question. A recent illumos push mildly affects people > who have, but in OmniOS's case, it's a mild change (not a horrible one). > > I'd appreciate some on-list feedback about who does and doesn't change > their /etc/profile or /etc/.login vs. what we ship. > This isn't OmniOS specific, but every production Solaris box I've managed for the last decade or two has had one or both fixed (usually .login, because tcsh was the default, now /etc/profile for other shells). Primarily to eradicate the completely unnecessary calls to quota and mail. I've seen some horrible monstrosities perpetrated by commercial software or well-meaning power users. PowerPath has this habit of whacking its stuff onto /etc/profile, that's the one 3rd-party application that I've seen on a routine basis. > The next bloody update (and r151014) will alter the default file, and > because of IPS, it won't be replaced IF the file was altered from the stock > configuration. Fortunately for OmniOS, it's mostly a don't-care. > > I appreciate your feedback! > > Thanks, > Dan McD -- OmniOS Engineering > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gate03 at landcroft.co.uk Fri Feb 27 22:04:15 2015 From: gate03 at landcroft.co.uk (Michael Mounteney) Date: Sat, 28 Feb 2015 08:04:15 +1000 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: <20150228080415.4a268256@dickless.landy.net> On Fri, 27 Feb 2015 16:27:18 -0500 Dan McDonald wrote: > This is a serious question. A recent illumos push mildly affects > people who have, but in OmniOS's case, it's a mild change (not a > horrible one). > > I'd appreciate some on-list feedback about who does and doesn't > change their /etc/profile or /etc/.login vs. what we ship. I leave as stock, but I'm a home user really. Michael. From sjorge+ml at blackdot.be Fri Feb 27 22:16:24 2015 From: sjorge+ml at blackdot.be (Jorge Schrauwen) Date: Fri, 27 Feb 2015 23:16:24 +0100 Subject: [OmniOS-discuss] =?utf-8?q?Who_alters_/etc/profile_or_/etc/=2Elog?= =?utf-8?q?in_on_their_boxes=3F?= In-Reply-To: References: Message-ID: <2bace3c6b17daa3e0d0415d3c3b770a6@blackdot.be> I made one small change: if [ -e /etc/motd.generated ]; then /bin/cat -s /etc/motd-* else /bin/cat -s /etc/motd fi motd-00-header -> header unique per server motd-01-info -> file generated every 10min with some basic info like disk usage for /export/home,... motd-02-footer -> symlink to /etc/motd Made no further changes. Regards Jorge On 2015-02-27 22:27, Dan McDonald wrote: > This is a serious question. A recent illumos push mildly affects > people who have, but in OmniOS's case, it's a mild change (not a > horrible one). > > I'd appreciate some on-list feedback about who does and doesn't change > their /etc/profile or /etc/.login vs. what we ship. > > The next bloody update (and r151014) will alter the default file, and > because of IPS, it won't be replaced IF the file was altered from the > stock configuration. Fortunately for OmniOS, it's mostly a > don't-care. > > I appreciate your feedback! > > Thanks, > Dan McD -- OmniOS Engineering > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From tim at multitalents.net Fri Feb 27 22:36:39 2015 From: tim at multitalents.net (Tim Rice) Date: Fri, 27 Feb 2015 14:36:39 -0800 (PST) Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: On Fri, 27 Feb 2015, Dan McDonald wrote: > This is a serious question. A recent illumos push mildly affects people > who have, but in OmniOS's case, it's a mild change (not a horrible one). > > I'd appreciate some on-list feedback about who does and doesn't change > their /etc/profile or /etc/.login vs. what we ship. I can't remember a distro that I didn't modify /etc/profile and /etc/.login (or their equivilants). > The next bloody update (and r151014) will alter the default file, and > because of IPS, it won't be replaced IF the file was altered from the > stock configuration. Fortunately for OmniOS, it's mostly a don't-care. Maybe IPS has a way to install it/them as .new if it/they have been modified. > I appreciate your feedback! > > Thanks, > Dan McD -- OmniOS Engineering > -- Tim Rice Multitalents tim at multitalents.net From danmcd at omniti.com Fri Feb 27 22:45:48 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 17:45:48 -0500 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: <711514CD-8A85-4B1F-8AC5-A8DD1521585D@omniti.com> > On Feb 27, 2015, at 5:36 PM, Tim Rice wrote: > >> The next bloody update (and r151014) will alter the default file, and >> because of IPS, it won't be replaced IF the file was altered from the >> stock configuration. Fortunately for OmniOS, it's mostly a don't-care. > > Maybe IPS has a way to install it/them as .new if it/they > have been modified. It does, and it will. A profile.new will be installed if you have an altered /etc/profile. Sorry I didn't mention that earlier. Dan From danmcd at omniti.com Fri Feb 27 23:00:00 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 18:00:00 -0500 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: <6CCB52C9-3873-4BB6-A071-72AFB879845F@omniti.com> > On Feb 27, 2015, at 4:27 PM, Dan McDonald wrote: > > The next bloody update (and r151014) will alter the default file, and because of IPS, it won't be replaced IF the file was altered from the stock configuration. Fortunately for OmniOS, it's mostly a don't-care. > > I appreciate your feedback! The change, BTW, is that upstream has ALSO removed machid(1) from illumos. Since we already had done it, our diffs are functionally the same, but I chose to inherit upstream's (modulo one copyright line) for future goodness: bloody(~)[1]% diff -u /mnt/etc/profile /etc --- /mnt/etc/profile Tue Feb 3 15:01:21 2015 +++ /etc/profile Thu Feb 26 23:24:53 2015 @@ -21,6 +21,7 @@ # Copyright 2010 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # Copyright 2012 OmniTI Computer Consulting, Inc. All rights reserved. +# Copyright 2015 Nexenta Systems, Inc. All rights reserved. # # The profile that all logins get before using their own .profile. @@ -30,9 +31,7 @@ if [ "$TERM" = "" ] then - CAN_I386=`/usr/bin/isalist | grep i386` - if [ -n "$CAN_I386" ] - then + if [ `uname -p` = "i386" ]; then TERM=sun-color else TERM=sun bloody(~)[1]% So those of you with altered /etc/profile or /etc/.login files will need to be aware you'll be getting .new ones installed on the next bloody, or with r151014 if you don't do bloody. FYI! Dan From sjorge+ml at blackdot.be Fri Feb 27 23:02:12 2015 From: sjorge+ml at blackdot.be (Jorge Schrauwen) Date: Sat, 28 Feb 2015 00:02:12 +0100 Subject: [OmniOS-discuss] =?utf-8?q?Who_alters_/etc/profile_or_/etc/=2Elog?= =?utf-8?q?in_on_their_boxes=3F?= In-Reply-To: <6CCB52C9-3873-4BB6-A071-72AFB879845F@omniti.com> References: <6CCB52C9-3873-4BB6-A071-72AFB879845F@omniti.com> Message-ID: <7dd8558e38d7e3d14857227692fcf699@blackdot.be> Add a note to the release notes when the time comes, should be good enough? On 2015-02-28 00:00, Dan McDonald wrote: >> On Feb 27, 2015, at 4:27 PM, Dan McDonald wrote: >> >> The next bloody update (and r151014) will alter the default file, and >> because of IPS, it won't be replaced IF the file was altered from the >> stock configuration. Fortunately for OmniOS, it's mostly a >> don't-care. >> >> I appreciate your feedback! > > The change, BTW, is that upstream has ALSO removed machid(1) from > illumos. Since we already had done it, our diffs are functionally > the same, but I chose to inherit upstream's (modulo one copyright > line) for future goodness: > > bloody(~)[1]% diff -u /mnt/etc/profile /etc > --- /mnt/etc/profile Tue Feb 3 15:01:21 2015 > +++ /etc/profile Thu Feb 26 23:24:53 2015 > @@ -21,6 +21,7 @@ > # Copyright 2010 Sun Microsystems, Inc. All rights reserved. > # Use is subject to license terms. > # Copyright 2012 OmniTI Computer Consulting, Inc. All rights > reserved. > +# Copyright 2015 Nexenta Systems, Inc. All rights reserved. > # > > # The profile that all logins get before using their own .profile. > @@ -30,9 +31,7 @@ > > if [ "$TERM" = "" ] > then > - CAN_I386=`/usr/bin/isalist | grep i386` > - if [ -n "$CAN_I386" ] > - then > + if [ `uname -p` = "i386" ]; then > TERM=sun-color > else > TERM=sun > bloody(~)[1]% > > > So those of you with altered /etc/profile or /etc/.login files will > need to be aware you'll be getting .new ones installed on the next > bloody, or with r151014 if you don't do bloody. > > FYI! > Dan > > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss From danmcd at omniti.com Fri Feb 27 23:03:41 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 18:03:41 -0500 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: <7dd8558e38d7e3d14857227692fcf699@blackdot.be> References: <6CCB52C9-3873-4BB6-A071-72AFB879845F@omniti.com> <7dd8558e38d7e3d14857227692fcf699@blackdot.be> Message-ID: <5916276A-8F80-436D-BF4A-7F52E66D834E@omniti.com> > On Feb 27, 2015, at 6:02 PM, Jorge Schrauwen wrote: > > Add a note to the release notes when the time comes, should be good enough? Yes, some of the myriad docs I need to write in the next month+. Also, it's a flag day for the next bloody I push out the door. Dan From omnios at citrus-it.net Fri Feb 27 23:16:33 2015 From: omnios at citrus-it.net (Andy) Date: Fri, 27 Feb 2015 23:16:33 +0000 (GMT) Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: On Fri, 27 Feb 2015, Dan McDonald wrote: ; This is a serious question. A recent illumos push mildly affects people who have, but in OmniOS's case, it's a mild change (not a horrible one). ; ; I'd appreciate some on-list feedback about who does and doesn't change their /etc/profile or /etc/.login vs. what we ship. We change /etc/profile here (and /etc/zshrc since that's our default shell). Nothing major or that can't be put back following the update. -- Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ Registered in England and Wales | Company number 4899123 From lotheac at iki.fi Sat Feb 28 00:59:45 2015 From: lotheac at iki.fi (Lauri Tirkkonen) Date: Sat, 28 Feb 2015 02:59:45 +0200 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: <20150228005945.GA3762@gutsman.lotheac.fi> On Fri, Feb 27 2015 16:27:18 -0500, Dan McDonald wrote: > This is a serious question. A recent illumos push mildly affects > people who have, but in OmniOS's case, it's a mild change (not a > horrible one). Which one, and how? > I'd appreciate some on-list feedback about who does and doesn't change > their /etc/profile or /etc/.login vs. what we ship. I would expect very nearly everyone to change them. That's why they're in /etc, right? > The next bloody update (and r151014) will alter the default file, and > because of IPS, it won't be replaced IF the file was altered from the > stock configuration. Fortunately for OmniOS, it's mostly a > don't-care. Sure - as I understand it, this is how user-modifiable configuration files are normally handled with IPS. What's the problem behind the seriousness? -- Lauri Tirkkonen | lotheac @ IRCnet From danmcd at omniti.com Sat Feb 28 01:02:02 2015 From: danmcd at omniti.com (Dan McDonald) Date: Fri, 27 Feb 2015 20:02:02 -0500 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: <20150228005945.GA3762@gutsman.lotheac.fi> References: <20150228005945.GA3762@gutsman.lotheac.fi> Message-ID: <047AEDCF-6511-4FA0-8E00-6BF601730738@omniti.com> Don't want people freaking out. If we had NOT ALREADY removed machid(1), it woiuld be a very noticeable problem. As it is, it's more of a "oh, there's a .new version." Dan Sent from my iPhone (typos, autocorrect, and all) > On Feb 27, 2015, at 7:59 PM, Lauri Tirkkonen wrote: > >> On Fri, Feb 27 2015 16:27:18 -0500, Dan McDonald wrote: >> This is a serious question. A recent illumos push mildly affects >> people who have, but in OmniOS's case, it's a mild change (not a >> horrible one). > > Which one, and how? > >> I'd appreciate some on-list feedback about who does and doesn't change >> their /etc/profile or /etc/.login vs. what we ship. > > I would expect very nearly everyone to change them. That's why they're > in /etc, right? > >> The next bloody update (and r151014) will alter the default file, and >> because of IPS, it won't be replaced IF the file was altered from the >> stock configuration. Fortunately for OmniOS, it's mostly a >> don't-care. > > Sure - as I understand it, this is how user-modifiable configuration > files are normally handled with IPS. What's the problem behind the > seriousness? > > -- > Lauri Tirkkonen | lotheac @ IRCnet From henson at acm.org Sat Feb 28 03:58:35 2015 From: henson at acm.org (Paul B. Henson) Date: Fri, 27 Feb 2015 19:58:35 -0800 Subject: [OmniOS-discuss] Who alters /etc/profile or /etc/.login on their boxes? In-Reply-To: References: Message-ID: <237901d0530a$d5f1b3f0$81d51bd0$@acm.org> > From: Dan McDonald > Sent: Friday, February 27, 2015 1:27 PM > > I'd appreciate some on-list feedback about who does and doesn't change their > /etc/profile or /etc/.login vs. what we ship. I will raise my hand here. I really wish upstream would've accepted the proposal for /etc/profile.d, as that would remove my need to edit the default system stuff . What a bike shed that turned into 8-/. Any thoughts on incorporating such a change into omnios? The final decision on the illumos discuss list as I recall was that distributions should just do it themselves if they wanted to... From sjorge+ml at blackdot.be Sat Feb 28 08:55:33 2015 From: sjorge+ml at blackdot.be (Jorge Schrauwen) Date: Sat, 28 Feb 2015 09:55:33 +0100 Subject: [OmniOS-discuss] =?utf-8?q?Who_alters_/etc/profile_or_/etc/=2Elog?= =?utf-8?q?in_on_their=09boxes=3F?= In-Reply-To: <237901d0530a$d5f1b3f0$81d51bd0$@acm.org> References: <237901d0530a$d5f1b3f0$81d51bd0$@acm.org> Message-ID: On 2015-02-28 04:58, Paul B. Henson wrote: >> From: Dan McDonald >> Sent: Friday, February 27, 2015 1:27 PM >> >> I'd appreciate some on-list feedback about who does and doesn't change > their >> /etc/profile or /etc/.login vs. what we ship. > > I will raise my hand here. I really wish upstream would've accepted the > proposal for /etc/profile.d, as that would remove my need to edit the > default system stuff . What a bike shed that turned into 8-/. > > Any thoughts on incorporating such a change into omnios? The final > decision > on the illumos discuss list as I recall was that distributions should > just > do it themselves if they wanted to... > +1 on this! > _______________________________________________ > OmniOS-discuss mailing list > OmniOS-discuss at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss