<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I’m seeing an annoying NFSv4 behaviour on OmniOS (also on Solaris 10 so this is probably an old bug).<div class=""><br class=""></div><div class="">Setup:</div><div class=""><br class=""></div><div class="">Server: </div><div class=""><br class=""></div><div class="">FreeBSD 11.0 joined to an AD with Kerberos and stuff serving NFSv4 _only_ with V4 “root” set to /export which is a ZFS pool/filesystem.</div><div class="">Under /export there are a number of additional filesystems with various “sharenfs” settings:</div><div class=""><br class=""></div><div class="">/etc/exports</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>V4:<span class="Apple-tab-span" style="white-space:pre"> </span>/export<span class="Apple-tab-span" style="white-space:pre"> </span>-sec=krb5:krb5i:krb5p:sys</div><div class=""><br class=""></div><div class=""> # zfs list -o name,sharenfs,mountpoint -r -d 1 DATA</div><div class="">NAME <span class="Apple-tab-span" style="white-space:pre"> </span>SHARENFS <span class="Apple-tab-span" style="white-space:pre"> </span>MOUNTPOINT</div><div class="">DATA <span class="Apple-tab-span" style="white-space:pre"> </span>ro <span class="Apple-tab-span" style="white-space:pre"> </span>/export</div><div class="">DATA/db <span class="Apple-tab-span" style="white-space:pre"> </span>ro <span class="Apple-tab-span" style="white-space:pre"> </span>/export/db</div><div class="">DATA/liu <span class="Apple-tab-span" style="white-space:pre"> </span>ro <span class="Apple-tab-span" style="white-space:pre"> </span>/liu</div><div class="">DATA/samarbete <span class="Apple-tab-span" style="white-space:pre"> </span>no <span class="Apple-tab-span" style="white-space:pre"> </span>/export/samarbete</div><div class="">DATA/software <span class="Apple-tab-span" style="white-space:pre"> </span>sec=krb5:sys,ro <span class="Apple-tab-span" style="white-space:pre"> </span>/export/software</div><div class="">DATA/staff <span class="Apple-tab-span" style="white-space:pre"> </span>ro <span class="Apple-tab-span" style="white-space:pre"> </span>/export/staff</div><div class="">DATA/students <span class="Apple-tab-span" style="white-space:pre"> </span>sec=krb5:krb5i:krb5p <span class="Apple-tab-span" style="white-space:pre"> </span>/export/students</div><div class="">DATA/test <span class="Apple-tab-span" style="white-space:pre"> </span>sec=sys,ro <span class="Apple-tab-span" style="white-space:pre"> </span>/export/test</div><div class="">DATA/users <span class="Apple-tab-span" style="white-space:pre"> </span>sec=krb5:krb5i:krb5p <span class="Apple-tab-span" style="white-space:pre"> </span>/export/users</div><div class=""><br class=""></div><div class="">Mounting the root of these filsystems on Linux (CentOS 7), Solaris 10/OmniOS and FreeBSD 11 (using NFSv4, sec=sys) gives slightly different behaviour, </div><div class="">but Solaris/OmniOS fails in a different way…</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Clients:</div><div class=""><br class=""></div><div class="">Linux (CentOS 7):</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><div class=""># mount -t nfs -o vers=4,sec=sys,ro testy:/ /mnt</div><div class=""># cd /mnt</div><div class=""># ls</div><div class="">db samarbete software staff students test users</div><div class=""># ls -l</div><div class="">total 7</div><div class="">drwxr-xr-x 3 root wheel 3 Jul 19 12:38 db</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 19 13:01 samarbete</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 18 13:40 software</div><div class="">drwxr-xr-x 2 root wheel 2 Jun 1 19:59 staff</div><div class="">drwxr-xr-x 88 root wheel 88 Aug 3 13:00 students</div><div class="">drwxr-xr-x 3 root wheel 4 Aug 19 13:15 test</div><div class="">drwxr-xr-x 4 root wheel 4 Jul 16 12:54 users</div><div class="">[root@filifjonkan mnt]# cd students</div><div class="">bash: cd: students: Operation not permitted</div></div><div class=""><wait some 30s or so></div><div class=""># ls -l</div><div class="">ls: cannot access students: Operation not permitted</div><div class="">ls: cannot access users: Operation not permitted</div><div class="">total 3</div><div class="">drwxr-xr-x 3 root wheel 3 Jul 19 12:38 db</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 19 13:01 samarbete</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 18 13:40 software</div><div class="">drwxr-xr-x 2 root wheel 2 Jun 1 19:59 staff</div><div class="">d????????? ? ? ? ? ? students</div><div class="">drwxr-xr-x 3 root wheel 4 Aug 19 13:15 test</div><div class="">d????????? ? ? ? ? ? users</div></blockquote></div><div class=""><br class=""></div><div class="">Ok, not perfect. But doable.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">FreeBSD 11.0:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""># mount -t nfs -o vers=4,sec=sys,ro testy:/ /mnt</div><div class=""># cd /mnt</div><div class=""># ls</div><div class="">db<span class="Apple-tab-span" style="white-space:pre"> </span>samarbete<span class="Apple-tab-span" style="white-space:pre"> </span>software<span class="Apple-tab-span" style="white-space:pre"> </span>staff<span class="Apple-tab-span" style="white-space:pre"> </span>students<span class="Apple-tab-span" style="white-space:pre"> </span>test<span class="Apple-tab-span" style="white-space:pre"> </span>users</div><div class=""># /bin/ls -l</div><div class="">nfsv4 err=10016</div><div class="">nfsv4 err=10016</div><div class="">ls: students: Input/output error</div><div class="">ls: users: Input/output error</div><div class="">total 3</div><div class="">drwxr-xr-x 3 root wheel 3 Jul 19 12:38 db</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 19 13:01 samarbete</div><div class="">drwxr-xr-x 3 root wheel 3 Aug 18 13:40 software</div><div class="">drwxr-xr-x 2 root wheel 2 Jun 1 19:59 staff</div><div class="">drwxr-xr-x 3 root wheel 4 Aug 19 13:15 test</div></blockquote></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">OmniOS (basically the same behaviour with Solaris 10):</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""># mount -F nfs -o vers=4,sec=sys,ro testy:/ /mnt</div><div class=""># cd /mnt</div><div class=""># /bin/ls</div><div class="">db samarbete software staff students test users</div><div class=""># /bin/ls -l</div><div class="">ls: can't read ACL on ./students: Not owner</div><div class="">ls: can't read ACL on ./samarbete: Not owner</div><div class="">ls: can't read ACL on ./users: Not owner</div><div class="">ls: can't read ACL on ./test: Not owner</div><div class="">total 3</div><div class="">drwxr-xr-x 3 root nobody 3 Jul 19 12:38 db</div><div class="">drwxr-xr-x 3 root nobody 3 Aug 19 13:01 samarbete</div><div class="">drwxr-xr-x 3 root nobody 3 Aug 18 13:40 software</div><div class="">drwxr-xr-x 2 root nobody 2 Jun 1 19:59 staff</div><div class="">drwxr-xr-x 88 root nobody 88 Aug 3 13:00 students</div><div class="">drwxr-xr-x 3 root nobody 4 Aug 19 13:15 test</div><div class="">drwxr-xr-x 4 root nobody 4 Jul 16 12:54 users</div><div class=""># cd students</div><div class="">students: Not owner.</div><div class=""># pwd</div><div class="">/mnt</div><div class=""># ls -l</div><div class="">.: Not owner</div><div class="">total 3</div></blockquote></div><div class=""><br class=""></div><div class="">After that any access to /mnt just gives “Not owner”. Only way out (that I’ve found so far) is to unmount and remount /mnt again.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">A “snoop port nfsd” gives this trace:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""># <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=409A GETATTR 10111a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=409A GETATTR 10011a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=69C4 GETATTR 10111a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=5E5D GETATTR 10111a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=7187 GETATTR 10111a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> NFS C 4 (getattr ) PUTFH FH=69D3 GETATTR 10111a b0a23a</div><div class=""><a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> -> <a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> NFS R 4 (getattr ) NFS4ERR_WRONGSEC PUTFH NFS4ERR_WRONGSEC</div><div class=""><a href="http://filur00.it.liu.se" class="">filur00.it.liu.se</a> -> <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a> TCP D=2049 S=1013 Ack=246526621 Seq=3432209847 Len=0 Win=32806 Options=<nop,nop,tstamp 37065137 4068396616></div></blockquote></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">and syslog contains:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">Aug 19 22:16:11 filur00 nfs: [ID 236337 <a href="http://kern.info" class="">kern.info</a>] NOTICE: [NFS4][Server: <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a>][Mntpt: /mnt]NFS op OP_GETATTR got error NFS4ERR_WRONGSEC causing recovery action NR_WRONGSEC.</div><div class="">Aug 19 22:16:11 filur00 nfs: [ID 581112 <a href="http://kern.info" class="">kern.info</a>] NOTICE: [NFS4][Server: <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a>][Mntpt: /mnt]NFS Starting recovery for mount /mnt (mi 0xffffd064166f5000 mi_recovflags [0x4]) on server <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a>, rnode_pt1 ./students (0xffffd0642f7083e0), rnode_pt2 <null string> (0x0)</div><div class="">Aug 19 22:16:11 filur00 nfs: [ID 711378 <a href="http://kern.info" class="">kern.info</a>] NOTICE: [NFS4][Server: <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a>][Mntpt: /mnt]NFS can't recover from NFS4ERR_WRONGSEC. error 1 for server <a href="http://testy.it.liu.se" class="">testy.it.liu.se</a>: rnode_pt1 ./students (0xffffd0642f7083e0) rnode_pt2 <null string> (0x0)</div></blockquote><br class=""></div><div class="">It seems that Solaris/OmniOS handles that “NFS4ERR_WRONGSEC” error a little bit “harder” than Linux/FreeBSD does which is annoying. And here I’ve been telling people that Solaris/OmniOS is the “Gold” standard when it comes to NFS :-). That it “fails” the whole mount is a bit annoying….</div><div class=""><br class=""></div><div class="">(I suppose I can always work around this by NFS-mounting the sub-shares directly (or always using sec=krb5) but it’s still annoying…)</div><div class=""><br class=""></div><div class="">—</div><div class=""><div class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">[L<span style="font-family: sans-serif; font-size: 13px; background-color: rgb(249, 249, 249);" class="">ı</span>.U] System Administrator ITI-NET IT.LiU.SE +46-13-28 2786</div></div></div> </div> <br class=""></div></body></html>