[OmniOS-discuss] NFSv4 (client) strange behaviour

Sat Aug 19 20:48:32 UTC 2017

I’m seeing an annoying NFSv4 behaviour on OmniOS (also on Solaris 10 so this is probably an old bug).

Setup:

Server: 

FreeBSD 11.0 joined to an AD with Kerberos and stuff serving NFSv4 _only_ with V4 “root” set to /export which is a ZFS pool/filesystem.
Under /export there are a number of additional filesystems with various “sharenfs” settings:

/etc/exports
	V4:	/export	-sec=krb5:krb5i:krb5p:sys

 # zfs list -o name,sharenfs,mountpoint -r -d 1 DATA
NAME            		SHARENFS              	MOUNTPOINT
DATA              		ro                    			/export
DATA/db         		ro                    			/export/db
DATA/liu         		ro                    			/liu
DATA/samarbete  	no                    		/export/samarbete
DATA/software   	sec=krb5:sys,ro       	/export/software
DATA/staff      		ro                    			/export/staff
DATA/students   	sec=krb5:krb5i:krb5p  	/export/students
DATA/test       		sec=sys,ro            		/export/test
DATA/users      	sec=krb5:krb5i:krb5p  	/export/users

Mounting the root of these filsystems on Linux (CentOS 7), Solaris 10/OmniOS and FreeBSD 11 (using NFSv4, sec=sys) gives slightly different behaviour, 
but Solaris/OmniOS fails in a different way…

Clients:

Linux (CentOS 7):

> # mount -t nfs -o vers=4,sec=sys,ro testy:/ /mnt
> # cd /mnt
> # ls
> db  samarbete  software  staff  students  test  users
> # ls -l
> total 7
> drwxr-xr-x  3 root wheel  3 Jul 19 12:38 db
> drwxr-xr-x  3 root wheel  3 Aug 19 13:01 samarbete
> drwxr-xr-x  3 root wheel  3 Aug 18 13:40 software
> drwxr-xr-x  2 root wheel  2 Jun  1 19:59 staff
> drwxr-xr-x 88 root wheel 88 Aug  3 13:00 students
> drwxr-xr-x  3 root wheel  4 Aug 19 13:15 test
> drwxr-xr-x  4 root wheel  4 Jul 16 12:54 users
> [root at filifjonkan mnt]# cd students
> bash: cd: students: Operation not permitted
> <wait some 30s or so>
> # ls -l
> ls: cannot access students: Operation not permitted
> ls: cannot access users: Operation not permitted
> total 3
> drwxr-xr-x 3 root wheel 3 Jul 19 12:38 db
> drwxr-xr-x 3 root wheel 3 Aug 19 13:01 samarbete
> drwxr-xr-x 3 root wheel 3 Aug 18 13:40 software
> drwxr-xr-x 2 root wheel 2 Jun  1 19:59 staff
> d????????? ? ?    ?     ?            ? students
> drwxr-xr-x 3 root wheel 4 Aug 19 13:15 test
> d????????? ? ?    ?     ?            ? users

Ok, not perfect. But doable.

FreeBSD 11.0:

> # mount -t nfs -o vers=4,sec=sys,ro testy:/ /mnt
> # cd /mnt
> # ls
> db		samarbete	software	staff		students	test		users
> # /bin/ls -l
> nfsv4 err=10016
> nfsv4 err=10016
> ls: students: Input/output error
> ls: users: Input/output error
> total 3
> drwxr-xr-x  3 root  wheel  3 Jul 19 12:38 db
> drwxr-xr-x  3 root  wheel  3 Aug 19 13:01 samarbete
> drwxr-xr-x  3 root  wheel  3 Aug 18 13:40 software
> drwxr-xr-x  2 root  wheel  2 Jun  1 19:59 staff
> drwxr-xr-x  3 root  wheel  4 Aug 19 13:15 test

OmniOS (basically the same behaviour with Solaris 10):

> # mount -F nfs -o vers=4,sec=sys,ro testy:/ /mnt
> # cd /mnt
> # /bin/ls
> db         samarbete  software   staff      students   test       users
> # /bin/ls -l
> ls: can't read ACL on ./students: Not owner
> ls: can't read ACL on ./samarbete: Not owner
> ls: can't read ACL on ./users: Not owner
> ls: can't read ACL on ./test: Not owner
> total 3
> drwxr-xr-x   3 root     nobody         3 Jul 19 12:38 db
> drwxr-xr-x   3 root     nobody         3 Aug 19 13:01 samarbete
> drwxr-xr-x   3 root     nobody         3 Aug 18 13:40 software
> drwxr-xr-x   2 root     nobody         2 Jun  1 19:59 staff
> drwxr-xr-x  88 root     nobody        88 Aug  3 13:00 students
> drwxr-xr-x   3 root     nobody         4 Aug 19 13:15 test
> drwxr-xr-x   4 root     nobody         4 Jul 16 12:54 users
> # cd students
> students: Not owner.
> # pwd
> /mnt
> # ls -l
> .: Not owner
> total 3

After that any access to /mnt just gives “Not owner”. Only way out (that I’ve found so far) is to unmount and remount /mnt again.

A “snoop port nfsd” gives this trace:

> # filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=409A GETATTR 10111a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK
> filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=409A GETATTR 10011a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK
> filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=69C4 GETATTR 10111a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK
> filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=5E5D GETATTR 10111a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK
> filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=7187 GETATTR 10111a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4_OK PUTFH NFS4_OK GETATTR NFS4_OK
> filur00.it.liu.se -> testy.it.liu.se NFS C 4 (getattr     ) PUTFH FH=69D3 GETATTR 10111a b0a23a
> testy.it.liu.se -> filur00.it.liu.se NFS R 4 (getattr     ) NFS4ERR_WRONGSEC PUTFH NFS4ERR_WRONGSEC
> filur00.it.liu.se -> testy.it.liu.se TCP D=2049 S=1013 Ack=246526621 Seq=3432209847 Len=0 Win=32806 Options=<nop,nop,tstamp 37065137 4068396616>

and syslog contains:

> Aug 19 22:16:11 filur00 nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: testy.it.liu.se][Mntpt: /mnt]NFS op OP_GETATTR got error NFS4ERR_WRONGSEC causing recovery action NR_WRONGSEC.
> Aug 19 22:16:11 filur00 nfs: [ID 581112 kern.info] NOTICE: [NFS4][Server: testy.it.liu.se][Mntpt: /mnt]NFS Starting recovery for mount /mnt (mi 0xffffd064166f5000 mi_recovflags [0x4]) on server testy.it.liu.se, rnode_pt1 ./students (0xffffd0642f7083e0), rnode_pt2 <null string> (0x0)
> Aug 19 22:16:11 filur00 nfs: [ID 711378 kern.info] NOTICE: [NFS4][Server: testy.it.liu.se][Mntpt: /mnt]NFS can't recover from NFS4ERR_WRONGSEC.  error 1 for server testy.it.liu.se: rnode_pt1 ./students (0xffffd0642f7083e0) rnode_pt2 <null string> (0x0)

It seems that Solaris/OmniOS handles that “NFS4ERR_WRONGSEC” error a little bit “harder” than Linux/FreeBSD does which is annoying. And here I’ve been telling people that Solaris/OmniOS is the “Gold” standard when it comes to NFS :-). That it “fails” the whole mount is a bit annoying….

(I suppose I can always work around this by NFS-mounting the sub-shares directly (or always using sec=krb5) but it’s still annoying…)

—
[Lı.U] System Administrator ITI-NET IT.LiU.SE +46-13-28 2786

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20170819/642a291c/attachment.html>