[OmniOS-discuss] ms.omniti.com Publisher tuntap manifest missing

Fri Jan 6 17:04:38 UTC 2017

Hi Jens!

> at first happy new year!

And a very good 2017 to you!

> Thanks Dale, but it seems that tuntap is not removed from the ms.omniti.com publisher. When I browse the webpage its still there, and a pkgrecv is still complaining about it. For testing I have also setup a new local repo for this:
> zfs create rpool/pkgrepo/omnios/ms.omniti.com2
> pkgrepo create /pkgrepo/omnios/ms.omniti.com2
> pkgrepo set -s /pkgrepo/omnios/ms.omniti.com2 publisher/prefix=ms.omniti.com
> pkgrecv -v -s http://pkg.omniti.com/omniti-ms/ -d /pkgrepo/omnios/ms.omniti.com2/ -m all-versions --clone -p ms.omniti.com
> Processing packages for publisher ms.omniti.com ...
> Retrieving and evaluating 2030 package(s)...
> Reading Manifests ( 881/2030) |pkgrecv: http protocol error: code: 404 reason: Not Found
> URL: 'http://pkg.omniti.com/omniti-ms/ms.omniti.com/manifest/0/omniti%2Fnetwork%2Ftuntap@1.3.3%2C5.11-0.151014%3A20161209T164618Z' (happened 4 times)

Yes, the ms.omniti.com repository has been a bit broken for quite some
time (for another example, see my mail to Dale from last Monday).

The only way to get a copy of that repository is a workaround:  Use the
 --raw flag for pkgrecv.  Then you will get the "raw" data from the 
repository in the location you specify with "-d".  You will also need
to explicitly specify the packages you want, leaving out all broken ones.

>From there, you can easily reconstruct the repository by recreating
the directory structure and moving the contents there.  For example:

  # cat MISSING
  pkg://ms.omniti.com/omniti/developer/versioning/subversion@1.9.4-0.151014:20161021T174720Z
  pkg://ms.omniti.com/omniti/developer/versioning/subversion@1.9.4-0.151014:20161102T145637Z

  # /usr/bin/pkgrecv -v --raw -s http://pkg.omniti.com/omniti-ms -d . `cat MISSING`
  Processing packages for publisher omniti-ms ...
  Retrieving catalog 1/1 omniti-ms 155.16 kB
  Unable to retrieve package data for publisher 'omniti-ms' from one
  of the following origin(s):

  http://pkg.omniti.com/omniti-ms/

  The catalog retrieved from one of the origin(s) listed above only
  contains package data for: ms.omniti.com.

  This is either a result of invalid origin information being provided
  for publisher 'omniti-ms', or because the wrong publisher name was
  provided when this publisher was added.

  Retrieving and evaluating 2 package(s)...

  Retrieving packages ...
          Packages to add:        2
        Files to retrieve:      414
  Estimated transfer size: 14.16 MB

  Packages to transfer:
  omniti/developer/versioning/subversion at 1.9.4,5.11-0.151014:20161021T174720Z
  omniti/developer/versioning/subversion at 1.9.4,5.11-0.151014:20161102T145637Z

  PROCESS                                         ITEMS    GET (MB)   SEND (MB)
  Completed                                         2/2   14.2/14.2     0.0/0.0

  Processing packages for publisher perl.omniti.com ...
  Processing packages for publisher ms.omniti.com ...
  Retrieving and evaluating 2 package(s)...

  Retrieving packages ...
          Packages to add:        2
        Files to retrieve:      414
  Estimated transfer size: 14.16 MB

  Packages to transfer:
  omniti/developer/versioning/subversion at 1.9.4,5.11-0.151014:20161021T174720Z
  omniti/developer/versioning/subversion at 1.9.4,5.11-0.151014:20161102T145637Z

  PROCESS                                         ITEMS    GET (MB)   SEND (MB)
  Completed                                         2/2   14.2/14.2     0.0/0.0

  Processing packages for publisher localhost ...
  Processing packages for publisher omnios ...

(You can safely ignore the warning about the broken publisher.)

Then you will get a directory structure like this:

  # ls -golFR
  drwxr-xr-x   4     311 Jan  6 17:46 omniti%2Fdeveloper%2Fversioning%2Fsubversion/

  ./omniti%2Fdeveloper%2Fversioning%2Fsubversion:
  total 80
  drwxr-xr-x   2   16547 Jan  6 17:46 1.9.4%2C5.11-0.151014%3A20161021T174720Z/
  drwxr-xr-x   2   16547 Jan  6 17:47 1.9.4%2C5.11-0.151014%3A20161102T145637Z/

  ./omniti%2Fdeveloper%2Fversioning%2Fsubversion/1.9.4%2C5.11-0.151014%3A20161021T174720Z:
  total 48360
  -rw-r--r--   1  813240 Jan  6 17:46 00370e03bd4b799dc893dce90752df6cfaf972e8
  -rw-r--r--   1    1714 Jan  6 17:46 013ab51d3c6c3356066642b7ed7388d8fc23fb12
  -rw-r--r--   1    3564 Jan  6 17:46 0539ba5fd5e1102801035f78ba0644e31423370b
  -rw-r--r--   1   61984 Jan  6 17:46 0927d0f5be0e344ed0ff22a8b08d1f003aca21b8
  [...]
  -rw-r--r--   1    1131 Jan  6 17:46 fc9d34774c2a4a21ace7eb1c1da3fb984090eb46
  -rw-r--r--   1   52457 Jan  6 17:46 manifest
  -rw-r--r--   1     927 Jan  6 17:46 manifest.depend
  -rw-r--r--   1    1085 Jan  6 17:46 manifest.dir
  -rw-r--r--   1     540 Jan  6 17:46 manifest.dircache
  -rw-r--r--   1   45036 Jan  6 17:46 manifest.file
  -rw-r--r--   1     142 Jan  6 17:46 manifest.license
  -rw-r--r--   1    4232 Jan  6 17:46 manifest.link
  -rw-r--r--   1       0 Jan  6 17:46 manifest.mediatorcache
  -rw-r--r--   1     477 Jan  6 17:46 manifest.set
  -rw-r--r--   1     677 Jan  6 17:46 manifest.signature

The files you want are the ones with the "hash" filenames and the
manifest file.  You need to move these into the "real" local copy of
your repository.  The hash files go into the "file" subdirectory but
must be compressed by gzip first.   For example:

  # gzip -c 00370e03bd4b799dc893dce90752df6cfaf972e8 > /my_repo/publisher/ms.omniti.com/publisher/ms.omniti.com/file/00/00370e03bd4b799dc893dce90752df6cfaf972e8

The manifest file goes into the "pkg" subdirectory with a special naming
structure.  In this case:

  # mv manifest /my_repo/publisher/ms.omniti.com/pkg/omniti%2Fdeveloper%2Fversioning%2Fsubversion/1.9.4%2C5.11-0.151014%3A20161021T174720Z

You may need to create the "omniti%2Fdeveloper%2Fversioning%2Fsubversion"
directory first.

Finally, you need to do a "pkgrepo rebuild" on the entire repository.

Yes, it is absolutely tedious, but it works.  One of these days I will
write a Perl script to do this.

The real fix would be on the repository side, but it seems that will 
take a while yet. :-)

Viele Grüße -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt               Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANY            Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 46
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"