[OmniOS-discuss] omnios-build: the build system, scripts and merging between branches

[Lauri Tirkkonen]

Please read this and comment if you maintain a fork of omnios-build.
Thanks.

Status quo
----------

The omnios-build repository currently versions more than one thing:
 - the build scripts which describe how to build certain software (under
   build/),
 - the build system (lib/ (mostly), build/buildctl and template/)
 - and some site-specific data (configuration).

The 'template' branch holds nothing site-specific and users are expected
to fork that one, adding their own stuff on top and pulling changes from
upstream's 'template'. This works in one direction, but in practice the
workflow is not "make your changes on top template"; it needs to work in
both directions. When the build scripts and configuration are in the
same repository, you cannot merge back into template - you must instead
be careful with cherry-picking. This isn't very maintainable: recently I
made a pull request that just ported build system changes from
omniti-labs:master to omniti-labs:template and that was 176 commits in
size (ie. after pruning all commits which only touched build scripts).

I maintain two forks of omnios-build (one for a publicly available repo,
and one for a private one) and syncing changes between them and upstream
is a royal pain. I'd wager OmniTI isn't particularly fond of porting
build system changes between different release version branches and
omniti-ms and friends.

I argue that making build system changes flow from one fork/branch to
another needs to be made easier. The way to do that is to split the
build system from the build scripts into another repository, so that
merges can be made freely. This obviously requires some rather large
changes, which I will propose below.

Proposal: submodules?
---------------------

At first, I was thinking "git submodules are the solution to this".  If
you're not familiar, submodules are a way to include a reference to
another repository at a certain commit. This is the sort of tree I was
thinking about:

    omnios-build-my-organization (your site-specific build repo)
    |-- lib (submodule: the build system, or: site-independent code)
    |-- build
    `-- (config.sh, site.sh and other site-specific data)

This would allow you to work on lib/ separately from your build scripts,
and make it clear which version of the build system they need to build.
However, git submodules are a bit unwieldy and not very intuitive (among
other things you need to manually use submodule commands to update the
submodule tree instead of git doing it for you; this is certain to get
annyoing when switching branches).

Better proposal
---------------

So, let's make this simpler by turning it around and ditching the
submodule idea. Let's just have omnios-build contain site-independent
data (ie. the build system), and make build/ another repository, but
let's not make it a submodule. Instead, .gitignore it and make ./new.sh
warn and initialize a new git repository into it if it doesn't exist.
This doesn't require any submodule trickery, but lets you have your
scripts in a separate repo.

Proof of concept is on github niksula/omnios-build split branch and
niksula/omnios-build-scrips repos:

    git clone -b split https://github.com/niksula/omnios-build.git
    cd omnios-build
    git clone https://github.com/niksula/omnios-build-scripts.git build
    cd build/ircii
    ./build.sh

In the PoC I moved buildctl out of build/ and {site,config}.sh inside it
(to the omnios-build-scripts repo). It's not a finished product, but
should demonstrate what I'm saying.

Thoughts?

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet