view · edit · print · history

Problems using branches in monotone

This is a step by step example of the problems which arise from the way monotone serves branches. Most of these problems come down to the fact that monotone does not serve (accept or receive) revisions unless they carry a certificate for a branch within the collection the server serves. Unfortunately it is very easy to create a branch within the collection which contains revisions "from" branches outside the collection.

A simple example

In this example bar.org is an organisation which produces software licensed under the GPL. The software is therefore freely available and may be modified and incorporated into other products. foo.com is a company which does just this - it takes the BAR development framework and modifies it in ways specific to the products foo.com produces.

Both bar.org and foo.com use monotone to maintain and to publish their source. Remember that the source is GPL, so they have to publish it, but they don't have the time to invest a whole lot of effort in this. monotone seems ideal. The (published) bar.org branch is bar.org.dev.

Initially Mr Foo of foo.com doesn't much expect to fold any changes back into the bar.org and he knows that the changes he's making are much more than bug fixes. Mr Foo does, however, need to track bar.org.dev - it's a work in progress. What is more he can't publish his changes as the branch bar.org.dev - that would lead to confusion and disaster. He could create bar.org.dev.bar, but partying on the bar.org namespace seems impolite at best. Consequently he creates the branch com.foo.foobar.

The problems start as soon as he creates this branch...

Initial development in bar.org

This story is told as a shell script - these commands can be copied into a bash shell and run. This will recreate the databases involved and demonstrate the problems.

Mr Bar of bar.org does the initial development of BAR:

 monotone --db=org.bar.db db init
 monotone --db=org.bar.db genkey mr.bar@bar.org
 monotone --db=org.bar.db --branch=org.bar.dev setup org.bar.dev
 cd org.bar.dev
 echo 'The BAR development framework' >bar.txt
 monotone add bar.txt
 monotone commit --message="BAR v1"
 cd ..

Mr Bar has completed his development, at this point he edits the bar.org monotonerc to allow anonymous read access and starts up a monotone server to release BAR to the world:

 monotone --db=org.bar.db serve localhost:5253 org.bar.dev

Mr Foo's branch development

Mr Foo learns about BAR and decides to use it for his own companies products. He finds it does a lot but a lot more needs to be done.

He decides to make what looks like it will be a new version of BAR with many changes, FoobAR?:

 monotone --db=com.foo.db db init
 monotone --db=com.foo.db genkey mr.foo@foo.com
 monotone --db=com.foo.db pull localhost:5253 org.bar.dev
 monotone --db=com.foo.db --branch=org.bar.dev co com.foo.foobar
 cd com.foo.foobar
 echo 'The FoobAR? project' >foo.txt
 monotone add foo.txt
 monotone --branch=com.foo.foobar commit --message="FoobAR? v1"
 cd ..

Mr Foo has completed his development and tested the result. He releases the product and also puts up a monotone server to satisfy the GPL requirements:

 monotone --db=com.foo.db serve localhost:5254 com.foo.foobar

An unpleasant user experience

joe@user.net finds out about FoobAR? and, after discovering the product is released under the GPL, goes to seek out the source. Mr Foo's web site directs him to the monotone server. Joe attempts to download the source:

 monotone --db=user.db db init
 monotone --db=user.db genkey joe@user.net
 monotone --db=user.db pull localhost:5254 com.foo.foobar
 monotone --db=user.db list branches

Joe sees something like this:

 $ monotone --db=user.db pull localhost:5254 com.foo.foobar
 monotone: setting default server to localhost:5254
 monotone: setting default collection to com.foo.foobar
 monotone: warning: doing anonymous pull
 monotone: connecting to localhost:5254
 monotone: rebuilding merkle trees for collection com.foo.foobar
 monotone: first time connecting to server localhost:5254
 monotone: I'll assume it's really them, but you might want to
 monotone: double-check their key's fingerprint: 0bb0268479d6446937dc5fb57ea7d0dadd28321f
 monotone: warning: saving public key for mr.foo@foo.com to database
 monotone: [bytes in: 2346] [bytes out: 373] [certs in: 4] [revs in: 1]
 monotone: successful exchange with localhost:5254
 $ monotone --db=user.db list branches

But list branches does not list anything.

Problem 1: remotely served branches don't work

Joe spends several days ;-) trying to work out why the monotone pull outputs nice hopeful messages about getting data yet fails to list any branches. Eventually, after Mr Bar has spent some time trying to work out what is going on, Mr Bar realises that his server isn't serving the base revisions which were in org.bar.dev.

So he tells Joe to get the base from foo.org - Mr Bar does not want to serve up what he knows to be an out-of-date revision of org.foo.dev:

 monotone --db=user.db pull localhost:5253 org.bar.dev
 monotone --db=user.db list branches
 monotone --db=user.db pull localhost:5254 com.foo.foobar
 monotone --db=user.db list branches

This is annoying, but hardly a show stopper. People who wish to see the GPL source of FoobAR? must first download the GPL source of BAR. (And it must be first - even though com.foo.foobar was successfully downloaded before org.bar.dev the list branches after the org.bar.dev pull shows only org.bar.dev.)

Mr Foo continues work

Mr Foo makes a change to FoobAR? and commits it:

 cd com.foo.foobar
 echo 'The new FoobAR? v2' >newfoo.txt
 monotone add newfoo.txt
 monotone commit --message="Untested FoobAR? v2"
 cd ..

Unfortunately at this point Mr Foo realises that v2 is not going to work without some major bugfixes in BAR. He talks to Mr Bar and Mr Bar makes the required changes:

 cd org.bar.dev
 echo 'The bugfixed BAR v2' >newbar.txt
 monotone add newbar.txt
 monotone commit --message="BAR v2"
 cd ..

Now Mr Foo can make v2 of FoobAR? work:

 cd com.foo.foobar
 monotone pull
 monotone propagate org.bar.dev com.foo.foobar
 monotone update
 echo 'The working FoobAR? v2' >newerfoo.txt
 monotone add newerfoo.txt
 monotone commit --message="Tested and working FoobAR? v2"
 cd ..

Joe attempts to update to v2

Now Joe attempts to pull the new v2 of FoobAR? from the foo.com server:

 monotone --db=user.db pull
 monotone --db=user.db --branch=com.foo.foobar co joe

This appears to have worked:

 $ monotone --db=user.db pull
 monotone: warning: doing anonymous pull
 monotone: connecting to localhost:5254
 monotone: rebuilding merkle trees for collection com.foo.foobar
 monotone: including branch com.foo.foobar
 monotone: [certs: 4] [keys: 1]
 monotone: [bytes in: 6196] [bytes out: 934] [certs in: 12] [revs in: 4]
 monotone: successful exchange with localhost:5254
 $ monotone --db=user.db --branch=com.foo.foobar co joe
 $ ls joe
 MT  bar.txt  foo.txt  newfoo.txt

Notice, however, that the changes do not include anything after Mr Foo pulled the changes from the latest BAR into his branch.

Problem 2: a pull may fail partially without error or warning

It's very difficult for Joe to know that the apparently updated tree is missing important changes. Eventually Joe might work it out but if, on the way there, he happens to pull the latest changes from bar.org the problem may well mysteriously disappear.

The problem here is that monotone pull is failing to indicate the failure. Despite this it leaves Joe's database with a error state:

 $ monotone --db=user.db db check
 monotone: [files: 5]
 monotone: [manifests: 5]
 monotone: [revisions: 3]
 monotone: [ancestry: 3]
 monotone: [revisions: 1]
 monotone: verifying new revisions (this may take a while)
 monotone: [revisions: 3]
 monotone: [keys: 3]
 monotone: [certs: 12]
 monotone: manifest 9b62be963a520d3b1257d1bc3ff3b493f66049b1 unreferenced
 monotone: manifest d75e5192bc1195c0a6bea714d926b27ec87579ed unreferenced
 monotone: warning: 2 unreferenced manifests
 monotone: check complete: 5 files; 5 manifests; 3 revisions; 3 keys; 12 certs
 monotone: total problems detected: 2 (0 serious)
 monotone: minor problems detected

This actually makes things worse - it looks like something has gone wrong and caused database damage. After both pulls (org.bar.dev then com.foo.foobar) have completed the problem goes away:

 monotone --db=user.db pull localhost:5253 org.bar.dev
 monotone --db=user.db pull localhost:5254 com.foo.foobar

Mr Foo and Mr Bar decide to cooperate

At this point Mr Foo and Mr Bar decide to fold the com.foo.foobar branch back in to the BAR main line of development. Mr Bar pulls Mr Foo's code and manages the merge:

 monotone --db=org.bar.db pull localhost:5254 com.foo.foobar
 monotone --db=org.bar.db propagate com.foo.foobar org.bar.dev
 cd org.bar.dev
 monotone update
 cd ..

That was easy - no merge was necessary.

Mr Foo syncs up his database too:

 monotone --db=com.foo.db pull localhost:5253 org.bar.dev
 cd com.foo.foobar
 monotone update
 cd ..

Joe updates too - no problems, in fact at this point all three working trees are identical (apart from the MT/options files).

Both Mr Foo and Mr Bar are very happy at this point with monotone. Unfortunately this is about to change, very drastically.

The new user

Jane is a regular BAR user, uninterested in the foo.com company:

 monotone --db=jane.db db init
 monotone --db=jane.db genkey jane@user.net
 monotone --db=jane.db pull localhost:5253 org.bar.dev
 monotone --db=jane.db list branches
 monotone --db=jane.db --branch=org.bar.dev co jane.bar

This looks fine, but the working copy only contains the changes up to the point where the foo.com changes were merged back in. Jane can "fix" this by pulling the foobar changes from foo.com, but in general each pull only moves up the revision tree to the extent permitted by revisions in that server.

Problem 3: mixing branches from different servers makes pulls increasingly difficult.

In the limit every single revision might require a separate pull from a separate server. NOTE: this may be wrong, it may just require one pull from every involved server. Worse, there is no way to know which server contains the next revision, because that information is not revealed by the logs even with --debug!

Mr Bar has suddenly found himself with an unintended support headache. If he had known this would happen we would never have used monotone to merge Mr Foo's changes back in - he would have just copied and pasted the changed files. In desparation Mr Bar tries:

 monotone --db=org.bar.db serve localhost ""

This does work - it simply means the server serves all the branches in the database.

This is probably what will happen with the current system - every monotone server will serve all the branches it contatins. Truely separate developments will be in separate databases. This seems highly undesireable.

A suggested solution

There are a number of ways of solving these problems. The simplest, however, would seem to be to complete the closure on a collection, so that the heads of all the branches in the collection plus all their ancestors are served regardless of the branch certificate on an individual revision.

An obvious extension is to permit particular revisions to be served or to allow a branch to be served without it's derived branches - for example com.foo.foobar but not com.foo.foobar.unstable.

This is, perhaps, a suggestion to change the underlying model - serve one or more revisions - which means the given revisions plus all revisions necessary to build them. This would allow a server to serve disjoint sets of branches - for example com.foo.foobar and com.foo.footools without serving everything named com.foo.*

The real world example

The actual databases which led to discovery of these problems are somewhat larger - the monotone database file is about 30MByte (and apparently growing quite rapidly). The original branch was org.openembedded.dev in the OpenEmbedded monotone database. The separate development branch was org.nslu2-linux.openembedded. After the base problem was encountered the latter was changed to org.openembedded.nslu2-linux, however the databases still show the original contents.

monotone --db=local-oe.db pull ewi546.ewi.utwente.nl org
monotone --db=local-nslu2.db pull mtn.nslu2-linux.org org

At present both database serves serve org, and both databases contain most of the same revisions.

monotone version information

 jbowler@marsco:~$ monotone --full-version
 monotone 0.19 (base revision: 168adf9537ff136c9b7fe7faad5991f92859390d)
 Running on: Linux 2.6.11-mm3-jbraid #2 SMP Sun Mar 13 10:24:42 PST 2005 i686
 Changes since base revision:

 new_manifest [52a617d908ac6c4bb5b837ce5306f75155dc59ef]

 old_revision [168adf9537ff136c9b7fe7faad5991f92859390d]
 old_manifest [a9ee1d741b855fdcc0d038d64d913cef70da72f5]

 patch "po/monotone.pot"
  from [10e6d7cbad87eaa0dbe35c803cafa371567f024b]
    to [0465f6ac8d09fac5938ac067747b179aca677b67]

   Generated from data cached in the distribution;
   further changes may have been made.

   Generated from data cached in the distribution;
   further changes may have been made.

   Generated from data cached in the distribution;
   further changes may have been made.
view · edit · print · history · Last edited by jbowler.
Originally by jbowler.
Page last modified on June 28, 2005, at 02:28 PM