by Melinda Green
This is a branching strategy meant to increase branch quality and predictability. It represents a small change to most existing strategies and can be applied on a case-by-case basis but should be especially helpful when applied to long-lived branches that are intended to merge back to a parent branch more than once (E.G. major feature-based branches, developer's branches, etc). The technique scales to multiple levels and may even increase product quality and predictability of the entire release process overall.
The diagram below shows the before and after change applied to one cycle of a long-lived development branch. The green lines represent direct QA activity and the dots represent check-ins.
In the top diagram, QA and continued development proceed in parallel. The only changes allowed to a QA branch are those that fix discovery problems. Once a QA branch is cut, all such fixes must also be applied to the development branch. Interleaving those fixes with continuing development work makes that process more difficult.
In the bottom diagram, QA work begins on the development branch while the developers halt new development on that branch for a short time. Ideally all the developers on a project would shift into a testing mode themselves and help fix the initial bugs found. This is especially helpful since the process ideally begins with a refresh from trunk which often brings new problems when changes to the trunk suddenly interact with the latest work in the development branch. The main idea here is that the obvious and most easily fixed problems tend to be discovered early in the QA process, and fixing them in only one place is easier and safer than performing parallel fixes. Developers not working on discovery bugs can help test and fix problems in their own work while it is fresh in their minds.
After the first discovery bugs have been found and fixed, an official QA branch is cut and everything proceeds as before. The decision as to exactly when to cut the QA branch is a bit of an art. It's similar to making popcorn where you don't want to wait too long otherwise you burn it, and you don't want to stop too early and leave too many unpopped kernels.
Isn't this similar to using sandbox and integration branches? Yes and no. Yes in that the mechanisms used are the same, but no in the ways that the QA and development teams interact. QA will begin testing directly on a development branch and carry over to the formal QA branch in preparation for merging back to trunk. The same fixes will occur so there is no need to retest anything once the shift happens. The overall QA effort should be roughly the same but QA can count on greater responsiveness from the developers, so it is a win for both groups.
The developers are the ones most affected by the change as this means they must hold off submitting new development work during stabilization phases. Best of all, QA and developers will work more closely together during stabilization phases and the testing and bug fixing work won't come as unexpected or unwanted interruptions. Of course development can still proceed with work saved in patch files or sub branches but the assumption will be that developers will occasionally shift gears to focus on fixing bugs and only need to fix them in a single branch.
One interesting possibility is to apply phased branching to the trunk itself. This may not be the best pattern for the trunk but it is interesting to note that the release candidate process is very similar to a QA/merge process, the difference being that official versions are released into the wild rather than merged into another branch. Because the pattern is similar, it's a natural question to ask whether it might be possible to improve the release process using the same technique.
The diagram above shows the result of applying phased branching to the RC process. Externally, beta testers will see fewer release candidates. The main visible internal change will be that many of the fixes made during lengthy release cycles will be felt more quickly in the development branches. This should cut down on the number of times that developers in multiple groups lose time by rediscovering the same problems inherited from the trunk. I expect it may also allow for even shorter release cycles if desired.
Developers know that the quality of the trunk (or any parent branch) tends to wax and wane and they therefore often want to know when will be a good time to branch, refresh, or merge. A common mantra of source control is to just "refresh early and often". The theory being that if everyone does this then most refreshes will be trivial and overall quality will be improved because fixes will propagate more quickly. The main problems with this strategy in practice are that
Even the smoothest update comes with non-trivial fixed costs (overhead), and
Humans tend to procrastinate when it comes to housekeeping duties.
The idea behind phased branching minimizes the pain of refreshing, and works with human tendencies rather than against them. So let's accept the fact that quality will wax and wane on long-lived branches and see what happens.
The diagram above shows that with a phased branch, it is very clear that the best time to branch, refresh, or merge is right after a stabilization phase when stability is at its highest. It is still perfectly fine to branch or refresh at any time but there should be benefits in being better able to plan these activities and to better predict those same activities by others.
One potential problem could be that the period of highest stability for a particular branch could degrade quickly when a lot of developers work on it. It therefore might be a good idea in those cases to announce the end of a stabilization phase a few days before opening it for new commits in order to give people a chance to branch and refresh from that branch before it begins to get polluted with new submissions (black arrows above). That amounts to a adding a third phase and may be a good option to consider on a branch-by-branch basis.
It is interesting to ask what would happen if this pattern is taken to it's logical extreme and applied to all branches. In that case I would expect the stabilization phases of all branches to begin to synchronize with each other as each project naturally attempts to maximize its own stability and quality. The following diagram gives a rough idea of the branch/refresh/merge patterns I would expect would naturally develop.
The result is roughly analogous to the way that people walking together tend to synchronize their steps. Note that I am not proposing any sort of systematic branch synchronization; this is simply what I expect would happen if the phased branching strategy ends up being used in most development branches.
Here are step-by-step instructions for implementing phasing in your branch. It assumes locking & tagging using SVN, but those steps can be implemented in other source control systems or skipped altogether. It is the other steps that are important:
Announce Stabilization Start - At the point that you would normally cut your QA branch, announce to all contributors of your development branch that it is now locked down for stabilization. It is highly recommended that you make sure in advance that this come as no surprise to any of the devs and that you have arranged for QA testing to begin shortly. Your announcement should include the lock keyword you will use in the following step.
Lock - Lock
your branch: cd top-of-source-tree;
svn propset LOCKED keyword
. (TortiseSVN users: right-click your branch root and select
TortiseSVN->Properties->New...) From this point on, you need
to include keyword in your commit comment. For example, if I
use the keyword ignore-featurettes-lock, then only commits
including that string in their comments will succeed.
Refresh - Synchronize with the Trunk or other parent branch.
Test - Encourage all interested parties to begin testing your development branch. The release team can assist you with automated or one-off builds if needed. The most important testers will be the developers who contributed to the branch. They should test their own changes as well as that of the other contributors, and non developers in the company should use these versions in their daily work.
Fix - Fix the bugs that you, QA, and internal users find. Work closely with QA. At this early point they may or may not mind if problems they find get fixed without filing formal discovery bugs, but you absolutely may not check-in unrelated changes during a stabilization phase. Note: if serious bugs are discovered here, this will be a great time to abort and go back to the drawing board with minimal wasted effort from anyone.
Branch - The real art here is figuring out just the right moment to create the official QA branch that will ultimately merge back to trunk or other parent branch. This is the pop-corn problem. You will want to wait long enough to deal with most of the obvious and easy discovery problems but not so long as to overly burden those who would like to commit new work to the development branch. Optionally refresh from trunk again, especially if the stabilization phase took longer than expected or you know that changes were made to the trunk that interact with the work in your branch. If you do that, be sure to coordinate with the QA testers and be sure they approve. Their previously QA'ed work would not need to be repeated but some double-checking might be a good idea.
Announce Stabilization End - Customers of your branch would be smart to take these opportunities to refresh or create sub-branches, grab stable binaries, etc. Developers on your branch will similarly want to know when it is OK to contribute new work into it. You may want to pause for a couple days before unlocking and say so in your announcement.
Unlock -
Remove lock: cd top-of-source-tree;
svn propdel LOCKED . Your development branch is now open for
normal business. If you chose to pause in the above step, you will
probably want to announce when you have unlocked.
Finish QA - Complete the QA and merge to trunk exactly as before.
This proposal can be applied to any development branch but should be especially helpful with long-lived branches that occasionally merge back to their parent branches. The expected benefits include
Higher overall code quality by reducing instances of untimely branch refreshes and merges resulting in fewer instances of duplicated debugging and related distractions.
Greater project predictability from well-defined phases.
Greater efficiency from increased developer-QA interaction.
Better intra-team effectiveness from synchronizing develop/debug efforts.
Fewer bugs introduced due to code drift, misapplied parallel fixes, and merge problems.