[Summary view] [Print] [Text view]

   1  @title Recommendations on Branching
   2  @group review
   3  
   4  Project recommendations on how to organize branches.
   5  
   6  This document discusses organizing branches in your remote/origin for feature
   7  development and release management, not the use of local branches in Git or
   8  queues or bookmarks in Mercurial.
   9  
  10  This document is purely advisory. Phabricator works with a variety of branching
  11  strategies, and diverging from the recommendations in this document
  12  will not impact your ability to use it for code review and source management.
  13  
  14  = Overview =
  15  
  16  This document describes a branching strategy used by Facebook and Phabricator to
  17  develop software. It scales well and removes the pain associated with most
  18  branching strategies. This strategy is most applicable to web applications, and
  19  may be less applicable to other types of software. The basics are:
  20  
  21    - Never put feature branches in the remote/origin/trunk.
  22    - Control access to new features with runtime configuration, not branching.
  23  
  24  The next sections describe these points in more detail, explaining why you
  25  should consider abandoning feature branches and how to build runtime access
  26  controls for features.
  27  
  28  = Feature Branches =
  29  
  30  Suppose you are developing a new feature, like a way for users to "poke" each
  31  other. A traditional strategy is to create a branch for this feature in the
  32  remote (say, "poke_branch"), develop the feature on the branch over some period
  33  of time (say, a week or a month), and then merge the entire branch back into
  34  master/default/trunk when the feature is complete.
  35  
  36  This strategy has some drawbacks:
  37  
  38    - You have to merge. Merging can be painful and error prone, especially if the
  39      feature takes a long time to develop. Reducing merge pain means spending
  40      time merging master into the branch regularly. As branches fall further
  41      out of sync, merge pain/risk tends to increase.
  42    - This strategy generally aggregates risk into a single high-risk merge event
  43      at the end of development. It does this both explicitly (all the code lands
  44      at once) and more subtly: since commits on the branch aren't going live any
  45      time soon, it's easy to hold them to a lower bar of quality.
  46    - When you have multiple feature branches, it's impossible to test
  47      interactions between the features until they are merged.
  48    - You generally can't A/B test code in feature branches, can't roll it out to
  49      a small percentage of users, and can't easily turn it on for just employees
  50      since it's in a separate branch.
  51  
  52  Of course, it also has some advantages:
  53  
  54    - If the new feature replaces an older feature, the changes can delete the
  55      older feature outright, or at least transition from the old feature to the
  56      new feature fairly rapidly.
  57    - The chance that this code will impact production before the merge is nearly
  58      zero (it normally requires substantial human error). This is the major
  59      reason to do it at all.
  60  
  61  Instead, consider abandoning all use of feature branching. The advantages are
  62  straightforward:
  63  
  64    - You don't have to do merges.
  65    - Risk is generally spread out more evenly into a large number of very small
  66      risks created as each commit lands.
  67    - You can test interactions between features in development easily.
  68    - You can A/B test and do controlled rollouts easily.
  69  
  70  But it has some tradeoffs:
  71  
  72    - If a new feature replaces an older feature, both have to exist in the same
  73      codebase for a while. But even with feature branching, you generally have to
  74      do almost all this work anyway to avoid situations where you flip a switch
  75      and can't undo it.
  76    - You need an effective way to control access to features so they don't launch
  77      before they're ready. Even with this, there is a small risk a feature may
  78      launch or leak because of a smaller human error than would be required with
  79      feature branching. However, for most applications, this isn't a big deal.
  80  
  81  This second point is a must-have, but entirely tractable. The next section
  82  describes how to build it, so you can stop doing feature branching and never
  83  deal with the pain and risk of merging again.
  84  
  85  = Controlling Access to Features =
  86  
  87  Controlling access to features is straightforward: build some kind of runtime
  88  configuration which defines which features are visible, based on the tier
  89  (e.g., development, testing or production?) code is deployed on, the logged in
  90  user, global configuration, random buckets, A/B test groups, or whatever else.
  91  Your code should end up looking something like this:
  92  
  93    if (is_feature_launched('poke')) {
  94      show_poke();
  95    }
  96  
  97  Behind that is some code which knows about the 'poke' feature and can go lookup
  98  configuration to determine if it should be visible or not. Facebook has a very
  99  sophisticated system for this (called GateKeeper) which also integrates with A/B
 100  tests, allows you to define complicated business rules, etc.
 101  
 102  You don't need this in the beginning. Before GateKeeper, Facebook used a much
 103  simpler system (called Sitevars) to handle this. Here are some resources
 104  describing similar systems:
 105  
 106    - There's a high-level overview of Facebook's system in this 2011 tech talk:
 107      [[http://techcrunch.com/2011/05/30/facebook-source-code/ | Facebook Push
 108        Tech Talk]].
 109    - Flickr described their similar system in a 2009 blog post here:
 110      [[http://code.flickr.com/blog/2009/12/02/flipping-out/ | Flickr Feature
 111        Flags and Feature Flippers]].
 112    - Disqus described their similar system in a 2010 blog post here:
 113      [[http://blog.disqus.com/post/789540337/partial-deployment-with-feature-switches |
 114        Disqus Feature Switches]].
 115    - Forrst describes their similar system in a 2010 blog post here:
 116      [[http://blog.forrst.com/post/782356699/how-we-deploy-new-features-on-forrst |
 117        Forrst Buckets]].
 118    - Martin Fowler discusses these systems in a 2010 blog post here:
 119      [[http://martinfowler.com/bliki/FeatureToggle.html |
 120        Martin Fowler's FeatureToggle]].
 121    - Phabricator just adds config options but defaults them to off. When
 122      developing, we turn them on locally. Once a feature is ready, we default it
 123      on. We have a vastly less context to deal with than most projects, however,
 124      and sometimes get away with simply not linking new features in the UI until
 125      they mature (it's open source anyway so there's no value in hiding things).
 126  
 127  When building this system there are a few things to avoid, mostly related to not
 128  letting the complexity of this system grow too wildly:
 129  
 130    - Facebook initially made it very easy to turn things on to everyone by
 131      accident in GateKeeper. Don't do this. The UI should make it obvious when
 132      you're turning something on or off, and default to off.
 133    - Since GateKeeper is essentially a runtime business rules engine, it was
 134      heavily abused to effectively execute code living in a database. Avoid this
 135      through simpler design or a policy of sanity.
 136    - Facebook allowed GateKeeper rules to depend on other GateKeeper rules
 137      (for example, 'new_profile_tour' is launched if 'new_profile' is launched)
 138      but did not perform cycle detection, and then sat on a bug describing
 139      how to introduce a cycle and bring the site down for a very long time,
 140      until someone introduced a cycle and brought the site down. If you implement
 141      dependencies, implement cycle detection.
 142    - Facebook implemented some very expensive GateKeeper conditions and was
 143      spending 100+ ms per page running complex rulesets to do launch checks for a
 144      number of months in 2009. Keep an eye on how expensive your checks are.
 145  
 146  That said, not all complexity is bad:
 147  
 148    - Allowing features to have states like "3%" instead of just "on" or "off"
 149      allows you to roll out features gradually and watch for trouble (e.g.,
 150      services collapsing from load).
 151    - Building a control panel where you hit "Save" and all production servers
 152      immediately reflect the change allows you to quickly turn things off if
 153      there are problems.
 154    - If you perform A/B testing, integrating A/B tests with feature rollouts
 155      is probably a natural fit.
 156    - Allowing new features to be launched to all employees before they're
 157      launched to the world is a great way to get feedback and keep everyone
 158      in the loop.
 159  
 160  Adopting runtime feature controls increases the risk of features leaking (or
 161  even launching) before they're ready. This is generally a small risk which is
 162  probably reasonable for most projects to accept, although it might be
 163  unacceptable for some projects. There are some ways you can mitigate this
 164  risk:
 165  
 166    - Essentially every launch/leak at Facebook was because someone turned on
 167      a feature by accident when they didn't mean to. The control panel made this
 168      very easy: features defaulted to "on", and if you tried to do something
 169      common like remove yourself from the test group for a feature you could
 170      easily launch it to the whole world. Design the UI defensively so that it
 171      is hard to turn features on to everyone and/or obvious when a feature is
 172      launching and this shouldn't be a problem.
 173    - The rest were through CSS or JS changes that mentioned new features being
 174      shipped to the client as part of static resource packaging or because
 175      the code was just added to existing files. If this is a risk you're
 176      concerned about, consider integration with static resource management.
 177  
 178  In general, you can start with a very simple system and expand it as it makes
 179  sense. Even a simple system can let you move away from feature branches.
 180  
 181  = Next Steps =
 182  
 183  Continue by:
 184  
 185    - reading recommendations on structuring revision control with
 186      @{article:Recommendations on Revision Control}; or
 187    - reading recommendations on structuring changes with
 188      @{article:Writing Reviewable Code}.