Maybe… But no doubt there are those of you out there who’ll one-up me with your CORBA car-crashes and DCOM disasters.
Then we had pickling and WSDL and DTD, now we have REST and thrift and Avro and gRPC and protobuf and GraphQL, but it’s all the same thing.
If you’ve ever wrestled with backwards compatibility across APIs, binary formats or schema versions, you have an advantage.
You need to wear the same “thinking hat” for trunk-based-development.
Adding new fields to a class/record/‘data transfer object’ [DTO] is backwards compatible.
Removing them is not, unless we can be 100% certain they’re unused.
Modifying a field is a ‘remove then add’ in one operation — which means the ‘remove’ rules apply.
One key insight towards making small commits each individually production safe is to realise there’s no difference between worrying about compatibility with something you persisted to disk earlier, or an older API, and compatibility with code from an ‘incomplete’ feature.
Additions are safe.
In place modifications are unsafe.
Removal is only safe once you can guarantee the subject is unused.
Following on from that, the following basic building blocks are safe:Adding an unused column to a tableAdding an unused field to a DTOAdding an unused parameter to a methodAdding an unused method to a classAdding an unused class to a moduleAdding an unused module to a repositoryNote that ‘unused’ means unused by production code.
Unit and integration tests should give us confidence that the code being added is going to work.
We can now use these building blocks to lay out some common playbooks for adding new features.
The following scenarios will assume a generic stack with a statically typed language, RPC based incoming end-points, a domain logic layer, an ORM and a relational storage layer.
The scenarios chosen all have a front-end/back-end split.
This is because I expect this boundary to be familiar to the majority of readers.
The ‘plays’ themselves are equally applicable across or within any architectural boundary, for example between two back-end services in a micro-services architecture.
Adding FeaturesBottom upScenario: A form that was previously capturing first name, surname and date-of-birth now needs to capture ‘favourite salad green’ as well.
Flow of commits:Assume after each bullet point below you have a code change that is safe to commit and push to master.
Add favouriteSaladGreen column to the User table (nullable)Add favouriteSaladGreen field to the User ‘entity’Add favouriteSaladGreen field to the ‘UserDTO’ transfer object that is being passed between the domain layer and the persistence layer[If applicable] Add favouriteSaladGreen field to the transfer object that is used to pass data between the service-end point and the domain layer.
(You might be reusing the ‘UserDTO’ from above)Start sending the favouriteSaladGreen field from the front-end to the back-end, if provided.
Add the favouriteSaladGreen field to the user form and bind the input to the data being sent to the back-end.
Job done!Pros:For simple changes you can avoid the complexity of adding a feature-flag.
(You might want one anyway though, for A/B testing or controlled roll-out.
)Straight forward development flowCons:For more complex changes, there is a risk that by the time the change has percolated up to the front-end there is a mismatch between what the UI can provide and what the back-end is expecting.
Similarly, if the change is to read and display new information on the front-end, the risk is that a back-end driven design may not meet the front-end requirements.
Top downScenario: Favourite green is too limiting.
We are would like to capture an entire ‘favourite salad recipe’ from our users.
The user interface for this will be moderately complex, requiring multiple form elements and far too much CSS.
It may take a while to develop and perfect.
But once the front-end is complete the additional data will be sent to an existing end-point.
Flow of commits:Assume after each bullet point below you have a code change that is safe to commit and push to master.
Create a feature flag for the ‘capture salad recipe’ user preference.
Add a stubbed form to capture the salad recipe, being sure it will not appear if the feature flag is off.
Complete development of the front-end UI [will likely require multiple commits, but these are safely hidden behind the feature flag].
Add saladRecipe to the data being sent to the existing back-end end-point.
The extra data is unused at this point.
On the back-end, parse the saladRecipe information in the end-point receiver (typically a REST controller, but adjust for your stack) … and throw it away.
Pass the saladRecipe data from the controller to the ‘domain logic’ layer.
Then either add any logic you might need to transform or manipulate it, being sure to unit test it … and throw it away.
Remember that because the UI is feature-flagged, you won’t actually be receiving this data in production.
As above, pass the saladRecipe from your domain layer to your persistence layer, throw it away.
(If you’re using something like JPA, you can mark fields as transient to achieve this)Alter the database to add the new table structure you need.
Stop throwing away the saladRecipe data, it can now be persisted.
Job done!.The feature is working end-to-end.
Pros:Eliminates the risk that the front-end requirements will be misaligned with the back-end implementationStraight forward development flowCons:Requires a feature flag, which is an added complexity, so consider whether the feature is small enough to develop ‘bottom up’ before instituting.
Middle outScenario: Having captured ‘favourite salad recipe’, we want to build a feature that allows connected users to see each others’ recipes and form opinions on whether they have enough superfoods in their diet.
Development Flow: Although the scenario described has many complicating factors, we could naively boil it down to a front-end/back-end split.
Personally, I usually deride such a lazy break-down and always prefer to look for vertical slices of value.
But let’s assume we tried that and we’ve already whittled our task down to something with a substantial amount of work on either side of a service boundary.
We could implement these tasks using the ‘top down’ approach, which would have the advantage of making sure when we get around to building our API the consumer’s requirements are all accounted for.
But what if we can’t do that?.What if want to parallelise the work?In the past I’ve seen these tasks tackled like “building a bridge.
” The front-end work proceeds ‘top down.
’ It starts by creating components or UI layouts with mocked data, which then drives the API design.
Meanwhile the back-end work starts and, lacking a defined API, begins the work ‘bottom up’.
Back-end work starts with anticipated data-later changes and proceeds upwards towards exposing what’s needed in an API.
Then you try and meet in the middle.
It never works.
With sufficient planning about what that API ought to look like you can minimise the damage, but speaking personally here — I’ve never seen this pulled off without a hitch.
At this point you’re probably thinking, “Swagger!.GraphQL!.Planning?.Just talk to each other…” And absolutely!.This is how we’ll play the game.
But let’s take it one step further.
Don’t just talk about and define the API.
Build it.
First.
This is what I mean by ‘middle-out’.
The front-end can safely build up (over the course of multiple commits) an API client if nothing is calling that client.
The back-end can safely implement (over multiple commits), and API end-point knowing that no production code will be invoking it.
The front-end then builds from the API up.
It is at this point that we would introduce a feature-flag.
The back-end builds from the API down.
It is likely that both sides may find they need to make small tweaks to the API as they proceed.
If you are using a RPC protocol with a schema such as GraphQL or protobuf, you may find that sufficient to keep you aligned.
Otherwise, consumer-driven contract tests such as Pact or Spring Cloud Contracts are great tools to put in place here.
They go a step beyond simply agreeing on the API and will check at build time that both sides are obeying the contract.
Modifying Existing FeaturesModifying existing features can be a little harder than simply adding new features.
I’ve heard the process likened to rebuilding the engine on a jet plane while it’s still in the air.
How we approach modifying an existing feature depends on whether the behaviour needs to change, or not.
If/ElseChanging the behaviour of a feature is similar in nature to modifying a field in a serialisable object or an API.
So far as version compatibility is concerned, what are really doing is a ‘remove’ then an ‘add’ in one atomic step.
For a tiny change, in a single commit — that’s fine.
For a prolonged change, across several commits, we need to break this down into three steps.
Add the new behaviourUse the new behaviourRemove the old behaviourFor implementing the first step, pick any of the appropriate methods from the playbooks in the previous section.
For the second step, you probably want to put a feature flag in place.
This allows you to roll-out the changes gradually.
There may be multiple consumers of our new behaviour and we can’t switch the feature on until they’ve all been cut over.
Thus the name of this playbook:if (flagEnabled) newBehaviour()else oldBehaviour()Depending on the complexity of what’s being lumped into the overloaded term ‘behaviour,’ and the facilities provided by your language, this could become a switch statement or a strategy pattern.
Either way — we’re branching in the code, not branching in the code-base.
Up to a point, that is simpler and easier to follow… Up to a point.
So step 3, remove the old behaviour, really is important.
After you implement ‘step 2,’ you can perform A/B testing and your feature can be put in front of customers.
If pressure to deliver is high it can be very tempting to call it a day at this point.
This, you must not do.
The feature is not ‘done’ until the flag is removed.
Measure usage, declare success, remove the flag.
Or declare failure and remove the feature.
Depending on how the team you’re in works — leave the ticket open, leave the card on the board, or just add a new card/ticket to remove the branch at the bottom of your queue when you commence work.
However you do it, make sure it’s done.
Replacing Existing FeaturesThe other main reason you may want to touch existing features is not to modify behaviour, but to improve non-functional requirements such as performance, or replace an integrated technology with another choice, e.
g.
swapping out Clickhouse for Druid.
Branch by Abstraction‘Branch by abstraction’ is not a new concept.
The phrase itself dates to 2007 and the technique itself is much older.
How much older is hard to say, but we needed these things in the days before Git and cheap branching.
How much of effective TBD is really just rediscovering a lost art, I wonder?Many other software engineering luminaries have discussed the branch by abstraction technique, for example Martin Fowler, Jez Humble and Paul Hammant.
Some of these write-ups are harder to digest than others.
I recommend Martin Fowler’s as an easy read.
The technique boils down to building an Adapter or other suitable abstraction layer in front of the thing you want to replace.
Point All The Things at your new abstraction layer.
Put a feature toggle inside the abstraction layer.
When the switch is flipped, the new behaviour kicks in.
Et voila!.You have changed the engine on your jet plane, while it was in the air.
Caveat: Not every change can be branched by abstraction.
Sometimes the thing you want to replace is too tightly entwined and too deeply coupled.
An example of this from past experience was an upgrade from Spring 3 to Spring 4 on a complex, monolithic code-base.
The two frameworks could not co-exist in the same project at the same time.
A branch was used.
That’s okay.
You can’t win them all.
ScientistScientist is a Ruby library for ‘carefully refactoring critical paths’ — to quote their own GitHub page.
It has ports in many other languages… and Perl.
In contrast to ‘branch by abstraction,’ the ‘scientist’ playbook involves running both the old and new code-paths simultaneously and reporting on the results.
The library was born out of GitHub’s refactoring of the “merge” feature, and any attempt of my own to summarize further would be an injustice.
I suggest rather to read of the original article (linked at the end of this section) to get a flavour for what the library is and how it was used.
All I will add to their write-up is to draw attention back to the first half of my own post.
Adding incomplete functionality that is not being used is harmless and does not require a feature branch.
The new code path can be built piece-meal on master.
When you think it’s ready, Scientist will build the trust that it is.
Move Fast and Fix Things – The GitHub BlogAnyone who has worked on a large enough codebase knows that technical debt is an inescapable reality: The more rapidly…github.
blogA Word on TestingDespite play-books, despite patterns, despite feature flags and branching by abstraction — some changes are inherently more risky than others.
Does your feature flag really truly work?.Is everything that is supposed to be behind it actually behind it.
Were your front-end style changes sufficiently contained?We need confidence that the changes we’re about to push to master will not break anything.
Part of this is knowing how to structure the changes and the code to reduce risk — which was the main focus of this article.
But only part.
The rest of your confidence should probably come from testing.
Ideally there are an appropriate amount of unit, integration and maybe even end-to-end tests that execute on every commit.
Hopefully you ran most of them before pushing!.Just having a passing build should give us a reasonable level of confidence.
For particularly risky changes it is nice to have a way to perform manual or exploratory testing in a production-like environment as early as possible.
In one work-place we had testing environments that were reset to track master every week or so — but we could do ad-hoc deploys of local code before we even pushed.
For the riskiest of changes, where we were concerned that multiple feature-flags might interact badly or wanted to be 110% sure a newly introduced block of code was flagged off , we would build locally and test before pushing.
In another workplace we had every push to master trigger a deployment to a staging environment, followed by a gamut of ‘smoke tests’ to ensure the system was basically up.
For changes we were reasonably confident about, we could push to master and then test on ‘staging’.
We also had all our services dockerised and our CI pipeline set up to tag docker images with the branch name.
We could deploy arbitrary tags of our docker services to personal testing environments.
Although feature-branches were discouraged, we did retain the option of using a short-lived branch to test a potentially dangerous change in our own environment before pushing to master.
Was testing branched-based images a good idea?.It certainly gave us the option to provide ourselves an extra level of comfort and confidence.
Perhaps it would have been better to invest in automated tests, lean on ‘staging’ as a quality gate and try to install a culture of ‘push but revert if staging breaks’?I’m not sure… But I can confidently say that a suitable testing strategy needs to be part of any plan to adopt or promote trunk-based-development.
To ConcludeAdditions are safe, removals are not.
Big tasks can be broken down into little tasks.
Little tasks can be split into a sequence of additions, then removals.
Feature Flags can be judiciously inserted to force logic to flow down the ‘old’ path while the additions are being, err, added.
It all sounds rather complicated.
This was a long article, after all.
Longer, if you include the bits where I took a short circuit and just linked to someone else’s post.
If you’re still asking “Why not just branch?” that’s a legitimate question.
Particularly if you’re already working ‘just fine’ with a branching model.
Let’s take this back to the three points raised right at the start.
Trunk Based Development:Keeps your code continuously integrated, reducing risk and improving the ‘code, test, deploy’ feedback loop.
Improves your ability to reason about backwards and forwards compatibility, resulting in a safer — more robust codebase.
Makes you think about introducing feature flags, which decouple deploy from release and liberates your company from planning everything around deploy dates.
It takes practice, commitment and absolutely changes the way you approach solving engineering problems and writing code.
For the better.
.. More details