The Next Primitives of Dev Tools

The Next Primitives of Dev ToolsAidan CunniffeBlockedUnblockFollowFollowingApr 8Like any field, the domain of software development has a rich and interesting ontology.

It’s easy to forget that the whole history of modern programming could fit within a human lifespan.

We’re lucky to have software veterans among us that can speak of a world without Commits, Dependencies, or Containers as the primitives of software development.

If you take a look at all the major step-changes in how programming is done, they’re almost all the result of a new primitive being introduced — seemingly out of thin air.

Then, in an instant, everything changes and cultural blindness sets into the way things were before.

This essay explores the most important primitives we discovered over the last 40 years that have become a part of nearly every programmer’s workflow.

We begin with a quick history lesson and then move on to predictions of what primitives are coming next:Primitives from the era of the single programmerThe first developers were soloists so it’s only natural the primitives that defined this era were all about making an individual’s work easier.

the TraceIn the earliest days of programming, code would run, or it would crash.

For the first few decades of programming, you didn’t get nice stacktraces or errors in sentence form.

When things broke it was up to you to look in the memory registers and figure out what went wrong.

This could have worked, except for another major change, programming languages were getting good — really good.

So good in fact that their nice English-like syntax looked nothing like the processor instructions that actually got executed.

That made an already difficult problem nearly impossible.

And all that may well have crumbled under its own weight if not for the trace: the linking of your code’s runtime behavior to the higher level abstractions you used when writing it.

This was and still is one of the most useful inventions in programming and it’s only gotten better thanks to Richard Stalman who showed us what a modern debugger could be in GNU.

Today interactive debuggers project runtime information back onto source code so you can hop between imaginary breakpoints and see the values of your variables at runtime.

All of this has made complex code manageable to us mere mortals.

the TestMost developers think of testing as a means of ensuring software quality.

We only recently began to realize their full potential.

Automated tests used primarily to aid the developer (not just to ensure quality) is an idea only about 20 years old.

But before we open that box, let’s get back to basics.

Programming is all about translating domain and business logic into instructions a machine can execute.

We’re mechanizing some expertise by pressing that knowledge into code.

Tests go the other way; they allow us to map the runtime characteristics of our code back onto concepts in our domain.

When the test for “messages are archived when user is deleted” passes, it’s telling us and all the developers who work with us something important about how the system works.

One of the first things a good developer does when they pick up a new codebase is read the tests and if they inherited a code base without tests they write them before changing anything.

The rapid feedback tests provide allow programmers to operate in sync with their codebase.

The evangelists of test driven development have been saying this for years, but often writing the test cases reveals something important about your domain long before you write any code.

Without automated testing, the total complexity of any piece of software would have a strict upper bound and there’s no way a team of developers could work together on anything of reasonable complexity.

the SignatureOnce we had solid higher level abstractions to program with like the class and function, developers realized that their language level abstractions enforced an implicit contract in their code.

It didn’t take long to realize that you could enforce that contract with compile-time feedback that verified that every routine was called correctly.

But where would they get the information they needed to enforce these contracts?Code takes a wild ride when it is compiled from a high level language to processor instructions.

Along the way, it’s parsed from text into structures representing the abstractions we program with such as classes, methods, calls and, expressions.

These signatures, which could be obtained after running the first few stages of the compiler, became the foundation of the IDEs we use today.

Suddenly our tools understood the structure of our code enough to power autocomplete, jump to definitions, search and even guide us towards fixing issues before they hit the compiler or runtime.

The signature ushered in the era of the IDE, the first commercially viable developer productivity tools.

Primitives from the era of the collaborationThe Trace, Signature and Test primitives were enabling software to become vastly more complex, but it was still mostly developed by individuals or small teams working in geographic proximity to one another.

Then the internet happened…the Package, the Version, the Manifest, the RegistryDevelopers used to load the software they depended on off a solid storage device like a CD or floppy disk, manually compile the code and then link the output to their project.

Oh…and if you were missing a peer dependency…you may have to order it in the mail.

This was, by the accounts of my veteran programming friends, the worst part of being a programmer in the 80s and 90s.

It was so hard to depend on other people’s code that much of the time, it was easier to roll the software yourself.

The idea that you could automate all the steps required to use other code in your project wasn’t new, but it took the introduction of several important primitives to bring about the world as we know it today:Package— the software you wanted to distribute and the script other people needed to build/link itVersion — a fixed notion of immutable versionManifest — the idea that every piece of software could define what other software it used so that a package manager could install it in a (mostly) deterministic wayRegistry — an online listing of all the other software you could include in your manifestThanks to these primitives it’s now easy to use other people’s code within your projects allowing teams to focus on the parts of their application that matter most.

Modern registries like Maven, npm, pypi, RubyGems, etc are used weekly by most programmers and have become vital infrastructure in the world of programming.

the CommitThe commit changed everything — I almost don’t need to write about it.

Before Linus released Git, the first (decent) distributed source control management tool, software was built by small teams.

The commit allowed developers from all over the world to work together with the common goal of building great software.

It’s responsible in large part for the proliferation of open source software and, when combined with the advancements in dependency management completely changed how software could be built.

Along the way companies like GitHub, Atlassian and GitLab became the master ledger of software development.

the TicketWith more and more people involved in software projects and a growing number of projects that were being depended on by other software, the old human communication problems became front and center.

How do you plan work and assign tasks on teams building software?.How do you track a team’s progress, or let someone know when their code is broken?It’s easy to ignore the ticket, especially in the company of so many other innovations, but I’ll argue that without it the software we use would be much less powerful with far less reliability.

As long as humans are writing the code we’ll need to organize ourselves effectively.

Innovations in software project management (think Agile + the tools that support it) have a ton of leverage on our industry’s productivity.

Primitives from the era of scaleSoftware ate the world, but oh shit…that was quite the meal.

the BuildBuilding used to be done locally by whichever developer was responsible for deploying the finished software, but now the build is a vital primitive to developers.

It’s become so much more than compiling/archiving code.

Builds are now triggered whenever you check in code to make sure your dependencies resolve, your tests run, regressions pass, and that your code can be deployed safely.

When builds are automatic, developers push code more often.

This lets teams iterate faster, fix bugs before they affect large numbers of users and embrace lean development workflows that make modern teams incredibly productive.

The need to properly “manage the build” created DevOps and set the stage for infrastructure as code to change the way we deploy our software.

Companies like CircleCI, Jenkins and Travis-CI were built atop the build.

the Infrastructure as Code, the Container, the ClusterOver the last decade, we’ve witnessed a profound change in the way we deploy software and manage resources in our infrastructure.

At the center of these developments was a shift in mindset away from managing individual servers to managing groups of them.

It began with tools like Vagrant and Ansible that automated provisioning/updating servers.

These quickly gave way to the container which offered even stronger guarantees around portability and ease of maintenance.

The Docker container quickly became the atom of the infrastructure universe and is being assembled into clusters that run our software across data centers all over the world.

It’s clear that infrastructure as code, the container and the cluster will be the primary primitives we think about as we deploy software, but I think we’re still in the early days of this transition.

The work here is far from finished and there’s still plenty of room for new players to change everything once again.

What’s Next?History shows that an important characteristic of the primitives that take off is that they make a manual process programmable.

It makes sense that the primitives that allow you to build tools, companies and global movements around them proliferate and become tenants of the industry.

So the logical place to look for what’s next are processes that are a) widely practiced by developers and b) just becoming possible to automate.

I think the next set of primitives that transform development will be discovered by applying machine learning and code analysis to our existing source code.

There are so many primitives I could write about, but the following two feel inventible and I’d bet they are going to play a crucial role on nearly every developer team.

the VulnerabilityOne of the emerging primitives that I expect will grow exponentially is the vulnerability.

Our code has always been full of exploits, and until recently it’s always been a manual task to stay current on what exploits have been discovered and check all your dependencies and sub-dependencies for those dangerous packages.

The technology is finally here to do this automatically and now millions of developers, myself included, are using Snyk to automatically find and fix vulnerabilities in their projects.

GitHub, npm and many other developer tools have started to follow suit with similar technologies.

Over the next few years, I suspect you’ll see these tools going beyond simple dependency analysis.

Before long, ML driven approaches to static analysis will be able to catch issues in your code itself.

Are there any places in your code that would allow a buffer overflow attack?.Or areas where you consume untrusted user input w/o proper protection?Vulnerability detection and reporting is poised to become the new code coverage — and it’s all going to be automated.

the APIDisclaimer: this is the space I’m working in.

I’m the founder of Optic and biased in my vision of the API primitive.

Developers rely on more APIs than ever before and with the rising popularity of microservice architectures, this number is only rising.

Meanwhile, developers spend a substantial amount of time each week reading and writing documentation and debugging the code they use to connect to an API.

It’s 2019, why isn’t publishing and consuming APIs more like using node modules?.We should be able to run ‘api document’ to generate documentation for any API and ‘api install’ to add the SDK needed to consume the API to your project.

The growing popularity of protocol buffer APIs powered by the likes of gRPC is definitely a step in the direction of making an API’s interface programmable.

Then there’s Optic which brings these new ideas to existing codebases by making it possible to automatically document any Restful API, publish it to your team, and then generate API clients to consume it.

Over the next few years the process of documenting and consuming APIs will be fully automated and become as important to developer workflows as the package manager.

Final ThoughtsMost major step-changes in programming have followed the discovery of new primitives.

I think the vulnerability and the API will be the next most impactful primitives…but I could well be wrong.

What do you think the next major developer tool primitives are?Better yet, how could you help create a novel one?.What process or data could you use to change the way we program?I’m looking forward to chatting more on Twitter and in the comments!.

. More details

Leave a Reply