Fox in the Henhouse

There is a renown rivalry in Software Development as polarized as the Hatfield and McCoy's. As long as there has been software, there has existed two factions: the Developers, and the Administrators (also known as Operations).

The Developers just want to write code and get it into Production. But Administrators are in charge of making sure Production works without breaking (keeping things operational). Where Developers love to write code, Administrators are more interested in knowing how a computer is put together and how to keep it running.

And herein lies the friction, because they often look down on each other. Stereotypically, Developers presume that people become Administrators because they couldn't cut it as a Developer. Yet, most Administrators look at Developers as neanderthal's who cobble together their code like a Rube Goldberg contraption with little understanding as to how the computers really work.

However, the two have to work together. For without software, there is no reason for an Administrator to run a server. And without servers to run software on, a Developer will not be able to provide their service.

But when they come together, things don't often go well. What worked perfectly on the developer's system doesn't always work well on the servers, giving rise to the popular meme:

The industry has struggled with this problem for decades. As technologies have improved, advances have come about including Server Automation, Test Driven Development, and Continuous Delivery--which has laid the groundwork allowing for the DevOps movement.

People have a variety of understandings for any word, especially so in the technology industry. With all of the rhetoric and excitement, it sounds like "DevOps" should solve all your IT and application headaches... but how often has the technology industry found a silver bullet?

The challenge we are faced with lies in really understanding what we are trying to solve. DevOps is the latest attempt to mend the friction between Developers and Administrators.

As tools have improved there are some that argue for the Holy Grail of Developers: remove Operations/Administrators entirely. To help separate this concept from DevOps, lets call it Dev-Owned Ops. Dev-Owned Ops is having no Operations Team—developers own the entire solution front to back... but there still are Administrators, they are just abstracted away.

So then, what is DevOps? It is a partnership between Developers and Administrators. This relationship is fundamentally what DevOps is addressing--although with DevOps comes a breadth of tooling and automation to assist in the processes. Both teams share a role in Architecture and Engineering their respective parts, and come together in supporting Operations.

The root of the problem we are faced with is the mindset difference between a Developer, and an Administrator. While they face many similar challenges, the result of a decision made by one or the other can be dramatically different because of this mindset.

And this is Good!

We should delight in the fact that not everybody thinks the same way, and that not everybody is challenged by the same things. This diversity makes us stronger as we work through the different perspectives.

A Developer may struggle for hours to go deep enough into the network layer to really understand a problem they are faced with, where an Administrator might be able help in just a few minutes. And a Developer may have a better understanding over how to deal with CORS and XSS configurations to better secure a web service, where an Administrator just wants to run the service. But if the two are not communicating, then everybody suffers because of it.

DevOps is a way of thinking. It is a shift in how we operate. It is about removing the friction between each side by marrying ownership of production as a partnership. Dev-Owned Ops, in contrast, side steps the issue by ignoring the root problem, which is inter-team relationships, and fails to empathize with the Administrator's concerns.

The problem is, Administrators feel ownership over the "henhouse" of production so strongly that on some level it feels like they are asking the Fox to guard it when bringing in DevOps concepts, and they struggle to trust that Developers can make the right choices.

Good Administrators have a grave fear of production. This is not something easily trained, but it comes with experience. However, Developers will never get that experience if they are not allowed to guard the henhouse.

But even deeper, this comes to a matter of priorities and the fundamental conflict of interest that exists between the two. A person can only serve one master. For the Developer, this is Customer Features, and for the Administrator this is Production Stability.

When push comes to shove, a Developer will always be more willing to sacrifice the security or stability of the system to get a feature in place--with honest and good intentions of fixing it in the future. It is easier to deal with the problem in your face (such as an irate project manager wanting features right now), than the problem you can put in the future (doing something more securely). But this technical debt is rarely met, and as soon as one feature is live, there is the next one, and it is challenging to look back. This has been demonstrated time and time again through the history of software.

So how do we build a team that can help address this fundamental challenge?

Consider what DevOps may describe:

• Automation - Servers and Systems can be managed by hand, as pets, but when you need to scale up to hundreds and thousands, this automation is critical. DevOps tools help with this scale by treating Infrastructure as Code. If Infrastructure is Code, then we should treat it completely the same as application changes and CI pipelines. No infrastructure changes happen without code that has been tested, reviewed by a second person, merged into the master branch, and then delivered into production with a delivery system.
• Ownership - Developers create problems in production with bad code, but don't have responsibility for the stability, so they are not invested in writing stable code. It is somebody else's problem. This creates friction between teams. DevOps is about bringing the ownership of Production to the Developers, to help them recognize the impact of their changes.
• Rapid Change - Historically the process of getting software changes into production has been lengthy, to say the least -- explored further in Continuous Delivery and SaaS. The automation and testing behind Delivery Systems is part of DevOps.
• Operations Intelligence - Knowing how your application runs is invaluable. The new OpInt tools are part of the DevOps toolbox.
• Harmony - It is easy to do something as a one-man band in a Dev-Owned Ops world. But bringing it together with many other services in concert, as an orchestra becomes much more challenging. DevOps is about this type of architecture. If you just have one app with a developer or two, then perhaps DevOps isn't as big of a deal for you.

The list could go on--but the scope continues to expand; and with the expanded scope we are reminded as to why people are so confused over what it really means.

There are four common ways of implementing Operations:

1. Isolated Ops Team -- classic approach, developers build software and "throw it over the wall" to Administrators. Not ideal.
2. Dedicated DevOps Team -- Similar to #1, but addressing modern automation and management tooling to help facilitate making it easier for developers to push changes. Working with developers but providing them tools.
3. Combined DevOps Engineers -- People within each development team are identified as DevOps engineers. They have greater responsibilities as both a developer and ownership when it comes to Operations.
4. Dev-Owned Ops -- You no longer need an Operations team, Developers are doing it all.

Whichever structure you use, at the end of the day, DevOps should be about turning the delivery of code to production into a "dial tone" process, while also reducing defects in the delivered code. This is done by creating a strong relationship and partnering Developers and Administrators to share in the operations role.

The key to DevOps is working on the relationship between the teams so that neither looks at the other as obstacles, but rather as facilitators.

When using Infrastructure as Code, and proper automation systems, developers can be given access into their production systems in ways that are fully auditable, using ISO/SOC2 standards, and still allowing for greater ownership on behalf of the developer.

And this is a good thing.