Centralised vs Decentralised DevOps Toolchain - The Differences

7th Feb 2021

Context:

To centralise or decentralise, that's the question. When enterprises adopt DevOps, one of the most important considerations is how internal teams consume a toolchain supporting the DevOps mindset.

In this article, I want to focus on the big question around how DevOps should (or could) be delivered within an enterprise. Should companies enable complete freedom and decentralisation of their toolchain and process? This lets teams put their own flavour on the core principles of DevOps. Or, do companies go for a more centralised route? This means more standardisation & consistency across the enterprise and an easier adoption avenue due to central capabilities. Let's discuss the pros and cons of each and come up with a recommendation.

What does centralisation mean?

Centralisation of DevOps tooling means offering a single toolchain and process for teams to follow across an enterprise. When you centralise, you provide multiple teams with a single avenue to adopt a DevOps mindset. There are many advantages to centralisation, some examples being:

• Adoption: Whenever you centralise, you usually have a higher adoption percentage off the bat. This is mainly due to teams who are new to DevOps, more specifically, the tools that come as part of the DevOps toolchain, have a place to start and consume something as a Service. If you rely on teams standing up their own tooling, the time to adoption is generally longer as there is more setup time.
• Consistency: If you centralise a process, an expected outcome of that is consistency. This means that if a developer is working on team X, and then moves to team Y; there is less "getting up to speed" time, as the processes would be the same for Team X and Team Y. This is a massive win if your company promotes rotation of developers and engineers across teams.
• Time to Value: Application teams new to DevOps will find it quicker and easier to deploy something to production if you centralise. Teams spread across an enterprise can always ask questions to the central team owning DevOps, so someone is around to help out and ask questions. There is short term success here and a quicker time to value.

As you can tell, there is a theme to the above. The newer your enterprise is to DevOps, the more value you will see from centralisation. However, there are some disadvantages or considerations to keep in mind.

• Slow turnaround of Change: If only one team are allowed to own and maintain tools, and a customer puts in a feature request, that request may be one request often already in the backlog. As that customer who logged that bug, you may have to wait a month, even longer to see that feature request in production. Whereas if you had control of the tools in your application DevOps toolchain, you make the decisions, so the power is in your hands.
• Reliance & Excuses: If you centralise, it is easy for application teams to rely heavily on the team's who own the central tools. Especially if there is something the application team is waiting on. There is an excuse to why they are behind or not moving at the pace they should be, "I can't do this, just waiting for the central team to deliver something". Too much reliance on one team means you don't move at the pace that you may want to move at.

There are lots of advantages to centralisation, especially for larger & legacy enterprises. Now let us discuss what decentralisation means.

What does decentralisation mean?

Decentralisation of DevOps tooling is the opposite of centralisation. It's where there is not central tooling, there is no central process, and you allow your enterprise to embrace the principles of DevOps themselves. There is a trust that they will see the value in DevOps, and have the creative and learning agile mindset to read, up-skill and apply. Some advantages of decentralisation are:

• Embracing Diverse DevOps Models: There isn't one way to adopt DevOps, and there most certainly isn't one way to use a DevOps Toolchain. If you centralise too much, you will find people "doing the same thing" and not getting the most out of the tools that they are using.
• Learning Agility: The value of decentralisation is that you are equipping your enterprise for learning success. Change has been, and always will be happening, especially within IT. If you fully decentralise, you are asking and almost enforcing your enterprise to upskill themselves. What this leads to is a longer-term success. If employees are used to changing and are used to learning new tools & processes, anything new that comes up in the future should require less time to adopt as employees are used to a changing environment.
• Innovation: This is a pivotal point to understand. You want your employees to be innovative and apply learnings to their working model. For some reason, there is a preconception that the centralised team knows best and what they say should go. This is not always the case. Just because the central team think they know what's right for the enterprise, doesn't mean they do. If you decentralise your DevOps process and toolchain, you are letting your employees innovate and think differently without the central team's restrictions. A different way of thinking about this is - would you rather a central team of 40 people making decisions, or would you like 1000 people (or everyone in your company) learning and deciding what's best. One thousand minds are better than 40.

However, for all these advantages, there are possible disadvantages to think about:

• Can be slow to get enterprise-wide adoption: Normally, leadership within an enterprise want quick results and want to see success straight away. If you have a highly skilled and learning agile company, this may not be so much of a problem. But companies who have been around a while may struggle with getting results quickly.
• You may not see shared learnings: There is a reality to decentralisation: some teams will get it quicker than others. As there is no central team, there may not be an easy way to share learnings across teams. This will lead to some teams racing ahead of others, with the high likelihood some teams will never get it and will be left behind. At the end of the day you are one company, so the success shouldn't be perceived as team success, it should be company success.

Similar to the centralisation section, there is a theme appearing. A newer company or a company with a highly agile culture is likely going to suit decentralisation more. I believe following this model will set yourself up for success longer-term; when done right.

So, what one should we pick?

The reality of this question is that there is no right answer. But there is a recommendation:

Centralise the DevOps Toolchain's foundations, whilst decentralising the journey on "how" teams adopt & consume the centralised tools, patterns, and processes.

So, what does this mean? Let's break down that sentence into two:

"Centralise the DevOps Toolchain's foundations".

It isn't time effective, or cost-effective to have multiple of the same tools, or similar tools spread across an enterprise. An example being source code management. Stand up a single source code management tool and offer it out to the enterprise as a Service. This will reduce operational overhead because instead of multiple teams managing that tool; it will be supported and maintained centrally. Additionally, it's typically cheaper contractually if multiple licences are purchased centrally vs distributed across numerous contracts. For centralisation to be a success, automation and end-user autonomy needs to be a core principle. Staying with the source code management example mentioned above, you should allow customers to create repositories themselves without putting a request in and waiting. Allow teams to create teams themselves, as well as administer their own repository. Ensure there is no friction in the way of the customer of the tool. The more "red tape" and "friction" in the way of the customer and the centralised tool, the less chance of success. If you're thinking, "I can't give administration access and full autonomy in my company, we have quality process and regulations we need to meet." Just because you give autonomy and allow self-service, doesn't mean you can't wrap process and rigour around the tool, ensuring quality and security compliance. Use API's to build the needed quality processes to ensure the centralised tool is "in compliance" of any in-company procedures. Then publish documentation of any processes, so customers know what to expect. Mimic this for every tool within your toolchain.

Additionally, understand that you will have different teams at different skill levels, so centralising foundational blueprints and patterns for teams to get started with makes sense. For example, within GitHub, create repository templates which follow best practice (locked branches, pull requests enforced, turned on security tools). This means teams new to GitHub have a place to start "out of the box" with everything needed pre-created. The same goes for your CI/CD tool. Build reusable workflows/pipelines with standard testing & deployment patterns across your enterprise. For example, you may be a predominately JavaScript shop, so building a single pipeline that does JavaScript testing/linting/security means teams new to CI/CD have a place to start (additionally teams with CI/CD experience may not want to build their own so will reuse what is on offer centrally). Thirdly, and finally, is the hosting platform chosen (AWS, Azure, GCP). Build central patterns for common software application architectures. e.g. if many of your software products built are web assets, creating a SPA template that comes with the needed infrastructure as code to deploy to AWS S3 & Cloud-front. This means teams have a place to start, especially people new to cloud-native. The same for API's would apply, an OpenAPI template that comes with AWS API Gateway & Lambda would be a great starter kit.

Okay, so now you may be thinking wow this is a lot of centralisation, let's shift to the second part of the above phrase which is:

"Whilst decentralising the journey on "how" teams adopt & consume the centralised tools, patterns, and processes."

You are not the police. You cannot dictate and enforce how teams use your tools. If you lock everything down to a single way of doing things, you will slow down teams. This will fail. Your company may have adopted a "DevOps Toolchain", but your teams won't be moving fast and won't be adopting the principles of DevOps. Past standing up the tools and foundational patterns/starter kits, let teams decide how they want to use the tools you have stood up.

An example being your hosting platform. Stand up an underlying hosting service that meets the needed quality & security processes for the enterprise, but then get out the application teams way. Do not centralise manual gates and reviews to one team as that team "they know best".

Doing this will massively slow down not just deployment time, but learning time. Give autonomy for teams to deploy from Dev -> Q.A. -> Prod. If teams fail, that's okay, let them learn from that failure and apply that learning for next time.

The same goes for CI/CD. In the above paragraph, I explained building example pipelines/workflows. Do not dictate this is the only way for teams to use CI/CD. Allow teams to see these as examples, and then customise to suit their needs. Teams may not even use these pre-built pipelines/workflows, and that's okay. They are there for reference and use if desired.

The main point to be conveyed here is that there is no one way of doing things. Do not centralise the process, just the foundations and tools. Doing this will set teams up for success by themselves. Additionally, the more centralisation you do and the more is done for teams, the less they will learn themselves. This puts enterprises at a further disadvantage because if centralised tools change, or there is an update in a procedure that requires a change on the application team, teams will be foreign to doing things themselves, leading to slower change adoption.

Conclusion

There is a misconception that DevOps should be a specific role, and only people who's job title includes "DevOps" needs to focus on DevOps. In my opinion, this isn't, and shouldn't be the case. DevOps is a mindset (process) which teams need to adopt when building and deploying software products. Everyone part of that software team somewhat contributes to the DevOps process. As mentioned above, there is no right answer when it comes to centralisation or decentralisation, it depends on the company and most importantly, the people. In most enterprises, a mixture, or somewhere in-between centralisation/decentralisation, will give the best experience for different teams. To conclude, centralisation will get you quicker success but can be short term thinking. Decentralisation will take longer to get there but in the long term can see more significant benefits.

The key takeaway though is centralisation works, but only when done at the correct level, and autonomy is vital.

Review of GitHub Advanced Security Code Scanning in an Enterprise (Part 1)

30th Jan 2021

Context:

GitHub recently introduced a suite of new products under an overarching service called GitHub Advanced Security (GHAS). There are three core features as part of GHAS: Code Scanning, Secret Scanning and Dependency Review. In this article, I want to focus on what I would consider the core capability, Code Scanning.

Code Scanning is a tool that allows developers to search for potential security vulnerabilities and coding errors in their code. People may compare this to static application security testing (SAST), or static analysis.

This article assumes some background knowledge of GHAS Code Scanning. The core two sections will focus on what makes Code Scanning stand out amongst similar products and the challenges found so far within an enterprise. Meaning it's recommended reading the following links to get a brief understanding if you are unfamiliar:

• Securing software, together
• GitHub Code Scanning - your automated security PR review

What's good?

Critical features when enabling new tools and services within an enterprise are ease of use, and how simple it is for teams to get up and running. This is even more important when introducing new capabilities within the developer ecosystem. It needs to be seamlessly integrated into the developer workflow and configurable via code, not clicks.

GHAS Code Scanning shines here, as all developers have to do is head to the security tab within a repository, click Setup this Workflow within the CodeQL card, and GitHub will auto-create a codeql-analysis.yml file. This file contains the process of running Code Scanning within your repository. The setup time per repository on average was 55 seconds.

Furthermore, where GHAS Code Scanning starts to come into its element is its flexibility in configuration. As mentioned above, GitHub will auto-create a code-analysis.yml template, with GitHub being smart enough to detect repository languages and core branches (main, dev, etc.) and put these into the template automatically. However, the development team have complete control over how Code Scanning runs, meaning they can change the code-analysis.yml to meet there needs. A few examples being:

• Triggers: Teams can decide what events Code Scanning triggers on. E.G. Pull Request, Tags, Branches, and even cron jobs. A good tip is to configure Code Scanning to run on all pull requests, and then have a cron job running nightly on the default branch (which usually is your production build). Doing this means if any new security queriers get added to the CodeQL pack running on your repository, but you didn't make any commits to your default branch, the cron job will ensure you are kept updated with any new security vulnerabilities.
• Auto Build Process: The way Code Scanning works is to auto-build your application (for compiled languages) automatically following standard build processes. This build process is all handled and abstracted for you using the GitHub Action - Auto Build. However, suppose your repository has some crazy build process which is entirely un-standard (yes, we all know a few of these applications). In that case, you can specify your application build process, and Code Scanning will honour it. Something that other security tools struggle with.
• Configure the type of scan that suits you: This is one key feature which I believe goes under the radar. Different types of applications require different levels of acceptance of false positives. For example, if your application is lower risk, you only really want your security tooling to return vulnerabilities that are near 100% accurate. On the other hand, if your application is higher risk, your toleration of possible false positives is higher. GHAS Code Scanning allows you to tailor different "suites" of queries, dependent on your toleration level. If you don't specify a pack, the standard one will be run, which will only return high accuracy vulnerabilities. More information can be found in the following article: Running additional queries. The value is enterprises has such a varied range of risk. Most security tools don't have a built-in feature that allows you to configure the type of scan to the application risk acceptance. GHAS Code Scanning changes the game in this aspect.

It isn't just configuration flexibility that makes GHAS Code Scanning an excellent tool; it's the scans' speed. One of the problems seen previously is that scans could take 20-25 minutes when you had a polyglot repository. We found a dual language repository (JavaScript and Python) has an average scan time of 2 minutes and 11 seconds. As applications move to the cloud, and multi-language support becomes the norm, having a security tool that runs at a speed that falls in line with other CI/CD tools will be essential.

The final point to touch on is the fact; it's GitHub. GitHub has built up such a good brand of delivering tooling that focuses on the developer, and we can't ignore that. Security tooling mostly has been very much focused on security, with the comprise of developer experience. GHAS Code Scanning so far has demonstrated that shouldn't be the case and doesn't have to be. There is a happy medium between developer autonomy and enabling a self-service nature, with security policies, processes and rigour.

What's missing?

If you are reading this and thinking Code Scanning sounds like the tool for you, keep in mind that it's still a maturing tool with feature gaps. Some of the critical problems found so far are:

• GitHub Contents API (No Support): One of the areas you may want to explore, to help teams adopt GHAS Code Scanning is building a tool that automatically creates a pull request on every repository with a template code-analysis.yml file. Doing this would enable two things, 1) reduce the time for teams to get up and running, as there is a template to get started with 2) encourage more teams to adopt GHAS, as there is an open pull request on their repository needed to be actioned. Both these areas lead to a better experience for application teams and a better security stance for the enterprise. The best way of doing this would be to use the GitHub Contents API to commit the template file. However, the .github/workflows directory (where the template needs to live) is protected from the Contents API, meaning it wouldn't work. GitHub says you need to use a native git client to perform any commits into the .github/workflow directory. The annoyance here is if you want to run this serverless, or without an underlying O.S., you can't. If you wanted to run this on AWS Lambda, you would need to do something like Containers and Lambda. Although this isn't architecturally a poor design, it's simpler to be consistent and use API commands only, versus having to use API calls and bash commands. When you think about unit testing, linting, etc., the moment you start writing bash, it becomes problematic. We overcame this by using the Child Process Module. Overall, if this sounds like something you would be interested in, a tool has been built, and we are in the process of open-sourcing the codebase; I'll post back here when the repository is open.
• Language Support: The languages that GitHub support today are the most common, and when they support a language, it is thorough. However, key languages are missing that I would consider critical to support enterprises that likely has a diverse range of languages used. Examples being R, Swift, Kotlin and Rust. R is being used extensively across the statistical analysis and machine learning application world, so I expect to see GitHub support R in the next few languages released. I also hope to see Swift & Kotlin support soon. Mobile apps are still essential, so interested to see support for mobile soon.
• Permission Model: GitHub has to change the way permissions are currently handled with GHAS Code Scanning. Most enterprises need a way to add certain security personnel to repositories to monitor, action and closeout vulnerabilities found. Right now, to achieve this, they would have to be given some form of administrative access, which shouldn't be required. Creating a dedicated permission, focused purely on the security section, would allow that de-coupled responsibility away from other features, such as committing, maintaining, etc.
• Connecting to the rest of the GitHub Ecosystem: GHAS Code Scanning still has minimal support when you think about how it connects with the wider GitHub Ecosystem. For example, the API support is minimal. There are no webhooks, so for large organisations with loads of repositories, this means lots of clicks. Additionally, there is no in-built way to reference vulnerabilities within GitHub Issues as you can with Pull Requests. I imagine a world where a vulnerability is found; a GitHub Issue is opened, and a conversation sparks between developers and security discussing vulnerability thoughts.

These are the essential items, there are a few more, but if thinking about enabling GHAS, keep in mind the above gaps.

Concluding Thoughts

I genuinely believe GHAS Code Scanning has the foundation and vision to become a leader within the security space, whilst ensuring developer experience stays a core principle. Code Scanning is built directly into GitHub, which means developers find it easy to locate results and therefore action them. This is a primary behaviour any enterprise wants when introducing a new security tool. I am excited to see Code Scanning mature over the coming months and years. I believe the roadmap is strong. To conclude, GHAS Code Scanning can disrupt the developer ecosystem's security tooling space with the right push from the open-source and enterprise community's. Watch this space!

I plan to update this blog as new features come out, and the service grows. Part 2 and Part 3 will come in the coming months focused on Secret Scanning & Dependency Reviews.

Building a simple website

21st Jan 2021

Context:

This dates back to December 2020 over the Christmas break when I decided it would be a good idea to start my own technology blog. It is something I have been doing myself for the last few years, but never published publicly. I am lucky enough to work for a company and a role, that gives me opportunities to work with diverse teams (from a technology perspective), meaning I am in a unique position where I see a broad range of technologies and how technology is applied/adopted. My role is leading DevOps & Developer Experience globally for a company called Lilly. Most of my day-to-day tasks involve improving developer momentum and building a community for developers to be successful. This blog represents my thoughts only, and no way represents the company's thoughts that I work for.

This leads me to this website, and the importance of this article's title, building a simple website. 

What I used:

"What I used" is somewhat the most crucial section, as it walks through the journey that I went on for building this website and why building "simple" is so important. 

JavaScript is by far my most comfortable language, and VueJS being my go-to front end framework. So naturally, I started there. It didn't take me long to have a simple blueprint in place. 

I started by using the VueJS Enterprise Boilerplate, that comes with some great tools and utils out of the box, and a stable build process with Nuxt. I built on-top of this blueprint, adding some simple components and style that made my application look like a blog. I was getting ready to publish when my DevOps side came out and thought I should run my blog through lighthouse to check performance, accessibility and best practice. The results were poor, 

• Accessibility was at 65%.
• Performance was at 71%.
• Best Practise was 73%.

The scores above came down to the application having so much going on. Specifically, the performance score was low for a few reasons. A few being: The network payload was 12,210kb, the JavaScript execution time was 1.9 seconds, and the DOM contained 210 elements! I spent a few hours fine-tuning the bundle, reducing a lot of JavaScript "bloat" that wasn't needed, and that did help improve the score, but that made me pause, take a step back, and think, why am I doing this. Why am I spending time fine-tuning an application, with so much bloat, that just shouldn't be needed for my use case? I wanted to build a site where I can share my experiences with others in a comfortable and consumable way. I was overcomplicating the simple. Taking a look at my code base, it already had over 40 files (a lot of these being configuration files, obviously), and the total LOC came over 3000 lines (ignoring the package-lock file). I only had 1 blog post.

This was when I decided to go back to basics and go "old school" with a simple, static HTML and CSS site. Fast forward 55 minutes, I had a simple website built, that meet my use case. The only technologies I used was HTML & CSS, I had no JS, as it just wasn't needed. When running it through lighthouse again, the difference was insane: 

• Accessibility was at 95%.
• Performance was at 100%.
• Best Practise was 100%.

The network payload was under 1,000kb, the JavaScript execution time went down to 0.4 seconds, and the DOM size was minimal. In all, there was just so much less pressure on the browser. The number of files totalled 19, and my overall LOC come out under 500 lines. I spent about 20 minutes fine-tuning, but that it. 

What I learned from this experience:

What have I learnt from this? In a few words, simplicity is vital. Always think about your use case and what you need to build. Ask yourself, what is the most straightforward path I can take to achieve success. Now, I agree that nowadays, personalised experiences are vital, and dynamic content is the norm. This naturally means front end frameworks like VueJS, React, etc. are the standard and the "go-to". This theory showed as my "go-to" for building this website was VueJS. I didn't even think of anything else, mostly plain HTML. But this isn't, and shouldn't always be the case. I recognise that instead of thinking about the product I was building, my focus went straight to the technology and what I was going to use.

To conclude, building this website has actually been fun. It has made me go back to my roots and write some vanilla HTML. It has made me realise I rely too heavily on third-party frameworks. Finally, it has made me realise that building a performant and optimal website is made easy by starting simple. If this blog expands, moving to a more progressive & dynamic application may be needed, but for now, my simple design meets what I needed to build.