A Brief Summary on DevOps in Ackee 2022

I know that the few people who will read this post are my colleagues, me next year (to prepare a new summary), and my mum. Thank you for being a pal and reading what I have to say. Anyway, I digress. Let me give you a small summary of what DevOps is in Ackee. Our team consists of three members. Our primary focus is to support the development operations of our developers. This widens from backend support, which is happening daily, to the support of mobile app developers, which is relatively rare. Somewhere in the middle, we have frontend developers. That means our focus is rather broad.

To make our support for developers sufferable, we have created unspoken boundaries between all the teams. They are unspoken because they are relatively fluid, and it won’t make a difference to specify them anyway. To give a glimpse of where the boundaries are: our Slack #support channel mainly consists of requests to gain permissions and to fix pipelines. It’s not like we don’t work on anything else than that. Work on projects is discussed directly with the team members. Our work on support is at most 10% of our work time. We mainly bootstrap infrastructure for the projects and improve the tools we support: Terraform modules, pipelines, databases, …

What’s new

Due to the end of one of our most significant projects, we had to switch the focus from large topologies to tools, which deploy a docker image and run it until the next deployment. That also means the projects got significantly smaller. For us, the supporting tools had to change slightly as well. E.g., we integrated infracost into our pipeline to give customers better insights on what we are deploying.

Next.js

One major improvement of our technology stack was the adoption of Next.js. From a DevOps perspective, it meant changing the pipeline setup slightly for frontend: The build docker image is deployed to the Cloud Run. That was mainly it. We also investigate possible setup in Firebase Hosting. The difference in the cost is not significant. The value for us is that Firebase hosting is relatively simple to work with – frontends do not usually need Cloud Run's features.

Being part of other projects

Since our presence on support got lower, and we have some capacities freed up, we decided to offer our services to others. We tried the Blockchain community since we have considerable know-how in supporting Ackee Blockchain. We also did a workshop at ELI, which is an academic institution. And finally, we got into package delivery in Shipmonk.

Overall, our value for other companies (cloud knowledge, pipelines mastering, support) is the same as in Ackee. We just had enough time to standardize our approach over time. Our priorities recur in different companies. Those could be the speed of deployment, the transparency of the Stack, and plenty more. In case you are wondering if your stack is good or not, the rule of thumb is how long it takes to onboard a new person to the team. This applies to developers and also to members of the platform teams.

It usually happens that developers have some kind of problem. That problem might be significant in their relevant context. Developers, as they always do, create a solution to the problem. That also applies to the automation of the everyday jobs they do. But context changes as well as the problems the developers face. That’s ok, and that’s the world we live in. The grand issue is that the developers are too nostalgic about the obsolete code to remove it. Or they bend the context to fit the tools they already have.

In the end, there is always somebody new who does not understand why things are the way they are. That’s where the value of the newcomers lay. By questioning the tooling, you can find that the intentions developers once had are no longer relevant. The context is gone, and tooling has to improve.

So far, that’s the feeling I’ve got since we started to help out in different companies. Also, the feeling that I am the awkward Ralph thinking I am helping. Instead of adding value, I question things and get credit for that.

Never-ending discussion of what’s DevOps

For some reason, people still can’t name the field they are employed in. There is no such thing as a DevOps engineer because DevOps is a culture. The same can be said for DevSecOps or any other related names. But you can be a Professional Cloud DevOps Engineer certified by Google.

The problem is that it is a bit hard to advertise the field if you can’t give one sentence of explanation. For example, we would rather discuss infrastructure with the German audience than DevOps. It is generally better understood there. I personally like using SRE, where you can simply say: “class SRE implements DevOps”. But that’s not really explaining anything to the people who do not know the field that much.

I also enjoy the term Platform engineering, which should describe people working on developing and deploying internal tools to improve the developer experience. Yet, again, this term might be already used for engineers working on the core platform of the business. Another term might be an Infrastructure engineer, which somewhat loses the whole development part.

I guess I started to notice the naming issue once we began advertising our DevOps services and could not clearly explain our scope of work. In the end, it seems that the general public better understands even Sysadmin.

Honestly, if your position generates value for the company you are working for, not that many people from management would need to rename it. Even in that case, the name is essentially wrong. I am a DevOps team lead here in Ackee. We offer DevOps services to others. If you are interested, leave me a message.

Lessons learned

This year, bad and ugly, was full of ups and downs. There were a few lessons learned along the way I would like to share with you. I try to be brief, and maybe you will find something you can use in the future:

People do not know what you do. They notice if you stop.

The horrible truth is that many people find your position (whatever the name) very hard to understand. I noticed they appreciate your job once you are not as available as you used to be. Take a two-week vacation, and let me know if I was right.

Nobody pays for Kubernetes if you don’t explain the value

Plenty of projects might start small and want to expand with the rise of their popularity. Kubernetes is a platform that offers a reasonable amount of customization without a large hustle. Yet, the abstraction, the names of resources, might sometimes be hard to explain. Customers would rather stick with predictable small services than put extra cash into a tool that sounds like it will be your playground.

Always think about the database as something which might change

If you ask me what’s the biggest value of GCP, it’s their battery of databases you can use. From Firestore to Bigtable, they all suit their designed use cases without feeling you have to bend something. The problem is that small projects tend to use the cheapest starting option. Once the project rises to popularity, the database usually won’t suffice. Changing the database is not simple. It poses a significant risk to project growth. Thinking about the database layer as an interchangeable item in the project's early days might save you time and money in the long run.

Onboarding is the most expensive item in the team

I tried to describe this point earlier. The main point is that the longer it takes to onboard a new developer into your team, the more expensive it is. When you introduce a new custom tool that is not a best practice or just not common in your field, your onboarding process will take longer and is more expensive. Keep that in mind. A lot of new colleagues will thank you for that.

And that’s all the wisdom I could come up with from the times I felt butthurt this year. It could be worse.

2023, please

Last year, I wrote: “Let's face it: there is hardly anything suggesting the year is not going to suck.” Oh boy, wasn’t I a visionary? Since that turned out to be true, I should be more positive this time: Next year will be great. New opportunities, new challenges. Covid will be gone, and Russians will be defeated in Ukraine.

With that said, I can’t wait for new problems to solve. For those who read till the end, thank you. Hopefully, it wasn’t a waste of time. Like & subscribe, it’s appreciated.