r/devops • • 2d ago

Weekly Self Promotion Thread

5 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • 1h ago

Discussion How do you prevent analytical queries from tanking your transactional database?

• Upvotes

I was checking out how to build a live operational dashboard for a logistics app and found out about composed vs. converged HTAP architectures.
I’ve always been warned never to run heavy analytical aggregations on the primary transactional DB because the contention spikes your transaction latency. But a guide I was reading explained how HTAP systems are handling this now. A “converged” system runs both through the same engine (which sounds risky for resource contention), while a “composed” system actually separates the compute engines and keeps them in sync through shared storage or replication, so each side stays optimised for its own workload.
Apparently, you’re supposed to measure p99 transaction latency while the analytical queries are running, not separately.
If you are running live analytics on operational data, how are you keeping the two from interfering with each other?


r/devops • • 6h ago

Discussion Preparing for OCTA and CCA

4 Upvotes

Hey guys,

I'm preparing to get the CCA and OCTA cert to become golden kubestronuat, i see that the kode cloud cuurses are 20 hrs each, is it really necessary to go with that courses? also i saw that kyverno kode cloud course is so deprecated, is this true? i need to get them fast because cks will expire and i dont want to do it again xD

Thank you so much.


r/devops • • 3h ago

Discussion Interview tomorrow (5+ YOE DevOps), CV deep-dive, but I’ve been caught in reorg purgatory doing nothing for months. How do I spin this?

0 Upvotes

I have an interview tomorrow for a DevOps Engineer position (I have 5+ YOE). The recruiter explicitly told me the interview will be a deep-dive through my CV.

I’m in a massive bind regarding my current/most recent role and need some urgent advice on how to frame it.

The Situation:

  • Early this year: I was the lead engineering resource tasked with getting a major product production-ready. The literal day it was ready, a massive reorg hit. The product was stripped from me, and most of its target end-users were laid off.
  • Immediately following: I was pivoted to head up another large project for a different internal product. I busted my ass, completed it successfully, and my manager was impressed. Then... another reorg hit. This product was also taken away.
  • The past 5+ months: My team was merged with another to form a new SRE team for a completely separate product. The scope was entirely undefined. In fact, one colleague's sole task for the last 4 months was just figuring out what our responsibilities are.

Because of this reorg limbo, I have had zero actual engineering work to do for months. I’ve repeatedly met with my manager and skip-level to voice concerns, but I just get generic corporate hand-waving about a "clear roadmap ahead."

The Dilemma for Tomorrow:
If they deep-dive into what I am doing right now, I see three options, and they all feel terrible:

  1. The Brutal Truth: Admit I’ve been sitting on my hands doing nothing due to corporate dysfunction. (Worried this looks toxic or makes me look unmotivated).
  2. The Total Lie: Pretend I’m still actively working on those two successful deployment projects and that they were never taken away. (Worried because my poor recall will make it obvious I haven't touched the code recently, and I'll have to fabricate metrics, current challenges, and future roadmaps).
  3. The Middle Ground: Pretend I just transitioned off those projects a few weeks ago (similar issue as option 2).

How do I handle this in a technical deep-dive without sounding like a liar or a slacker? Has anyone successfully spun "reorg purgatory" to a hiring manager?

Thanks in advance!

TL;DR: 5+ YOE DevOps engineer with a CV deep-dive interview tomorrow. This year, I built and completed two major projects, but back-to-back corporate reorgs stripped them away. I've spent the last 4 months in team-merger purgatory doing absolutely nothing. Do I tell the truth, lie and pretend I'm still on the projects, or find a middle ground without looking like a slacker?


r/devops • • 3h ago

Ops / Incidents The $87K API key leak had nothing to do with .env

0 Upvotes

Every time someone posts about a leaked API key, the top advice is the same. Put it in .env, add .env to .gitignore, done.

After the OpenCode breach and the LiteLLM mess this month I went back through the leaked key stories people posted this year. Found 10, with bills from $2 to about $87K. Not one was "I pushed my .env to GitHub".

Where the keys actually were:

> In the JS bundle. A VITE_ variable in a gitignored .env, repo scanner said clean, then the build shipped it to every visitor. One scan of 978 AI-built apps found 37 doing this.

> In an agent config file. LiteLLM's master key sat in opencode.json, got force pushed over but never revoked, and was used 136 days later.

> In someone else's tool. Keys saved in OpenCode's console got drained, one person lost $5,200 of company credits.

> In a WordPress backup plugin. One root AWS key reused across sites, tested on Aug 23, used 56 days later for about $8K of Bedrock.

> In old Google keys that quietly gained generative AI access. That's the $87K one.

> In nobody's leak at all. Someone got into an Anthropic console through email login and created 3 new keys, $5,473 in a few hours.

And then last week Railway got hit. Attackers decrypted customer environment variables straight from the platform, and Railway told everyone to rotate every key. Gitignored, never pushed, stolen anyway.

What actually helped was boring:

> Hard spending caps. The one person who had one lost a few minutes of usage and that was it.

> An alert isn't a cap. The $87K team's budget alert fired at $550, five hours in.

> Revoke, don't clean up. Force pushing doesn't un-leak anything.

> Delete keys you don't use, and know where every key lives (Infisical, Vault, whatever you already have).

> Don't count on a refund. The $87K bill got $55K off, they still owe about $32K.

I'm not saying stop using .env locally. I'm saying "it's gitignored" isn't the end of the conversation.


r/devops • • 4h ago

Ops / Incidents I recreated a Node2Shell RCE attack in the form of an interactive exercise

1 Upvotes

On 21 October 2026, Google released patched versions of its Brotli compression library for CVE-2026-111497, a vulnerability nicknamed Node2Shell. It is an out-of-bounds write in the Brotli decoder. Node and many Python stacks bundle their own copy of Brotli, so patching libbrotli on the host doesn't fix them.

I made an interactive, hands-on lab exercise on this vulnerability. The goal is to give engineers a general overview of the attack vector and the response. You'll exploit a service as an attacker, then switch roles to the on-call engineer: upgrade the runtimes, rebuild and redeploy the images, verify the fix, and rotate the deploy token the attacker read from memory.

I did my best to highlight all the details, but if you notice any issues or inconsistencies, please let me know!

https://ransomleak.com/exercises/http2shell-rce/


r/devops • • 4h ago

Discussion CloudFlare Tunnel with pre-auth (and multiple IdPs)

1 Upvotes

I am working on setting up CloudFlare tunneling for a SaaS product in order to get WAF in front of the application and potentially use the CDN for faster distribution.

In the process there was also a requirement to move over to using pre-authentication in front of our SaaS solution, so that all requests are authenticated before they even hit our servers (similar to Azures application proxy).

It was suggested to use CloudFlare access, but this seems to more targetted towards internal apps and replacing / addition to a VPN in order to get ZTNA.

What would be a good way be to setup this pre-authentication? To me it seems there is no CloudFlare solution that handles this, so either I have to setup something to handle this check on CloudFlare or find a different vendor.


r/devops • • 1h ago

Discussion building is so easy and verification becomes bottleneck now !!

• Upvotes

now we all know .. building/coding is easy with lot of ai tools/coding agents.. and pressure we are seeing during the deployment cycles verification and human in the loop cycle..

and wondering how are you guys handling it? we have team of 15+ and multiple teams and using Asana agents they have long runing agents and produce features/bug fixes super fast.. but determinstic verification lacks..


r/devops • • 1d ago

Discussion Anyone interested for DevOps discussions/sessions?

45 Upvotes

As i always wanted to help and teach people, I wanted to do my part by teaching, by the way I'm working as a devops enginner.

FYI without money , for the first time I'm gonna do what I wanted to , let me know which topic has to be picked and interested people ping the comments.

I want to start the basics on kubernetes cuz I like kubernetes I want to start with this. Thank you. Not sure if this post will be retained or delayed.


r/devops • • 6h ago

Security Devops with AI..

0 Upvotes

I've recently met a former colleague who currently works as a DevOps engineer at a company that forces its employees to use AI (Claude, I guess). Each employee has an AI budget that he HAS TO use every month. I believe the goal is increasing productivity per employee.

I have a lot of concerns about this approach to work, especially from the perspective of security, privacy, data leaks, and having a non-open-source AI app installed locally. What if the tool is hacked? What if my credentials are leaked? What if AI hallucinates and deletes non-recoverable data?

On the other hand, I understand this might be the new industry standard. If yes, I don't think there's a point of resisting change since the train will leave without me.

I would like to see what the perspective of other industry experts is and how they navigate this topic. Are you an early adopter? Or do you prefer to wait until the storm passes?


r/devops • • 5h ago

Ops / Incidents Suggest a good laptop for DevOps under ₹90,000 in India

0 Upvotes

Hi everyone!

I'm looking to buy a new laptop with a budget of ₹90,000 for learning and working with DevOps.

My primary use cases will be:

Docker and Docker Compose

Kubernetes and Minikube/kind

AWS (EKS, EC2, CI/CD, etc.)

Jenkins, Terraform, Ansible and Linux

Running multiple containers and virtual machines

Programming and general development

I'm not looking for a gaming laptop specifically. My priority is performance, multitasking, Linux compatibility, cooling and long-term reliability.

I'd appreciate recommendations based on real-world experience, especially from people working in DevOps or cloud engineering in India.

Thanks!


r/devops • • 13h ago

Discussion At what point does monitoring become so noisy that engineers stop trusting it?

0 Upvotes

You add alerts because you want to catch problems early.

CPU spikes. Memory usage. Failed requests. Slow queries. Queue delays.

Eventually, the team starts receiving notifications throughout the day.

Some are important. Most aren't.

And that's when a dangerous habit develops: people start assuming the next alert probably doesn't matter either.

I'm curious how teams deal with this.

Have you ever deliberately removed a large number of alerts to make monitoring more useful?

How do you decide which alerts deserve to interrupt an engineer and which should just be visible on a dashboard?


r/devops • • 1d ago

Discussion Any beginner-friendly DevOps Discord servers or Slack communities?

2 Upvotes

Hey Everybody!

Does anyone know of any active Discord servers, Slack channels or study groups where beginners can connect, share resources, ask questions and support each other along the way?


r/devops • • 2d ago

Security Making attacking agents pay with Context Bombs

Thumbnail
agentic.tracebit.com
185 Upvotes

We just published new research on using deception against AI attackers.

In controlled AWS tests, agents took up to 8× longer and cost 23× more per objective. With all nine techniques combined, Kimi, Qwen, DeepSeek and GPT-5.6 Cyber reached zero objectives across 40 runs.


r/devops • • 1d ago

Vendor / market research What APM tool does your team actually rely on, and why?

0 Upvotes

We build a patch management product. Right now it uses Dynatrace to pull health signals from managed assets (things like whether a service or host is healthy before and after a patch is applied).

Before we commit further, I'd like to hear from people who use these tools every day. Which APM or monitoring tool does your team use (Dynatrace, Datadog, New Relic, AppDynamics, Elastic, Grafana/Prometheus, something else)?

I'm most curious about:

- How reliable the health and alerting data is in practice (false positives, gaps, lag)

- How comfortable your team is using it day to day

- How easy it was to integrate with other tools through its API

- Anything you regret or wish you'd known before choosing it

I'm not selling anything. I just want to understand what teams trust so we can support the right tools. Thanks in advance!


r/devops • • 1d ago

AI content why is everyone still using cloud llms just to route tool calls? tested local cpu embeddings vs bm25 on 100k tools

0 Upvotes

honestly i dont get why current multi-agent setups are built like this.

if you connect 4 or 5 mcp servers (github, postgres, filesystem, slack etc) you easily end up with 40-50+ tool schemas. right now the standard pattern in langchain or claude code is dumping all 4,000+ tokens of json schemas into the prompt on every single turn just so the llm can pick which function to run.

you end up paying token costs and waiting ~2 seconds of network latency just for what is basically an L7 routing lookup.

i spent the last few weeks testing if we can just decouple tool discovery completely from the llm using a local cpu vector mesh (all-MiniLM-L6-v2 + chromadb hnsw index) and a split execution daemon.

instead of embedding just the function name (which fails on synonyms), i combined the tool name, description, schema keys, and tags into a single profile vector. when a prompt comes in, it embeds the query on cpu and pulls the top match before ever touching the llm.

ran a cold-cache benchmark across 100,000 synthetic tool endpoints on my local machine (standard cpu, no gpu):

method top-1 acc avg latency p95 latency

---------------------------------------------------------------

naive keyword scan 38.1% 46.41 ms 57.60 ms

bm25 lexical index 40.4% 194.05 ms 247.25 ms

local vector mesh 70.7% 9.56 ms 11.44 ms

in-memory fast path n/a 0.082 ms 0.15 ms


r/devops • • 2d ago

Discussion Open Source Contribution Recommendations for projects related to DevOps and Platform Engineering

57 Upvotes

I'm interested in Open Source Contribution and looking for recommendations for projects that are related to DevOps and Platform Engineering.

I'd like my contribution to be relevant to my everyday work.

I'm good with python, go, nodejs and typescript.

This will be my first contribution so, I'm also looking for something that's not so difficult to work on.

I'd appreciate any advice or recommendations


r/devops • • 1d ago

Discussion First-year DevOps student, 5 months to finish an individual project, feeling overwhelmed. Any advice?

0 Upvotes

Hi everyone,

I’m a student in the early stages of a DevOps degree, and I have to complete an individual project within 5 months. My lecturer asked for a unique, new topic that is purely DevOps, not system engineering. I searched for a long time without luck, then used Perplexity to explore ideas and found one that my lecturer approved. He told me to start with a literature review and continue from there.

The topic is: I don't want to reveal it here, but I'll let you know when you are willing to help me.

Now that I’m starting, I’m honestly panicking. DevOps is new to me; the degree has only just begun, and many of the tools and terms are unfamiliar. I’m comfortable with practical work, but I’m worried about how much I need to learn in such a short time and whether I can finish on schedule.

I’d really appreciate help with:

  1. How should I approach the literature review when I don’t know the terminology yet?
  2. What core DevOps concepts and tools should I learn first (CI/CD, containers, IaC, etc.)?
  3. Any tips for planning a 5-month project as a beginner?
  4. Good free resources, communities, or books you’d recommend?

If you’ve been through something similar, I’d love to hear how you handled it. Thanks so much in advance!


r/devops • • 2d ago

Discussion Two widely used npm packages had their security advisories withdrawn. How should we handle false positives in dependency audits?

16 Upvotes

Both the braces and sprintf-js security advisories have been withdrawn, despite previously being marked as GitHub Reviewed.

braces: CVE-2026-93687 / GHSA-vfj7-8cjw-p6xm
sprintf-js: CVE-2026-97058 / GHSA-hp3w-g68c-fv3c

I understand that third-party security risks are a pressing concern. Last week was particularly rough, with incidents and critical vulnerabilities involving LiteLLM, PHP, intermediate CAs, Brotli, and more. Security teams everywhere have been scrambling to assess their exposure and mitigate potential risks.

However, these two widely used npm packages, commonly pulled in through linters, test frameworks, and other development tools, have once again caused unnecessary alarm across the ecosystem.

Both advisories have now been withdrawn, but not before triggering security alerts and forcing developers to investigate issues that turned out not to be practically exploitable.

How much can we trust automated dependency audits when even reviewed security advisories can turn out to be false positives?

More importantly, what should change in how vulnerability reports are vetted, published, and propagated to downstream security tools?


r/devops • • 2d ago

Ops / Incidents HA AI Clusters - SRE Engineer - General questions

15 Upvotes

Hey guys, quick question for anyone in the AI or infrastructure space.

I was never really interested in running AI models on my homelab cluster, but this job offer looked too good to pass up.

I have a technical interview coming up soon for an SRE role on High Availability AI clusters, and honestly I have zero clue what to expect or what kind of questions they might throw at me.

The position is completely remote, focusing on monitoring, troubleshooting, and keeping things running via SSH without any hands-on access to physical hardware.

Core stack is mostly standard Linux, Docker, Kubernetes, and networking.

If anyone here works with HA AI infrastructure or went through a similar interview process, what do they actually ask about?

Do they stick to standard Linux and networking troubleshooting, or do they push hard on AI-specific tech like GPU drivers, Infiniband, NCCL, or CUDA failures?

Any heads up or real-world examples would be awesome. Thanks in advance!


r/devops • • 4d ago

Discussion Seems someone is interested in AWS Golden Jacket 🧥

Post image
207 Upvotes

r/devops • • 4d ago

Discussion CKA exam stress

32 Upvotes

Hello community,

I am really stressed about my cka exam. I have bought it but have not prepared yet. My goal is to give exam in december. I have udemy monthly subscription and thrive subscription for this year. I am watching mumshad course on udemy.

I have 4 years of experience as a devops engineer. Done ccna training and comptia security plus. I have a very good command on linux(ubuntu) because my career started with it.

But the uncertain and pressure of work is giving me hard time. I am also works as a consultant so have to give time on that as well. I get exhausted after work. I consumed half of the course with labs on udemy and kodekloud but having no experience of production kubernetes. I understand the concepts but still thinks something is missing. The last topic of admission controllers gave me hard time to consume.

Do you guys know anything like tips and tricks to pass this exam. Anyone studying or prepping for the same can also collaborate. Because it is better to clear the concepts.

Dont ignore this post.

Regards

Worried and stressed person

Edit: thankyou everyone for the kind advices. Maybe I have many sources that is why I overthink it. Because I switch from udemy to thrive to multiple youtube channels. I will stick to 1 course and complete it.


r/devops • • 4d ago

Observability What are some use cases for an internal system monitoring bot?

9 Upvotes

​

Hey everyone! I’m exploring ways to extend an existing internal production health-check bot that monitors systems checks whether the important pages loads or not and sends status updates to Microsoft Teams.

I’m looking for ideas beyond basic health checks, such as proactive alerts, log analysis, incident detection, automated troubleshooting, or AI-powered features.

What useful automations have you built around monitoring bots? Which ones have made the biggest difference in real-world scenarios?

Would love to hear your suggestions and experiences!


r/devops • • 4d ago

Ops / Incidents Railway's Connected GitHub Accounts Were Compromised

50 Upvotes

Got this email from them this morning. My understanding is you can't fully audit GitHub access through the API, which is how accounts were connected, so it's best to assume the worst. Most connected accounts gave full read and write access, so if true, this constitutes serious IP theft concerns.

I personally don't feel like they have been fully transparent with the breach. This was essentially a footnote and buried under a misleading statement "Review provider activity and audit logs; especially AI keys". It should be flashing and unambiguous "GitHub code is possibly compromised".

Furthmore, a simple, nonchalant "we're sorry" and that's it!? Not "we're deeply sorry that we messed up and we take full responsibility, it won't happen again and we are doing x, y, z for our customers..."

I'm switching away permanently. I highly recommend Fly instead, took 30 mins to migrate. Not sure what to do about to do about the IP theft though? I would expect legal action to come of this, possibly a Class Action Lawsuit regarding IP theft.


r/devops • • 4d ago

Ops / Incidents AWS Administrators of Reddit, How are you securing yourself against AI?

3 Upvotes

With Black October happening and all.