Nahidul Kibria

🔭 OpenTelemetry Collector: The Architecture Review I have just published the second article in my OpenTelemetry series about design, architecture and interesting implementation spots in the OTel Collector including: - 🔗 The Signal Processing Pipeline Architecture -📡 OTel Receivers. Prometheus-style Scrapers - ⚙️ OTel Processors. The Memory Limiter & Batch Processor. Multi-tenant Signal Processing - 🚚 OTel Exporters. The Exporting Pipeline & Queues. The implementation of persistent queues - 🔭 How observability is done in the OTel Collector itself. Logging, metrics, and traces - 🔌 OTel Extensions Design. Authentication & ZPages -👷Custom Collectors & OTel Collector Builder - 🚧 Feature Gates Design & The Feature Release & Deprecation Process Enjoy the article & let me know how do you like it 🙌 #observability #opentelemetry #golang #distributedsystems

18

3 Kommentare

Terraform's Import block functionality will help give a lot of companies the ability to bring their existing infrastructure under a unified code base. Now with the ability to use for_each inside their import block this video shows a cool way to utilize that feature to build on top of your existing pipelines without having to re-engineer anything.

4

How does kubelet fetch images from private ECR repos on EKS (v1.26+)? Can we stop for a second to appreciate how come we don't specify any credentials/secrets (imagePullSecrets) whatsoever and yet we can launch Pods in our EKS cluster without any problems? how does this magic work? Let's break it down. kubelet doesn't know at the time of compilation on which cluster it is provisioned, so just like the old adage from David A. Wheeler, it shells it to an external plugin. The kubelet credential provider plugin has become GA in Kubernetes v1.26. And to validate what I just said, if you poke at how kubelet is launched on EKS worker nodes, you will see 2 interesting parameters: --image-credential-provider-config /etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir /etc/eks/image-credential-provider But that's just only half of the story. So how does the EKS credential provider securely fetch the credentials? Let's see how it works end to end: 1. When a Pod is scheduled to run on a Pod, kubelet looks for a matching credentials helper 2. A match is found and kubelet executes the ECR credentials helper 3. The ecr-credential-provider uses the AWS SDK (aws-sdk-go, under the hood) to obtain AWS auth token, which in turn contacts the link-local IMDSv2 to actually obtain that token 4. It uses the obtained AWS token to obtain yet another auth token from ECR using GetAuthorizationToken 5. It returns a CredentialProviderResponse JSON (on stdout) to kubelet with a Username and Password 6. The kubelet uses the credentials from the CredentialProviderResponse document and passes it to its registry client 7. The image is fetched to the node To me it's mind boggling how so many moving pieces are perfectly orchestrated together at a stroke of a pen (with either eksctl or CF or TF). In the next post we'll break down how similarly Pods and workload identities work in concert to access container registries without requiring any secrets whatsoever. Resources: https://lnkd.in/du7Ks--q https://lnkd.in/dhZgZGz4 #eks #cloudsecurity #kubernetes #kubernetessecurity #cloudnativesecurity #sigauth

33

2 Kommentare

"This is one of the exceptions where you have basically no choice, but do it and use a shared local disk state, you should however instruct your users that have problems while scaling to at least offload all the data away to a CDN or an self hosted S3 Storage like minio and setup the executor of the code files to cache them in memory instead of rereading them from disk." https://2.wx1.in/wnc

1

Poorly informed posts alert! #kubernetes #memoryThrottlingFactor #kubernetes 1.27 is introducing memoryThrottlingFactor. This is not a new feature, but finally, features of #cgroup2 are reaching Kubernetes. We have been using this cgroup2 features already quite some time, so we have first-hand experience. Let's #share! You will find currently very poor posts hyping this new feature but don't really explain what is going on. You will hear the claim: It will slow down the process. This is not correct, or let's say it is misleading. What really is happening is that the process is being put under reclaim pressure, which is a #Linux #kernel feature. What happens here is that the kernel will start to, obviously, reclaim memory. It does that by freeing buffers and caches first and next by freeing so-called anonymous memory (allocated but unused memory, i.e., your heap and stack). This will already slow the process down, as it may have to reread data from the disk, which is not in memory anymore. When this is no longer possible, Linux will try to swap out those memory pages and, at the same time, start to stall allocation requests. That means your program that just issued a memory allocation request waits for it to be fulfilled. Depending on your program, this can mean your process is not slowed down; it may be freezing. So you can call this a slowdown, but effectively the process may be waiting, depending on how the memory allocation is implemented. TL;DR; The point is, you can't slow down memory, ok, you could by controlling the latency, but that has nothing to do with managing available memory, but access speed; that may be a topic for another day, let me know if you want to learn something about this. It's stalled, not running at all, until there is free memory again, or it eventually still gets OOM killed. What you can learn from this: - Yes, the new feature will help you to utilize the total available memory better, but there will still be OOM kills, but there is now a proactive management of the memory, trying to avoid OOM kills. - Another important fact that has yet to be mentioned. To actually make cgroup2 features meaningful, you have to activate #swap! For most of the years, the swap was always recommended to be disabled in #k8s. If you still deactivate the swap with these new features introduced, you will gain nothing from them. So swap is **mandatory**, and so is a #fast #SSD that can hold it! - You never want to use this feature for your #database and instead give your database still access priority to this #memory. However, high-performance dbs anyways want a mostly interrupt-free system to run on. So here goes that...

3

Keycloak based on the MYSQL database with Prometheus monitoring that visualize data in Grafana This repository is modified version of https://lnkd.in/dpEdWYma in order to use MYSQL instead of PostgreSQL. https://lnkd.in/dbqDdxUm

6

3 Kommentare

Even 𝗡𝗩𝗜𝗗𝗜𝗔 𝗡𝗜𝗠 𝗶𝘀 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗼𝗻 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀, what is 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 and why is it worth learning 𝗮𝘀 𝗠𝗟𝗢𝗽𝘀/𝗠𝗟/𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿? Today we look into the Kubernetes system from a bird's eye view. 𝗦𝗼, 𝘄𝗵𝗮𝘁 𝗶𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 (𝗞𝟴𝘀)? 𝟭: It is a container orchestrator that performs the scheduling, running and recovery of your containerised applications in a horizontally scalable and self-healing way. Kubernetes architecture consists of two main logical groups: 𝟮: Control plane - this is where K8s system processes that are responsible for scheduling workloads defined by you and keeping the system healthy live. 𝟯: Worker nodes - this is where containers are scheduled and run. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗵𝗲𝗹𝗽 𝘆𝗼𝘂? 𝟰: You can have thousands of Nodes (usually you only need tens of them) in your K8s cluster, each of them can host multiple containers. Nodes can be added or removed from the cluster as needed. This enables unrivaled horizontal scalability. 𝟱: Kubernetes provides an easy to use and understand declarative interface to deploy applications. Your application deployment definition can be described in yaml, submitted to the cluster and the system will take care that the desired state of the application is always up to date. 𝟲: Users are empowered to create and own their application architecture in boundaries pre-defined by Cluster Administrators. ✅ In most cases you can deploy multiple types of ML Applications into a single cluster, you don’t need to care about which server to deploy to - K8s will take care of it. ✅ You can request different amounts of dedicated machine resources per application. ✅ If your application goes down - K8s will make sure that a desired number of replicas is always alive. ✅ You can roll out new versions of the running application using multiple strategies - K8s will safely do it for you. ✅ You can expose your ML Services for other Product Apps to use with few intuitive resource definitions. ✅ … ❗️Having said this, while it is a bliss to use, usually the operation of Kubernetes clusters is what is feared. It is a complex system. ❗️Master Plane is an overhead, you need it even if you want to deploy a single small application. Are you deploying your ML applications in Kubernetes? What are the main pain points you are facing? Let me know in the comments 👇 #MachineLearning #GenAI #LLM #LLMOps

608

33 Kommentare

We just published a new interactive roadmap for learning Terraform. Learning resources and details are to be added shortly. Check it out https://lnkd.in/dtZfGWU7

147

4 Kommentare

🚀 In my latest blog post I explore practical techniques to troubleshoot containerized applications effectively. From `kubectl debug` to ephemeral containers, get the insights you need to keep your environment running smoothly. #Kubernetes #k8s #DevOps #Debugging #CloudComputing #java

3

𝐒𝐭𝐨𝐩 𝐥𝐨𝐬𝐢𝐧𝐠 𝐭𝐢𝐦𝐞 (𝐚𝐧𝐝 𝐬𝐚𝐧𝐢𝐭𝐲) 𝐛𝐲 𝐟𝐨𝐫𝐠𝐞𝐭𝐭𝐢𝐧𝐠 𝐭𝐡𝐞 -𝐧 𝐟𝐥𝐚𝐠!! If you ever need to use namespaces in Kubernetes - please immediately install the kubens cli. I get annoyed with how people introduce bugs in the system just because they forgot to add -n flag. Checkout kubectx + kubens here: https://lnkd.in/eHPHpYYK #devops #kubernetes

25

1 Kommentar

How To Build A Platform Engineering Environment On Kubernetes 👇 ✅ Part 1: https://buff.ly/3u58qhA Part one dives into building the cluster, automating the workflows, and securing the environment. ✅ Part 2: https://buff.ly/47NDG2k Part two dives into platform capabilities like monitoring, observability, and overall cluster management. ✅ Part 3: https://buff.ly/4939GRg Part three dives into various tools and platforms you can use to make platform engineering easier on Kubernetes. #kubernetes #devops #platformengineering

119

1 Kommentar

C# vs Rust vs Go. A performance benchmarking in Kubernetes #csharp #rustlang #golang #comparsion #performancebenchmarking https://lnkd.in/etb4TFwa

15

Without Etcd properly running in your k8s cluster, it’s useless. Etcd is where anything state related is stored for your cluster. It’s the Kubernetes database (well, technically a key store, but it’s the k8s database). Because of that, you’re going to want to not only secure it as much as possible, but ensure you can always restore it. A quick way is with the Etcd command-line tool. ‘Etcdctl snapshot save’ #kubernetes #devops #platformengineering

74

3 Kommentare

🔍 AWS Donates $60,000 to Jenkins: A Commitment to Open-Source 🔍 Recently, I came across a blog on Jenkins.io sharing that AWS has donated $60,000 to Jenkins. This was news to me, and it sparked my curiosity. For those who might not know(Like Me), Jenkins core and its plugins are hosted on AWS, utilizing services like #AWS #EC2 and #EKS. It made me wonder why AWS would donate to Jenkins when Jenkins already uses AWS services. Here's the reasoning: 🌟 Innovation and Collaboration: Open-source projects like Jenkins drive innovation and foster collaboration among developers, resulting in the creation of robust and versatile tools. 🌍 Community Support: By supporting open-source projects, AWS strengthens the community and ecosystem that many of its customers rely on. 💼 Customer Value: Open-source tools like Jenkins are widely used by AWS customers. Supporting these projects enhances the overall customer experience and improves integration with AWS services. 🏆 Reputation and Leadership: Contributing to open-source initiatives boosts AWS's reputation as a leader and strong supporter of the software development community. This donation underscores AWS's commitment to the open-source ecosystem and its continuous efforts to support and enhance the tools that developers around the world rely on. Reference: https://lnkd.in/g6H4G2Zv Hope it was an interesting thing to learn! #AWS #Jenkins #OpenSource #DevOps #Innovation #Collaboration #CommunitySupport #Leadership

1

Streamline Your Kubernetes Workflows with https://lnkd.in/gWnqV5Xc Tired of scouring through Kubernetes documentation for YAML files and concepts? This repository provides a one-stop shop for everything you need to get started with Kubernetes resource creation! Simplified Learning: Ditch the technical jargon. Our repository offers clear and concise explanations of Kubernetes concepts, making it easy for beginners to grasp core principles. Ready-to-Use YAML Files: No need to write YAML from scratch! Find a wide variety of pre-configured YAML files for common Kubernetes resources, saving you valuable time and effort. Contributor Friendly: We welcome contributions from the Kubernetes community! If you have expertise to share or want to help improve the repository, we'd love to hear from you. Clone the repository today and kickstart your Kubernetes journey! https://lnkd.in/gWnqV5Xc Become a Contributor! If you're passionate about Kubernetes and want to give back to the community, consider contributing to this repository. We value all contributions, from bug fixes and code improvements to new YAML files and documentation enhancements. Feel free to connect with me to discuss contribution opportunities! githublink: https://lnkd.in/gWnqV5Xc

14

The State of Ingress: Why Do We Need Gateway API? #WorkSmartWithK8s #kubernetes #ingress #gatewayapi #cncf https://lnkd.in/encUPwE9

9