Andrew Lamb

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

Included in this article is the mention of vector databases. Unsurprisingly, Michael Stonebraker and Andy Pavlo feel there is not a need for vector-specific databases. They believe relational databases will adopt their features, which is already happening. Check out the details here: https://lnkd.in/gyq_4WjP

27

1 Comment

Possibly the best explanation of direct-style Scala that I have read till date. Personally I still favor the more FP, more compositional effect based programming model. But looks like direct-Scala is a definite step towards democratising functional programming without the learning curve of monads and applicatives. ZIO had done a lot in abstracting away the algebraic structures from the programming model through use of powerful combinators. Direct-style may be the next logical step .. and definitely Scala 3 may prove to be the right vehicle to drive this adoption.

19

How we index timeseries data at Datadog with Rust and RocksDB https://lnkd.in/e8x_RUmz

56

3 Comments

My favorite part about working at Amplitude is we aren't afraid of making big, bold bets, all in service of our customers and today is one of those exciting launch days! We were one of the early adopters of Snowflake at Amplitude (early 2017). We made a big, bold decision of transitioning over from Redshift, which was a well-established data-warehouse to Snowflake, which was a new product and probably had a few hundred customers. It felt like the right decision because the technology was miles ahead of any other competitor (Google Bigquery, Amazon Redshift, etc.) and we had conviction that this is where the world is headed. Fast forward a few years, and Snowflake has completely redefined the modern-day cloud data stack. Today, we are announcing a first of it's kind integration with Snowflake. You can get all the power of Amplitude without moving your data, without any engineering effort for FREE. We are just getting started. In this age of #ArtificialIntelligence, there's so much we can accomplish by combining Amplitude's behavioral data with the customer and business data that sits in Snowflake. #productanalytics #datawarehouse #snowflake #amplitude #analytics #ai

28

In part 2 of the ‘Inside Elastic’ blog series… Cory Mangini offers a step-by-step view into how Elastic's own Field Engineering team built the knowledge library of 300k documents required to power a RAG- based Support Assistant. Dive into this blog to learn more → https://gag.gl/6P4RzW #RAGAI #Elasticsearch #GenerativeAI

3

Introducing Structured Outputs: By popular demand, we are introducing the ability to provide your own JSON schema to customize the output structure of the OpenAI API response https://lnkd.in/g7g3xmvb

653

25 Comments

Tiered storage for Kafka is a classic tarpit idea. It makes all the sense in the world, but it doesn't work in practice. Check out our latest blog post to learn why. https://lnkd.in/eG9BJ95t

107

8 Comments

I just realized: we should start saying "build memoization" rather than "build caching." We aren't really talking about least-recently-used or time-to-live algorithms, we're talking about "don't bother recompiling this piece if none of its inputs changed." (It has been puzzling me that some people are so deeply fearful of "build cache" as a concept, and honestly I think it's mostly because most build systems have non-hermetic state scattered all over the place. Maybe a slightly more precise term for the type of caching we're concerned with would clarify the semantics in question.)

83

9 Comments

This overview of Generative AI and Retrieval-Augmented Generation by Gwen (Chen) Shapira of Nile Database is fantastic. Highly recommend it for other infrastructure nerds like me who want to know what's going on in the AI world. #generativeai #rag https://lnkd.in/gJum7_qa

10

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Are you someone who finds distributed systems challenging? If so, I highly recommend checking out Martin Kleppmann's distributed systems lecture series on YouTube. It's an absolute goldmine of information! In this series, Kleppmann covers a wide range of fundamental concepts in depth, including: 💻 Physical clocks and logical clocks (Lamport and Vector clocks) ⚡ Causality and the happens-before relationship 📡 Broadcasting algorithms 🔄 State Machine Replication ☑️ Consensus algorithms like Raft Whether you're a student, a professional, or just someone curious about distributed systems, these lectures are an invaluable resource. Kleppmann's clear explanations and real-world examples make even the most complex topics accessible. Don't struggle with distributed systems any longer. Leverage this incredible resource and gain a solid understanding of the core concepts. Trust me, it'll be a game-changer! #DistributedSystems #MartinKleppmann #LecturesSeries #TechEducation

6

We released a new update of our Visual Studio Code Java extension. Go try it! Bunch of new improvements and already supporting JDK 23 Early Access builds as the extension is directly leveraging javac. https://lnkd.in/g8BKsNjS #java #IDE #OpenJDK #VSCode

61

1 Comment

This article provides a helpful, straightforward explanation of the options for improving the output of LLMs for specific types of use cases - worth a read.

2

Great SRE Interview Questions 

4

Enjoy reading the Datadog blog about how they adopted a new indexing strategy for time-series data, which aligns closely with what GreptimeDB is developing: 1. Written entirely in Rust. 2. Always build inverted indexes for time-series data to avoid full table scans. 3. Shards data and indexes by ranges. 4. Accelerates queries using parallel processing combined with partitioning and indexing. Furthermore, GreptimeDB aims to unify logs, metrics, and events, supporting log-searching capabilities as well. Additionally, all of this data and indexing can be stored on object storage. The Datadog blog is here: https://lnkd.in/gr36NfQr Indexing Techniques in GreptimeDB: https://lnkd.in/gjwp3hHw #Datadog #GreptimeDB #Monitoring #Observability

4

Imagine this - you're onboarding a new engineer onto a codebase, and they hit every error/exception after every CLI command. They're trapped in the proverbial "config hell". Then a security vulnerability for their toolchain comes out, and they have to go through "config hell" all over again! And then when it's finally working, they realize they need more compute, load their application onto some beefy cloud hardware --- but forget to turn it off ($$$!!) Multiply this problem by the number of engineers you're onboarding/supporting and this is the nightmare scenario of wasted time and money for Enterprise CTOs/CIOs. Coder solves these problems with it's Cloud IDE -- getting engineers to their 'flow state' faster. Thrilled to enable developer productivity with the Coder team.

19

2 Comments

You too can follow along with my new article and load 1 TiB of data / 2 billion rows into a SingleStore Trial cluster in under 20 minutes. In the case of data loads, impatience is a virtue. https://lnkd.in/grFBJvrD

45

4 Comments

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

19