Jared Rosoff

The afternoon Keynote @ Stripe Sessions was Fireside Chat between Patrick Collison (CEO of Stripe) and Jensen Huang (CEO of NVIDIA). Patrick was an excellent moderator — asked Jensen really probing, nuanced questions that Jensen answered with his usual sharp wit and unique (and impressively well rationalized) perspective. A couple of quotes among many that stuck with me: - “Current CPU installed market is $1 trillion of hardware, all of which will be upgraded to GPUs over the next 5 years. And I would estimate about another $1 trillion will be spent on top of that during the same time period as companies try to do more with AI. Add those together — and I think at least $2 trillion will be spent on GPUs in the next 5 years.” - “I like to go after Zero-Billion Dollar Markets — nonexistent markets that I envision & can rationalize well, but will be very hard, expensive & time-consuming to achieve.”

1

The past two years at StarTree, working with the Apache Pinot community, have been fantastic. I’ve been able to get to know some incredibly talented people, and it’s validated my perspective that the advent of event-driven architecture is real and pressing, and even more interesting than I realized. Looking back, I’d never spent that much time focusing on analytics and the huge set of tools and concerns that attend it. It’s a critical ecosystem with a history deeper than digital computing itself, and I anticipate a lot of fruitful work with the Pinot community in the months and years to come. That being said, it’s ironic that my time working with this community has crystallized an important idea I’d been developing since before I joined it. The disruption introduced into our world by the perfect storm of the cloud, microservices, and ubiquitous Kafka has created an entirely new approach to building business software. It’s new, uncharted territory to be explored—and I want to help map out as much of it as possible, together with the hundreds and thousands developers who are doing the work of building applications in this new world. That’s why I’ve decided to transition from StarTree, and return to Confluent as VP of Developer Relations. The most valuable thing we can do as developers is write differentiated code that provides meaningful value to our customers or users. That code sits atop a huge stack of what is, with respect to their businesses, undifferentiated software—various kinds of frameworks and infrastructure components—that solves hard problems that application developers shouldn’t have to spend time on. They shouldn’t build that software, but they need to know how to navigate the terrain and make choices in the space they can trust. The problem is that the event-driven revolution has thrown a somewhat stable stack into relative disarray, so some of us are lured into creating little bits of data infrastructure in that unknown area between our code and Kafka. Those of us who are able to avoid that temptation still aren’t totally clear on what all that terrain looks like, and what infrastructure should even live there. History suggests this could take another five to ten years to stabilize. It will require the input of many communities, vendors, and a world full of developers. In my opinion, Confluent is the company that is best positioned to significantly move the needle on this problem, and that’s why I’m excited to (re)join them. One company can’t do it all, but their vision of the Data Streaming Platform is exactly what I’m talking about here. The opportunity to lead a world-class developer relations team at the company which has pioneered the journey so far, and is building the most comprehensive vision for its next stage, is a tremendous one. I’m delighted to be back, and eager to do my part to help chart our course to the future. It is, as I am wont to say, a wonderful time to be alive.

915

291 Comments

On April 18th, just last week, Meta unveiled Llama 3, lauded as "the most capable openly available LLM to date". This remarkable achievement by the Meta team comes hot on the heels of Llama 2's release last July. Notably, the new models surpass their predecessors and competing offerings from other providers in terms of performance. Meta's commitment to open sourcing these powerful models while prioritizing model safety and responsible usage is commendable. (https://lnkd.in/ghv3RJRz) Yesterday, Microsoft introduced Phi-3 Mini, a member of the Phi-3 family of models, promoted as "the most capable and cost-effective small language models available". According to Microsoft, the key to achieve high performance in such a small package lies in the quality of the training data. The model has been open sourced and is now available on Ollama and Hugging Face. (https://lnkd.in/gKgSmrBn)   These compact yet powerful models serve as invaluable tools for creators eager to bring their visions to life or simply experiment with cutting-edge technology, particularly for those with limited access to resources.   Shortly after the release of Llama 3, reports began surfacing of enthusiasts running the 8B model on a Raspberry Pi 5 equipped with just 8GB of RAM—a modest $80 single-board computer. (https://lnkd.in/gbeQPhQ2) While the performance understandably reflects the limitations of such a platform, the fact that the model runs and generates an output is nothing short of magical. Similarly, the Phi-3 Mini model can run on ubiquitous devices such as smartphones.   Reflecting on my own journey, I recall the pivotal role that early access to a low-cost computer played in shaping my passion for coding. The gift of an 8-bit Commodore 64 home computer ignited my curiosity at a young age. Later, with the guidance of an outstanding Computer Science teaching staff at my high school IISS Marconi-Hack Bari, I embarked on ambitious projects like a networked multiplayer version of Battleship written in Turbo Pascal and 8086 Assembly, complete with custom sprites and lo-res graphics. These formative experiences paved the way for my career, eventually leading me to various engineering and leadership roles.   It all began with a simple, low-cost, modestly capable computer used for experimentation—a gateway to discovering my life's passion.   I encourage educators and parents alike to consider setting up similar low-cost experimentation environments. By introducing younger generations to open-source AI technologies, we can nurture their creativity and help them uncover their true passions. It's not just a weekend project—it's an investment in their future and the future of innovation and discovery. #education #youth #AI #LLM #future #innovation

24

Performance is key! As we cannot share the detailed work with (former) customers of Rockset (acquired by OpenAI), we decided to implement the Rockbench benchmark to check how streaming ingest between CrateDB and Rockset (acquired by OpenAI) compares. The results are very insightful. We see: ✅ CrateDB achieves 6-9x lower latencies than Rockset for streaming ingest. ✅ CrateDB outperforms Rockset on the same hardware while saving about 20% costs. ✅ When volumes increase, the latency increased linearly in Rockset, while remaining mostly flat in CrateDB. As a plus: 1️⃣ Achieve full high availability due to the distributed nature of CrateDB (benchmark results are achieved including replication in CrateDB) 2️⃣ Achieve higher throughput on the same amount of CPUs as Rockset (at 50% of memory needs) 3️⃣ Full open-source solution that prevents you from market shifts On side note: If you are thinking to switch to Elastic/OpenSearch, this benchmark has been created to compare Rockset vs. Elastic, so CrateDB is also faster than Elastic for streaming ingest cases! Read the full comparison here: https://lnkd.in/d_nEMDkx

29

1 Comment

I ran across this post in the Changelog newsletter: https://lnkd.in/eCdjVekq - I like the name slop for unwanted AI-generated content. I love LLMs. They are part of my daily workflow, and I've been exploring them more seriously while on my sabbatical. I wouldn't misrepresent generated content without critical review as my own, but the web seems to be filled with more generated garbage (or 'slop')—content that looks fine at first but isn't accurate or helpful. LLMs (and other generative AI) should help us to make things of higher quality, not just churn out slop more quickly.

11

1 Comment

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

19

I had a great discussion with Abi Noda on his Engineering Enablement podcast! Among other things, we dove deep into how our Platform Engineering team at SiriusXM prioritizes projects to boost developer speed and align with business objectives. Listen to the full episode here: https://lnkd.in/eN57Hync

34

🍜Anybody up for some ramen? I am excited to share that I am back in the garage, building again (it’s actually a house in the Berkeley Hills). We are taking the wraps off of Supercog AI, a new startup focused on solving Application Integration using LLM powered agents. My co-founder in this new venture is a great friend of mine (and a former co-founder partner), Alex Osborne. Alex and I have worked around the application integration space for a long time, and we are unrealistically excited about applying the GenAI stack to this problem. Not many people relish the task to get some bits to move from system A to system B. But it’s a critical job to be done, and that’s led to a lot of brittle scripts, a myriad of inflexible “no code” tools, and a lot of hand crafted SQL. But the power of the Large Language Model, trained on a huge corpus of information, offers a remarkable tool to solve this problem. The LLM you use today already knows the APIs of hundreds of popular systems. It knows the SQL dialects of every major database. It understands the semantics and data schemas of many popular SaaS systems. It understands much of the specific domain in which your business operates. I like to call the AI revolution “the last platform shift”. That may turn out to be hyperbole, but I definitely believe we are in the very early stages of seeing what this new stack can enable. I know we are hardly the first folks to claim that “amazing stuff is coming!”. But this is why Alex and I have decided to focus on a real and hard problem. If we can prove that LLMs can power a new way to solve this problem, it will be the first successful *new* approach in 15 years.. We will have much more to share about this new platform soon. In the meantime, if you’re interested in getting a peek - or even better if you have application or data integration tasks that you would like some help with - please reach out because we want to help. #genai #startups #backinthegarage

163

37 Comments

We are excited to announce the launch of Ursa engine, our Kafka API-compatible data streaming engine that is built on top of Lakehouse. Ursa is not something entirely new; it is built on our many years of experience building and operating #ApachePulsar and #ApacheKafka. To learn more about the #Ursa engine, I recommend our #PulsarSummit keynote presentation and announcement blog post. Here, I’d like to delve deeper into some of its features: 1. Making Kafka API a first-class citizen: In #ApachePulsar, we developed a unified messaging model combining queuing and streaming semantics into one protocol, aimed at enhancing data sharing between business applications. Although powerful and increasingly adopted by enterprises and unicorns, the requirement for application rewrites is impractical in today’s economy. Instead of building data gravity around a protocol, we open up access to the robust stream storage layer by extending support to the Kafka protocol, transitioning from a single protocol to a multi-protocol platform, facilitating greater flexibility and integration. 2. Achieve Stream-Table duality via Lakehouse Storage: Data streams commonly end up in data lakehouses, where integrating and maintaining them can be complex and resource-intensive. Ursa simplifies this by storing data streams as lakehouse tables in object storage. This integration not only streamlines operations but also allows seamless access to operational data for analytical processes. Thanks to its multi-modal storage engine which fosters data sharing across teams and departments. 3. Object storage as the primary storage solution: To better align with cloud economics, particularly for large workloads, Ursa makes ZooKeeper and BookKeeper optional and utilizes a cost-optimized write-ahead log based solely on object storage with leaderless and stateless brokers. This architecture leverages object storage for both data replication and storage, minimizing inter-AZ data transfers and reducing costs for latency-relaxed, high-volume workloads. By integrating these features at the multi-tenant level rather than at isolated cluster levels, Ursa eliminates costly data duplication and movement across clusters, allowing teams to map directly to tenants and be able to share data across teams and departments more efficiently. We believe that the future of data streaming platforms lies in being open and multi-faceted, fundamentally different from traditional data platforms. They are designed not just to store and process data, but to enhance organizational capabilities by fostering data sharing across the organization and integration with other systems. More and more data streaming platforms will look more like Ursa, with multi-protocol, multi-tenant, and multi-modal capabilities. Check out our announcement blog post about Ursa: https://lnkd.in/ghEKTYE6 #ApacheKafka #Lakehouse #DataStreaming

100

1 Comment

A very interesting article on DSPy and CoT. As complex queries becomes more popular the need to build pipelines is becoming more of a necessity. "When we are building a pipeline, it is not just me trying to convince an LLM to give an output in a certain way, but more of the output should be restricted in such a way that it can work as input for other modules in the bigger pipeline." The shift from prompting to programming to LLM as a module adapting its behavior automatically is going to be very interesting.

9

Protecting users and developers on #GooglePlay is a top priority. In 2023 alone, we prevented 2.28M policy-violating apps from being published on Google Play in part thanks to our investment in new and improved security features, policy updates, and advanced machine learning and app review processes. Learn more about our initiatives that help keep you safe on Play and #Android: https://lnkd.in/g4-b2xYq

93

1 Comment

Recently, Eric Ignasik and Michael Sols from The Software House asked me about my first steps as a VP of Technology at Habito. It's been a fun 8 months in a supportive environment with fantastic people! 😉 This interview discusses a few improvements ‌I made with a big emphasis on why servant leadership is important to me. Hope you’ll learn from it, ➡️ https://hubs.la/Q02Bw-Bv0 #softwaredevelopment #cto #softwaredelivery

52

4 Comments

I'm really intrigued by this idea (not mine) that some bitcoin mining companies (Iris, Cleanspark, etc) have successfully built low-cost power shells that can now be leveraged by AI compute needs -- where power itself is becoming the gating factor in the continuing growth of AI. I think this will become a real 2025 story -- as the gating factor of "chip shortage" lessens, and the gating factor of "cheap power" rises in significance. There's not really a venture capital angle here, but there is a public market one. Perhaps more importantly, it becomes a touchpoint between "crypto" and AI. Something to ponder and keep an eye on (full disc: I own a bunch of bitcoin miner stocks, including those mentioned above).

9

2 Comments

Exciting announcing the release of PyTorch/XLA 2.4, with several notable improvements to address developer challenges that build on the last release. Improvements to Pallas, a custom kernel language that supports both TPUs and GPUs. #ai #pytorch #googlecloud #tpc #gpc

1

A successful systems design and delivery strategy begins by assuming that change is likely the only constant. Be adaptable, or fail.

2

pgvectorscale: A Game-Changer for Vector Similarity Search 🚀 Timescale's #pgvectorscale extension for PostgreSQL has achieved a remarkable milestone, matching the performance of Pinecone, a dedicated vector database, while being 75% more cost-effective. 💰 In their latest blog post, Timescale showcases the impressive speed and efficiency of pgvectorscale, which leverages the power of PostgreSQL to deliver lightning-fast vector similarity search. This breakthrough opens up new possibilities for applications like semantic search, recommendation systems, and more. 🔍 Key takeaways: 🌟 pgvectorscale now matches Pinecone's performance in vector similarity search 💸 pgvectorscale is 75% more cost-effective compared to Pinecone 🔧 Easy integration with existing PostgreSQL databases 🌐 Scalable solution for handling billions of vectors ⚡ Optimized for fast and efficient search results pgvectorscale's seamless integration with PostgreSQL allows developers to incorporate vector similarity search into their applications with ease, while benefiting from the robustness and scalability of a proven database system. 🗄️ If you're looking for a high-performance, cost-effective solution for vector similarity search, pgvectorscale is definitely worth exploring. 🔍💡 #pgvectorscale #postgresql #vectorsimilaritysearch #semanticsearch #recommendationsystems #timescale https://lnkd.in/gsfgCK32

5