Ken Krugler

𝐀𝐠𝐞𝐧𝐭𝐢𝐜 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰: 𝐏𝐢𝐨𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐭𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 Recently I happened to see this nice video ( 𝘩𝘵𝘵𝘱𝘴://𝘸𝘸𝘸.𝘺𝘰𝘶𝘵𝘶𝘣𝘦.𝘤𝘰𝘮/𝘸𝘢𝘵𝘤𝘩?𝘷=𝘴𝘢𝘭78𝘈𝘊𝘵𝘎𝘛𝘤 ) by Andrew Ng in which he talks about Agentic workflows. In Language Model (LLM) development, we provide a zero-shot prompt and receive an answer. We may review it and add more prompts to get it to a desired shape. This is exactly what the agentic workflow does. Basically, 𝘢 𝒃𝒖𝒏𝒄𝒉 𝒐𝒇 𝑨𝑰 𝒂𝒈𝒆𝒏𝒕𝒔 𝒘𝒐𝒓𝒌 𝒕𝒐𝒈𝒆𝒕𝒉𝒆𝒓 𝒊𝒏 𝒂𝒏 𝒊𝒕𝒆𝒓𝒂𝒕𝒊𝒗𝒆 𝒘𝒂𝒚 𝒕𝒐 𝒑𝒓𝒐𝒅𝒖𝒄𝒆 𝒂 𝒅𝒆𝒔𝒊𝒓𝒆𝒅 𝒐𝒖𝒕𝒑𝒖𝒕. Moreover, he even claims the output produced by agentic workflow is much better than the output produced single LLM which might sound obvious but what surprised me was, as per the “𝑯𝒖𝒎𝒂𝒏𝑬𝒗𝒂𝒍” coding benchmark, agents-based approach beating plain LLM by quite a margin (see the picture below). The plain 𝐆𝐏𝐓 3.5 was at 48% but 𝑳𝑫𝑩+𝑹𝒆𝒇𝒍𝒆𝒄𝒕𝒊𝒐𝒏 (𝘈 𝘓𝘢𝘳𝘨𝘦 𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘔𝘰𝘥𝘦𝘭 𝘋𝘦𝘣𝘶𝘨𝘨𝘦𝘳 𝘷𝘪𝘢 𝘝𝘦𝘳𝘪𝘧𝘺𝘪𝘯𝘨 𝘙𝘶𝘯𝘵𝘪𝘮𝘦 𝘌𝘹𝘦𝘤𝘶𝘵𝘪𝘰𝘯 𝘚𝘵𝘦𝘱-𝘣𝘺-𝘴𝘵𝘦𝘱) fared at impressive 95.1% with the same GPT 3.5. The plain 𝐆𝐏𝐓 4 was at 67% but 𝑨𝒈𝒆𝒏𝒕𝑪𝒐𝒅𝒆𝒓 (𝘔𝘶𝘭𝘵𝘪-𝘈𝘨𝘦𝘯𝘵-𝘣𝘢𝘴𝘦𝘥 𝘊𝘰𝘥𝘦 𝘎𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘰𝘯 𝘸𝘪𝘵𝘩 𝘐𝘵𝘦𝘳𝘢𝘵𝘪𝘷𝘦 𝘛𝘦𝘴𝘵𝘪𝘯𝘨 𝘢𝘯𝘥 𝘖𝘱𝘵𝘪𝘮𝘪𝘴𝘢𝘵𝘪𝘰𝘯) fared at whooping 96.3% with the same GPT 4.  You can get more information about the evaluation and the models at 𝘩𝘵𝘵𝘱𝘴://𝘱𝘢𝘱𝘦𝘳𝘴𝘸𝘪𝘵𝘩𝘤𝘰𝘥𝘦.𝘤𝘰𝘮/𝘴𝘰𝘵𝘢/𝘤𝘰𝘥𝘦-𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘰𝘯-𝘰𝘯-𝘩𝘶𝘮𝘢𝘯𝘦𝘷𝘢𝘭. He also mentioned the following are the 4 design patterns that are emerging in the agentic workflow. 1.    Reflection 2.    Tool use 3.    Planning 4.    Multi-agent collaboration If you are interested in learning more, instead of me explaining each of these patterns, I would encourage watching the video :) #ai #agent #agenticworkflow #gpt

22

Symflower Launches DevQualityEval: A New Benchmark for Enhancing Code Quality in Large Language Models Symflower has recently introduced DevQualityEval, an innovative evaluation benchmark and framework designed to elevate the code quality generated by large language models (LLMs). This release will allow developers to assess and improve LLMs’ capabilities in real-world software development scenarios. DevQualityEval offers a standardized benchmark and framework that allows developers to measure & compare the performance of various LLMs in generating high-quality code. This tool is useful for evaluating the effectiveness of LLMs in handling complex programming tasks and generating reliable test cases. By providing detailed metrics and comparisons, DevQualityEval aims to guide developers and users of LLMs in selecting suitable models for their needs. The framework addresses the challenge of assessing code quality comprehensively, considering factors such as code compilation success, test coverage, and the efficiency of generated code. This multi-faceted approach ensures that the benchmark is robust and provides meaningful insights into the performance of different LLMs. Read our full take on 'DevQualityEval': https://lnkd.in/guRuBjaB GitHub: https://lnkd.in/grssvCVR Symflower #artificialintelligence #ai #datascience #llms

44

3 Comments

Nvidia Publishes A Competitive Llama3-70B QA / Retrieval-Augmented Generation (RAG) Fine-Tune Model Llama3-ChatQA-1.5-8B and Llama3-ChatQA-1.5-70B are the two versions of this state-of-the-art model that come with 8 billion and 70 billion parameters, respectively. These models, which were first trained with Megatron-LM, have been converted to the Hugging Face format for accessibility and convenience. Building on the success of ChatQA, a family of conversational QA models with performance levels comparable to GPT-4, Llama3-ChatQA-1.5 was developed. ChatQA greatly improves zero-shot conversational QA outcomes with Large Language Models (LLMs) by introducing a unique two-stage instruction tweaking strategy. Full read: https://lnkd.in/guPxrkjX Llama3-ChatQA-1.5-8B: https://lnkd.in/gzfRhXPr Llama3-ChatQA-1.5-70B : https://lnkd.in/gh42ji9y #artificialintelligence #rag #llms #ai NVIDIA NVIDIA AI

55

3 Comments

Today's dispatches from quantum: ⚆ Quantum computing for protein structure prediction as demonstrated by framework that accurately predicts Zika Virus NS3 Helicase ⚆ QAOA gets a makeover — QIRO bypasses locality issues, matches classical heuristic performance ⚆ Central New Mexico Community College to offer first of its kind quantum computing bootcamp amidst concerns for related US programs Plus, IBM releases execution modes, QTunisia hosting certificate-awarding workshop for all levels, and Massachusetts Institute of Technology technique for atom-proximity has big implications for quantum. We're THE quantum computing newsletter, bringing you daily insights into the latest news & research. With leaders in academia, founders, and subscribers from top companies like Xanadu, SandboxAQ, and Qunnect, you're in good company. Join now at www.thedailyqubit.com and never miss a single qubit. #quantumcomputing #quantumphysics #innovation

27

BONDS RAIDERS EPISODE XIII: YIELDS CRASH UP WATCH THE RECORDING IN AUGMENTED WEB FREE LINK (NOT GATED - the usual mind-map format) https://t.co/8lcGyRjgkv https://lnkd.in/dWSExq68 WHY CURRENCY PLUNGING VS COMMODITIES RESULTS IN YIELDS CRASHING UP AND RATES SENSITIVE ASSETS TO CRASH WITH IT -THOMAS TOOKE - FITCH IS ON SOMETHING The Concept of ccy falling & impacting rates is probably best explained by T. Tooke. In his masterful book of 1826 he differentiates between return in kind & return in money explaining that return in money could be positive while return in kind (commos) could be negative. The “term premium” is simply in Tooke’s parlance the compensation for the alteration of the value of money. In Emerging Markets, if a government does bad policies, the market smells “repaid in confettis” and this forces the yield to go higher on the long end of the curve. The very large plunge of the USD versus commodities means we are going to be repaid in confettis, and the "returns in kind" are hugely negative. This is what Tooke explained. The rates rise for the compensation of the alteration of the value of money. Means of circulation are not meant to speculate with US treasury, it is meant to purchase goods and services and more importantly basic necessities. A key concept to understand the “alteration” of the value of money is illustrated. If you make 5% (in fiat USD money) but even AFTER the compounding of interest you buy a lot less commodities (return in kind) your return in kind is very negative and that is the case today. The return in kind or in commodity for people who experienced a partially convertible system prior to 1797 in the UK is quite obvious and the same thing as the return in money since money is pegged to commodities via convertibility in Gold. A return in money disconnected from commodities was experienced by in the fiat episode between 1797 to1821 in the UK and back to convertibility (this time fully convertible). Return in commodity is this: I buy a Gold bar and I get 4% on top of my Gold bar in GOLD weight. It is a tall order today, but that is RETURN IN KIND and you can understand easily why it is very negative today and forces rates higher. Evidently during periods of war countries have a propensity to shift to fiat currency and the return in money can be positive while in kind or in commodity quite negative. I let you ponder the current situation of Gold prices and interest rates and figure for yourself if we have positive return in commodity today. Rest of my analysis in augmented media available here... https://t.co/8lcGyRjgkv Michael Sell Nicholas Stuart Jim Wylie Adnan Khan Tony Orlov Kamal Masud, CFA Carmen Law, AICPA, CICPA, CFA Mark Jacobs Leonard Katz Brooks O'Neil Thomas Cardwell Timothy William Boardman Bader Al Monawer Jonathan Fouts Christopher Lewis Bruce Blechman Willy Sistig Barry C. Mohr Rafael Elias Brandon Rossetti Per-Arne Weiner

1

I remember learning category theory in undergrad and wondering: when would this ever be useful? Turns out it's very useful for program synthesis and software modeling -- thanks Kristopher for the great talk!

9

OpenPipe Introduces a New Family of ‘Mixture of Agents’ MoA Models Optimized for Generating Synthetic Training Data: Outperform GPT-4 at 1/25th the Cost Quick read: https://lnkd.in/gfxdiDT5 OpenPipe’s MoA models have excelled in rigorous benchmarking tests, achieving notable scores on LMSYS’s Arena Hard Auto and AlpacaEval 2.0. The MoA model scored 84.8 on Arena Hard Auto and 68.4 on AlpacaEval 2.0, indicating its superior performance in generating high-quality synthetic data. These benchmarks are critical as they represent challenging user queries that test the robustness and adaptability of AI models. The MoA model has been benchmarked against various GPT-4 variants in real-world scenarios. Results showed that OpenPipe’s MoA model was preferred over GPT-4 in 59.5% of the tasks evaluated by Claude 3 Opus. This is a significant achievement, highlighting the model’s effectiveness and practical applicability in diverse tasks encountered by OpenPipe’s customers...... OpenPipe

43

3 Comments

A great example around #semantics - love the treasure map analogy. 🤨 Imagine being trying to find treasure by following directions in a completely different language, with different references (Km instead of miles), and wanting to do this effectively? Ooh - and perhaps you have a Parrot (Generative AI) giving you 'wisdom' in your treasure hunt by spewing out interesting, but non-grounded facts about your journey? 🏅 Having an agreed to language reference that allows you to connect the dots and understand the people, technology, places and data you interact with - regardless of system or format is the best way to get to that treasure. It's all about outcomes!

8

In the spirit of sharing more code, I have posted an example of computer generated hologram rendering using C++. This computation expensive process sees the RGB light/wavefront contribution of every point in a scene on every pixel to construct holographic interference patterns. This example loads a point cloud as a source model and generates rows of binary float images. I have not yet verified the results with a true holographic projection system. The github repo is https://lnkd.in/gPkjZZy6.

15

Scale3's LangTrace.ai SDK now supports native open telemetry(o11y) tracing for LlamaIndex This allows developers to utilize o11y tracing to ship, debug, and evaluate AI applications more efficiently across various observability tools such as SigNoz, Grafana Labs Labs, Datadog, and honeycomb.io 🔍 Why is o11y tracing crucial? With open telemetry (a CNCF project), engineering teams gain a standard data model for telemetry that supports high cardinality. This is vital as it provides flexibility, prevents vendor lock-in, and helps in quicker debugging and insights, especially important in systems utilizing LLMs and VectorDBs. 👨‍💻 Meet Langtrace.ai  Langtrace.ai from Scale3 helps you gain confidence with LLMs before taking your application to production. If you are using OpenAI's APIs or vector DB's for any of your products, this could help with: **Tracking the responses made by the LLM **Evaluating and measuring the accuracy of responses **Identify bottlenecks. Best part - It's full Open Source and has native Open Telemetry support. https://langtrace.ai/ https://lnkd.in/grYfWsbP We’re thrilled to roll out native o11y support for LlamaIndex, making it easier for developers to harness the power of AI while maintaining transparency and control over their applications. Interested in o11y tracing for LLM applications or want to contribute to Langtrace? Let’s connect! 🚀 https://langtrace.ai/ # OpenTelemetry #ai

16

♊️Pairwise comparisons - our newest evaluation feature that I'm super excited to unveil This is NOT the same as typical comparison views ✨This is native pairwise comparison functionality - brand new, pretty unique, and pretty powerful ❓How is it different and why am I so excited about it? 1⃣The previous way we had for comparing runs would evaluate each run individually, and then let you view the results side by side For example, if you were evaluating summarization, you'd create an evaluator to score a summary on a scale of 1-10, and then compare the results 2⃣This is fine/best for a lot of use cases. That's why we implemented it first, and why most (all?) other eval tools focus on this 3⃣However, it does have some downsides. Oftentimes it's hard to come up with a prompt to perfectly score summarization on a scale of 1-10. This task is hard for even humans to do! 4⃣It's often easier to not score an LLM generation in a vacuum, but rather compare it to another generation and say whether it is better or worse. ⭐️There's a reason the most trusted LLM eval (LLMsys arena) does this type of "pairwise comparison" - it's easier and more intuitive to do⭐️ 5⃣We've taken this idea and added in native support for "pairwise comparisons" in LangSmith. Specify two runs, as well as a comparison metric. We show how to do this using LLM-as-a-judge (with prompts from the LLMsys chatbot arena) Check out the blog and Lance Martin's video in the tweet below for more information and a great walkthrough of how to use

33

4 Comments

Introducing Pairwise Evaluations in LangSmith Evaluating LLM application outputs can be challenging when done in isolation. It's often more effective to rank outputs preferentially, a method beneficial not only for human evaluators but also for LLM-as-a-judge systems. Example: instead of grading an LLM output's 'vagueness' on a 0-1 scale, rank two outputs by which one is more vague. With this in mind, I'm pretty excited about this powerful new feature in LangSmith: native support for running and visualizing pairwise experiments! Docs here: https://lnkd.in/dGHgc9MZ

24

New Methods for Efficient Finetuning of LLMs!

10

SB 1047: A balanced approach to AI safety or a misguided attempt at regulation? I examine the pros and cons of California's controversial AI bill. 🧠💥 https://lnkd.in/gV6vChHZ

6

A few takeaways from SuperAI conference day 2: On open models: - Emad Mostaque emphasized the safety of open models and how releasing open source pushes safety standards and avoids single points of failure. Small edge models can suffice for many use cases. - Vipul Ved Prakash noted increased investment in inference efficiency, enabling more cost-efficient production apps. - Sean (Xiang) Ren highlighted that value comes from applications (the full software and data stack), not just the models themselves. On AI venture trends: - Harshjit Sethi predicts the rise of regional foundation model players as governments push for sovereignty and foreign companies won't collect local datasets for smaller markets. - Vishal Harnal sees AI becoming table stakes for software companies, as opposed to a distinct vertical. He uses AI personally with diverse applications from deal analysis to generating hip-hop album art for birthday party guests. - Prayank Swaroop observes the paradigm of using specialized models for different tasks emerging. He advises focusing on unique datasets. - Geetali Kumar is most interested in AI use cases in healthcare. Looking ahead, Girish Dilip Patil from AWS identified 3 key trends: natively multimodal models, autonomous agents, and small on-device models for privacy-sensitive use cases ( 👋 Opilot). SuperAI also had great food and drinks, cool cyberpunk exhibits, EDM music, top-notch people-watching, some unhinged speakers, and of course it was a great opportunity to meet interesting people. Overall I had a great time, thanks Prahlad Jaya for the ticket!

26

Evaluations of LLM applications is a big buzzword right now... but when do you really need new tools? Our latest blog post from Wenzhe Xue cuts through the noise to show with examples how built-in libraries such as pytest are often enough to get started. As complexity grows along the following dimensions, we're here to help! Metric based ➡ Human review Off the shelf eval or LLM as a judge ➡ Custom eval models trained on your data Offline ➡ Online, realtime https://lnkd.in/gGb3Sjah

22

I was chatting about translation with Andrew Ng at an event several weeks ago. It's fun to see him post a workflow with the seed of the AI Review concept that's live in LILT today. === This is along the path to what many in the industry are doing. Two comments: 1. Replace step 2 with a formalized reflection approach such as AutoMQM. 2. BLEU isn't discriminative at the accuracy levels modern MT systems achieve. Use a trained metrics like BLEURT or COMET to see real differences between systems. 20 years ago we called this idea automatic post-editing, where you'd train a second string transducer to rewrite initial MT output. However, those systems were very difficult to train and cascade. Using LLMs in cascades is significantly more effective.

42