Google Cloud’s Post

💡 "⚡ Up to 4X lower input price than GPT-3.5 Turbo, with context caching enabled for inputs larger than 32,000 characters ⚡ On average, 40% faster than GPT-3.5 Turbo when given input of 10,000 characters ⚡ 1 million token context window, which is approximately 60x bigger than the context window provided by GPT-3.5 Turbo"

Gemini 1.5 Flash 3 key differentiations: - 1 million token context window, which is approximately 60x bigger than the context window provided by GPT-3.5 Turbo - On average, 40% faster than GPT-3.5 Turbo when given input of 10,000 characters - Up to 4X lower input price than GPT-3.5 Turbo

Research by Gregory Kamradt suggests that GPT4 Turbo looses retrieval accuracy - after 72k context tokens - and when the factoid is placed at a document depth of < 50%. That suggests that switching the first and second half of a given text would impact the outcome. It also suggests that flipping the halfs, running the call for both orderings and merging the outcomes will likely yield good results (and double - triple the costs). It also suggests that the previously observed "lost in the middle" problem evolves into a "lost in the middle of the top half" problem. Lastly, I'm personally rather surprised, the accuracy is overall rather high. The effective context window is now at 72k, having started this year at just 4k. Progress! Thanks to Gregory for his insights! Lost in the middle problem: https://lnkd.in/eVvw_aVA

GPT-4o Benchmark Results Showing that it is Truly a Next-Generation Model We tested the performance of GPT-4 Omni (model name: gpt-4o) on our finRAG dataset, the results show that this is truly a next generation model that does not seem to have some common issues that previous generation models had, making it possibly the first model suitable for reliable enterprise use. https://lnkd.in/ehNvXSzF

GPT-4o Benchmark Results Showing that it is Truly a Next-Generation Model We tested the performance of GPT-4 Omni (model name: gpt-4o) on our finRAG dataset, the results show that this is truly a next generation model that does not seem to have some common issues that previous generation models had, making it possibly the first model suitable for reliable enterprise use. https://lnkd.in/e7WRQGM9

👀 Just saw phi-3 drop. It's outperforming other models across the board. This is big news on a few different levels: * Phi-3 mini: Edging out on benchmarks versus Llama 3 8B while crushing Mistral and Gemma at 3.8B parameters. * Phi-3 small: Solid across tasks Nearly matches GPT-3.5 at 7B parameters. * Phi-3 medium: Crushing the Mixtral at 14B parameters versus the MoE 56B total parameters with about 14B active ones for a given response! 🔄P.S. Planning to give phi-3 a try? GPT-5 coming soon? What do you think?

🚀 Meta's Llama 3.1 is here, shaking up the AI world! 🌋 The open-source community just got a massive boost with Llama 3.1 405B matching (and even beating) the big closed-source players like OpenAI's GPT-4 and Anthropic's Claude 3.5 Sonnet. Key highlights: • Multilingual support for 8 languages • 128K context window • Trained on a whopping 15 trillion tokens • MMLU scores: 405B (85.2), 70B (79.3), 8B (66.7) My guess is an open-source 10x8B model based on the llama 3.1 family of models will surpass the last predecessors of Closed Source Model Ig Closed Source Models won't be sitting ducks either, Man it's gonna be a fun year ahead. Why open-source is winning: • Faster development through community feedback • Collaborative building process • Quicker iterations and improvements As Mark Zuckerberg said having an approach like how Linux OS got developed making it one of the safest Operating Systems in the world to maintain alignment and stuff OpenSource Collab is a must. Hardware requirements (prepare your wallet 💸): • 405B model: 810 GB VRAM (fp16) • 70B model: 140 GB VRAM (fp16) • 8B model: 16 GB VRAM (fp16) Is this the beginning of the end for closed-source AI dominance? 🤔 What do you think? Is open-source the future of AI, or are there still advantages to keeping things under wraps? For More details check out this blog: https://lnkd.in/gBrdb4-c by Vaibhav Srivastav, Omar Sanseviero, Pedro Cuenca and Daniel Vila Suero Robert Ranson Reuven Cohen Alexander Gross Nick Ruest Scott Howard 🍁/acc John Petty Jordan Legg Nazim Girach Sergio Villanueva-Meyer Joe Anandarajah Haiyang L. James Rowe Jan Daniel Semrau

Stumbled upon this tweet this morning where Greg Kamradt had completed a comprehensive study to evaluate the recall performance of GPT-4 with 128K tokens. Here's a summary of what he found: 📉 Recall Performance Degradation: Above 73K tokens, GPT-4's ability to recall information started to diminish significantly. This works out at about ~110 pages (if you do about 500 words a page) - so again we are still somewhat limited to what can be ingested in one go. 📍 Importance of Document Positioning: Facts placed at the beginning of documents were recalled more effectively. However, facts in the middle portion (7%-50% of the document) showed lower recall rates. 🔑 Key Insights: No Guarantees on Recall: Essential to recognise that there's no certainty in GPT-4's ability to retrieve specific facts. Less is More for Context: Reducing the context sent to GPT-4 can improve accuracy, reaffirming the principle that less context can lead to better performance. Fact Placement Matters: The study underscores the impact of fact positioning within a document on recall efficiency. #legaltech #openai #contextwindow https://lnkd.in/eYNrF5X2

Finally an open-source model has caught up to GPT-4-0314. We have new open-source models in town. Thanks to the team at WizardLM Here are some benchmarks for their three models respectively: 8x22B 70B 7B

I combined GPT and Gemini on one screen. An easy way to compare output from different models. One message - two answers. Check it out: https://bit.ly/gpt_arena