Renato Marroquín Mogrovejo

💬 Short video interview with Flavio Junqueira, CTO at Qbeast, at #JOTB24. Flavio is a renowned computer scientist and researcher known for his significant contributions to the field of distributed systems. #data #bigdata #datalakehouse #distributedsystems 👇👇👇

10

📹 Check out the video of the presentation "A case for Data Lakehousekeeping" by Flavio Junqueira, CTO at Qbeast, at #JOTB24 ▶️ https://lnkd.in/eK2yRvFZ #data #datalakes

18

Discover the Cutting-Edge Consensus Algorithms in Apache IoTDB! In the last part of the #distributed architecture trilogy, we explore: 🔍 The unique unified #consensus protocol framework of IoTDB. ⚙️ How users can flexibly choose different consensus algorithms based on their needs. 💡 The innovative IoTConsensus algorithm tailored for IoT scenarios, offering high performance and availability. Compared to other #timeseries #database, Apache IoTDB is the first and only system to propose and apply a unified consensus protocol framework. Users can flexibly choose different consensus algorithms based on performance, #availability, #consistency, and storage cost requirements. We also introduce the upcoming FastIoTConsensus, which enhances performance and availability by leveraging Apache TsFile for data synchronization. 👉 Read the full article here: https://lnkd.in/eHuc-gY2 🚀 Stay tuned for the release of FastIoTConsensus and more exciting developments! #timeseries #database #iot #opensource #distributedsystems #consensusalgorithms

4

🤝 A union of the analytical and operational data domains and a breaking down of boundaries between streaming and batch data have long been sought-after ideals. ✳️ Tableflow represents a milestone in the bridging of the operational/analytical and streaming/batch divides, as it allows the easy materialization of streaming #ApacheKafka topics into #ApacheIceberg tables in a data lake or warehouse—yielding one coherent dataset without ETL. 💡 In his blog post, Jack Vanlightly explores the full ramifications of the Tableflow release. He goes into detail on Tableflow's place within the stream processing industry and describes its technical implementation in depth. 📖 Read more. ⬇️ https://cnfl.io/49WMTXy

97

🤝 A union of the analytical and operational data domains and a breaking down of boundaries between streaming and batch data have long been sought-after ideals. ✳️ Tableflow represents a milestone in the bridging of the operational/analytical and streaming/batch divides, as it allows the easy materialization of streaming #ApacheKafka topics into #ApacheIceberg tables in a data lake or warehouse—yielding one coherent dataset without ETL. 💡 In his blog post, Jack Vanlightly explores the full ramifications of the Tableflow release. He goes into detail on Tableflow's place within the stream processing industry and describes its technical implementation in depth. 📖 Read more. ⬇️ https://cnfl.io/49WMTXy

86

1 Comment

Category of the week: Procedural Languages Select extension: PL/R 💭 It's not uncommon for analysts and researchers to leverage an IDE (Integrated Development Environment) for incidental statistical or aggregate processing. This tends to involve tremendous manual effort, and is difficult to scale. 💡 PL/R grants the user the power of the R programming language, which allows for example, statistical analysis on the data where it lives; this workflow can be further supported by materialized views, for example. Source: https://lnkd.in/gBrZ7esP Link: https://lnkd.in/er_G7dbY 🧰 Check PL/R out and similar extensions within the Procedural Languages category on Trunk; a Postgres extension registry! https://lnkd.in/g7pvNg2c 📊 Tembo is powered by Trunk's ever-growing collection of Postgres extensions, and Tembo users have the freedom to choose. Try out this extension and more on Tembo Cloud, for free! https://lnkd.in/gDQjj8ur

11

New on the #IDUG blog - Calisto Zuzarte discusses how the #Db2 optimizer is evolving into the next generation Db2 AI optimizer. (From, heuristic rule-based algorithm to a cost-based algorithm; next step in the evolution is AI-based.) Read all about it https://buff.ly/3wMJ0Xy

18

“Just Use Postgres For Everything” It feels like it’s everywhere at the moment, the recent surge from developers isn’t just a battle cry; it’s a revolution. People often talk about “how” to use PostgreSQL for everything, but no one has actually covered the “why.” Until now. Why is this mantra being widely adopted? Why would I want a single database? Why does this affect the future of data? 🐯 Ajay Kulkarni tackles these questions head-on in our latest blog. 💪 https://lnkd.in/g-6Xw5im

57

2 Comments

May marked the release of Apache Flink CDC 3.1.0, part of a significant evolution in stream processing and data integration. In a recent collaboration, Giannis Polyzos spoke with Leonard Xu, PMC of Apache Flink and Flink CDC Lead at Alibaba Cloud, about the details of this release, culminating in a comprehensive blog post that dives into Apache Flink CDC 3.0's capabilities. https://lnkd.in/grpAgyzj In essence, Apache Flink CDC is maturing into a sophisticated streaming data integration solution, and your feedback is crucial to its continuous improvement. After reading, testing, or implementing, consider contributing to the project's progress by starring it on GitHub: https://lnkd.in/g4-pQa9E Flink CDC announcement: https://lnkd.in/g3HJEbBT #ApacheFlinkCDC #StreamProcessing #DataEngineering #Flink #CDC #DataIntegration #OpenSourceDevelopment #ReleaseNote

27

Meet the StarTree All-Stars: Mateus Henrique Cândido de Oliveira 🌟 Mateus is a data architect at One Way Solution. He has worked with data for 8+ years, developing streaming data pipelines for 3+ years, and has been a champion for the Apache Pinot community in Brazil! Learn more about Mateus, and let us know which movie his favorite quote is from: 𝗤: 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘆𝗼𝘂𝗿 𝗳𝗮𝘃𝗼𝗿𝗶𝘁𝗲 𝗺𝗼𝘃𝗶𝗲 𝗾𝘂𝗼𝘁𝗲? A: “The body cannot live without the mind.” 𝗤: 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝗳𝗮𝘃��𝗿𝗶𝘁𝗲 𝗣𝗶𝗻𝗼𝘁 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗮𝗻𝗱 𝘄𝗵𝘆? A: Ingestion, we can have both Streaming and Batching ingestion in a single place. 𝗤: 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 𝗽𝗿𝗼𝗷𝗲𝗰𝘁 𝗼𝗿 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 𝘆𝗼𝘂’𝗿𝗲 𝗽𝗿𝗼𝘂𝗱 𝗼𝗳? A: Kafka Summit APAC 2021 - Tim made a recap, and my session was highlighted in the blogpost of Confluent 𝗤: 𝗪𝗵𝗮𝘁'𝘀 𝘀𝗼𝗺𝗲 𝗮𝗱𝘃𝗶𝗰𝗲 𝗳𝗼𝗿 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝘁𝗼 𝗺𝗮𝗸𝗲 𝗮𝗻 𝗶𝗺𝗽𝗮𝗰𝘁 𝗶𝗻 𝘁𝗵𝗲 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆? A: Don’t worry if you can’t contribute with code, make content about Pinot and show the world this incredible technology #StarTree #Community #ApachePinot

22

1 Comment

Learn how to seamlessly integrate a data-streaming stack: Augusto de Nevrezé explains how he Dockerized Apache Flink, Kafka, and PostgreSQL in order to handle real-time data processing and storage efficiently.

53

2 Comments

Why is StarTree the best alternative to Rockset for real-time analytics? ✨ Apache Pinot is an open source distributed columnar database that forms the backbone of mission-critical analytical systems at companies like Uber, DoorDash, Stripe, LinkedIn, Cisco, and others. StarTree Cloud provides a fully-managed, cheaper, faster offering of Apache Pinot across all 3 cloud providers. Apache Pinot and StarTree come closest to Rockset in terms of feature parity, with features like: ✨ Support for real-time upserts ✨ Support for handling unstructured or semi-structured data ✨ Intelligent materialized views using the star-tree index ✨ Cloud tiered storage Check out the full blog from Chinmay Soman (Head of Product @ StarTree) and see a detailed feature comparison of Rockset and StarTree Cloud: https://lnkd.in/gndDzu_x #Rockset #ApachePinot #StarTree

33

3 Comments

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Comments

You can now use SingleStore to seamlessly ingest #data from and write back to #Iceberg tables — in real time, with no additional tooling required. https://lnkd.in/eBuprpJ4

4

Apache IoTDB 1.3.2 is out! Check out the new features in query module and scripts: 📈 Storage Module: Improved insertRecords performance 🔍 Query Module: - New Explain Analyze statement for detailed monitoring - New UDAF framework - MaxBy/MinBy functions for timestamped values - Better performance for value filtering queries 🔄 Data Synchronization: - Path pattern matching - Metadata synchronization 🔧 Stream Processing: New Alter Pipe statement for plugin updates 📊 System Module: Data point stats include TsFile imports 🛠️ Scripts and Tools: - Local upgrade backup tool - New export/import scripts for CSV, TsFile, SQL - Windows environment improvements Discover more: https://lnkd.in/g-R55BEp The full release note: https://lnkd.in/ejKDVTv5 Try it now 👉 https://lnkd.in/edyn7ya7 #opensource #timeseries #database #datamanagement #UDAF #streamprocessing #ratis #csv #tsfile #sql

8

PostgreSQL and pgvector: Now Faster than Pinecone, 75% cheaper, 100% open-source. Introducing pgvectorscale, an open-source PostgreSQL extension that builds on pgvector, enabling greater performance and scalability. Here’s how pgvectorscale helps pgvector outperform specialized vector database like Pinecone: 1️⃣ StreamingDiskANN:  A new vector search index that overcomes limitations of in-memory indexes like HNSW the index on disk, making it more cost-efficient to run and scale as vector workloads grow. Inspired by the DiskANN paper from Microsoft. 2️⃣ Statistical Binary Quantization (SBQ): Developed by researchers at Timescale, this technique improves on standard binary quantization techniques by improving accuracy when using quantization to reduce the space needed for vector storage 3️⃣ Written in Rust, giving the PostgreSQL community to contribute to vector support. 📈The result? On our benchmark of 50 million Cohere embeddings (768 dimensions each), PostgreSQL with pgvector and pgvectorscale achieves 28x lower p95 latency and 16x higher query throughput compared to Pinecone for approximate nearest neighbor queries at 99 % recall, all at 75 % less cost when self-hosted on AWS EC2. We also tested it against Pinecone’s p2 high performance index, see the blog post at the end of this post for full results (spoiler: It’s just as impressive). Pgvectorscale is open-source under the PostgreSQL license and free for you to use on any PostgreSQL database for your AI projects. To get started, see the pgvectorscale github repo: https://lnkd.in/ghXj2e-U Or try it on Timescale Cloud on any new database service. Eager to learn more about pgvectorscale and how it works? Head over to our blog post with all the details: https://lnkd.in/gcMcxrVb

83

3 Comments