Apache Iceberg

Looking for a way to start your training path to learn #Iceberg? Alex Merced provides a list of high quality content 👇

🚨 Thats huge! Processing streaming data and saving them into a Data Lakehouse just got much easier! Any real-time data stream that we push through Amazon Kinesis Data Streams or directly into Amazon Kinesis Firehose can be now (in preview) saved in Apache Iceberg format. Apache Iceberg adds ACID (atomicity, consistency, isolation, and durability) transactions, snapshots, time travel, schema evolution and more. It is designed to provide efficient and scalable data storage and analytics capabilities — particularly for big data workloads. Previously you could save streaming data on S3 only in plain file formats (like JSON, Parquet) under a specified partition (or dynamic partition) that often resulted with huge number of small files (it makes queries on it really inefficient). Now, being able to write in Apache Iceberg we can overcome this issues with AWS Glue Data Catalog automatic compaction of your Iceberg tables. AWS took a bet on Apache Iceberg as a leading table format that they now spread across many services (Amazon Security Lake, AWS HealthLake...) and to be honest i really like that! Remember: It's still in preview (since August 1st), so it's not recommended yet for production workloads. Fingers crossed for Kinesis team for a quick GA! ♻️ Recycle this post by like or share #aws #awscommunity #awshero #data #dataengineering

Interested in learning how you can leverage Apache Iceberg to optimize overhead and software costs? Join us for a free 2-Hour Virtual Hands-On Lab on August 14th. https://lnkd.in/gjdnYmai

Lakehouse Interoperability with Apache XTable (Incubating)

Heard about Apache Iceberg and curious why it's so popular all of a sudden? Wondering why enterprises (and data platforms) are moving quickly to adopt Iceberg? I have some hot takes on a cool topic. https://lnkd.in/gjDYNHsB

Polaris Catalog!

🧑💻 Hands-on Workshop: Build Real-Time AI Apps on Apache Iceberg 🔷 Apache Iceberg is a high-performance format for managing massive analytic tables. It simplifies big data management, enabling engines like Apache Spark, Trino , Flink, Presto, Hive, and Impala to work safely with the same tables. It supports easy data merging, updates, and targeted deletes, ensuring flexible and efficient data handling. 🔷 Iceberg serves as an open table format for massive analytic datasets, acting as a middle layer between the computing (Flink, Spark) and storage layers (ORC, Parquet, Avro). This setup allows high-performance SQL table functionalities within these engines. 🔷 Coupled with a local deployment via a virtual private cloud (VPC) solution, Iceberg enables secure, reliable, real-time analytics, providing subsecond analytics and powering low-latency applications while ensuring top-notch security and data integrity. 📅 Workshop Details: - Date: Tuesday, July 30 - Time: 10-11am PT / 10:30-11:30pm IST 🔗 https://lnkd.in/gqgXuPK6 What You'll Learn: 🔷 Unfreeze your lakehouse for low-latency applications with native Apache Iceberg integration. 🔷 Faster vector search and improved full-text search techniques. 🔷 Strategies to scale your apps and reduce complexity with Autoscaling. 🔷 How to deploy SingleStore in your own VPC. ✅ Includes a live demo and code-share session. Register to receive the code and webinar recording even if you can't attend live.

Breaking down a Lakehouse Architecture - [New Blog] Lakehouse architectures are gaining a lot of traction. The promise is - - reliability & performance of data warehouses - scalability & cost-effectiveness of data lakes. Most importantly though, it is the open foundation & flexibility offered by table formats such as Apache Hudi, Apache Iceberg & Delta Lake on top of file formats like #Parquet. Since your data is stored in an 'independent data tier' in a lakehouse, it is open to pretty much any compute engine that is compatible with the formats. So, depending on the type of workload (BI, ML, streaming), you can bring your compute engine to the 'independent data tier'. This is actually the opposite of how we have been working traditionally until this point with analytical workloads. We load data to a proprietary database (OLAP/warehouse) & can use only that database's compute engine to process the data. What if we need to use any other compute in such environment? Well, we make data copies, i.e. export a subset of it and then load it to say data lakes (cloud storage) to hand over to other data teams. While lakehouses are becoming ubiquitous, there is also the hype and marketing jargons that comes with it. We often use different terms to refer to the same thing. Or make a complete different thing out of it. In the Hadoop world, this was the same with Apache Hive - There is a Hive query engine, Hive table format & Metastore (HMS). Nebulous right? In this new blog, I go back to the basics and explain what is a lakehouse and how it works. Things covered: ✅ Evolution of data systems ✅ Lakehouse components & advantages ✅ Implementation guide ✅ Use cases/applications ✅ Real-world examples ✅ Open Table Formats ✅ Vendor Platforms ✅ Future of lakehouse Check out the link in comments & reach out for any questions/clarifications. #dataengineering #softwareengineering

How Z-Ordering in Apache Iceberg Helps Improve Performance by Dremio https://lnkd.in/d4uXGwiW

Amazing article published by Dave Eyler on how SingleStore’s bidirectional integration to Apache Iceberg can help you get the most of the data stored into your data lakehouse to power your mission critical applications and still guaranteeing sub-second SLA. https://lnkd.in/e9XWJwya