Apache Doris’ Post

📣 Apache Doris 2.1.5 is released! ✨ Highlights: 🌟 Support complex type output format that is compatible with #Presto. This can enable easy migration from Presto. 🌟 Support non-deterministic functions in materialized view building. 🌟 Support atomically replacing definitions of asynchronous materialized views. 🌟 Support transparent rewrites for various types of aggregation queries. 🌟 Support partial column updates in tables using the VARIANT type. 🌟 Support exporting VARIANT type to #CSV format. https://lnkd.in/geR-wevV #database #dataplatform #OLAP #opensource #semistructureddata #dataanalysis

Apache XTable provides users with the ability to translate metadata from one #lakehouse table format to another omni-directionally. What exactly happens after the XTable "Sync" Process is run? The sync process provides users with the following: ✅ Syncs the data files along with their column-level statistics and partition metadata information ✅ All the schema-level updates in the source table are reflected on to the target format metadata ✅ Metadata maintenance for the target table format. - If the target format is Apache Hudi, unreferenced files will be marked as ‘cleaned’ to control metadata table size - If the target format is Apache Iceberg, snapshots will be expired after a configured amount of time - If the target format is Delta Lake, the transaction log will be retained for a configured amount of time ⭐️ Want to try out XTable? - Here is a link to the getting started page: https://lnkd.in/gHMBQeqV #dataengineering #softwareengineering

Awesome and easier to create chart using plug-in Apache ECharts and can retrieve real-time data using Data Source there selected and configure on Grafana!

Learn How to Use Apache Hudi Streamer with DataHUB An Open Source Metadata Platform Exercise Files https://lnkd.in/eP3869r8 Apache Hudi

Get Started with Hudi CLI Locally Using Docker in Minutes and Connect to Your S3 Data Steps and Instructions  https://lnkd.in/eHTANrBJ Apache Hudi

Previously known as the open-source Apache ECharts panel for Grafana, it is now called the Business Charts, powered by the Apache ECharts library. The latest available version is 6.0.0. It has a new name with the same fantastic flexibility and mighty functionality.  Release blog post: https://lnkd.in/eyre3m7Z #Grafana #BusinessCharts #ECharts

https://lnkd.in/dpWEbzCi << ...DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format... >>

** USING Rust/Actix to Auto-Schedule Apache Iceberg Table Maintenance with Dremio ** If your Apache Iceberg tables are in a Dremio Arctic catalog, the Dremio UI can allow you to auto-schedule table maintenance with a few clicks. However, if you are using catalogs like AWS Glue, Hive, etc. you can still run table maintenance with Dremio's OPTIMIZE and VACUUM commands. In the code below, we show a POST route written in Rust using Actix that would allow you to run both maintenance procedures on the table with a simple post request using Dremio's REST API. Schedule a cron job to hit this endpoint on a set schedule and Dremio can make auto-optimizing your table in any catalog easy. That is what Dremio does for Data Lakehouses, makes them easier, faster and open. Learn more about getting started here: https://lnkd.in/eiEuhigc

Apache Doris 2.1.4 is available now! 📣 It is optimized in terms of data lakehouse capabilities, query optimizer, and asynchronous materialized view, with higher usability and stability in query execution, semi-structured data analysis, Unique Key model, data ingestion/control, memory management, and privilege management. Just to list a few of the key improvements: 🎈The new query optimizer now supports high-concurrency point queries. 🎈Support native reader of Apache Paimon deletion vector, which largely improves data update/deletion efficiency. 🎈Access controller supports Data Mask with Hive Ranger plugin 🎈Support for transparent rewriting in single-table asynchronous materialized views 🎈Support fetching file lists of Hive/Hudi tables in batches (The list fetching latency for 1.2 million files has reduced from 390s to 46s.)  ... Read more: https://lnkd.in/gr2nAd5F #opensource #database #dataengineer #bigdata #datalakehouse #OLAP #ApacheSoftwareFoundation

It seems like every week I learn about a new data processing framework 🙂. This time it’s Apache Arrow DataFusion https://lnkd.in/gtVFv5dQ. It appears to be more batch-oriented, but extremely powerful - written in Rust on top of Apache Arrow format, and it supports SQL and DataStream APIs. Kamu has recently compared DataFusion with Spark and Flink - https://lnkd.in/g2i4NQ9A