Joe Minichino

🎉 #VLDB2025 has published the paper “Disaggregated State Management in Apache Flink® 2.0”! 🎉 🤝 Who made it happen A true community effort—co-authored by the #ApacheFlink community, Alibaba Cloud, Boston University, and KTH Royal Institute of Technology. The work decouples state and compute, slashing snapshot cost and speeding recovery—a major leap toward cloud-native, high-scale stream processing. 📜 Why it matters • Exactly 10 years after Flink’s first VLDB paper defined consistent streaming state, this new work charts the next leap: disaggregated, cloud-native state. • Shows continued academic trust in Flink and Alibaba’s decade-long contribution to open innovation. • Kicks off a new chapter—Generic Incremental Compute: ForSt + batch push-down promise lower latency AND lower cost, bringing near-real-time analytics to everyone. 🗓️ VLDB 2025 is LIVE NOW in London! Join us at the 51st International Conference on Very Large Databases (VLDB) as we explore the future of distributed systems. 📅 When: September 1–5, 2025 📍 Where: London, UK Don’t miss Industry Session 1 on Tuesday, September 2nd (9/2)—Disaggregated State Management in Apache Flink® 2.0. This session is a must-attend for data engineers building next-gen architectures! 🔗 Learn More: Download the Paper: https://lnkd.in/gqcd9BYW Read the Technical Deep Dive: https://lnkd.in/gnwB48kn Watch the full recording on Youtube: https://lnkd.in/gV-n45sq #VLDB2025 #ApacheFlink #DataEngineering #StreamingSystems #DistributedComputing Yuan Mei

62

1 Comment

Claude Sonnet 4.5 is available in GitLab Duo Pro, Duo Enterprise, and Duo Agent Platform, giving teams access to Anthropic’s most advanced model for coding and agents. Learn more in our blog: https://lnkd.in/efQpr-kU

196

4 Comments

📰 Building a personalized newsletter with Apache Airflow 3, AWS Developers, and AWS Open Source! Kenten Danas joins Darko Mesaroš to demonstrate leveraging Airflow 3's new features like DAG versioning, assets, and backfills. Learn how to build an automated personalized newsletter system using Airflow 3, AWS Bedrock for AI generation, and Amazon SQS for event-driven scheduling. https://bit.ly/3JbDzXT

24

1 Comment

O'Reilly Media has just released Apache Polaris: The Definitive Guide—the first-ever book dedicated to the open-source, Iceberg-native catalog powering modern lakehouses. Authored by Dremio’s Alex Merced, this comprehensive guide offers hands-on strategies for building scalable, cloud-native data architectures using Apache Polaris. It covers everything from metadata management and governance to integrating with tools like Apache Spark, Flink, and Dremio. Whether you're a data engineer, architect, or analyst, this book is your roadmap to mastering the open lakehouse ecosystem. https://lnkd.in/givFEUgV

27

MinIO Brings Iceberg Tables to AIStor: https://lnkd.in/g6QsFmjR MinIO is introducing Iceberg Tables, bringing native Apache Iceberg support directly into AIStor to unify enterprise data for AI. By integrating the Iceberg Catalog API directly into MinIO AIStor, MinIO is helping enterprises unlock more value from AI on their own terms with an AI data store that brings together structured and unstructured data under a single, open standard. Apache Iceberg | #DataStore #Iceberg #AIStor #AI

91

2 Comments

A GUI is nice—but real agility comes from code. That’s why we built Graphiant SDKs in Python and Go, plus Terraform and Ansible playbooks. Now, developers can programmatically spin up cloud gateways and integrate Graphiant into their CI/CD pipelines. Consistent. Repeatable. Scalable. Multi-cloud networking should be as flexible as your application stack. With Graphiant, it finally is. 👉 Read the full blog here: https://hubs.ly/Q03DTYH20

25

The ClickHouse JSON type was already fast, but v25.8 made it 58x faster and use 3,300x less memory for complex JSON structures. We heard two common challenges when working with JSON documents: 1. Individual documents have many thousands of paths 2. Document structure is variable, resulting in 10,000+ unique paths across documents ClickHouse v25.8 introduces two new serialization formats for JSON data that solve the "thousands of paths" problem. Bucketed shared data: JSON paths are grouped and stored into deterministic buckets. When querying a specific path, ClickHouse knows which bucket to read and skips all the others. Advanced shared data: Builds on bucketed, adding granular indexing. Advanced metadata allows skipping within buckets to improve read performance even further. The results? 📊 200k rows of JSON with 10,000 paths 📊 ⚡ Query time: 3.63s → 0.063s 💾 Memory usage: 12.53 GiB → 3.89 MiB Read more about how it works → https://lnkd.in/es-NKWhi

179

4 Comments