AWS Glue 6.0 Unleashed: A New Era of Serverless Data Processing, Apache Iceberg v3 Support, and Cost Reduction

SEATTLE — Amazon Web Services (AWS) has officially announced the general availability of AWS Glue 6.0, marking one of the most substantial architectural leaps for its fully serverless managed Apache Spark service. Designed to meet the escalating demands of enterprise-scale data engineering, analytics, and machine learning pipelines, the new release introduces a modernized runtime stack, robust implementation of Apache Iceberg v3, and—crucially for cost-conscious enterprise architectures—a sweeping 30% price reduction compared to previous iterations.

As organizations grapple with petabytes of streaming logs, complex semi-structured JSON events, and the operational overhead of maintaining modern data lakes, AWS Glue 6.0 positions itself as a cornerstone technology. By melding cutting-edge open-source standards with enterprise-grade management, AWS aims to simplify ETL (Extract, Transform, Load) authoring, boost PySpark performance, and enable real-time streaming with single-digit millisecond latency.

Main Facts

AWS Glue 6.0 is built from the ground up on a modernized software runtime stack featuring Apache Spark 4.1, Python 3.13, and Scala 2.13. This foundational update grants data engineering teams access to the latest performance optimizations, security updates, and developer APIs natively available in the open-source ecosystem.

The headline feature of the release is its comprehensive integration with Apache Iceberg v3 (built on Iceberg version 1.11.0). Most notably, AWS Glue 6.0 introduces native support for the VARIANT data type complete with shredding capabilities. This allows organizations to ingest, store, and query semi-structured data—such as nested JSON, application logs, and telemetry event streams—without the traditional requirement of flattening schemas or writing custom parsing logic.

Key highlights of the AWS Glue 6.0 release include:

• 30% Lower Pricing: A significant cost reduction across AWS Glue jobs, passing infrastructure efficiencies directly to customers.
• Complete Apache Iceberg v3 Implementation: Making Glue the most complete Iceberg v3-compliant serverless Spark service on the market.
• VARIANT Data Type with Shredding: Dramatically accelerates query read performance for semi-structured data by avoiding duplicate data copies and pipeline breakages caused by upstream schema evolution.
• Modernized Runtime Engine: Harnesses the latest performance milestones of Apache Spark 4.1.
• Real-Time Capabilities: Supports ultra-low latency, single-digit millisecond streaming data processing.
• Seamless Upgrades: Zero API changes required; jobs can be migrated effortlessly via the AWS CLI, SDKs, or automated upgrade agents in AWS Glue Studio.

Chronology

The journey toward AWS Glue 6.0 represents years of incremental runtime modernization, open-source community collaboration, and user feedback integration within the AWS analytics ecosystem.

• Early Evolution (2017–2020): AWS Glue established itself as a foundational serverless data integration service, abstracting away the provisioning hassles of underlying Spark and Hadoop clusters. Early versions focused heavily on automated schema discovery via crawlers and managed ETL job orchestration.
• The Rise of Open Table Formats (2021–2023): As the industry shifted decisively away from proprietary formats toward open table formats like Apache Hudi, Delta Lake, and Apache Iceberg, AWS systematically incorporated these standards into Glue. Previous versions introduced foundational Iceberg support, allowing users to perform ACID transactions on data lakes stored in Amazon Simple Storage Service (Amazon S3).
• The Modernization Push (2024–2025): With data volumes growing exponentially and generative AI driving demand for clean, semi-structured vector and text data pipelines, customers increasingly voiced the need for lower operating costs and faster query speeds on complex JSON workloads. AWS responded by aligning its internal serverless roadmaps with the rapid advancement of Apache Spark 4.1 and Iceberg v3 specifications.
• August 2026: AWS officially announces the general availability of AWS Glue 6.0 across all commercial AWS regions where the service operates, pairing the modernized runtime with a structural 30% price reduction.

Supporting Data

To contextualize the technical and financial impact of AWS Glue 6.0, it is helpful to examine the operational mechanics of the new features and the underlying pricing structure.

Performance and the VARIANT Data Type

Historically, processing semi-structured data in data lakes involved storing JSON payloads as giant strings or flattening them into wide, brittle tabular schemas. String columns require heavy computational lifting during query time to parse nested fields, while schema flattening introduces massive data duplication and pipeline failures whenever an upstream application changes a single field type.

The introduction of the VARIANT data type with shredding in Iceberg v3 changes this paradigm:

1. Columnar Shredding: The database engine automatically "shreds" frequent paths and structures within the semi-structured payload into optimized columnar storage blocks.
2. Read Acceleration: Queries targeting specific nested attributes read only the relevant shredded columns, bypassing the overhead of scanning or parsing raw string blobs.
3. Resilience to Change: When application schemas evolve, downstream pipelines do not break, drastically lowering maintenance overhead for data engineering teams.

Pricing and Billing Architecture

AWS Glue 6.0 maintains a transparent, usage-based billing model while slashing costs by 30%:

• ETL Jobs and Crawlers: Billed on a per-second basis with a one-minute minimum, calculated against the number of Data Processing Units (DPUs) consumed during execution.
• AWS Glue Data Catalog: Uses a simplified monthly fee structure for storing and querying metadata. Notably, the first million objects stored and the first million metadata accesses remain entirely free of charge.

Official Responses

Speaking on the release, AWS Principal Developer Advocate Channy Yun emphasized the seamless nature of the upgrade path for existing customers, noting that no breaking API modifications are required.

"Today, we are announcing the general availability of AWS Glue 6.0, delivering 30% lower pricing than previous AWS Glue versions and introducing full support for Apache Iceberg v3 features," Yun stated in the official release documentation. "AWS Glue 6.0 is built on a fully modernized runtime, Apache Spark 4.1, Python 3.13, and Scala 2.13, delivering faster performance."

Engineering leaders across the AWS ecosystem have echoed the sentiment, highlighting that the combination of lower cost and advanced open table format support removes long-standing financial barriers to adopting modern data lakehouse architectures. Furthermore, AWS has integrated support for the new release directly into modern developer toolchains, including the AWS MCP (Model Context Protocol) Server and specialized AI plugins, allowing developers to query documentation, check regional availability, and troubleshoot migrations using their preferred AI assistants.

Implications

The launch of AWS Glue 6.0 carries far-reaching implications for data engineers, enterprise architects, and the broader data analytics software market.

1. Accelerated Adoption of Apache Iceberg

By offering the most complete Apache Iceberg v3 implementation on a fully serverless managed Spark service, AWS is signaling that Iceberg has won the mindshare battle for open table formats on AWS. Enterprises hesitant to manage their own Iceberg catalog maintenance, compaction, and snapshot management now have a turnkey, serverless option backed by enterprise-grade SLAs and a 30% cost discount.

2. Democratization of Real-Time Analytics

With single-digit millisecond latency capabilities for real-time streaming pipelines, Glue 6.0 bridges the traditional gap between batch ETL and real-time event processing. Organizations can now unify their streaming ingestion and batch transformation pipelines under a single service umbrella, reducing the architectural sprawl of maintaining separate streaming and batch infrastructure.

3. Economic Relief for Data-Heavy Enterprises

Data gravity and escalating cloud storage/processing bills have forced many CFOs to scrutinize data lake expenditures. The 30% price reduction across AWS Glue 6.0 directly addresses these budgetary pressures. Organizations can either pocket the cost savings or reallocate those cloud budgets toward intensive workloads like generative AI model fine-tuning and real-time analytics.

4. Simplified Migration Paths

Because AWS designed Glue 6.0 with backward-compatible API structures, adopting the new runtime does not require a complete rewrite of existing data pipelines. Teams can utilize the built-in Spark upgrade agent within AWS Glue Studio or configure automated upgrade settings to transition seamlessly from older versions to Glue 6.0.

Getting Started

Data engineering teams can begin utilizing AWS Glue 6.0 immediately. In the AWS Glue Studio console, developers simply navigate to the Job Details tab and select the version labeled Glue 6.0 – Supports Spark 4.1, Scala 2, Python 3. For notebook users, setting 6.0 in the %glue_version magic command initializes an interactive session on the new runtime, paving the way for faster, cheaper, and more robust cloud-scale data processing.