Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake | Amazon Web Services

SEATTLE — In a major development for enterprise database management and data architecture, Amazon Web Services (AWS) has announced a powerful new capability for Amazon Aurora PostgreSQL. Organizations can now directly query operational data residing within their relational databases alongside historical data stored in data lakes—specifically formatted in Apache Iceberg and Apache Parquet—using standard PostgreSQL applications, tools, and syntax.

By eliminating the traditional requirement to build, maintain, and monitor complex Extract, Transform, Load (ETL) pipelines or reverse-ETL mechanisms, AWS is dismantling the historical barrier separating transactional systems from analytical data stores. Enabled by embedding the high-performance DuckDB query engine directly within Aurora PostgreSQL, this feature promises to streamline application development, reduce infrastructure overhead, and provide seamless access to unified data for real-time dashboards, transactional enrichment, and next-generation artificial intelligence applications.

Main Facts: Unifying Operational and Analytical Data Stores

The newly launched capability fundamentally changes how applications interact with fragmented data landscapes. Historically, enterprises faced a distinct architectural divide: operational databases like Amazon Aurora handled fast, real-time transactional workloads (OLTP), while data lakes on Amazon S3 stored vast amounts of historical, analytical data (OLAP) structured in formats like Apache Parquet or managed via Apache Iceberg catalogs.

To bridge this divide, engineering teams routinely deployed ETL pipelines to copy data back and forth. These pipelines introduced severe operational friction:

• Data Duplication: Storing identical data across multiple systems inflated storage and infrastructure costs.
• Synchronization Latency: Real-time applications could not easily access historical trends without delays caused by batch processing or sync schedules.
• Engineering Overhead: Maintaining pipelines demanded dedicated developer hours and ongoing debugging.

With the integration of DuckDB directly into Amazon Aurora PostgreSQL, query processing occurs locally within the database cluster. There are no additional network hops, no data duplication, and no rigid ETL jobs. A single SQL query can now seamlessly join live operational tables—including uncommitted writes—with massive historical archives residing in Amazon S3, S3 Tables, or external catalogs compatible with the Iceberg REST Catalog (IRC) standard via AWS Glue Data Catalog federation.

Furthermore, AWS has ensured that developers do not need to learn a new language or adopt unfamiliar tooling. The capability utilizes standard PostgreSQL syntax, meaning existing applications, reporting tools, and database clients (psql, BI dashboards, and custom microservices) work out of the box.

Chronology: The Journey to Embedded Analytics

The path to this architectural milestone reflects a broader industry trend toward unified query engines and strategic technological acquisitions.

• The Historical Divide: For decades, relational database management systems (RDBMS) focused intensely on ACID compliance, row-level locking, and rapid transactional throughput. Meanwhile, the explosive growth of big data birthed data lakes, columnar file formats like Parquet, and open table formats like Apache Iceberg. Enterprises were forced to build expensive, fragile bridges between these two worlds.
• The Rise of DuckDB: In recent years, DuckDB emerged as an open-source analytical data management engine renowned for its vectorised query execution, remarkable speed, and lightweight footprint. It became the de facto standard for in-process analytical data processing.
• The DuckLabs Acquisition: Demonstrating a commitment to embedding cutting-edge performance features directly into AWS managed services, Amazon welcomed DuckLabs—the core team maintaining the DuckDB project—to the company.
• Integration and Release: Leveraging the architectural efficiency of DuckDB, AWS engineers embedded the engine directly into the core runtime of Amazon Aurora PostgreSQL.
• General Availability: AWS officially announced the capability for Aurora PostgreSQL versions 17 (starting with version 17.11) and 18 (starting with version 18.6). The feature is rolling out across all commercial AWS Regions and AWS GovCloud (US) Regions at no additional software charge.

Supporting Data, Architecture, and Technical Implementation

Under the hood, the new capability is engineered for high performance, efficiency, and scale. Rather than blindly dragging entire datasets across the network, Aurora PostgreSQL leverages advanced optimization techniques native to modern analytical engines.

Key Optimization Mechanisms

1. Predicate Pushdown: Aurora evaluates filters and conditions at the storage layer, ensuring that only relevant rows matching the query criteria are scanned and processed.
2. Column Pruning: Because formats like Apache Parquet and Apache Iceberg are columnar, Aurora reads exclusively the columns requested in the query, ignoring extraneous data fields.
3. Smart Caching: Frequently accessed data blocks from the data lake are cached within the Aurora instance. Subsequent queries targeting the same data return significantly faster.
4. Granular Diagnostics: Developers can inspect execution metrics per query using the aurora_analytics_stat_statements() function, which reports critical performance data such as rows scanned, bytes read from Amazon S3, and cache hit ratios.

Step-by-Step Implementation Guide

Enabling and utilizing this capability requires minimal configuration. The setup process can be executed via the Amazon RDS Console or any standard PostgreSQL client:

1. Prerequisites and Permissions: The feature is supported on Aurora PostgreSQL versions 17.11+ and 18.6+. Administrators must attach an AWS Identity and Access Management (IAM) role containing the AuroraAnalytics feature policy to the Aurora cluster. This grants the database permissions to read objects from Amazon S3 and query the AWS Glue Data Catalog.
2. Enabling the Extension: Connect to the Aurora PostgreSQL database using a client such as psql and enable the extension:

   CREATE EXTENSION aurora_analytics;

3. Creating Foreign Tables: Instead of manually mapping column definitions, Aurora can automatically infer schemas directly from file metadata. For instance, pointing a foreign table to a historical Parquet file in S3 is executed as follows:

   CREATE FOREIGN TABLE transaction_history ()
   SERVER aurora_analytics_server
   OPTIONS (
      location 's3://<my-bucket>/finance/transaction_history.parquet',
      format 'parquet'
   );

   For larger data lakes containing numerous tables, engineers can bypass individual definitions entirely by utilizing an IMPORT FOREIGN SCHEMA statement, which bulk-creates foreign tables for every table in an AWS Glue Data Catalog database automatically.

4. Executing Unified Queries: With both operational tables (e.g., recent transactions stored natively in Aurora) and foreign tables (historical data in S3) in place, developers can execute federated queries using standard UNION or JOIN operations:

   SELECT merchant, category, amount, transaction_date, 'recent' AS source
   FROM recent_transactions
   WHERE customer_id = 'C-1001'
   UNION ALL
   SELECT merchant, category, amount, transaction_date, 'historical' AS source
   FROM transaction_history
   WHERE customer_id = 'C-1001'
    AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
   ORDER BY transaction_date DESC
   LIMIT 15;

5. Materialization for Ultra-Low Latency: For query patterns requiring single-digit-millisecond operational latency, data can be materialized from the data lake into native Aurora PostgreSQL tables using standard commands such as CREATE TABLE AS SELECT, INSERT INTO ... SELECT, or MERGE INTO. This allows read queries to run against local tables across any writer or read replica instance, completely offloading heavy analytical scans from the primary operational workload.

Official Responses and Strategic Perspective

While AWS has introduced the feature directly through its engineering and product updates, the broader architectural community has responded with intense interest regarding what this means for enterprise data strategy.

Industry analysts note that the blurring lines between transactional databases and analytical data lakes represent a major paradigm shift. For years, vendors pushed specialized, siloed systems, arguing that transactional databases and analytical data lakes must remain strictly segregated. However, the operational reality for software developers—particularly those building modern artificial intelligence applications—demands a frictionless, unified view of data.

In commentary accompanying the release, AWS engineering representatives emphasized that artificial intelligence agents, in particular, accelerate the need for this capability. When building intelligent agents that reason over live enterprise workflows, it is fundamentally impractical to predict, anticipate, and pre-replicate every disparate dataset an agent might query. By placing DuckDB directly inside Aurora PostgreSQL, AWS provides agents and human developers alike with an immediate, unified interface to explore both live transactions and deep historical archives without architectural bottlenecks.

Implications: Reshaping Application Development and Cloud Economics

The introduction of direct data lake querying within Amazon Aurora PostgreSQL carries profound implications for software engineering teams, database administrators, and cloud financial management:

• Drastic Reduction in Operational Complexity: Engineering organizations no longer need to allocate valuable developer bandwidth to build, monitor, and troubleshoot reverse-ETL pipelines. This frees up technical talent to focus on core product features and business logic rather than data plumbing.
• Cost Optimization: By eliminating the storage duplication inherent in traditional data warehousing and ETL synchronization workflows, enterprises can significantly trim their cloud storage bills. Furthermore, AWS is offering this capability at no additional software charge; customers pay only for the incremental Aurora compute resources consumed by the analytical queries and standard Amazon S3 request costs.
• Empowering Modern AI Workloads: Artificial intelligence and Large Language Model (LLM) architectures thrive on rich, contextual data. Giving AI systems real-time access to live operational context fused with years of historical telemetry—all through a standard SQL interface—significantly lowers the barrier to building sophisticated, context-aware autonomous agents.
• Flexibility in Read Scalability: Because analytical queries can be offloaded to read replicas or processed efficiently via DuckDB’s vectorized execution engine, mission-critical operational workloads remain protected from performance degradation, ensuring high availability and consistent response times for core user-facing transactions.

Conclusion

Amazon Web Services has redefined the boundaries of traditional relational database engines. By synthesizing the transactional reliability of Amazon Aurora PostgreSQL with the analytical velocity of DuckDB and the vast, open storage standards of Apache Iceberg and Parquet on Amazon S3, AWS has delivered a unified data architecture designed for the demands of modern cloud applications and AI-driven enterprises. The capability is available today across all commercial and GovCloud AWS regions, inviting organizations to simplify their architectures and unlock new dimensions of data utility.