AWS Glue S3 create_dynamic_frame.from_catalog Exclude: A Comprehensive Guide

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load your data for analytics. One of the key functions in AWS Glue is create_dynamic_frame.from_catalog, which allows you to create a DynamicFrame from data catalog entries. The exclude parameter in this function provides a powerful way to filter out specific data while loading. This blog post will delve into the core concepts, typical usage scenarios, common practices, and best practices related to using the exclude parameter with create_dynamic_frame.from_catalog when working with S3 data in AWS Glue.

Table of Contents#

  1. Core Concepts
  2. Typical Usage Scenarios
  3. Common Practice
  4. Best Practices
  5. Conclusion
  6. FAQ
  7. References

Article#

1. Core Concepts#

create_dynamic_frame.from_catalog#

This is a method in the AWS Glue Python API (PyGlue) that retrieves metadata from the AWS Glue Data Catalog and creates a DynamicFrame. A DynamicFrame is a collection of data similar to a DataFrame, but it can handle semi - structured data more effectively. It can infer the schema from the data in the catalog and load the data from the specified data source, which in this case is S3.

exclude Parameter#

The exclude parameter is used to specify a list of columns that you want to exclude from the DynamicFrame being created. When you use this parameter, the specified columns will not be loaded into the DynamicFrame, which can save memory and processing time, especially when dealing with large datasets.

Here is a basic example of using the exclude parameter:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
 
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
 
dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
    database="your_database",
    table_name="your_table",
    exclude=['column1', 'column2']
)

2. Typical Usage Scenarios#

Reducing Memory Usage#

When working with large datasets, loading all columns into memory can be resource - intensive. By using the exclude parameter, you can exclude columns that are not needed for your analysis, reducing the memory footprint of your Glue job.

Data Security#

In some cases, certain columns may contain sensitive information such as personal identifiable information (PII). You can use the exclude parameter to ensure that these columns are not loaded into the DynamicFrame, protecting the privacy of your data.

Simplifying Data Processing#

If your data processing logic only requires a subset of columns, excluding unnecessary columns can simplify your code and make it easier to maintain.

3. Common Practice#

Identifying Unnecessary Columns#

Before running your Glue job, you need to carefully analyze your data and determine which columns are not needed. This can be done by reviewing the data schema and understanding the requirements of your data processing task.

Testing with Excluded Columns#

It's a good practice to test your Glue job with the exclude parameter on a small subset of data first. This allows you to verify that excluding the specified columns does not affect the correctness of your data processing logic.

# Test on a small sample
sample_dynamic_frame = dynamic_frame.limit(100)
# Process the sample data
transformed_sample = ApplyMapping.apply(frame = sample_dynamic_frame, mappings = [("col1", "string", "col1", "string")])
# Check the results
transformed_sample.show()

4. Best Practices#

Use Column Aliasing#

If you need to refer to columns in your data processing code, it's a good practice to use column aliasing to make your code more readable. This can be especially useful when you are excluding columns and the original column names may not be present in the DynamicFrame.

from awsglue.dynamicframe import DynamicFrame
 
# Create a new DynamicFrame with column aliasing
aliased_frame = dynamic_frame.select_fields(['col3', 'col4']).rename_field('col3', 'new_col3').rename_field('col4', 'new_col4')

Keep a Record of Excluded Columns#

Maintain a record of the columns that you are excluding in your Glue job. This can be useful for auditing purposes and for future reference if you need to change your data processing logic.

Conclusion#

The exclude parameter in create_dynamic_frame.from_catalog is a powerful tool for optimizing data loading and processing in AWS Glue when working with S3 data. By understanding the core concepts, typical usage scenarios, common practices, and best practices, software engineers can effectively use this parameter to reduce memory usage, enhance data security, and simplify data processing.

FAQ#

Q1: Can I exclude nested columns using the exclude parameter?#

A1: As of now, the exclude parameter works at the top - level column level. If you need to exclude nested columns, you may need to use other data transformation techniques such as ApplyMapping to selectively extract the required nested fields.

Q2: Will excluding columns affect the performance of my Glue job?#

A2: In most cases, excluding columns can improve the performance of your Glue job by reducing the amount of data that needs to be loaded and processed. However, the actual performance improvement may vary depending on the size of your dataset and the complexity of your data processing logic.

Q3: Can I use wildcards in the exclude parameter?#

A3: The exclude parameter does not support wildcards. You need to specify the exact column names that you want to exclude.

References#

2025-10