AWS Glue Cross - Account S3: A Comprehensive Guide

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load your data for analytics. Amazon S3, on the other hand, is an object storage service offering industry - leading scalability, data availability, security, and performance. When dealing with data across multiple AWS accounts, the ability to access S3 buckets from one account using AWS Glue in another account (AWS Glue cross - account S3) becomes crucial. This blog post will provide a detailed overview of the core concepts, typical usage scenarios, common practices, and best practices related to this powerful combination.

Table of Contents#

  1. Core Concepts
  2. Typical Usage Scenarios
  3. Common Practices
  4. Best Practices
  5. Conclusion
  6. FAQ
  7. References

Article#

1. Core Concepts#

AWS Glue#

AWS Glue consists of a central metadata repository called the AWS Glue Data Catalog, an ETL engine that can generate Python or Scala code, and a flexible scheduler. It simplifies the process of building ETL jobs by automatically discovering data sources, inferring schemas, and generating the necessary code to transform and load data.

Amazon S3#

Amazon S3 stores data as objects within buckets. Each object has a unique key within the bucket. S3 provides high - durability and availability, and it can be used to store a wide variety of data types, from small text files to large multimedia files.

Cross - Account Access#

Cross - account access in the context of AWS Glue and S3 means that an AWS Glue job running in one AWS account can access an S3 bucket located in another AWS account. This is achieved through proper configuration of IAM (Identity and Access Management) roles and policies. IAM roles act as a set of permissions that can be assumed by trusted entities, allowing them to perform specific actions on AWS resources.

2. Typical Usage Scenarios#

Data Centralization#

A large organization may have multiple business units, each with its own AWS account. One unit might be responsible for collecting raw data and storing it in an S3 bucket in its account. Another unit, which is focused on data analytics, can use AWS Glue in its own account to access the S3 bucket in the data - collecting unit's account, transform the data, and load it into a central data warehouse for analysis.

Regulatory and Security Compliance#

Some industries have strict regulatory requirements that mandate data isolation. By using multiple AWS accounts, an organization can isolate sensitive data in one account while still being able to perform ETL operations on that data using AWS Glue in another account. This way, they can ensure compliance with regulations while still leveraging the power of AWS services.

Cost Management#

Different AWS accounts may have different cost structures and usage patterns. By separating data storage in one account and ETL processing in another, an organization can optimize costs. For example, the account used for data storage can be configured with long - term storage options, while the account used for ETL can be optimized for compute - intensive tasks.

3. Common Practices#

Step 1: Create an IAM Role in the S3 Bucket Account#

In the account where the S3 bucket is located, create an IAM role that has permissions to access the S3 bucket. The role should also have a trust policy that allows the AWS Glue service in the other account to assume the role.

{
    "Version": "2012 - 10 - 17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::TARGET_ACCOUNT_ID:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step 2: Attach an S3 Access Policy to the IAM Role#

Attach a policy to the IAM role that allows the necessary S3 actions, such as s3:GetObject, s3:ListBucket, etc.

{
    "Version": "2012 - 10 - 17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your - bucket - name",
                "arn:aws:s3:::your - bucket - name/*"
            ]
        }
    ]
}

Step 3: Configure AWS Glue in the Other Account#

In the AWS account where the AWS Glue job will run, create an IAM role for the Glue job. This role should have permissions to assume the role created in the S3 bucket account. Then, configure the Glue job to use this role.

import boto3
 
sts_client = boto3.client('sts')
 
assumed_role_object = sts_client.assume_role(
    RoleArn="arn:aws:iam::SOURCE_ACCOUNT_ID:role/your - cross - account - role",
    RoleSessionName="AssumeRoleSession1"
)
 
credentials = assumed_role_object['Credentials']
 
s3_client = boto3.client(
    's3',
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken']
)

4. Best Practices#

Least Privilege Principle#

When creating IAM roles and policies, follow the least privilege principle. Only grant the minimum permissions necessary for the AWS Glue job to access the S3 bucket. This reduces the risk of unauthorized access and potential security breaches.

Regular Auditing#

Regularly audit the IAM roles and policies associated with the cross - account access. Check for any unnecessary permissions or outdated configurations. AWS provides tools like AWS IAM Access Analyzer to help with this process.

Encryption#

Enable encryption for the S3 bucket. AWS S3 supports server - side encryption (SSE - S3, SSE - KMS) and client - side encryption. Encrypting data at rest and in transit adds an extra layer of security, especially when dealing with cross - account access.

Monitoring and Logging#

Use AWS CloudWatch to monitor the AWS Glue jobs and S3 access. Set up appropriate alarms to notify you of any unusual activity, such as excessive data access or failed ETL jobs. Also, enable AWS CloudTrail to log all API calls related to the cross - account access for auditing and compliance purposes.

Conclusion#

AWS Glue cross - account S3 access is a powerful feature that enables organizations to efficiently manage and process data across multiple AWS accounts. By understanding the core concepts, typical usage scenarios, common practices, and best practices, software engineers can leverage this feature to build robust and secure ETL pipelines. It provides flexibility, cost - optimization, and compliance benefits, making it an essential tool for modern data - driven organizations.

FAQ#

Q1: Can I use AWS Glue cross - account S3 access for real - time data processing?#

A1: While AWS Glue is primarily designed for batch processing, you can use it in combination with other AWS services like Amazon Kinesis for real - time data processing. However, the cross - account access setup remains similar, but you need to ensure that the necessary permissions are granted for real - time data sources and sinks.

Q2: What if the IAM role in the S3 bucket account is deleted?#

A2: If the IAM role in the S3 bucket account is deleted, the AWS Glue job in the other account will no longer be able to access the S3 bucket. You will need to recreate the role with the appropriate permissions and update the Glue job configuration to use the new role.

Q3: Are there any additional costs associated with cross - account access?#

A3: There are no additional costs specifically for cross - account access. However, you will be charged for the normal usage of AWS Glue and S3, such as compute time for Glue jobs and storage costs for S3 buckets.

References#

2025-10