AWS CLI S3 Pagination: A Comprehensive Guide

When working with Amazon S3 using the AWS Command Line Interface (AWS CLI), you may encounter situations where you need to retrieve a large number of objects from an S3 bucket. By default, AWS S3 API operations return a limited number of results per request. To handle cases where the number of objects exceeds this limit, AWS provides a mechanism called pagination. This blog post will delve into the core concepts of AWS CLI S3 pagination, explore typical usage scenarios, discuss common practices, and share best practices to help software engineers effectively manage large datasets in S3.

Table of Contents#

  1. Core Concepts of AWS CLI S3 Pagination
  2. Typical Usage Scenarios
  3. Common Practices
  4. Best Practices
  5. Conclusion
  6. FAQ
  7. References

Core Concepts of AWS CLI S3 Pagination#

What is Pagination?#

Pagination is a technique used to divide a large dataset into smaller, more manageable "pages" of data. In the context of AWS S3, when you make a request to list objects in a bucket, the S3 API returns a maximum of 1000 objects per page by default. If your bucket contains more than 1000 objects, you need to use pagination to retrieve the remaining objects.

How Pagination Works in AWS CLI S3#

When you use the aws s3api list-objects-v2 command to list objects in an S3 bucket, the response includes a NextContinuationToken if there are more objects to retrieve. This token acts as a pointer to the next page of results. To get the next page, you need to pass this token as the ContinuationToken parameter in the next list-objects-v2 request.

Here is an example of a basic list-objects-v2 request:

aws s3api list-objects-v2 --bucket my-bucket

If the response contains a NextContinuationToken, you can get the next page like this:

aws s3api list-objects-v2 --bucket my-bucket --continuation-token <NextContinuationToken>

Typical Usage Scenarios#

Bulk Object Operations#

Suppose you need to perform a bulk operation on all objects in an S3 bucket, such as deleting all objects or updating their metadata. Since you can't retrieve all objects in a single request, you need to use pagination to iterate through all objects in the bucket.

Data Analysis#

When analyzing data stored in S3, you may need to access all objects in a bucket. For example, you might want to calculate the total size of all objects in a bucket or count the number of objects with a specific prefix. Pagination allows you to process the data in manageable chunks.

Common Practices#

Using a Script to Automate Pagination#

To simplify the process of retrieving all objects in a bucket, you can write a script to automate pagination. Here is an example of a Bash script that lists all objects in a bucket:

#!/bin/bash
 
bucket_name="my-bucket"
continuation_token=""
 
while true; do
    if [ -z "$continuation_token" ]; then
        response=$(aws s3api list-objects-v2 --bucket $bucket_name)
    else
        response=$(aws s3api list-objects-v2 --bucket $bucket_name --continuation-token $continuation_token)
    fi
 
    # Extract object keys from the response
    object_keys=$(echo $response | jq -r '.Contents[].Key')
    for key in $object_keys; do
        echo $key
    done
 
    # Check if there are more pages
    continuation_token=$(echo $response | jq -r '.NextContinuationToken')
    if [ "$continuation_token" = "null" ]; then
        break
    fi
done

Limiting the Number of Results per Page#

You can use the MaxKeys parameter to limit the number of results returned per page. This can be useful if you want to control the amount of data retrieved in each request. For example:

aws s3api list-objects-v2 --bucket my-bucket --max-keys 500

Best Practices#

Error Handling#

When using pagination, it's important to handle errors properly. Network issues or API rate limits can cause requests to fail. You should implement retry logic in your script to handle these errors gracefully.

Performance Optimization#

To improve performance, you can parallelize the processing of pages. For example, you can use multi-threading or multi-processing techniques to process multiple pages simultaneously. However, be aware of AWS API rate limits when doing this.

Security#

When handling sensitive data in S3, make sure to follow security best practices. For example, use IAM roles with the minimum necessary permissions to access the bucket.

Conclusion#

AWS CLI S3 pagination is a powerful feature that allows you to handle large datasets stored in S3. By understanding the core concepts, typical usage scenarios, common practices, and best practices, software engineers can effectively manage and process data in S3 buckets. Whether you're performing bulk operations or data analysis, pagination ensures that you can access all objects in a bucket without being limited by the default result size.

FAQ#

Q: What is the maximum number of objects that can be returned per page by default?#

A: By default, the list-objects-v2 operation returns a maximum of 1000 objects per page.

Q: Can I change the maximum number of objects per page?#

A: Yes, you can use the MaxKeys parameter to specify the maximum number of objects to return per page. The valid range for MaxKeys is between 1 and 1000.

Q: How do I know if there are more pages of results?#

A: If the response from the list-objects-v2 operation contains a NextContinuationToken, it means there are more pages of results. You need to use this token to get the next page.

References#

2025-10