AWS CLI Sync S3 Missing Files Only

Amazon Simple Storage Service (S3) is a widely used cloud storage service that offers scalable, durable, and highly available storage. The AWS Command Line Interface (CLI) provides a powerful way to interact with S3 and perform various operations, including syncing files between a local directory and an S3 bucket. One common requirement is to sync only the missing files, which can save time and bandwidth, especially when dealing with large datasets. In this blog post, we will explore how to use the AWS CLI to sync only the missing files between a local directory and an S3 bucket.

Table of Contents#

  1. Core Concepts
  2. Typical Usage Scenarios
  3. Common Practice
  4. Best Practices
  5. Conclusion
  6. FAQ
  7. References

Core Concepts#

AWS CLI#

The AWS CLI is a unified tool that allows you to manage your AWS services from the command line. It provides a consistent interface for interacting with various AWS services, including S3. You can use the AWS CLI to perform tasks such as creating buckets, uploading files, and syncing data between local and S3 storage.

S3 Sync#

The aws s3 sync command is used to synchronize files between a local directory and an S3 bucket, or between two S3 buckets. By default, the sync command compares the source and destination files based on their size and last modified time. It then copies or deletes files as necessary to make the destination match the source.

Syncing Only Missing Files#

To sync only the missing files, you can use the --size-only option in combination with the --delete option. The --size-only option tells the sync command to compare files based only on their size, ignoring the last modified time. The --delete option is used to delete files in the destination that are not present in the source. By using these options together, you can ensure that only the missing files are synced.

Typical Usage Scenarios#

Incremental Backups#

When performing incremental backups of your local data to an S3 bucket, you may only want to upload the files that have been added or changed since the last backup. Syncing only the missing files can significantly reduce the backup time and bandwidth usage.

Disaster Recovery#

In a disaster recovery scenario, you may need to restore your data from an S3 bucket to a local directory. Syncing only the missing files can help you quickly restore the data that is not already present on the local system.

Data Replication#

If you need to replicate data between different S3 buckets or between a local directory and an S3 bucket, syncing only the missing files can ensure that only the necessary data is transferred, saving time and resources.

Common Practice#

Prerequisites#

Before you can use the AWS CLI to sync files with S3, you need to have the following:

  • AWS CLI installed and configured on your system. You can follow the official AWS CLI installation guide to set it up.
  • Appropriate permissions to access the S3 bucket. You can manage your IAM (Identity and Access Management) policies to grant the necessary permissions.

Syncing Only Missing Files#

Here is an example of how to use the aws s3 sync command to sync only the missing files between a local directory and an S3 bucket:

aws s3 sync /path/to/local/directory s3://your-bucket-name --size-only --delete

In this example, /path/to/local/directory is the path to your local directory, and s3://your-bucket-name is the URI of your S3 bucket. The --size-only option ensures that only files with different sizes are considered for syncing, and the --delete option removes any files in the S3 bucket that are not present in the local directory.

Syncing in the Opposite Direction#

If you want to sync files from an S3 bucket to a local directory, you can simply reverse the source and destination:

aws s3 sync s3://your-bucket-name /path/to/local/directory --size-only --delete

Best Practices#

Test in a Staging Environment#

Before performing a sync operation in a production environment, it is recommended to test the sync command in a staging environment. This can help you identify any potential issues and ensure that the sync operation works as expected.

Monitor the Sync Process#

You can use the --dryrun option to perform a dry run of the sync command. This will show you what files will be copied, deleted, or modified without actually making any changes. You can also monitor the progress of the sync operation using the --no-progress option to disable the progress bar or the --progress option to enable it.

Error Handling#

When performing a sync operation, it is important to handle errors gracefully. You can use try-catch blocks in scripts or check the exit code of the aws s3 sync command to handle errors and take appropriate actions.

Conclusion#

Syncing only the missing files between a local directory and an S3 bucket using the AWS CLI can save time and bandwidth, especially when dealing with large datasets. By using the --size-only and --delete options, you can ensure that only the necessary files are synced. Understanding the core concepts, typical usage scenarios, common practices, and best practices can help you effectively use the AWS CLI to manage your S3 data.

FAQ#

Q: Can I sync only specific file types?#

A: Yes, you can use the --exclude and --include options to specify which file types to include or exclude from the sync operation. For example, to sync only .txt files, you can use the following command:

aws s3 sync /path/to/local/directory s3://your-bucket-name --size-only --delete --include "*.txt" --exclude "*"

Q: What if I don't want to delete files in the destination?#

A: If you don't want to delete files in the destination, you can omit the --delete option from the sync command. This will only copy the missing files to the destination without deleting any existing files.

Q: How can I resume a interrupted sync operation?#

A: The aws s3 sync command will resume from where it left off if the operation is interrupted. You can simply run the same sync command again, and it will continue the sync process.

References#

2025-10