AWS CLI Copy in Place S3 Skip Folder

The AWS Command - Line Interface (CLI) is a powerful tool that allows developers and system administrators to interact with various AWS services directly from the command line. One common use case is working with Amazon S3, a scalable object storage service. When using the aws s3 cp command for in - place copying within an S3 bucket, there are scenarios where you might want to skip certain folders. This blog post will provide a comprehensive guide on how to achieve this, including core concepts, typical usage scenarios, common practices, and best practices.

Table of Contents#

  1. Core Concepts
  2. Typical Usage Scenarios
  3. Common Practice
  4. Best Practices
  5. Conclusion
  6. FAQ
  7. References

Article#

1. Core Concepts#

AWS CLI#

The AWS CLI is a unified tool that provides a consistent interface for interacting with AWS services. It uses a simple syntax to perform a wide range of operations on services like S3.

S3#

Amazon S3 is an object storage service that offers industry - leading scalability, data availability, security, and performance. It stores data as objects within buckets, and these objects can be organized in a way that mimics a file system with folders, although S3 doesn't have a true folder structure.

aws s3 cp#

The aws s3 cp command is used to copy files and objects between local file systems and S3 buckets, or between S3 buckets. The "in - place" copying refers to copying objects within the same S3 bucket.

Skipping Folders#

When performing an in - place copy in S3, you may want to skip certain folders. This can be achieved by using filters based on the object keys. S3 object keys are the unique identifiers for objects in a bucket, and they can include a path - like structure that resembles folders.

2. Typical Usage Scenarios#

Data Migration#

Suppose you have an S3 bucket with a large amount of data, and you want to migrate some of it to a different "location" (i.e., a different prefix) within the same bucket. However, there are some folders that you don't want to include in the migration, such as temporary or test data folders.

Data Archiving#

You might be archiving old data within an S3 bucket. You want to move all objects except those in specific folders that are still in active use.

Data Organization#

If you are reorganizing the data in your S3 bucket, you may want to copy objects from one prefix to another while skipping certain folders that should remain in their original location.

3. Common Practice#

To skip folders when using aws s3 cp for in - place copying, you can use the --exclude option.

Here is an example:

aws s3 cp s3://your - bucket/ s3://your - bucket/new - prefix/ --recursive --exclude "folder - to - skip/*"

In this command:

  • s3://your - bucket/ is the source location in the S3 bucket.
  • s3://your - bucket/new - prefix/ is the destination location in the same S3 bucket.
  • --recursive is used to copy all objects under the source prefix recursively.
  • --exclude "folder - to - skip/*" tells the aws s3 cp command to skip all objects whose keys start with folder - to - skip/.

You can also exclude multiple folders by using multiple --exclude options:

aws s3 cp s3://your - bucket/ s3://your - bucket/new - prefix/ --recursive --exclude "folder1/*" --exclude "folder2/*"

4. Best Practices#

Testing#

Before performing a large - scale in - place copy operation, it's a good idea to test the command on a small subset of data. You can use the --dryrun option to see which objects would be copied without actually performing the copy:

aws s3 cp s3://your - bucket/ s3://your - bucket/new - prefix/ --recursive --exclude "folder - to - skip/*" --dryrun

Error Handling#

Make sure to handle errors properly. The AWS CLI will return an error code if the copy operation fails. You can use shell scripting to catch these errors and take appropriate action, such as retrying the operation or logging the error details.

Monitoring#

Monitor the progress of the copy operation. You can use tools like Amazon CloudWatch to monitor the performance of S3 operations and ensure that the copy is proceeding as expected.

Conclusion#

The aws s3 cp command with the --exclude option provides a simple and effective way to perform in - place copying in S3 while skipping specific folders. By understanding the core concepts, typical usage scenarios, common practices, and best practices, software engineers can efficiently manage their S3 data and perform data migration, archiving, and organization tasks.

FAQ#

Q: Can I use wildcards other than * in the --exclude option? A: Yes, the --exclude option supports more advanced wildcards. For example, you can use ? to match a single character.

Q: What if I want to include only certain folders and skip the rest? A: You can use the --include option in combination with the --exclude option. First, exclude everything with --exclude "*" and then include the folders you want with --include "folder - to - include/*".

Q: Does the --exclude option work for partial folder names? A: Yes, you can use partial folder names in the --exclude option. For example, --exclude "partial - name*" will exclude all folders and objects whose keys start with partial - name.

References#

2025-10