AWS Golang S3: Delete a Folder

Table of Contents#

1. Core Concepts
2. Typical Usage Scenarios
3. Prerequisites
4. Common Practice: Deleting a Folder in S3 using Go
5. Best Practices
6. Conclusion
7. FAQ
8. References

Core Concepts#

Amazon S3 Buckets and Objects#

• Buckets: Buckets are the top - level containers in Amazon S3. They are used to store objects and must have a globally unique name across all AWS accounts.
• Objects: Objects are the fundamental entities stored in S3. Each object has a key (a unique identifier within the bucket) and a value (the data). The key can be structured to mimic a folder hierarchy using a delimiter, typically /. For example, mybucket/path/to/object.txt where path/to simulates a folder structure.

Deleting a "Folder" in S3#

Since S3 doesn't have actual folders, deleting a "folder" means deleting all objects whose keys start with the folder's prefix. For example, to delete the "folder" path/to, we need to delete all objects with keys like path/to/object1.txt, path/to/subfolder/object2.txt, etc.

Typical Usage Scenarios#

• Data Cleanup: When you no longer need a set of related data stored in a specific "folder" in an S3 bucket, you can delete it to free up storage space.
• Testing and Development: During development or testing, you may create temporary "folders" to store test data. Once the testing is complete, you can easily clean up these "folders".
• Data Lifecycle Management: As part of a data lifecycle strategy, you may want to delete old data stored in specific "folders" after a certain period.

Prerequisites#

• AWS Account: You need an active AWS account with appropriate permissions to access and modify S3 buckets.
• Go Installation: Install the Go programming language on your machine. You can download it from the official Go website (https://golang.org/dl/).
• AWS SDK for Go: Install the AWS SDK for Go using the following command:

go get github.com/aws/aws-sdk-go/aws
go get github.com/aws/aws-sdk-go/aws/session
go get github.com/aws/aws-sdk-go/service/s3

Common Practice: Deleting a Folder in S3 using Go#

Here is a sample Go code to delete a "folder" in an S3 bucket:

package main
 
import (
	"fmt"
	"log"
 
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)
 
func deleteFolder(svc *s3.S3, bucket, folder string) error {
	// List all objects with the given folder prefix
	input := &s3.ListObjectsV2Input{
		Bucket: aws.String(bucket),
		Prefix: aws.String(folder),
	}
 
	result, err := svc.ListObjectsV2(input)
	if err != nil {
		return err
	}
 
	// Create a batch delete input
	deleteInput := &s3.DeleteObjectsInput{
		Bucket: aws.String(bucket),
		Delete: &s3.Delete{
			Objects: []*s3.ObjectIdentifier{},
		},
	}
 
	for _, item := range result.Contents {
		deleteInput.Delete.Objects = append(deleteInput.Delete.Objects, &s3.ObjectIdentifier{
			Key: item.Key,
		})
	}
 
	// Delete objects in batches of 1000 (S3 limit)
	for i := 0; i < len(deleteInput.Delete.Objects); i += 1000 {
		end := i + 1000
		if end > len(deleteInput.Delete.Objects) {
			end = len(deleteInput.Delete.Objects)
		}
		batch := &s3.DeleteObjectsInput{
			Bucket: deleteInput.Bucket,
			Delete: &s3.Delete{
				Objects: deleteInput.Delete.Objects[i:end],
			},
		}
		_, err := svc.DeleteObjects(batch)
		if err != nil {
			return err
		}
	}
 
	return nil
}
 
func main() {
	// Create a new session
	sess, err := session.NewSession(&aws.Config{
		Region: aws.String("us-west-2"),
	})
	if err != nil {
		log.Fatalf("Failed to create session: %v", err)
	}
 
	// Create an S3 service client
	svc := s3.New(sess)
 
	// Bucket and folder to delete
	bucket := "my-bucket"
	folder := "path/to/folder/"
 
	err = deleteFolder(svc, bucket, folder)
	if err != nil {
		log.Fatalf("Failed to delete folder: %v", err)
	}
 
	fmt.Printf("Folder %s in bucket %s deleted successfully.\n", folder, bucket)
}
 
 

In this code:

1. We first list all objects with the given folder prefix using ListObjectsV2.
2. Then we create a batch delete input and add all the objects to be deleted to it.
3. Since S3 has a limit of 1000 objects per DeleteObjects request, we split the objects into batches of 1000 and delete them one by one.

Best Practices#

• Error Handling: Always handle errors properly when working with the AWS SDK. This ensures that you can identify and troubleshoot issues effectively.
• Permissions: Make sure your AWS IAM (Identity and Access Management) user or role has the necessary permissions to list and delete objects in the S3 bucket.
• Testing: Before deleting a large number of objects, test your code in a development or staging environment to avoid accidental data loss.
• Logging: Implement proper logging in your code to track the progress of the deletion process and any potential errors.

Conclusion#

Deleting a "folder" in an S3 bucket using Go involves listing all objects with the given folder prefix and then deleting them in batches. By understanding the core concepts, typical usage scenarios, and following best practices, software engineers can effectively manage the deletion of S3 "folders" in their applications.

FAQ#

Q: Can I directly delete a "folder" in S3 without deleting all its contents?#

A: No, S3 doesn't have actual folders. You need to delete all objects whose keys start with the folder's prefix to achieve the effect of deleting a "folder".

Q: Is there a limit to the number of objects I can delete in a single request?#

A: Yes, S3 has a limit of 1000 objects per DeleteObjects request. You need to split your deletion requests into batches of 1000 objects if you have more than 1000 objects to delete.

Q: What if an error occurs during the deletion process?#

A: You should handle errors properly in your code. If an error occurs, you can log the error and take appropriate action, such as retrying the operation or notifying the administrator.

References#

• Amazon S3 Documentation
• AWS SDK for Go Documentation