AWS CodePipeline Not Detecting New Push in S3

AWS CodePipeline is a powerful continuous delivery service that helps you automate your software release processes. It allows you to build, test, and deploy your applications with ease. Amazon S3 (Simple Storage Service) is a scalable object storage service that can be used as a source for CodePipeline. However, sometimes you may encounter an issue where CodePipeline fails to detect new pushes in an S3 bucket. This blog post will delve into the core concepts, typical usage scenarios, common practices, and best practices to address this problem.

Table of Contents#

  1. Core Concepts
  2. Typical Usage Scenarios
  3. Common Reasons for CodePipeline Not Detecting New Pushes in S3
  4. Common Practices to Troubleshoot
  5. Best Practices to Avoid the Issue
  6. Conclusion
  7. FAQ
  8. References

Core Concepts#

AWS CodePipeline#

AWS CodePipeline is a fully managed continuous delivery service that enables you to automate your release pipelines. It orchestrates the build, test, and deployment phases of your application. CodePipeline can integrate with various AWS services and third - party tools, such as AWS CodeBuild, AWS CodeDeploy, and GitHub.

Amazon S3#

Amazon S3 is an object storage service that offers industry - leading scalability, data availability, security, and performance. You can use an S3 bucket as a source for CodePipeline. When new objects are uploaded to the S3 bucket, CodePipeline can be configured to detect these changes and trigger the pipeline.

S3 Event Notifications#

S3 event notifications are used to notify other AWS services when certain events occur in an S3 bucket. For CodePipeline to detect new pushes in an S3 bucket, it relies on S3 event notifications. When a new object is created in the bucket, S3 can send a notification to an Amazon Simple Notification Service (SNS) topic or an Amazon Simple Queue Service (SQS) queue, which can then trigger CodePipeline.

Typical Usage Scenarios#

  1. Automated Deployment: You have a development team that pushes new application artifacts to an S3 bucket. CodePipeline is configured to detect these new pushes and automatically deploy the updated application to a production environment.
  2. Continuous Integration: Developers push code changes to an S3 bucket, and CodePipeline triggers a build and test process. If the tests pass, the pipeline can proceed with the deployment.
  3. Data Processing: New data files are uploaded to an S3 bucket, and CodePipeline is used to trigger a data processing pipeline, such as running ETL (Extract, Transform, Load) jobs.

Common Reasons for CodePipeline Not Detecting New Pushes in S3#

Incorrect S3 Event Configuration#

  • Missing Notifications: The S3 bucket may not be configured to send event notifications for object creation. Without these notifications, CodePipeline has no way of knowing when new objects are added to the bucket.
  • Incorrect Destination: The event notifications may be sent to the wrong SNS topic or SQS queue, or the destination may not be properly configured to trigger CodePipeline.

Permissions Issues#

  • Bucket Policy: The S3 bucket policy may not grant the necessary permissions for CodePipeline to access the bucket and receive event notifications.
  • IAM Roles: The IAM roles associated with CodePipeline may not have the appropriate permissions to access the S3 bucket and the SNS topic or SQS queue used for event notifications.

SNS or SQS Issues#

  • Subscription Problems: If SNS is used for event notifications, the subscription to the SNS topic may be inactive or misconfigured.
  • Queue Delays: In the case of SQS, there may be delays in message processing, or the queue may be full, preventing CodePipeline from receiving timely notifications.

Versioning and Object Key Issues#

  • Versioning Enabled: If S3 versioning is enabled on the bucket, CodePipeline may have issues detecting new versions of objects if the configuration is not set up correctly.
  • Object Key Filtering: If CodePipeline is configured with object key filters, new objects may not match the filter criteria, causing the pipeline not to trigger.

Common Practices to Troubleshoot#

Check S3 Event Configuration#

  • Verify Notifications: Log in to the AWS Management Console and navigate to the S3 bucket. Check the event notifications settings to ensure that object creation events are configured to send notifications to the correct SNS topic or SQS queue.
  • Test Notifications: You can manually upload a test object to the S3 bucket and check if the SNS topic or SQS queue receives the notification.

Review Permissions#

  • Bucket Policy: Review the S3 bucket policy to ensure that it allows CodePipeline to access the bucket and receive event notifications.
  • IAM Roles: Check the IAM roles associated with CodePipeline and ensure that they have the necessary permissions to access the S3 bucket, SNS topic, and SQS queue.

Check SNS and SQS#

  • SNS Subscriptions: Verify that the subscription to the SNS topic is active and that the endpoint is correctly configured to trigger CodePipeline.
  • SQS Queue Status: Check the SQS queue for any messages and ensure that there are no delays or issues with message processing.

Review Versioning and Object Key Configuration#

  • Versioning: If S3 versioning is enabled, review the CodePipeline configuration to ensure that it is set up to handle new object versions correctly.
  • Object Key Filters: Check the object key filters in the CodePipeline configuration and ensure that new objects match the filter criteria.

Best Practices to Avoid the Issue#

Proper Initial Configuration#

  • Set Up Event Notifications Correctly: When creating the S3 bucket and configuring CodePipeline, ensure that S3 event notifications are set up correctly from the start.
  • Define Clear Permissions: Create well - defined IAM roles and bucket policies that grant the necessary permissions for CodePipeline to access the S3 bucket and receive event notifications.

Regular Monitoring#

  • Monitor SNS and SQS: Use AWS CloudWatch to monitor the SNS topic and SQS queue for any issues or delays in message processing.
  • Check Pipeline Status: Regularly check the status of CodePipeline to ensure that it is functioning as expected.

Documentation and Testing#

  • Document Configuration: Keep detailed documentation of the S3 bucket configuration, CodePipeline settings, and IAM roles.
  • Conduct Regular Tests: Periodically test the pipeline by uploading new objects to the S3 bucket to ensure that CodePipeline can detect the changes and trigger the pipeline.

Conclusion#

The issue of AWS CodePipeline not detecting new pushes in an S3 bucket can be frustrating, but by understanding the core concepts, typical usage scenarios, common reasons, and following the troubleshooting and best practices outlined in this blog post, you can effectively address and prevent this problem. Proper configuration, permissions management, and regular monitoring are key to ensuring that CodePipeline functions smoothly and automatically detects new pushes in S3.

FAQ#

Q: Can I use S3 event notifications without SNS or SQS?#

A: No, CodePipeline relies on SNS or SQS to receive event notifications from S3. You need to configure either an SNS topic or an SQS queue to trigger CodePipeline when new objects are created in the S3 bucket.

Q: How can I check the S3 bucket policy?#

A: You can check the S3 bucket policy in the AWS Management Console. Navigate to the S3 bucket, click on the "Permissions" tab, and then click on "Bucket policy".

Q: What should I do if the SNS subscription is inactive?#

A: You can re - activate the SNS subscription in the AWS Management Console. Navigate to the SNS service, select the topic, and then click on the subscription to re - subscribe.

References#

2025-10