Uploading files to AWS s3

For you to be able to upload to AWS S3 and sync your database with your S3 bucket, you need to set up S3 Bucket and SNS. Let's get started.

Configuration

1. Sign in or sign up on AWS
2. Go to your AWS console
3. Account menu in the upper-right (has your name on it) then click Security Credentials
4. Create your access key
5. Update your .env.local

AWS_ACCESS_KEY_ID= # Your AWS Key ID
AWS_SECRET_ACCESS_KEY= # Your AWS ACCESS Key

2. Update config.ts

const config = {
  //...
  // AWS S3 configuration for storing media files.
  aws: {
    region: "my-bucket-region", // AWS region where the S3 bucket is hosted.
    bucketName: "my-bucket-name", // Name of the S3 bucket.
    hostname: "my-bucket-name.s3.my-bucket-region.amazonaws.com", // Hostname for accessing the bucket.
    bucketUrl: "https://hostname", // Full URL to the S3 bucket.
  },

  // Configuration for file uploads.
  upload: {
    imageExtensions: [".jpg", ".jpeg", ".png", ".gif", ".webp"], // Allowed image file extensions.
    videoExtensions: [".mp4"], // Allowed video file extensions.
  },
};

SNS (Simple Notification Service)

SNS is a service offered by AWS one of its features is notify you when an s3 object is created/updated/deleted.

We will use it so when we upload or delete any s3 object in the admin dashboard our database will update itself based on the actions we take.

You can think of it as a webhook, in fact I have added the API route for handling these notifications under /api/webhooks/s3

Testing Locally

Because SNS does not work with localhost we need a service that will act as a proxy to redirect traffic from AWS to localhost.

The service I have used is called ngrok

Setup ngrok (skip for production)

1. Sign up here
2. On successful sign up you should be able to access their guide here
3. Choose your OS if not auto selected and follow their installation steps
4. Then in your terminal/cmd run ngrok http http://localhost:3000

Setup SNS

1. Go to your AWS console
2. Enter SNS in the search bar and select it
3. Create a topic by giving it a name and make sure the type is Standard
4. Click Create Topic then open it if it didn't automatically
5. Copy the topic (ARN) and Topic owner then click Edit
6. Expand Access Policy and enter this policy

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sns:Publish",
      "Resource": "your_topic_arn_here",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "topic_owner_number"
        }
      }
    }
  ]
}

6. Save changes
7. Under Subscriptions click Create subscription
8. Select Topic ARN you just created and the https Protocol
9. For localhost type the URL you got from ngrok and for production your app url followed by /api/webhooks/s3
   • e.g. https://your_domain/api/webhooks/s3
10. Click create subscription and then wait for your subscription status to become Confirmed

You can repeat steps 7-10 for both localhost and production

Update the S3 SNS logic

You can find the S3 SNS under src/app/api/webhooks/s3/route.ts feel free to explore.

Setup AWS S3 bucket

1. Go to your AWS console
2. Create S3 bucket by typing s3 in the search bar and select it
3. Click Create bucket
4. Pick a name for your bucket and uncheck Block all public access then click Create bucket
5. Open your new bucket and click on Properties tab then copy the bucket (ARN) code (keep it aside you will need it later)
   • Click on Create event notification
   • Pick a name
   • Add a prefix media/ (highly recommended so you can manually update your s3 to this bucket without triggering notification on the dashboard upload folder and causing a crash)
   • Check All object create events and All object removal events
   • Select SNS Topic as Destination
   • Choose the SNS topic that you created earlier.
   • Save the changes
6. Click on Permissions tab and Edit the Bucket policy and enter this policy instead to allow public access to view s3 objects then save

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "paste_your_ARN_here/media/*" # only allow public access to /media folder in the s3 bucket
        }
    ]
}

7. Edit Cross-origin resource sharing (CORS) and enter these rules then save

[
    {
        "AllowedHeaders": [
            "Authorization",
            "x-amz-date",
            "x-amz-content-sha256",
            "content-type"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST"
        ],
        "AllowedOrigins": [
            "http://localhost:3000",
            "https://your_app_domain"
        ],
        "ExposeHeaders": [
            "ETag",
            "Location"
        ],
        "MaxAgeSeconds": 3000
    },
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

🎉 S3 and SNS are now ready! Start by testing uploading under /dashboard

Useful Components

• FileUploader