MERN Secure File Uploads with AWS S3

Before diving into the code, let's first compare uploading a file directly to your server and then to AWS, versus using a presigned URL to upload and retrieve from AWS. This comparison will help us understand the need for presigned URLs.

Uploading a File to Your Server, Then to AWS

Process

Client Uploads to Server: The client uploads the file to your server.
Server Processes the File: The server receives the file, may process it, and then uploads it to AWS S3.
Server Stores and Manages Files: The server might keep a copy or metadata of the file for reference.

Pros

Control: Full control over the file during the upload process. we can perform any necessary processing, validation, or transformations on the file.
Security: Direct interaction with our server allows us to implement custom security measures and access controls.

Cons

Bandwidth and Load: High server load and bandwidth usage due to the need to handle file uploads and then transfer to AWS.
Scalability: Can become a bottleneck with high traffic or large files, requiring significant infrastructure to handle scale.
Latency: Additional latency is introduced due to the double hop (client to server, then server to AWS).

Using a Presigned URL to Upload and Retrieve Files to/from AWS

Process

Generate Presigned URL: There are two API calls for uploading. First, we need to get the presigned URL. To do this, send metadata and your authorization header for authentication. After successful authentication, the server will call the function to generate the presigned URL, and the user will receive the URL in the response.
Client Uploads: The client uses the presigned URL to upload the file.

Note: We will discuss the retrivel part in the below sections.

Pros

Reduced Server Load: Offloads the bandwidth and processing burden from your server, as the file transfer happens directly between the client and AWS.
Scalability: Easily handles high traffic and large files since AWS infrastructure manages the upload/download process.
Performance: Lower latency due to the direct client-to-AWS connection, improving upload and download speeds.
Security: Presigned URLs have time-limited permissions, reducing the risk of unauthorized access. The URLs can also be configured to allow only specific actions (upload or download).

Cons

Control: Limited control over the file during the upload/download process.

In summary, the best way to upload and retrieve files can be using a presigned URL because it reduces the server's workload by allowing direct uploads to AWS. Additionally, if you are concerned about users uploading viruses or infected files to your S3, you can create a Lambda function (a serverless function) that triggers whenever an upload happens on S3.

Note: AWS does not cost for upload.

Now before going to the project set up and code follow this step on AWS:

Create an S3 bucket in your preferred region (ideally where your most users are located).

After click on the create bucket give the bucket name and it must be unique like your domain name.

After naming the bucket, review the settings. If you want to change any settings, you can do so; otherwise, you can leave them as default. Finally, click the "Create" button.

Note: We are creating a secure upload and retrieval process, so there is no need to make the bucket policies public.

Now your bucket is created, and you can upload and retrieve your files. Yes, AWS makes this part easy, but directly accessing the bucket from your account is not a good approach. Let's create an IAM role (for application-level access) and grant that role access to your bucket.

Provide only the necessary access to the role. Since we are creating this role for S3 access, there is no need to provide AWS console access.

In the next step, attach the policy by searching for "S3" and selecting "S3 Full Access." Then, click the "Next" button.

In the final step, review your settings and click the "Create" button. Congratulations, you have successfully created the IAM user.

Now we will create an access key and secret key to access the bucket from the user we just created.

After this, you will see a page where you need to click "Create" to get your access key and secret key.

Now, to access the S3 bucket from your application, you need to configure the CORS policy.

Scroll down to find Cross-origin resource sharing (CORS) and click on "Edit." Then, define the CORS policy and hit the save button.

Note: In the "Allow Origins" section, replace the asterisk (*) with the URL you are using to request uploads and retrievals.

Wonderful, your setup on AWS S3 is done. Now, let's move on to the project setup.

Let's first set up the back-end. Create an empty folder and open it in VS Code or your preferred editor. Run the command npm init -y to initialize the project.

below are the folders structure.

server/
├── models/
│   └── Image.js
├── routes/
│   └── upload.js
├── .env
├── package.json
├── server.js
└── ...

Let's first install the dependencies by running the following command:

npm i express mongoose dotenv cors body-parser aws-sdk/s3-request-presigner aws-sdk/client-s3

Now, we will create the files and add the following code to each respective file. I will include comments to explain the code.

//server/.env 
MONGO_URI=mongodb+srv:your mongo url
AWS_ACCESS_KEY_ID= your access key
AWS_SECRET_ACCESS_KEY=your seceret key
AWS_BUCKET_NAME=bucket name
AWS_REGION=regin
PORT=5000

//server/models/Image.js
const mongoose = require('mongoose');

// Define the schema for an image document in MongoDB
const imageSchema = new mongoose.Schema({
  fileName: String,
  fileType: String,
  fileSize: Number,
  url: String,
});

module.exports = mongoose.model('Image', imageSchema);

upload.js file code:

// server/routes/upload.js
require('dotenv').config();
const express = require('express');
const router = express.Router();
const { S3Client, PutObjectCommand, GetObjectCommand } = require('@aws-sdk/client-s3');
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
const Image = require('../models/Image');



// Configure AWS
const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});

// Middleware to check if user is verified
const isAuthenticated = (req, res, next) => {
  // your authentication logic here 
  next();
};

// Route to generate a presigned URL for uploading an image
router.post('/generate-presigned-url', isAuthenticated, async (req, res) => {
  const { fileName, fileType, fileSize } = req.body;

  try {
    // Create a command to put an object in the S3 bucket
    const command = new PutObjectCommand({
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: 'uploads/' + fileName,
      ContentType: fileType,
    });

    // Generate a presigned URL for the command with a 1-hour expiration
    const presignedUrl = await getSignedUrl(s3Client, command, { expiresIn: 3600 });

    // Create a new Image document and save it to the database
    const newImage = new Image({
      fileName,
      fileType,
      fileSize,
      url: presignedUrl,
    });

    await newImage.save();

    // Send the presigned URL and the image URL in the response
    res.json({ presignedUrl, url: presignedUrl });
  } catch (err) {
    console.error('Error generating presigned URL:', err);
    res.status(500).send('Server Error');
  }
});

router.get('/image/:fileName', isAuthenticated, async (req, res) => {
  const { fileName } = req.params;

  try {
    const command = new GetObjectCommand({
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: 'uploads/' + fileName,
    });

    // Generate a pre-signed URL
    const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 }); // URL expires in 1 hour

    res.json({ url });
  } catch (err) {
    console.error('Error getting the image:', err);
    res.status(500).send('Server Error');
  }
});


module.exports = router;

//server/server.js
const express = require('express');
const mongoose = require('mongoose');
const cors = require('cors');
const bodyParser = require('body-parser');
require('dotenv').config();

const app = express();
app.use(cors());
app.use(bodyParser.json());

const PORT = process.env.PORT || 5000;

// Connect to MongoDB
mongoose.connect(process.env.MONGO_URI, { useNewUrlParser: true, useUnifiedTopology: true })
  .then(() => console.log('MongoDB connected'))
  .catch(err => console.error(err));

// Import routes
const uploadRoute = require('./routes/upload');
app.use('/api', uploadRoute);

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

Now that you have successfully set up the back-end code, let's move on to the front-end setup.

Run npx create-react-app my-app to create the React app. Then, install Axios to fetch URLs by running npm i axios.

client/
├── src/
│   ├── App.css  
│   ├── App.js
│   ├── index.js
│   └── ...
├── package.json

App.js file code:

import React, { useState } from 'react';
import axios from 'axios';
import './App.css'; 

const App = () => {
  const [file, setFile] = useState(null);
  const [imageUrl, setImageUrl] = useState('');

  // Handle file input change
  const handleFileChange = (e) => {
    setFile(e.target.files[0]);
  };

  // Handle file upload
  const handleUpload = async () => {
    if (!file) return;

    try {
      // Get presigned URL from backend
      const { data } = await axios.post('http://localhost:5000/api/generate-presigned-url', {
        fileName: file.name,
        fileType: file.type,
        fileSize: file.size,
      });

      const { presignedUrl, url } = data;

      // Upload file to S3 using the presigned URL
      await axios.put(presignedUrl, file, {
        headers: {
          'Content-Type': file.type,
        },
      });

      // Fetch the image using the URL
      const imageResponse = await axios.get(`http://localhost:5000/api/image/${file.name}`);

      // Set image URL to display
      setImageUrl(imageResponse.data.url);
    } catch (error) {
      console.error('Error uploading file:', error);
    }
  };

  return (
    <div className="container">
      <input type="file" onChange={handleFileChange} className="file-input" />
      <button onClick={handleUpload} className="upload-button">Upload</button>
      {imageUrl && (
        <div className="image-container">
          <img src={imageUrl} alt="Uploaded" className="uploaded-image" />
        </div>
      )}
    </div>
  );
};

export default App;

.container {
  display: flex;
  flex-direction: column;
  align-items: center;
  margin-top: 50px;
}

.file-input {
  margin-bottom: 20px;
}

.upload-button {
  padding: 10px 20px;
  background-color: #007bff;
  color: white;
  border: none;
  border-radius: 5px;
  cursor: pointer;
}

.upload-button:hover {
  background-color: #0056b3;
}

.image-container {
  margin-top: 20px;
}

.uploaded-image {
  width: 300px;
  border: 2px solid #ddd;
  border-radius: 10px;
}

Now run the frontend and backend, and you will see the output after uploading, as shown below.

Conclusion:

Using presigned URLs significantly reduces server load, enhances scalability, and ensures secure, efficient file uploads and retrievals. This approach leverages AWS infrastructure for better performance and security.

Thank you for following along! If you have any questions or need further assistance, feel free to reach out. Happy coding!

Linkedin: View LinkedIn | GitHub: View GitHub | Email: ikarankhatik@gmail.com

Implementing Scalable and Secure File Uploads and retrieve in MERN with AWS S3

Uploading a File to Your Server, Then to AWS

Process

Pros

Cons

Using a Presigned URL to Upload and Retrieve Files to/from AWS

Process

Pros

Cons

Create an S3 bucket in your preferred region (ideally where your most users are located).

Conclusion: