Have you ever needed to rename a file on Amazon S3? Officially, the only way to do so is to download the file, change it's name, upload it again, then delete the original file.

Well, it turns out there is a way to do it using the AWS CLI without the need of any third party tools, and it is quite simple. Moreover, this technique can also be used to rename a bunch of files with little trouble.

EDIT: As Reddit user Neres28 pointed out, the download operation does not need to be done to your computer: it can be done from S3 to S3 directly with the PUT copy API operation. This was not clear in the first version of this post but was, actually, the point in it. Thanks again to Neres28 for the feedback

Assuming you have the AWS Command Line Interface correctly installed and set up with your credentials, you should be able to list your S3 buckets like this:

$ aws s3 ls

2015-06-16 12:47:13 sample-bucket-1
2015-06-16 17:36:05 sample-bucket-2

Renaming a single file

Now, let us assume there is a file on sample-bucket-1 called incorrectly-named-file.txt and that you would like to rename it to correctly-named-file.txt. In this scenario, a simple mv operation between two S3 objects (the original file and the new one) would suffice:

$ aws s3 ls sample-bucket-1
2016-04-12 15:13:20          0 incorrectly-named-file.txt

$ aws s3 mv s3://sample-bucket-1/incorrectly-named-file.txt s3://sample-bucket-1/correctly-named-file.txt
move: s3://sample-bucket-1/incorrectly-named-file.txt to s3://sample-bucket-1/correctly-named-file.txt

$ aws s3 ls sample-bucket-1/
2016-04-12 15:14:39          0 correctly-named-file.txt

Renaming multiple files

Sometimes, you will have a list of files that follow some kind of pattern that you need to replace. Let us see an example:

$ aws s3 ls sample-bucket-1/
2016-04-12 15:21:04          0 file-replaceme-1.txt
2016-04-12 15:21:05          0 file-replaceme-2.txt
2016-04-12 15:21:06          0 file-replaceme-3.txt
2016-04-12 15:21:07          0 file-replaceme-4.txt
2016-04-12 15:21:08          0 file-replaceme-5.txt

Here, we would like to change replaceme for replaced for all files. First, we are going to need a list of just the filenames:

$ aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "/" | grep replaceme | cut  -f 3
file-replaceme-1.txt
file-replaceme-2.txt
file-replaceme-3.txt
file-replaceme-4.txt
file-replaceme-5.txt

Wait, what have we done here?

  • We have used the s3api subcommand to interact directly with bucket API operations as opposed to file operations.
  • We have asked for the list-objects operation because we want a list of S3 objects.
  • We have set the prefix to empty ("") because our objects happen to be in the root of the bucket.
    • If objects were located under a prefix (S3's concept of folders, so to speak), we would have used the prefix name instead: --prefix parentfolder/ or --prefix parentfolder1/parentfolder2, etc.
  • We have set a / delimiter to prevent recursion. Since prefixes always end with a /, we are effectively discarding them from the search results.
  • We grep the results to match our search string ("replaceme"). This is done to filter out some output lines that we do not need right now.
  • We finally cut that output to select the third tab-separated field of each line, because there's where the file name is.

Now we are ready to create a for loop that iterates over these file names in order to execute an s3 mv operation:

$ for f in $(aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "/" | grep replaceme | cut  -f 3);
 do aws s3 mv s3://sample-bucket-1/$f s3://sample-bucket-1/${f/replaceme/replaced}
done

Again, what have we done here?

  • We have created a for loop in bash that iterates over the results of the command before
  • On each iteration, we call aws s3 because we want to operate at the S3 file level
  • We issue mv commands to rename the files just as we learned at the beginning of this post
  • We use bash's native substitution syntax to replace replaceme with replaced: ${f/replaceme/replaced}, where:
    • ${var/str1/str2} is the replace command syntax
    • var (f) is the variable name that comes from the for loop
    • str1 (replaceme) is the name of the substring in var that we want to replace
    • str2 (replaced) is the value we want to have instead of str1 when it is found

And the results are as expected:

$ aws s3 ls sample-bucket-1
2016-04-12 15:53:42          0 file-replaced-1.txt
2016-04-12 15:53:43          0 file-replaced-2.txt
2016-04-12 15:53:45          0 file-replaced-3.txt
2016-04-12 15:53:46          0 file-replaced-4.txt
2016-04-12 15:53:48          0 file-replaced-5.txt

I hope this is useful to you.