amazon web services - Can we copy the files and folders recursively between aws s3 buckets using boto3 Python?

Question

Is it possible to copy all the files in one source bucket to other target bucket using boto3. And source bucket doesn't have regular folder structure.

Source bucket: SRC
Source Path: A/B/C/D/E/F..
where in D folder it has some files,
E folder has some files

Target bucket: TGT
Target path: L/M/N/

I need to copy all the files and folders from above SRC bucket from folder C to TGT bucket under N folder using boto3.

Can any one aware of any API or do we need to write new python script to complete this task.

Answer 1

S3 store object, it doesn't store folder, even '/' or '' is part of the object key name. You just need to manipulate the key name and copy the data over.

import boto3
old_bucket_name = 'SRC'
old_prefix = 'A/B/C/'
new_bucket_name = 'TGT'
new_prefix = 'L/M/N/'
s3 = boto3.resource('s3')
old_bucket = s3.Bucket(old_bucket_name)
new_bucket = s3.Bucket(new_bucket_name)

for obj in old_bucket.objects.filter(Prefix=old_prefix):
    old_source = { 'Bucket': old_bucket_name,
                   'Key': obj.key}
    # replace the prefix
    new_key = obj.key.replace(old_prefix, new_prefix, 1)
    new_obj = new_bucket.Object(new_key)
    new_obj.copy(old_source)

Optimized technique of defining new_key suggested by zvikico:

new_key = new_prefix + obj.key[len(old_prefix):]