Using python boto3 Check if a folder is in an S3 bucket and if not in...

Using python boto3

Check if a folder is in an S3 bucket and if not in the bucket then it creates a folder in the right directory

Solutions

Expert Solution

import boto3
import os

s3_client = boto3.client('s3')

def download_dir(prefix, local, bucket, client=s3_client):
"""
params:
- prefix: pattern to match in s3
- local: local path to folder in which to place files
- bucket: s3 bucket with target contents
- client: initialized s3 client object
"""
keys = []
dirs = []
next_token = ''
base_kwargs = {
'Bucket':bucket,
'Prefix':prefix,
}
while next_token is not None:
kwargs = base_kwargs.copy()
if next_token != '':
kwargs.update({'ContinuationToken': next_token})
results = client.list_objects_v2(**kwargs)
contents = results.get('Contents')
for i in contents:
k = i.get('Key')
if k[-1] != '/':
keys.append(k)
else:
dirs.append(k)
next_token = results.get('NextContinuationToken')
for d in dirs:
dest_pathname = os.path.join(local, d)
if not os.path.exists(os.path.dirname(dest_pathname)):
os.makedirs(os.path.dirname(dest_pathname))
for k in keys:
dest_pathname = os.path.join(local, k)
if not os.path.exists(os.path.dirname(dest_pathname)):
os.makedirs(os.path.dirname(dest_pathname))
client.download_file(bucket, k, dest_pathname)

milcah answered 2 years ago

Next > < Previous

Related Solutions

Write a program in Python that walks through a folder tree and searches for files with...

Write a program in Python that walks through a folder tree and searches for files with a certain file extension (such as .pdf or .jpg). Copy these files from whatever location they are in to a new folder. The user can enter an absolute path for the start folder, or if the user does not enter a folder, the current directory is used. Likewise, the user can enter extensions to copy but if the user does not enter an extension,...

Using python: Given a string x, write a program to check if its first character is...

Using python: Given a string x, write a program to check if its first character is the same as its last character. If yes, the code should output "True"

Using the income statement from Folder Factory Financials Folder Factory Financials , prepare a pro forma...

Using the income statement from Folder Factory Financials Folder Factory Financials , prepare a pro forma income statement for 2016 if the company projects sales to be $60,000 in 2016. Balance Sheet Folder Factory, Inc. December 31, 2014 and 2015 2015 2014 Assets Cash $ 1,000 $ 500 Accounts receivable 5,000 4,500 Inventory 7,000 6,000 Total current assets $13,000 $11,000 Gross fixed assets $20,000 $15,000 Less: Accumulated depreciation 10,000 9,000 Net fixed assets $10,000 $ 6,000 Total assets $23,000...

Find the center of S3 and the centralizer of each element of S3

language: python Create a text file in your project folder with at least 20 "quirky sayings"/fortunes...

language: python Create a text file in your project folder with at least 20 "quirky sayings"/fortunes (the only requirement is that they be appropriate for display in class), If I use my own file though, you should handle as many fortunes as I put in. Make each fortune its own line, •in your main function ask the user for the name of the fortunes file.•Create a function which takes the name of the fortunes file as a parameter, open that...

In the root of the project create a folder called res. In that folder create a...

In the root of the project create a folder called res. In that folder create a file called DemoFile.txt Create a class with a static main that tests the ability to resolve and print a Path: Create an instance of a FileSystem class. Create an instance of the Path interface for the DemoFile.txt file. Print the constructed Path with System.out.println() method. Create a class that does the following: Using a pre-Java 7 solution, create a class that tests streams in...

JAVA There is a folder named Recursive folder at the same location as these below files,...

JAVA There is a folder named Recursive folder at the same location as these below files, which contains files and folders. I have read the data of the Recursive folder and stored in a variable. Now they are required to be encrypted and decrypted. You may copy past the files to three different files and see the output. I am confused on how to write code for encrypt() and decrypt() The names of files and folders are stored in one...

I. At the beginning, create a folder named "test" and add the folder into the version...

I. At the beginning, create a folder named "test" and add the folder into the version control tool tracking system. Then create one file named "001" under this folder. Commit any change if necessary. II. Next, add another file named "002" in the same folder and commit the change ("adding") in the version control tool. III. Followed by adding that file, create a branch (namely, "branch-A") in the version control tool and make some changes to the contents in file...

Using a large bucket whose volume is known and measuring the time it takes to fill...

Using a large bucket whose volume is known and measuring the time it takes to fill the bucket with water from a garden hose, determine the mass flow rate and the average velocity of water through the hose. ***Note: It is an experiment you need to do it at home. Instead of a garden hose, you can use the same method on a faucet in your kitchen or in your bathroom. Explain the steps that you follow with actual photographs...

Create this on Python Write a program that asks for three numbers. Check first to see...

Create this on Python Write a program that asks for three numbers. Check first to see that all numbers are different. If they’re not different, then exit the program. Otherwise, display the largest number of the three. Outputs: - Enter the first number: 4 - Enter the second number: 78 - Enter the third number: 8 (The largest number is 78.) Tasks - Complete the algorithm manually without using any built-in functions to find the largest number on list. -...

Subjects