Question

PLEASE WRITE CLEAR AND CONCISE CODE

Subject is Unix system programming

Write code that downloads all of the log files ending in `.log` in an S3 bucket and counts the total number of HTTP errors in those logs.

Log lines are in the format `{"path": "/", status: 200}` or `{"path": "/", status: 404}`, for example. (This is JSON, and you can process it as such if you choose.)

Use the bucket `class6-logs`, the access key ID "AKIASUMBPHIPY6DLZ4C5", and the secret access key "JQdQIbxsRcipnoKFnsfse44SMRGouzz4tbAzTYbe".

Use the code below  to get started:

Code-

import boto3

client = boto3.client(

's3',

aws_access_key_id="AKIASUMBPHIPY6DLZ4C5",

aws_secret_access_key="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",

)

resp = client.list_objects(Bucket='class6-logs')

client.download_file('class6-logs', resp['Contents'][0]

['Key'], 'downloaded_file')

# for object in resp['Contents']:

# print(object['Key'])

# print(resp['Contents'][0]['Key'])

Answer 1

import boto3
import os

def download_dir(client, resource, dist, local='/tmp', bucket='your_bucket'):
    paginator = client.get_paginator('list_objects')
    for result in paginator.paginate(Bucket=bucket, Delimiter='/', Prefix=dist):
        if result.get('CommonPrefixes') is not None:
            for subdir in result.get('CommonPrefixes'):
                download_dir(client, resource, subdir.get('Prefix'), local, bucket)
        for file in result.get('Contents', []):
            dest_pathname = os.path.join(local, file.get('Key'))
            if not os.path.exists(os.path.dirname(dest_pathname)):
                os.makedirs(os.path.dirname(dest_pathname))
            resource.meta.client.download_file(bucket, file.get('Key'), dest_pathname)

The function is called that way:

Try this code with specifying your values. This will work well with your systems.

def _start():
    client = boto3.client('s3')
    resource = boto3.resource('s3')
    download_dir(client, resource, 'clientconf/', '/tmp', bucket='my-bucket')