Max retires exceeded when dowloading data from AWS

  • 26 April 2024
  • 3 replies
  • 2 views

I try to download data from AWS, using sentinelhub-py, with following code:

import os
from datetime import datetime

from sentinelhub import CRS, BBox, DataCollection, SentinelHubCatalog, SHConfig
from sentinelhub.aws import AwsDownloadClient

boto_params = {"RequestPayer": "requester"}
config = SHConfig()
config.sh_client_id = "my client id"
config.sh_client_secret = "my client secret"
config.instance_id = "my instance id"
config.aws_access_key_id = "my access key id"
config.aws_secret_access_key = "my secret access key"
s3_client = AwsDownloadClient.get_s3_client(config)

def download(s3_tile_path, download_dir, objects_to_download=None):
    os.makedirs(download_dir, exist_ok=True)
    all_files = list_tile_objects(s3_tile_path)

    for file in all_files:
        file_name = file["Key"].split("/")[-1]
        if not objects_to_download or file_name in objects_to_download:
            out_path = os.path.join(download_dir, file_name)
            _, _, bucket_name, _ = s3_tile_path.split("/", 3)
            s3_client.download_file(Bucket=bucket_name, Key=file["Key"], Filename=out_path, ExtraArgs=boto_params)

files_to_download = ["B04.jp2", "B07.jp2"]
download("s3://sentinel-s2-l2a/tiles/51/R/UH/2023/4/10/0/", "images/", files_to_download)

The code output error:

RetriesExceededError                      Traceback (most recent call last)
File ~/anaconda3/envs/wbcrawler/lib/python3.9/site-packages/boto3/s3/transfer.py:326, in S3Transfer.download_file(self, bucket, key, filename, extra_args, callback)
    325 try:
--> 326     future.result()
    327 # This is for backwards compatibility where when retries are
    328 # exceeded we need to throw the same error from boto3 instead of
    329 # s3transfer's built in RetriesExceededError as current users are
    330 # catching the boto3 one instead of the s3transfer exception to do
    331 # their own retries.

File ~/anaconda3/envs/wbcrawler/lib/python3.9/site-packages/s3transfer/futures.py:103, in TransferFuture.result(self)
     99 try:
    100     # Usually the result() method blocks until the transfer is done,
    101     # however if a KeyboardInterrupt is raised we want want to exit
    102     # out of this and propagate the exception.
--> 103     return self._coordinator.result()
    104 except KeyboardInterrupt as e:

File ~/anaconda3/envs/wbcrawler/lib/python3.9/site-packages/s3transfer/futures.py:266, in TransferCoordinator.result(self)
    265 if self._exception:
--> 266     raise self._exception
    267 return self._result

File ~/anaconda3/envs/wbcrawler/lib/python3.9/site-packages/s3transfer/tasks.py:139, in Task.__call__(self)
...
    331 # their own retries.
    332 except S3TransferRetriesExceededError as e:
--> 333     raise RetriesExceededError(e.last_exception)

RetriesExceededError: Max Retries Exceeded

Anyone can help? Thank advance!


3 replies

Hi @hminhtri19 ,

Unfortunately, I can’t reproduce your error. Could you please make sure the configuration is correct and the list_tile_objects is defined correctly, then try to run the code again?

I solved it. It’s my error on aws keys.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Reply


Planet Monitoring
Investors

© 2024 Planet Labs PBC. All rights reserved.
| Privacy Policy | California Privacy Notice | California Do Not Sell
Your Privacy Choices | Cookie Notice | Terms of Use