SHRateLimitWarning - Statistical API

I wrote a Python script that uses SentinelHubStatisticalDownloadClient (using sentinelhub version 3.5.0), in the same way as described in this tutorial.

When I run it I get SHRateLimitWarning: Download rate limit hit warnings.warn("Download rate limit hit", category=SHRateLimitWarning) and the script seems to never finish execution.

So I changed the script in this way, thinking that I could slow down the script and avoid multiple parallel calls to the API:
client = SentinelHubStatisticalDownloadClient(config=config, n_interval_retries=3, max_retry_threads=1)
client.rate_limit = SentinelHubRateLimit(num_processes=client.config.number_of_download_processes, minimum_wait_time=0.5)
index_stats = client.download(download_requests)

But I still get the same behaviour.

Any help greatly appreciated.

Page 1 / 1

Hi,

For start I suggest to turn on logging to see what is happening in the background of your download process. You can find logging instructions here in docs. Especially the last part about formatting messages of a multi-threaded process should be useful.

Let us know if you find out anything unexpected in logs. I suspect that the download process is simply waiting for a sufficient number of processing units and requests before starting the next download call. You can check your currently available number of processing units and requests also in Sentinel Hub Dashboard under Rate limits.

Hi,
Thank you for your advice.
I managed to save the log and I’m observing an exponential increase in wait time, which leads to the client not being able to proceed.
The problem could be related to the fact that I’m asking for LandSat data for many years (from 2000 to 2017) over different bbox polygons. When testing one bbox at a time, I had issues with the resolution setting and ended up leaving the function default (i.e., I’m not providing any resolution variable as input to the aggregation function).

Here is the log (I replaced some codes with XXX, not knowing which ones were account secrets):

DEBUG:sentinelhub.download.session:ThreadPoolExecutor-0_0:Creating a new authentication session with Sentinel Hub service
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Encoding `client_id` "XXXXXXXXXXXXXXX" with `client_secret` as Basic auth credentials.
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Requesting url https://services.sentinel-hub.com/oauth/token using method POST.
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'client_credentials'}
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x0000014B06474D48>, 'verify': True, 'proxies': None, 'cert': None}.
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_0:Starting new HTTPS connection (1): services.sentinel-hub.com:443
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_0:https://services.sentinel-hub.com:443 "POST /oauth/token HTTP/1.1" 200 931
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Request to fetch token completed with status 200.
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Request url was https://services.sentinel-hub.com/oauth/token
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Request headers were {'User-Agent': 'python-requests/2.27.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8', 'Content-Length': '29', 'Authorization': 'Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Request body was grant_type=client_credentials
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Response headers were {'Date': 'Thu, 21 Apr 2022 09:10:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '931', 'Connection': 'keep-alive', 'cache-control': 'no-cache', 'set-cookie': 'shs1=XXXXXXXXXXXXXXXXXXXfab;Version=1;Domain=services.sentinel-hub.com;Path=/oauth/;Secure;HttpOnly;SameSite=None', 'access-control-allow-origin': '*', 'access-control-allow-headers': 'origin,content-type,accept,accept-crs,authorization,cache-control', 'access-control-allow-credentials': 'true', 'access-control-allow-methods': 'GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH', 'access-control-max-age': '3600'} and content {"access_token":"ABCXXXXXXXXX","expires_in":3599}.
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Invoking 0 token response hooks.
DEBUG:requests_oauthlib.oauth2_session:ThreadPoolExecutor-0_0:Obtained token {'access_token': 'ABCXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'expires_in': 3599, 'expires_at': 1650535802.3620963}.
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_2:Sending POST request to https://services-uswest2.sentinel-hub.com/api/v1/statistics. Hash of sent request is 05b12fea7b798020624dc0fd9bc9d035
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Request needs to wait. Sleeping for 0.05
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_4:Request needs to wait. Sleeping for 0.04
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_1:Request needs to wait. Sleeping for 0.04
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_0:Starting new HTTPS connection (1): services-uswest2.sentinel-hub.com:443
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_2:Starting new HTTPS connection (1): services-uswest2.sentinel-hub.com:443
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_1:Sending POST request to https://services-uswest2.sentinel-hub.com/api/v1/statistics. Hash of sent request is ddf91a9af4a7b5d59c21327144e0b008
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Request needs to wait. Sleeping for 0.05
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_4:Request needs to wait. Sleeping for 0.05
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_1:Starting new HTTPS connection (1): services-uswest2.sentinel-hub.com:443
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_4:Sending POST request to https://services-uswest2.sentinel-hub.com/api/v1/statistics. Hash of sent request is 8f85019c6d15d1395906f36b00ed63d9
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Request needs to wait. Sleeping for 0.05
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_4:Starting new HTTPS connection (1): services-uswest2.sentinel-hub.com:443
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Sending POST request to https://services-uswest2.sentinel-hub.com/api/v1/statistics. Hash of sent request is 1ccaa8bee8ea03a03cb18da8668f497e
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_3:Starting new HTTPS connection (1): services-uswest2.sentinel-hub.com:443
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_4:https://services-uswest2.sentinel-hub.com:443 "POST /api/v1/statistics HTTP/1.1" 429 293
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_4:Request needs to wait. Sleeping for 89.22
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_3:https://services-uswest2.sentinel-hub.com:443 "POST /api/v1/statistics HTTP/1.1" 429 293
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Request needs to wait. Sleeping for 89.18
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_0:https://services-uswest2.sentinel-hub.com:443 "POST /api/v1/statistics HTTP/1.1" 429 296
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_0:Request needs to wait. Sleeping for 11952.99
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_1:https://services-uswest2.sentinel-hub.com:443 "POST /api/v1/statistics HTTP/1.1" 429 296
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_1:Request needs to wait. Sleeping for 11943.50
DEBUG:urllib3.connectionpool:ThreadPoolExecutor-0_2:https://services-uswest2.sentinel-hub.com:443 "POST /api/v1/statistics HTTP/1.1" 429 296
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_2:Request needs to wait. Sleeping for 12597.55
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_4:Request needs to wait. Sleeping for 12529.09
DEBUG:sentinelhub.download.sentinelhub_client:ThreadPoolExecutor-0_3:Request needs to wait. Sleeping for 12529.05

I see you are making Statistical API requests for very long time intervals. This means the data will have to be collected from many time frames and the total processing units (PU) cost will be a sum of PU costs of all time frames in the interval. Therefore, the download process is waiting longer for your available PU to recharge before making the next request.

I recommend splitting the time interval into smaller parts and making requests for each of them. The total download time might not decrease that much but at least you will be getting responses back more frequently.

Also note that the amount of your available PU and therefore the speed of download depends on you pricing plan of Sentinel Hub account. If you are planning a large-scale processing then upgrading the pricing plan would be the only way to really speed up the download.

Thank you, I solved by splitting the request into smaller time chunks, as you suggest.
I am aware of the limit and am considering changing plan indeed.

Reply

Sign up.

Login with Planet Account

Login to the Planet community

Login with Planet Account

Scanning file for viruses.

This file cannot be downloaded