Batch Statistics API sending failed responses

Hi,

Thanks for the question. Are you able to share the payload that you are using in your request? Then we can help you debug your issue. In addition, you say you are using different dates, have you checked whether data is available for those dates?

Hi,

I am using somewhere between 100 - 300 Processing Units per request. And yes, data is available between those dates.

OK again, can I ask you to share the request payload (including the AOI and time range you are using) so that I can help you debug your code.

Thanks

The area we are using to analyze is attached here. The number of polygon requests that are being sent are approximately 6000, and the processing units being used are about 125. The dates I am trying to get data for are between 2022 Jam - 2022 July with a 1D interval

boundary_aoi.kml (16.5 KB)

can you please check this?

Hi, can I ask you to share the request payload including the AOI and time range you are using (this means the code you are using to make your request), so that I can help you debug your code. Preferably if you can share the code that worked and the code that doesn’t work then that would be really useful 🙂

The code is the same as before - it worked well in the past. It doesn’t work now. Here it is:

 aggregation = SentinelHubStatistical.aggregation(
        evalscript=evalscript,
        time_interval=time_int,
        aggregation_interval="P30D",
        resolution=(10, 10),
    )
    calculations = {"default": {"statistics": {"default": {"percentiles": {"k": [5, 50, 95]}}}}}
    input_data = [SentinelHubStatistical.input_data(DataCollection.SENTINEL2_L1C, maxcc=0.1)]
    input_features = SentinelHubBatchStatistical.s3_specification(input_rp_gpkg, aws_access_key_id, aws_secret_access_key)
    output = SentinelHubBatchStatistical.s3_specification(out_folder_s3_junk, aws_access_key_id, aws_secret_access_key)
    client = SentinelHubBatchStatistical(config)
    request = client.create(
        input_features=input_features,
        input_data=input_data,
        aggregation=aggregation,
        calculations = calculations,
        output=output,
    )

    client.start_job(request)
    client.get_status(request)

    logging.basicConfig(level=logging.INFO)
    monitor_batch_statistical_job(request, config=config, analysis_sleep_time=60)

Problem is, I am checking the sentinel hub dashboard and it is not updated status of request for a long while either. When it does, it sends responses with Failed status. I just sent a request for a single polygon and it has been on the same status for 15 minutes now.

OK thank you, and at which stage is the request failing? The ANALYSE stage or PROCESSING stage? And what is the exact error message you receive back?

Currently it is stuck in ANALYSIS_DONE stage, last few days it went through PROCESSING (that took a very long time) and then was showing DONE on the dashboard. However, the json had no data. A sample response would be:

{“id”:1,“identifier”:“1”,“response”:{“data”:null,“status”:“FAILED”}}

an update - it is now executed, but it took 100 Processing Units for a single polygon for dates between 2023-01-01 and 2023-01-30 for a single index. Can you tell me if this is an issue with the API?

That is good to hear. Please bear in mind that the minimum PUs for a Batch Statistical API request is 100 processing units [1].

Okay, understood. But can you please comment on the speed issue and failed requests?

Hi,

Batch Statistical API is Asynchronous, which means that the request isn’t returned instantly. If it was slow, it may have been queued in the cloud instance used for processing. In addition, please note the API is still in beta so does misbehave sometimes.