Statistical API Lambda execution error

Hello,

Thank you for highlighting the error, we will have a look into it for you.

To help us replicate the error, are you able to provide the evalscript and the full request too. This can be the curl request or the python code you may have been using.

Thanks!

Hi, today the script return all data and the error was not there… in any case, this is the script:

evalscript =’’’
//VERSION=3
function setup() {
return {
input: [“B01”,“B02”,“B03”,“B04”,“B05”,“B06”,“B07”,“B10”,“BQA”,“dataMask”],
output: [
{id:“coastal_blue”,bands:1,sampleType:“UINT16”},
{id:“blue”,bands:1,sampleType:“UINT16”},
{id:“green”,bands:1,sampleType:“UINT16”},
{id:“red”,bands:1,sampleType:“UINT16”},
{id:“NIR”,bands:1,sampleType:“UINT16”},
{id:“SWIR1”,bands:1,sampleType:“UINT16”},
{id:“SWIR2”,bands:1,sampleType:“UINT16”},
{id:“TIR”,bands:1,sampleType:“UINT16”},
{id:“NDVI”,bands:1,sampleType:“UINT16”},
{id:“NDMI”,bands:1,sampleType:“UINT16”},
{id:“cloud_cover”,bands:1,sampleType: “UINT16”},
{id: “dataMask”,bands:1,sampleType:“UINT8”}
],
};
}

function evaluatePixel(sample) {

return {
coastal_blue: [10000sample.B01],
blue: [10000sample.B02],
green: [10000sample.B03],
red: [10000sample.B04],
NIR: [10000sample.B05],
SWIR1: [10000sample.B06],
SWIR2: [10000sample.B07],
TIR: [sample.B10],
NDVI: [10000index(sample.B05,sample.B04)],
NDMI: [10000*index(sample.B05,sample.B06)],
cloud_cover: [sample.BQA],
dataMask: [sample.dataMask]
}
}
‘’’

stats_request = {
“input”: {
“bounds”: {“geometry”: poly,“properties”: {“crs”: f"http://www.opengis.net/def/crs/EPSG/0/{EPSG_code}"}},
“data”: [{“type”: “landsat-ot-l2”,“dataFilter”: {“mosaickingOrder”: “leastRecent”}}]
},
“aggregation”:{
“timeRange”:{“from”: f"{sd}T00:00:00Z",“to”: f"{ed}T00:00:00Z"},“aggregationInterval”:{“of”: “P1D”},
“evalscript”: evalscript,“resx”: 30,“resy”: 30
}
}

headers = {‘Content-Type’: ‘application/json’,‘Accept’: ‘application/json’}
url = “https://services-uswest2.sentinel-hub.com/api/v1/statistics”
response = oauth.request(“POST”, url=url , headers=headers, json=stats_request)

poly = {‘type’: ‘Polygon’,
‘coordinates’: [[[543524.2066699625, 4913672.475454833],
[543301.7360189715, 4914182.25130983],
[543585.5990637002, 4914305.715457292],
[543636.2003406108, 4914324.679479339],
[543636.5445322505, 4914324.827200528],
[543662.5099398704, 4914337.441878019],
[543662.6089071627, 4914337.491664552],
[543676.3631063607, 4914344.651227288],
[543678.5419234377, 4914345.355938862],
[543902.1623488602, 4913826.227753871],
[543524.2066699625, 4913672.475454833]]]}

So the error is back again… I’ll appreciate it if you can check it.
Is it on my side or on the sentinel-hub side?

‘error’: {‘type’: ‘EXECUTION_ERROR’,
‘message’: “Lambda execution error: ‘2022-08-09T13:52:14.356Z 8c76915e-5e5b-43a3-99c1-f51b9a7f3a71 Task timed out after 10.02 seconds\n’”}},

We will look into this, is the error still occurring or has it disappeared again? Do you know what changed to cause the error? A different date/time or AOI?

e will look into this, is the error still occurring or has it disappeared again? Do you know what changed to cause the error? A different date/time or AOI?

I tried again just now (the same script) and the error is still occurring. I did not change anything in the script.
Some dates have data and some have the error.

It happens (as far as I know) only with Landsat data, it does not occur with Sentinel-2, even though the indices I am calculating are the same.

Maybe there is a problem with the Landsat end-point

Hi,

I couldn’t reproduce the error from your script, but I tried to modify the script a bit and the output looked good to me. Below is the script:

//VERSION=3
function setup(){
  return {
    input: [{
      bands:["B01", "B02", "B03", "B04", "B05", "B06", "B07", "B10", "BQA", "dataMask"],
    }],
    output: [
      {
        id: "data",
        bands: 10,
      },
      {
        id: "cloud_cover",
        bands: 1,
      },
      {
        id: "dataMask",
        bands: 1
      }
    ]
  };
}

function evaluatePixel(sample) {
    const c = 10000
    let band_arr = [sample.B01, sample.B02, sample.B03, sample.B04, sample.B05, sample.B06, sample.B07, sample.B10, index(sample.B05, sample.B04), index(sample.B05, sample.B06)]

    return {
      data: band_arr.map(band_val => band_val * c),
      cloud_cover: [sample.BQA],
      dataMask: [sample.dataMask],
    };
}

Note:

• I remove the sampleType: "UINT16" from the output in setup function. It is because the typical range of TIR band is 250-320, which exceeds the maximum value of UINT16 after multiplied by 10000. You can find out more info about the data here.
• The BQA band is in bit-packed combination, meaning that calculating its statistics does not make much sense. Even after decoding , the value is a flag and the statistics of the “flag value” does not make sense either.

Thank you for the answer but that misses the point of my problem.

• I would like to set the sample type to save PU, so keeping the sample type as default is not an option for me

• Your answer does not explain what “Lambda execution error” means. This problem keeps repeating also when I don’t multiply by 10000, so the UINT16 might not be the problem

• Calculating the BQA does make sense because each pixel has a value (which can be traced to a bit-packed combination), for example, the value 21824 = clear pixel

So, I hope the sentinel-hub team will be able to explain what is the source of the problem and how to fix it.

I also would like to add that worries me, besides the fact that there is an error, is that the dates with the error are not consistent, meaning, for every time I run the script, the number of dates with an error is different. This is very worrying because I can’t trust Sentinel-Hub to give all the results, it seems that it depends on the time of day.

Hi,

The error is from AWS. Lambda functions can, by default, only run 10 seconds. I’ve contacted our developer to look into the error, and will get back to you as soon as possible.

We apologise for the inconvenience.

Hi,

This randomly appeared error should be mitigated, it’s not perfect though. It’s going to take a bit more time to fully solve the issue.

We apologise for the inconvenience.

Am I to understand that this error/bug is on the Sentinel-Hub side and there is nothing I can do but wait for you to resolve it?

Hi,

Yes, the error is on our side and we appreciate your error report which help us improve our services.

We would be grateful if you could report if you observe any more such issues, and we would fix them as soon as possible.

I actually got another error message (same script) but did not report it as I think it might be related… but since you asked 🙂

‘java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Total timeout 20000 ms elapsed’

I tested the original script you provided but I couldn’t reproduce the error. I used the AOI and the time period you mentioned in this post. Do you have a specific AOI and time period that I can reproduce the error with the script?

Thank you.

In the last couple of days, I did many tries so I’m not sure I remember exactly the setting which caused this error. If I’ll encounter it again I’ll update it here.
Thanks

It would be the best if the issue is reported with a complete request that can reproduce the error.

One of the best ways is to build the request with Requests Builder and copy the CURL command from the Request Preview window. This would give us thorough info about the problematic request and would be really helpful for us to detect the issue and fix it.

Thank you very much for your help.