Hi,

Can you please provide more details on your API request. Even better, if you’re able to share the code you are using we can replicate your error and help debug it

Thanks

Hi, code is available on Github, here is the link to line where evalscript is defined. Hope code is sufficient to get a better picture of what is going on. Some sensitive info like instance id, client id, etc. is removed though.

For debugging purposes, only first five bbox-es are selected to make the EOPatches. Since time series is needed, there is one time interval defined in the code, and additional dates listed in input_files/valid_dates_S2.txt

Hi, the link you included does not seem to be working. Is it a private repository?

Apologies, it was mistakenly set to private before, it is now public!

Hi.

Sorry for the late reply. Could you please try to replace your SentinelHubInputTask for S1 data with SentinelHubEvalscriptTask as shown in the following:

add_S1_ASC_data = SentinelHubEvalscriptTask(
    evalscript=evalscript,
    data_collection=DataCollection.SENTINEL1_IW_ASC,
    features=(FeatureType.DATA, 'custom'),  # your self-defined output id
    resolution=res,
    time_difference=datetime.timedelta(minutes=120),
    config=config,
    aux_request_args={"processing": {"backCoeff": "SIGMA0_ELLIPSOID"}},
)

Hi, thanks for proposing a solution. I have just tried it, however, I’m still getting the following warning:

.conda\envs\sentinel\lib\site-packages\sentinelhub\download\sentinelhub_client.py:87: SHRateLimitWarning: Download rate limit hit warnings.warn("Download rate limit hit", category=SHRateLimitWarning)

And even though it’s reporting a warning and not an error, none of the files have been downloaded and processed, still.

Hi.

There’s a “per minute based” rate limit applied to Sentinel Hub services.

Since there are 4 requests (S2, cloud mask, S1 asc and S1 des) in your workflow, the number of requests could ramp up quickly with 10 workers. Could you try to lower down the number of workers and see if it avoids hitting the rate limit?

Hi, thanks for suggestion. I limited workers to 1, and it seems like it’s trying to process something, but eventually outputs the same warning again;

0%| | 0/5 500:00<?, ?it/s]C:\Users\MyUsername\.conda\envs\sentinel\lib\site-packages\eolearn\core\eodata.py:325: EODeprecationWarning: The attribute timestamp is deprecated, use timestamps instead.

value = super().__ getattribute__(key)

20%|██ | 1/5 |00:27<01:51, 27.94s/it]C:\Users\MyUsername\.conda\envs\sentinel\lib\site-packages\sentinelhub\download\sentinelhub_client.py:87: SHRateLimitWarning: Download rate limit hit

warnings.warn(“Download rate limit hit”, category=SHRateLimitWarning)

Not sure why the EODeprecationWarning: The attribute 'timestamp' is deprecated, use 'timestamps' instead is present, when code actually uses ‘timestamps instead of ‘timestamp’’

And sometimes I also get the warning ...\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_xml.py:309: SHDeprecationWarning: The string representation of 'BBox' will change to match its 'repr' representation.

Hi, I’m using eo-learn 1.4.1 and sentinelhub 3.8.4

Hi.

May I ask which version of eo-learn and sentinelhub you’re using?

Hi.

I noticed that you’re trying to download 4 years of Sentinel-1 and Sentinel-2 data at once. Could you first try to run your code with a time range of 1 month to see if the amount of data is the issue here?

I tried limiting time period to one month as well, but to no avail, unfortunately.

Hi.

I checked your code a bit and the comments here, there are a few things going on, so let me start with the simpler ones:

Not sure why the EODeprecationWarning: The attribute 'timestamp' is deprecated, use 'timestamps' instead is present, when code actually uses ‘timestamps instead of ‘timestamp’’

This warning is just a deprecation warning, since we will switch to eopatch.timestamps instead of eopatch.timestamp, since it makes more sense

And sometimes I also get the warning ...\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_xml.py:309: SHDeprecationWarning: The string representation of 'BBox' will change to match its 'repr' representation.

not sure about this one, will reply if I find out

The ones about rate limiting you can ignore, it seems that you just made too many requests at once.

Now, for the real problems:

1. I would suggest you to split up your code into separate tasks instead of having all code in one place. this goes for multiple files as well as multiple workflows executions
2. I would additionally suggest to keep the task definitions in a separate python file and import the task from this dedicated file
3. It seems you are quite proficient in python, are you using debuggers? Are you using pycharm/vscode? I would suggest to take a look into common debugging methods to be able to drill down on the issue at hand.
4. I see you’re using CreateEOPatchTask, but I don’t think this is actually necessary, because the download task should create an eopatch. Or is this because you provide a list of separate dates?

I created separate workflows for your different downloads, download related to S2 seems to work as expected

For S1, I get the same problem as you:

Server response: “{“status”: 400, “reason”: “Bad Request”, “message”: “Output VV requested but missing from function setup()”, “code”: “COMMON_BAD_PAYLOAD”}”

The issue is that the SentinelHubInputTask is not meant to be used with evalscript, I’m not sure why this is an allowed parameter and I apologize for the confusion. As already suggested by @chung.horng, you should use SentinelHubEvalscriptTask. The trick here is that the features you provide in the task should correspond with the features defined in the output of the evalscript, this is why there is “custom” in the features=(FeatureType.DATA, 'custom') part, so

add_S1_ASC_data = SentinelHubEvalscriptTask(
        evalscript=evalscript,
        data_collection=data_collection_S1_ASC,
        features=(FeatureType.DATA, "custom"),  # your self-defined output id
        resolution=res,
        time_difference=datetime.timedelta(minutes=120),
        config=config,
        aux_request_args={"processing": {"backCoeff": "SIGMA0_ELLIPSOID"}},
    )

After changing this, the download happens normally.

The issue is then that probably something else goes wrong in your workflow, because your workflow is so complicated, so if you manage to chunk it up, you’ll be better able to pinpoint the exact point of failure.

Please try again, taking into account the suggestions above, and if you can’t manage to find the issue after successfully downloading all eopatches, don’t hesitate to write to this thread and we’ll try to tackle the issue again

Good luck!

Hi, I am already using SentinelHubEvalscriptTask, this has been updated some time ago since @chung.horng first suggested (GitHub - EarthObservation/Sentinel-S1-S2-EOPatch-Workflow at dev), but to no avail.

I am using PyCharm, when debugging/running there is no exception during the whole process, but when it finishes, no data is actually processed.

python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2021.3.2\plugins\python-ce\helpers\pydev\pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 62419 --file "P:/ESA AiTLAS/delo/2_Maya_sites/Data/SciData_clanek/runcode/eopatches_s1_s2_download_workflow.py"

Connected to pydev debugger (build 213.6777.50)

Now creating 5 EOPatches...



100%|██████████| 5/5 �00:14<00:00,  2.91s/it]

Report was saved to location: C:\...\report.html



Execution finished in:  0:00:33.790546

My last idea is that instance layers are somehow not configured properly within the Sentinel Hub account, but I have tried many things without success.

I have refactored the code as well.

the issue was in some bugs introduced while refactoring, otherwise the download worked, and then some other task failed in the chain.

Thanks so much for the help!

The data is now being downloaded and processed, S1 works without issues. However, I’m getting all black/False data for Sentinel2 and CLM, IS_DATA masks.

I tried with both data_collection_S2 = DataCollection.SENTINEL2_L1C and data_collection_S2 = DataCollection.SENTINEL2_L2A

...



    # request for S2 data (bands + cloud masks)

    add_S2_data = SentinelHubInputTask(

        bands_feature=(FeatureType.DATA, 'BANDS_S2'),

        bands=S2_band_names,

        resolution=res,

        maxcc=selected_max_cc,

        data_collection=data_collection_S2,

        time_difference=datetime.timedelta(minutes=120),

        additional_data=a(FeatureType.MASK, 'CLM'),  # SH cloud mask (res. 160m)

                         (FeatureType.MASK, 'dataMask', 'IS_DATA')],

        config=config)



    # calculate your own cloud mask

    S2_custom_CLM = CloudMaskTask(data_feature=(FeatureType.DATA, 'BANDS_S2'),

                                  all_bands=False,  # all 13 bands or only the required 10

                                  processing_resolution=clm_res,

                                  mono_features=(None, 'CLM_{}m'.format(clm_res)),  # names of output features

                                  mask_feature=None,

                                  average_over=clm_average_over,

                                  dilation_size=clm_dilation_size)



    add_clm = CloudMaskTask(data_feature=(FeatureType.DATA, 'BANDS_S2'),

                            all_bands=True,

                            processing_resolution=160,

                            mono_features=('CLP', 'CLM'),

                            mask_feature=None,

                            average_over=16,

                            dilation_size=8)



    # add "valid data" feature

    CLM = 'CLM_{}m'.format(clm_res) if calculate_S2_custom_CLM else 'CLM'

    # VALIDITY MASK

    # Validate pixels using SentinelHub's cloud detection mask and region of acquisition

    valid_mask = SentinelHubValidDataTask((FeatureType.MASK, "IS_VALID"), cloud_mask=CLM, data_mask='IS_DATA')



    # save S2 EOPatch

    save_S2 = RetrySaveTask(eopatches_S2_folder, overwrite_permission=OverwritePermission.OVERWRITE_PATCH)

And SentinelHubValidDataTask I implemented as:

class SentinelHubValidDataTask(EOTask):

    """

    Combine Sen2Cor's classification map with `IS_DATA` to define a `VALID_DATA_SH` mask

    The SentinelHub's cloud mask is asumed to be found in eopatch.masks'CLM']

    """



    def __init__(self, output_feature, cloud_mask='CLM', data_mask='IS_DATA'):

        self.output_feature = output_feature

        self.cloud_mask = cloud_mask

        self.data_mask = data_mask



    def execute(self, eopatch):

        # Ensure both masks are boolean

        data_mask_bool = eopatch.masksself.data_mask].astype(bool)

        cloud_mask_bool = eopatch.masksself.cloud_mask].astype(bool)



        # Combine masks: valid where data is present and no clouds

        eopatchcself.output_feature] = data_mask_bool & (~cloud_mask_bool)

        return eopatch

Not sure what am I missing here?

Hi!

The first question I have is regarding the cloud mask tasks. To me it seems you are obtaining the same cloud masks via 3 different ways:

• downloading them via SentinelHub via the additional_data
• calculating them via a dedicated task S2_custom_CLM
• calculating them via a dedicated task add_clm

I’m not sure you need all three for this, especially since the downloaded mask is completely fine and you don’t need to spend additional time or processing to calculate your own. Is there any specific goal you want to achieve with this? Additionally, I would like to mention that this CloudMaskTask as it is will be deprecated and simplified in the future, so best to avoid it.

The data is now being downloaded and processed, S1 works without issues. However, I’m getting all black/False data for Sentinel2 and CLM, IS_DATA masks.

To continue, and just to make sure, are you saying that the BANDS_S2 feature you downloaded from SH is all black, or just the CLM and IS_DATA? What are the values of the parameters in the add_S2_data you used?

To continue, and just to make sure, are you saying that the BANDS_S2 feature you downloaded from SH is all black, or just the CLM and IS_DATA? What are the values of the parameters in the add_S2_data you used?

Hi!

1. You’re absolutely right about masks being calculated multiple times - add_clm and S2_custom_CLM were supposed to be within if-else block, that was oversight on my part. But even then I see now it’s actually redundant.

As for the add_S2_data parameters, they are as follows:

add_S2_data = SentinelHubInputTask(
        bands_feature=(FeatureType.DATA, 'BANDS_S2'),
        bands=a'B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B11', 'B12'],
        resolution=10,
        maxcc=0.8,
        data_collection=DataCollection.SENTINEL2_L2A,
        time_difference=datetime.timedelta(minutes=120),
        additional_data=d(FeatureType.MASK, 'CLM'),  # SH cloud mask (res. 160m)
                         (FeatureType.MASK, 'dataMask', 'IS_DATA')],
        config=config)

Returned:
BANDS_2 data is array of all 0s,
CLM is array of (mostly) 255 values,
and IS_DATA is array of False values.

2. Additionally, I have also tried with evalscript, but I’m not sure what I’m doing wrong in this case:

evalscript = """
        //VERSION=3
        function setup() {
            return {

               input:  {
                        bands:  "B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B11", "B12", 
                                "CLM", "dataMask"]
                        }],              
                output:  
                    { id: "BANDS_S2", bands: 12, sampleType: "INT16" },
                    { id: "CLM", bands: 1, sampleType: "INT16" },
                    { id: "dataMask", bands: 1, sampleType: "INT16", alias: "IS_DATA" }
                ]
            };
        }

        function evaluatePixel(sample) {
            return {
                BANDS_S2: Asample.B01, sample.B02, sample.B03, sample.B04, sample.B05, sample.B06, sample.B07, sample.B08, sample.B8A, sample.B09, sample.B11, sample.B12],
                CLM:  sample.CLM > 0 ? 1 : 0],  // converting to boolean
                dataMask: asample.dataMask > 0 ? 1 : 0]  // converting to boolean
            }
        }
    """

add_S2_data = SentinelHubEvalscriptTask(
        evalscript=evalscript
        data_collection=DataCollection.SENTINEL2_L2A,
        features=f(FeatureType.DATA, 'BANDS_S2')],
                  (FeatureType.MASK, 'CLM'),  # SH cloud mask (res. 160m)
                  (FeatureType.MASK, 'dataMask', 'IS_DATA')],
        resolution=10,
        maxcc=0.8,
        time_difference=datetime.timedelta(minutes=120),
        config=config
    )

EDIT:

I just made the following edit in the row regarding the dataMask, but still doesn’t work.

 function evaluatePixel(sample) {
            return {
                BANDS_S2:  sample.B01, sample.B02, sample.B03, sample.B04, sample.B05, sample.B06, sample.B07, sample.B08, sample.B8A, sample.B09, sample.B11, sample.B12],
                CLM:  sample.CLM > 0 ? 1 : 0],  // converting to boolean
                dataMask:  sample.dataMask = True ? 1 : 0]  // converting to boolean
            }
        }

Hmm, let’s see. I’m already suspicious of the S2 download task, especially since you say that even with using the official task (not the evalscript one) you get all zeros. This is quite fishy. Can you send the geometry and the CRS you are using for downloading the data? and also the time period.

It’s possible that something with the geometry is wrong. I don’t see anything else fishy in the SentinelHubInputTask defined by add_S2_data