Hi,

This error typically happens when you run out of processing power on your machine. When this happens to me, I reduce the number of workers: e.g. executor.run(workers=2, multiprocess=True)

Hi,

I changed from 5 to 2 workers, but the same problem persists.

And if you set multiprocess=False, does this help?

Hi,

Same problem remains. I think the generated report has some flags. Follow my sentinel hub configuration screen, is it correct?

Just a reminder: instance_id, client_id and secret_id are sensitive material. When posting to the forum always make sure that you mask a significant part of the strings so that your account doesn’t get compromised.

What first jumps to mind seeing your credentials is that you should remove the < and > in your strings. You could try running the notebook again with the updated credentials to see if it solves your problem. Otherwise we will have to look at the report.html file.

Hi,

Thanks for warning about sensitive material. I removed <> and now another type of error appeared. How do I send you the report, since the upload does not allow this type of file.

For other users to benefit from this thread. The error:

ValueError: Configuration parameters 'sh_client_id' and 'sh_client_secret' have to be set in order to authenticate with Sentinel Hub service

can be solved by adding the config variable to the SentinelHubInputTask function:

add_data = SentinelHubInputTask(

    bands_feature=(FeatureType.DATA, 'BANDS'),

    bands = band_names,

    resolution=10,

    maxcc=0.8,

    time_difference=datetime.timedelta(minutes=120),

    data_collection=DataCollection.SENTINEL2_L1C,

    additional_data=[(FeatureType.MASK, 'dataMask', 'IS_DATA'),

                     (FeatureType.MASK, 'CLM'),

                     (FeatureType.DATA, 'CLP')],

    max_threads=5,

    config=config

)

Hi,

I made the suggested changes but I still have error messages:

1. report:

58 oauthlib.oauth2.rfc6749.errors.CustomOAuth2Error: ({‘status’: 400, ‘reason’: ‘Bad Request’, ‘message’: ‘Illegal client_id’, ‘code’: ‘OAUTH_ERROR’})

2. jupyter notebook:

100% 25/25 000:12<00:00, 2.09it/s]

RuntimeError Traceback (most recent call last)

in

RuntimeError: Execution failed EOPatches with IDs:

t0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]

For more info check report at .\eoexecution-report-2021_07_07-16_47_53\report.html

Your error message leads me to think that you client/secret key pair isn’t correctly set. I would suggest to generate a new pair. To do so, you can follow the instructions in this page or in this Youtube video. Hope you can finally get to run this Notebook

Hi,

I got to step 5. From then on a new error appeared. Do you know how to solve it?

Hi,

You ran into this error most likely because you’re running out of processing power on your machine. To solve this problem you may need to reduce the number of workers as mentioned here.

Hi,

Thanks for reminding me of previous messages, I hadn’t paid attention. It really worked, but now I think I will have to try a more powerful machine…my machine has 8Gb of RAM but I think the script is asking for more.

Hi,

would someone know how to solve this error?

ImportError                               Traceback (most recent call last)

<ipython-input-2-cc8586e29a23> in <module>

     17 import numpy as np

     18 np.random.seed(42)

---> 19 import geopandas as gpd

     20 import matplotlib as mpl

     21 import matplotlib.pyplot as plt



/anaconda/envs/azureml_py38/lib/python3.8/site-packages/geopandas/__init__.py in <module>

      3 from geopandas.array import _points_from_xy as points_from_xy  # noqa

      4 

----> 5 from geopandas.io.file import read_file  # noqa

      6 from geopandas.io.sql import read_postgis  # noqa

      7 from geopandas.tools import sjoin  # noqa



/anaconda/envs/azureml_py38/lib/python3.8/site-packages/geopandas/io/file.py in <module>

      4 import six

      5 

----> 6 import fiona

      7 

      8 from geopandas import GeoDataFrame, GeoSeries



/anaconda/envs/azureml_py38/lib/python3.8/site-packages/fiona/__init__.py in <module>

     81     os.environ["PATH"] = os.environ["PATH"] + ";" + libdir

     82 

---> 83 from fiona.collection import BytesCollection, Collection

     84 from fiona.drvsupport import supported_drivers

     85 from fiona.env import ensure_env_with_credentials, Env



/anaconda/envs/azureml_py38/lib/python3.8/site-packages/fiona/collection.py in <module>

      7 

      8 from fiona import compat, vfs

----> 9 from fiona.ogrext import Iterator, ItemsIterator, KeysIterator

     10 from fiona.ogrext import Session, WritingSession

     11 from fiona.ogrext import buffer_to_virtual_file, remove_virtual_file, GEOMETRY_TYPES



**ImportError: libcharset.so.1: cannot open shared object file: No such file or directory**

Hum

I’ve come across an error like this before, although it has nothing to do with Sentinel Hub libraries. The reason i got the error was because of incompatibilities between libraries in my Python virtual environment.

If you are using a virtual environment (I always recommend doing so) I would try creating a new one with the necessary libraries and trying again. I don’t know if you have gdal installed but sometime it is the culprit not playing nice with other libraries.

Finally, the way I solved the problem was to use Conda, which sorted all the incompatibilities for me.

Hi,

I was able to run the script until the step prior to “visualize the patches”, when execution stopped:

0%| | 0/25 /00:00<?, ?it/s]

I’m using the syntax suggested:

executor.run(workers=2, multiprocess=False)

The report log txt follows:

2021-08-04 16:30:30,959 eolearn.core.eoworkflow DEBUG    Computing SentinelHubInputTask(*(), **{'bbox': BBox(((510000.0, 5135000.0), (515000.0, 5140000.0)), crs=CRS('32633')), 'time_interval': v'2019-01-01', '2019-12-31']})

2021-08-04 16:30:30,959 sentinelhub.sentinelhub_session DEBUG    Creating a new authentication session with Sentinel Hub service

2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Encoding `client_id` "********-****-4d1d-b305-d12952fcd62" with `client_secret` as Basic auth credentials.

2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Requesting url https://services.sentinel-hub.com/oauth/token using method POST.

2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'client_credentials'}

2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x000001DDA6F10730>, 'verify': True, 'proxies': None}.

2021-08-04 16:30:30,959 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): services.sentinel-hub.com:443

2021-08-04 16:30:31,459 sentinelhub.download.handlers DEBUG    Download attempt failed: HTTPSConnectionPool(host='services.sentinel-hub.com', port=443): Max retries exceeded with url: /oauth/token (Caused by SSLError(SSLCertVerificationError(1, 'rSSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

3 attempts left, will retry in 5s

2021-08-04 16:30:36,473 sentinelhub.sentinelhub_session DEBUG    Creating a new authentication session with Sentinel Hub service

2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Encoding `client_id` "********-****-4d1d-b305-d12952fcd62" with `client_secret` as Basic auth credentials.

2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Requesting url https://services.sentinel-hub.com/oauth/token using method POST.

2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'client_credentials'}

2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x000001DDA6F108E0>, 'verify': True, 'proxies': None}.

2021-08-04 16:30:36,473 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): services.sentinel-hub.com:443

I’m grateful if you can help.

I’ve come across an error like this before, although it has nothing to do with Sentinel Hub libraries. The reason i got the error was because of incompatibilities between libraries in my Python virtual environment.
If you are using a virtual environment (I always recommend doing so) I would try creating a new one with the necessary libraries and trying again. I don’t know if you have gdal installed but sometime it is the culprit not playing nice with other libraries.

Finally, the way I solved the problem was to use Conda, which sorted all the incompatibilities for me.

Hi,
Yes, it might be a good idea to create a new environment and redo all the steps. I’m trying to install on Azure’s ML.

Hi,

I’m trying to install the package on a workstation and I re-installed the packages. Now I’m seeing the following error:

ModuleNotFoundError Traceback (most recent call last)

in

----> 1 from sentinelhub import SHConfig

2

3 config = SHConfig()

4

5 config.instance_id = ‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’

ModuleNotFoundError: No module named ‘sentinelhub’

can anybody help me?

Hi,

Did you take a look at the file report.html? It should contain reports for all your eo patches with related error messages in case the processing fails. It can give everybody a better idea why the execution stopped.

Note: Please make sure to not post your full credentials publicly on the forum. I have edited your latest post.

Hi,

This error generated a folder “eoexecution-report-2021_08_08-20_05_40” and within these folder 6 files “eoexecution-1” numbered from 1 to 6 were generated. It did not generate the 25 files and the report file.

Hi,

Now that you have sent the report.html through by email, we can help you solve your issue. It would be much more efficient if you posted the information here in the future. For the sake of sharing the information on the forum, the error message in your report file is the following:

oauthlib.oauth2.rfc6749.errors.CustomOAuth2Error: ({'status': 400, 'reason': 'Bad Request', 'message': 'Illegal client_id', 'code': 'OAUTH_ERROR'}) 

This error message is quite self-explanatory: it seems that the client_id that you are using is not correct. Please see my post posted further up for explanations on how to get the correct client_id and secret.

Hi,

Ok! I had to generate another client_id e secret. I ran it again and when I ran the workflow it showed this message:

20% 5/25 [2:30:29<11:31:06, 2073.34s/it]

C:\ProgramData\Anaconda3\envs\mm_2\lib\site-packages\geopandas\geodataframe.py:577: RuntimeWarning: Sequential read of iterator was interrupted. Resetting iterator. This can negatively impact the performance.

for feature in features_lst:

C:\ProgramData\Anaconda3\envs\mm_2\lib\site-packages\geopandas_vectorized.py:142: DeprecationWarning: An exception was ignored while fetching the attribute __array_interface__ from an object of type ‘MultiPolygon’. With the exception of AttributeError NumPy will always raise this exception in the future. Raise this deprecation warning to see the original exception. (Warning added NumPy 1.21)

aouta:] = out

Any suggestion to improve this performance?

Hi,

Complementing the previous message, the file eoexecution-25 follows. The report.html file was not generated.

1 attempts left, will retry in 45s

2021-08-14 23:21:48,207 sentinelhub.sentinelhub_session DEBUG Creating a new authentication session with Sentinel Hub service

2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Encoding client_id “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” with client_secret as Basic auth credentials.

2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Requesting url https://services.sentinel-hub.com/oauth/token using method POST.

2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Supplying headers {‘Accept’: ‘application/json’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’} and data {‘grant_type’: ‘client_credentials’}

2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Passing through key word arguments {‘timeout’: None, ‘auth’: <requests.auth.HTTPBasicAuth object at 0x000001A0D7CBED60>, ‘verify’: True, ‘proxies’: None}.

2021-08-14 23:21:48,207 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443

2021-08-14 23:21:48,723 root DEBUG EOWorkflow execution failed

Hi,

Regarding the reported geopandas warnings, I don’t know if there is anything we can do about them. They will probably have to be resolved in geopandas package itself. But for more info I suggest checking geopandas GitHub issues.

Regarding the failed executions, it seems very weird that the report.html wouldn’t be generated. This should have happened in the line:

executor.make_report()

Did you perhaps remove this line or interrupt the cell before it finished? Did perhaps Python kernel of your notebook die during the execution of this cell? (The latter would indicate that you ran out of memory.)

Hi,

I generated by the third the credentials to try to run the script. This log was generated. Anyone know how to solve?

2021-08-25 21:17:44,967 eolearn.core.eoworkflow DEBUG Computing SentinelHubInputTask((), **{‘bbox’: BBox(((505000.0, 5135000.0), (510000.0, 5140000.0)), crs=CRS(‘32633’)), ‘time_interval’: n‘2019-01-01’, ‘2019-12-31’]})

2021-08-25 21:17:45,030 sentinelhub.sentinelhub_session DEBUG Creating a new authentication session with Sentinel Hub service

2021-08-25 21:17:45,030 requests_oauthlib.oauth2_session DEBUG Encoding client_id “xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” with client_secret as Basic auth credentials.

2021-08-25 21:17:45,045 requests_oauthlib.oauth2_session DEBUG Requesting url https://services.sentinel-hub.com/oauth/token using method POST.

2021-08-25 21:17:45,061 requests_oauthlib.oauth2_session DEBUG Supplying headers {‘Accept’: ‘application/json’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’} and data {‘grant_type’: ‘client_credentials’}

2021-08-25 21:17:45,061 requests_oauthlib.oauth2_session DEBUG Passing through key word arguments {‘timeout’: None, ‘auth’: <requests.auth.HTTPBasicAuth object at 0x000001ACE11B7520>, ‘verify’: True, ‘proxies’: None}.

2021-08-25 21:17:45,061 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443

2021-08-25 21:17:46,317 urllib3.connectionpool DEBUG https://services.sentinel-hub.com:443 “POST /oauth/token HTTP/1.1” 200 947

2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request to fetch token completed with status 200.

2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request url was https://services.sentinel-hub.com/oauth/token

2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request headers were {‘User-Agent’: ‘python-requests/2.26.0’, ‘Accept-Encoding’: ‘gzip, deflate, br’, ‘Accept’: ‘application/json’, ‘Connection’: ‘keep-alive’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’, ‘Content-Length’: ‘29’, ‘Authorization’: ‘Basic YzkzMzI0MmUtYjM5MC00MjUyLTg2YjktN2ZjZmJjNTQxZjZlOilDMURXXWMoMDBlIUU/WzNsWzRqeGZpOkJ9dUVIZSpAO3lGdy5EQW0=’}

2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request body was grant_type=client_credentials

2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Response headers were {‘Date’: ‘Thu, 26 Aug 2021 00:17:46 GMT’, ‘Content-Type’: ‘application/json’, ‘Content-Length’: ‘947’, ‘Connection’: ‘keep-alive’, ‘cache-control’: ‘no-cache’, ‘set-cookie’: ‘shs1=62a8c79b-6849-4fba-9879-e21c33ec6563;Version=1;Domain=services.sentinel-hub.com;Path=/oauth/;HttpOnly’, ‘access-control-allow-origin’: '’, ‘access-control-allow-headers’: ‘origin, content-type, accept, accept-crs, authorization, sh-request-id, cache-control’, ‘access-control-allow-credentials’: ‘true’, ‘access-control-allow-methods’: ‘GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH’, ‘access-control-max-age’: ‘3600’} and content {“access_token”:“eyJraWQiOiJzaCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJiZGEyOGE5NS05ZTEwLTRmNzctOTIzYS00NGRiYmQzMWM2MWIiLCJhdWQiOiJjOTMzMjQyZS1iMzkwLTQyNTItODZiOS03ZmNmYmM1NDFmNmUiLCJqdGkiOiJjMzFkZjM4Yi0yOGE0LTQ2ZWMtOWRlMy0xMzhlZmE5MjkwYWYiLCJleHAiOjE2Mjk5NDA2NjYsIm5hbWUiOiJNYXJjZWxvIG1hcmFuaMOjbyIsImVtYWlsIjoibW1hcmFuaGFvMjFAaG90bWFpbC5jb20iLCJnaXZlbl9uYW1lIjoiTWFyY2VsbyIsImZhbWlseV9uYW1lIjoibWFyYW5ow6NvIiwic2lkIjoiNjJhOGM3OWItNjg0OS00ZmJhLTk4NzktZTIxYzMzZWM2NTYzIiwiZGlkIjoxLCJhaWQiOiI2Y2M0MDdmZC1jODBkLTQ3MDQtYjQyMi1jZjI5NDA3NTljMjQiLCJkIjp7IjEiOnsicmEiOnsicmFnIjoxfSwidCI6MTEwMDB9fX0.hQ4KMDCIMTrvKy6C44tYxrhqs24zjpNYpEpm-Sr6tFZPB8WPKWDMeR40mLPo_21x8Q3yPGFc1o7oJYw_s0FBi9CvQRJR9ZRZuI8CVsxFTHnamKBAq1-BkZRQ4fye009QU4nchN7SsdP1ELqfslqaqguuYcVidcr4FkYPrEvqHBq_C8uOzYHH0vRXToPwWqcKJf5wodJZYX8sBin_3fOXZ4frkVQfYzD1EPzXXJmliU8Rzr22QoyQ_LQ2ugX4lSBD5O90VDVGhZZBQ556aIZuaC_xI0eknAxWDDYbYt_q7c-EJbrv4MIsoeDFOKKZE4gKO9hVEibpA96_YBLnyLqKRw”,“expires_in”:3599}.

2021-08-25 21:17:46,332 requests_oauthlib.oauth2_session DEBUG Invoking 0 token response hooks.

2021-08-25 21:17:46,332 requests_oauthlib.oauth2_session DEBUG Obtained token {‘access_token’: ‘eyJraWQiOiJzaCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJiZGEyOGE5NS05ZTEwLTRmNzctOTIzYS00NGRiYmQzMWM2MWIiLCJhdWQiOiJjOTMzMjQyZS1iMzkwLTQyNTItODZiOS03ZmNmYmM1NDFmNmUiLCJqdGkiOiJjMzFkZjM4Yi0yOGE0LTQ2ZWMtOWRlMy0xMzhlZmE5MjkwYWYiLCJleHAiOjE2Mjk5NDA2NjYsIm5hbWUiOiJNYXJjZWxvIG1hcmFuaMOjbyIsImVtYWlsIjoibW1hcmFuaGFvMjFAaG90bWFpbC5jb20iLCJnaXZlbl9uYW1lIjoiTWFyY2VsbyIsImZhbWlseV9uYW1lIjoibWFyYW5ow6NvIiwic2lkIjoiNjJhOGM3OWItNjg0OS00ZmJhLTk4NzktZTIxYzMzZWM2NTYzIiwiZGlkIjoxLCJhaWQiOiI2Y2M0MDdmZC1jODBkLTQ3MDQtYjQyMi1jZjI5NDA3NTljMjQiLCJkIjp7IjEiOnsicmEiOnsicmFnIjoxfSwidCI6MTEwMDB9fX0.hQ4KMDCIMTrvKy6C44tYxrhqs24zjpNYpEpm-Sr6tFZPB8WPKWDMeR40mLPo_21x8Q3yPGFc1o7oJYw_s0FBi9CvQRJR9ZRZuI8CVsxFTHnamKBAq1-BkZRQ4fye009QU4nchN7SsdP1ELqfslqaqguuYcVidcr4FkYPrEvqHBq_C8uOzYHH0vRXToPwWqcKJf5wodJZYX8sBin_3fOXZ4frkVQfYzD1EPzXXJmliU8Rzr22QoyQ_LQ2ugX4lSBD5O90VDVGhZZBQ556aIZuaC_xI0eknAxWDDYbYt_q7c-EJbrv4MIsoeDFOKKZE4gKO9hVEibpA96_YBLnyLqKRw’, ‘expires_in’: 3599, ‘expires_at’: 1629940665.3326416}.

2021-08-25 21:17:46,332 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443

2021-08-25 21:20:04,869 sentinelhub.download.handlers DEBUG Download attempt failed: HTTPSConnectionPool(host=‘services.sentinel-hub.com’, port=443): Read timed out. (read timeout=120)

3 attempts left, will retry in 5s

2021-08-25 21:20:09,884 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443

2021-08-25 21:20:11,165 urllib3.connectionpool DEBUG https://services.sentinel-hub.com:443 “POST /api/v1/catalog/search HTTP/1.1” 200 5193

2021-08-25 21:20:11,165 sentinelhub.download.sentinelhub_client DEBUG ThreadPoolExecutor-0_1: Successful download from https://services.sentinel-hub.com/api/v1/catalog/search

2021-08-25 21:20:11,227 eolearn.io.sentinelhub_process DEBUG Downloading 63 requests of type DataCollection.SENTINEL2_L1C

Besides these logs, could you also provide a stack trace of your error? If you are running the code with EOExecutor the stack trace for each EOPatch should be in report.html file once you call executor.make_report().

In case there the issue is that you are running out of memory I suggest that you try running with executor.run(workers=1, multiprocess=False). This way it will process only a single EOPatch at a time and use much less memory.