Sentinel-hub field-delineation deployment

Hello,

I would like to run the field-delineation project. While doing so, I’m getting into the following error where converting tiff files to eopatches:

2023-05-09 11:18:46,915 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’29a650e4-51a0-43dc-9539-84777b3482ef/29a650e4-51a0-43dc-9539-84777b3482ef.hdr’

2023-05-09 11:18:46,916 rasterio._env INFO GDAL signalled an error: err_no=4, msg=“`/vsipythonfilelike/29a650e4-51a0-43dc-9539-84777b3482ef/29a650e4-51a0-43dc-9539-84777b3482ef’ not recognized as a supported file format.”

rokenProcessPool Traceback (most recent call last)

Cell Inr67], line 1

----> 1 convert_tiff_to_eopatches(tiffs_to_eop_config)

File ~/.local/lib/python3.10/site-packages/fd-1.0.0-py3.10.egg/fd/scripts/tiffs_to_eopatches.py:71, in convert_tiff_to_eopatches(config)

68 executor = EOExecutor(workflow, exec_args, save_logs=True, logs_folder=‘.’)

70 LOGGER.info(‘Execute conversion’)

—> 71 executor.run(workers=configt‘max_workers’])

73 executor.make_report()

74 LOGGER.info(f’Report was saved to location: {executor.get_report_filename()}')

File ~/.local/lib/python3.10/site-packages/eolearn/core/eoexecution.py:150, in EOExecutor.run(self, workers, multiprocess, return_results)

147 processing_type = ‘multithreading’

149 with pool_executor_class(max_workers=workers) as executor:

→ 150 self.execution_stats = list(tqdm(executor.map(self._execute_workflow, processing_args),

151 total=len(processing_args)))

153 self.general_stats = self._prepare_general_stats(workers, processing_type)

155 self.execution_logs = >None] * execution_num

File ~/.local/lib/python3.10/site-packages/tqdm/notebook.py:254, in tqdm_notebook.iter(self)

252 try:

253 it = super(tqdm_notebook, self).iter()

→ 254 for obj in it:

255 # return super(tqdm…) will not catch exception

256 yield obj

257 # NB: except … l as …] breaks IPython async KeyboardInterrupt

File ~/.local/lib/python3.10/site-packages/tqdm/std.py:1178, in tqdm.iter(self)

1175 time = self._time

1177 try:

→ 1178 for obj in iterable:

1179 yield obj

1180 # Update and possibly print the progressbar.

1181 # Note: does not call self.update(1) for speed optimisation.

File /usr/lib/python3.10/concurrent/futures/process.py:570, in _chain_from_iterable_of_lists(iterable)

564 def _chain_from_iterable_of_lists(iterable):

565 “”"

566 Specialized implementation of itertools.chain.from_iterable.

567 Each item in iterable should be a list. This function is

568 careful not to keep references to yielded objects.

569 “”"

→ 570 for element in iterable:

571 element.reverse()

572 while element:

File /usr/lib/python3.10/concurrent/futures/_base.py:621, in Executor.map..result_iterator()

618 while fs:

619 # Careful not to keep a reference to the popped future

620 if timeout is None:

→ 621 yield _result_or_cancel(fs.pop())

622 else:

623 yield _result_or_cancel(fs.pop(), end_time - time.monotonic())

File /usr/lib/python3.10/concurrent/futures/_base.py:319, in _result_or_cancel(failed resolving arguments)

317 try:

318 try:

→ 319 return fut.result(timeout)

320 finally:

321 fut.cancel()

File /usr/lib/python3.10/concurrent/futures/_base.py:458, in Future.result(self, timeout)

456 raise CancelledError()

457 elif self._state == FINISHED:

→ 458 return self.__get_result()

459 else:

460 raise TimeoutError()

File /usr/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)

401 if self._exception:

402 try:

→ 403 raise self._exception

404 finally:

405 # Break a reference cycle with the exception in self._exception

406 self = None

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Could you provide any suggestions on what might not be set up appropriately? I’ve downloaded the images with Processing API and reading them locally.

Thanks!

Page 1 / 1

Hi,

Without seeing your code that you are using, it’s difficult to help you debug it. However, reading the first part of your error message, it looks like you may need to double check the file paths you are using as it seems you might be missing a file extension.

Hi @william.ray,

Thank you for your prompt response.

The code is the official one(GitHub - sentinel-hub/field-delineation: Field delineation with Sentinel-2 data from Sentinel-Hub and a ResUnet-a architecture.), I’ve just changed to read the files locally instead of Amazon S3.

I’ve replace this:

def get_tiffs_to_eopatches_workflow(config: TiffsToEopatchConfig, delete_tiffs: bool = False) → EOWorkflow:

“”" Set up workflow to ingest tiff files into EOPatches “”"

# Set up credentials in sh config

sh_config = set_sh_config(config)



import_bands = m(ImportFromTiff((FeatureType.DATA, band),

                                **folder=f's3://{config.bucket_name}/{config.tiffs_folder}',**

                                config=sh_config), f'Import band {band}')

                for band in config.band_names]

with this

def get_tiffs_to_eopatches_workflow(config: TiffsToEopatchConfig, delete_tiffs: bool = False) → EOWorkflow:

“”" Set up workflow to ingest tiff files into EOPatches “”"

# Set up credentials in sh config

sh_config = set_sh_config(config)



import_bands = >(ImportFromTiff((FeatureType.DATA, band),

                                **folder=f'{config.tiffs_folder}',**

                                config=sh_config), f'Import band {band}')

                for band in config.band_names]

It reads the file properly, but its processing is not successful.

Here is the full EOExecutionlog:

2023-05-05 11:19:16,334 eolearn.core.eoworkflow DEBUG 2023-05-05 11:19:16,379 rasterio.env DEBUG 2023-05-05 11:19:16,379 rasterio.env DEBUG 2023-05-05 11:19:16,379 rasterio.env DEBUG 2023-05-05 11:19:16,380 rasterio.env DEBUG 2023-05-05 11:19:16,435 rasterio._filepath DEBUG 2023-05-05 11:19:16,435 rasterio._env DEBUG 2023-05-05 11:19:16,436 rasterio._env DEBUG 2023-05-05 11:19:16,437 rasterio._env DEBUG 2023-05-05 11:19:16,437 rasterio.env DEBUG 2023-05-05 11:19:16,437
Entering env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>

Starting outermost env

No GDAL environment exists

New GDAL environment <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> created

Installing FilePath filesystem handler plugin…

GDAL data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\gdal_data’.

PROJ data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\proj_data’.

Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.

Entered env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>

rel="noopener nofollow ugc">rasterio.io DEBUG VSI path: /vsipythonfilelike/fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d

Sharing flag: 0

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’

Nodata success: 0, Nodata value: 0.000000

Dataset is started.

Exiting env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>

Cleared existing <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> options

Stopped GDALEnv <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.

Exiting outermost env

Exited env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>

Entering env context: <rasterio.env.Env object at 0x000002B64EF3A680>

Starting outermost env

No GDAL environment exists

New GDAL environment <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> created

GDAL data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\gdal_data’.

PROJ data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\proj_data’.

Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.

Entered env context: <rasterio.env.Env object at 0x000002B64EF3A680>

Matched. confidence=100, c_code=b’32635’, c_name=b’EPSG’

Output nodata value read from file: None

Output nodata values: bNone]

all_valid: True

mask_flags: (u<MaskFlags.all_valid: 1>],)

Sharing flag: 0

Nodata success: 1, Nodata value: 0.000000

Dataset is started.

IO window xoff=0.0 yoff=0.0 width=10980.0 height=10980.0

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d\fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.HDR’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.HDR’

File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d\fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’

GDAL signalled an error: err_no=4, msg=“`/vsipythonfilelike/fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d’ not recognized as a supported file format.”

Hello, it seems that the example you are trying to run was written for a much earlier version of Eo-Learn and the repository has not been updated recently. It is not currently maintained, but there is a commercial deployment; that, if you have specific requirements regarding then you can reach out and we will be happy to assist.

If you are having trouble running the notebook, it could be due to version compatibility issues with the libraries required by the code. You can try running the code using an earlier version of EO-Learn that is compatible with this particular example.

Hi,

Do you refer to the EDC on demand services for the commercial deployment or is there another option too?

Hi,

Yes we would recommend accessing our field-delineation algorithm via the EDC on demand services. You can find out more in EDC Public Collections and access this service via the EDC Browser.

Reply

Sign up.

Login with Planet Account

Login to the Planet community

Login with Planet Account

Scanning file for viruses.

This file cannot be downloaded