PlanetScope search and subscription deliveries discrepancy

Hi @Roman ,

This sounds related to the fact that the time range filter in PlanetScope subscriptions is applied to “published” time. Please see the post answered by my colleague for more details.

Hi @chung.horng ,
It does not seem to explain the discrepancy. I believe we use the acquisition times, not the publish ones. Here are more details how we get the data:

Subscription deliveries:

• Url used is https://services.sentinel-hub.com/api/v1/dataimport/subscriptions/{subscription_id}/deliveries.
• subscription id is ‘e0b8d5ea-b44c-4ef2-9397-4e0d7530f014’.
• the timRange used for creating the subscription is 5 years ago, so there are deliveries since March 2020 till yesterday.
• on each page of the paginated response we look into ‘data’ (which contain a list of deliveries), and from each delivery we collect ‘itemId’. I believe that ‘itemId’ is made of the capture time. One example of the delivery item looks like this:

{
  "id": "e994e27e-acf9-44cd-bb1e-c1a3ea7dfe1f",
  "itemId": "20231004_093546_13_2473",
  "provider": "PLANET",
  "created": "2023-10-05T05:48:41.176777Z",
  "status": "DONE"
}

Moreover, the subscription was created a yesterday, so the timeRange field should already refer to the acquisition time, if I understand well this.

Simple search

• Url used is https://services.sentinel-hub.com/api/v1/dataimport/search
• The same query (below) is used except that due to the result limit we sequentially query for periods of 150 days starting with 2020-01-01 (first timeRange is from ‘2020-01-01T00:00:00Z’ to ‘2020-05-30T00:00:00Z’, then from ‘2020-05-30T00:00:00Z’ to ‘2020-10-27T00:00:00Z’, etc.)

We end up with a close number of items, but in this particular subscription there are more items returned by search (777) than the number of deliveries (775), and one date included in the search is completely missing from delivery items ‘2022-04-15’.

Subscription query:

{
  "bounds": {
    "geometry": {
      "type": "Polygon",
      "coordinates": polygon_shape
    },
    "properties": {}
  },
  "data": [
    {
      "dataFilter": {
        "timeRange": {
          "from": "2018-10-05T09:31:13.880402Z"
        },
        "maxCloudCoverage": 100,
        "nativeFilter": {
          "type": "StringInFilter",
          "config": [
            "standard"
          ],
          "field_name": "quality_category"
        }
      },
      "itemType": "PSScene",
      "productBundle": "analytic_8b_sr_udm2",
      "harmonizeTo": "Sentinel-2"
    }
  ],
  "provider": "PLANET"
}

Simple search query:

query = {
    "provider": "PLANET",
    "bounds": {
        "geometry": {
            "type": "Polygon",
            "coordinates": polygon_shape,
        }
    }, "data": [
        {
            "itemType": "PSScene",
            "productBundle": "analytic_8b_sr_udm2",
            "dataFilter": {
                "timeRange": {
                    "from": date_from,
                    "to": date_to
                },
                "maxCloudCoverage": 100,
                "nativeFilter": {
                    "type": "StringInFilter",
                    "field_name": "quality_category",
                    "config": [
                        "standard"
                    ]
                }
            },
            "harmonizeTo": "Sentinel-2"
        }
    ]
}

Hi @Roman ,

Thank you for providing detailed information. We will investigate the issue and let you know as soon as we find anything.

Thank you, looking forward to your findings!

Hi @chung.horng , Is there any update on this?

Hi @Roman ,

We’re still investigating.

Hi @Roman ,

It seems the issue came from the Planet side. I’ve forwarded the issue to Planet support and asked if they can investigate why the item on 2022-04-15 is missing in your subscription.

I will update you as soon as we get a reply from them. Sorry for the inconvenience.

Hi @chung.horng ,
Ok, thank you!

Hi @Roman ,

I’m still communicating with Planet’s support to investigate the missing asset. Meanwhile, I’d like to provide some important information:

• The Subscriptions API supports delivery of both archive imagery and future imagery collections. Forwardfill subscriptions are subscriptions with an end time in the future and Backfill subscriptions are subscriptions with a start and end time in the past.
• The subscription delivery logic of backfill/forwardfill prioritizations is that data in the forwardfill option being prioritized as stated in our documentation.
• It is hard to tell the common ETA of a 5-year backfill delivery (which is the case of your subscription) precisely because of the backfill/forwardfill prioritization and Planet’s internal capacity. But it is reasonable to say that it might take up to 2 weeks.
• The use will notice when Planet stops ingesting images dated before subscription confirmed (which is 04.10.2023 in your case). Once you start receiving only one image per day, it will indicate that the backfill has stopped. We cannot determine the exact number of items through the subscription API. However, we can use the simple search endpoint to obtain the items by applying the same filter that we used when creating the subscription. That is to say, your approach is the right way to check if the backfill share of your subscription is complete.

We’ve confirmed that the asset 20220415_094542_98_240c and 20220418_092927_29_2498 which are in the result of simple search were not delivered in your subscription. But we’re still investigating why they were missing. I will keep you updated.

Sorry for the inconvenience and thank you for your patience.

Hi @chung.horng , Thank you for the update and all the information!

Hi @Roman ,

After a thorough investigation, we confirmed that the missing items, 20220415_094542_98_240c and 20220418_092927_29_2498, were under a republishing process and they had "quality_category": "test" when your subscription was confirmed. As a result, they were excluded in the export list of your subscription.

We’d recommend creating another subscription with a time range covering these two items to trigger another archive search and delivery. Sorry for the inconvenience.

Hi @chung.horng ,

Thank you for the clarification.

Do you plan to create a mechanism that would add such items after being republished, or should we check that all items have been delivered and create new subscription always when the situation occur in the future?

Hi @Roman ,

It would be great if you could post this idea on Planet Product Ideas, where Planet collaborate with the user community to improve data and platforms.

At the moment, please do check again after backfill share of the subscription is done.

Hi @chung.horng ,

Ok, thanks.