{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Search" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Searching for products is one of the most important features of `eodag`. This page describes the different methods available to search for products and the parameters that these methods accept." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`eodag` is set here to search for *Sentinel 2 Level-1C* products with *PEPS*." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-02-19 12:00:29,455 eodag.config [INFO ] Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n", "2024-02-19 12:00:29,545 eodag.core [INFO ] Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml\n" ] } ], "source": [ "from eodag import EODataAccessGateway\n", "from eodag import setup_logging\n", "setup_logging(2)\n", "\n", "dag = EODataAccessGateway()\n", "dag.set_preferred_provider(\"peps\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "A default set of search criteria is defined, the area of interest is in the South-West of France." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "default_search_criteria = {\n", " \"productType\": \"S2_MSI_L1C\",\n", " \"start\": \"2021-03-01\",\n", " \"end\": \"2021-03-31\",\n", " \"geom\": {\"lonmin\": 1, \"latmin\": 43, \"lonmax\": 2, \"latmax\": 44}\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Pagination" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "A [Google search](https://www.google.com/search?q=eodag) displays the first result page, which has 10 items (i.e. URLs). An action is required to get the next results, i.e. the second page, which would also contain 10 items. This process has the advantage of sending less data through the web, as a consequence the results are displayed faster. And after all, in most cases the URLs obtained from the first page are enough.\n", "\n", "Pagination is what is described above. Most EO product providers operate this way, they return by default the first page result with a given number of items (i.e. products). Since pagination is ubiquitous among providers, it is built deep into `eodag` too and its search capabilities." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Search methods" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The three search methods introduced below accept pretty much the same parameters, which are described further down the page." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "[search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) was the first search method implemented in `eodag`. It returns a tuple with:\n", "\n", "* a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) that stores the products obtained from a given **page** and a given maximum **number of items per page**\n", "* an integer that is the **estimated** total number of products matching the search criteria\n", "\n", "By default, [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) **returns the products from the first page with a maximum of 20 products**. This means\n", "that it is often expected to obtain a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) that contains 20 products and a much larger estimated\n", "total number of products available.\n", "\n", "
\n", "\n", "Warning\n", "\n", "The second element returned by [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) is the **estimated** total number of products matching the search criteria, since, unfortunately, all the providers do not return the **exact** total number. For example,\n", "*theia* returns the number of products available in the whole collection instead of the number of products that match the search criteria.\n", "\n", "
\n", "\n", "Pagination can be controlled with two optional parameters: `page` and `items_per_page`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:03:30,087 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:03:30,089 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:03:32,525 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1\n", "2023-01-13 11:03:33,810 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "products_first_page, estimated_total_number = dag.search(**default_search_criteria)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got a hand on 20 products and an estimated total number of 48 products available.\n" ] } ], "source": [ "print(f\"Got a hand on {len(products_first_page)} products and an estimated total number of {estimated_total_number} products available.\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:03:38,323 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:03:38,326 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:03:38,824 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=10&page=2\n", "2023-01-13 11:03:39,756 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "products_another_second_page, estimated_total_number = dag.search(page=2, items_per_page=10, **default_search_criteria)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got a hand on 10 products and an estimated total number of 48 products available.\n" ] } ], "source": [ "print(f\"Got a hand on {len(products_another_second_page)} products and an estimated total number of {estimated_total_number} products available.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Warning\n", "\n", "To get all the products available, it would seem natural to set `items_per_page` to a very high value\n", "(e.g. 10000). However, the providers usually have set a maximum number of products/items that can be\n", "requested in a single query. If `items_per_page` is set to a value higher than this provider's limit, the\n", "search may either return an empty [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) or fail and raise an error.\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The `raise_errors` parameter controls how errors raised internally during a search are propagated to the user. By default this parameter is set to `False`, which means that **errors are not raised**. Instead, errors are logged and a null result is returned (empty [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) and 0)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "bad_search_criteria = default_search_criteria.copy()\n", "bad_search_criteria[\"start\"] = \"malformed_start_date\"" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:04:01,626 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:04:01,629 eodag.core [INFO ] No result from provider 'peps' due to an error during search. Raise verbosity of log messages for details\n", "2023-01-13 11:04:01,631 eodag.core [ERROR ] Error while searching on provider peps (ignored):\n", "Traceback (most recent call last):\n", " File \"/home/sylvain/workspace/eodag/eodag/api/core.py\", line 1389, in _do_search\n", " res, nb_res = search_plugin.query(count=count, **kwargs)\n", " File \"/home/sylvain/workspace/eodag/eodag/plugins/search/qssearch.py\", line 384, in query\n", " self.search_urls, total_items = self.collect_search_urls(\n", " File \"/home/sylvain/workspace/eodag/eodag/plugins/search/qssearch.py\", line 582, in collect_search_urls\n", " for collection in self.get_collections(**kwargs):\n", " File \"/home/sylvain/workspace/eodag/eodag/plugins/search/qssearch.py\", line 812, in get_collections\n", " match = re.match(\n", "AttributeError: 'NoneType' object has no attribute 'groupdict'\n" ] } ], "source": [ "products_first_page, estimated_total_number = dag.search(**bad_search_criteria)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got a hand on 0 products and an estimated total number of 0 products available.\n" ] } ], "source": [ "print(f\"Got a hand on {len(products_first_page)} products and an estimated total number of {estimated_total_number} products available.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Setting this parameter to `True` does propagate errors." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:04:42,214 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:04:42,217 eodag.core [INFO ] No result from provider 'peps' due to an error during search. Raise verbosity of log messages for details\n" ] }, { "ename": "AttributeError", "evalue": "'NoneType' object has no attribute 'groupdict'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", "Input \u001b[0;32mIn [10]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m products_first_page, estimated_total_number \u001b[38;5;241m=\u001b[39m \u001b[43mdag\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msearch\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mbad_search_criteria\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mraise_errors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/workspace/eodag/eodag/api/core.py:902\u001b[0m, in \u001b[0;36mEODataAccessGateway.search\u001b[0;34m(self, page, items_per_page, raise_errors, start, end, geom, locations, **kwargs)\u001b[0m\n\u001b[1;32m 897\u001b[0m search_kwargs\u001b[38;5;241m.\u001b[39mupdate(\n\u001b[1;32m 898\u001b[0m page\u001b[38;5;241m=\u001b[39mpage,\n\u001b[1;32m 899\u001b[0m items_per_page\u001b[38;5;241m=\u001b[39mitems_per_page,\n\u001b[1;32m 900\u001b[0m )\n\u001b[1;32m 901\u001b[0m search_plugin\u001b[38;5;241m.\u001b[39mclear()\n\u001b[0;32m--> 902\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_do_search\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 903\u001b[0m \u001b[43m \u001b[49m\u001b[43msearch_plugin\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcount\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mraise_errors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mraise_errors\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43msearch_kwargs\u001b[49m\n\u001b[1;32m 904\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/workspace/eodag/eodag/api/core.py:1389\u001b[0m, in \u001b[0;36mEODataAccessGateway._do_search\u001b[0;34m(self, search_plugin, count, raise_errors, **kwargs)\u001b[0m\n\u001b[1;32m 1387\u001b[0m total_results \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m\n\u001b[1;32m 1388\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1389\u001b[0m res, nb_res \u001b[38;5;241m=\u001b[39m \u001b[43msearch_plugin\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcount\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcount\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1391\u001b[0m \u001b[38;5;66;03m# Only do the pagination computations when it makes sense. For example,\u001b[39;00m\n\u001b[1;32m 1392\u001b[0m \u001b[38;5;66;03m# for a search by id, we can reasonably guess that the provider will return\u001b[39;00m\n\u001b[1;32m 1393\u001b[0m \u001b[38;5;66;03m# At most 1 product, so we don't need such a thing as pagination\u001b[39;00m\n\u001b[1;32m 1394\u001b[0m page \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpage\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", "File \u001b[0;32m~/workspace/eodag/eodag/plugins/search/qssearch.py:384\u001b[0m, in \u001b[0;36mQueryStringSearch.query\u001b[0;34m(self, items_per_page, page, count, **kwargs)\u001b[0m\n\u001b[1;32m 382\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mquery_params \u001b[38;5;241m=\u001b[39m qp\n\u001b[1;32m 383\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mquery_string \u001b[38;5;241m=\u001b[39m qs\n\u001b[0;32m--> 384\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msearch_urls, total_items \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcollect_search_urls\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 385\u001b[0m \u001b[43m \u001b[49m\u001b[43mpage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpage\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mitems_per_page\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mitems_per_page\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcount\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcount\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\n\u001b[1;32m 386\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 387\u001b[0m provider_results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdo_search(items_per_page\u001b[38;5;241m=\u001b[39mitems_per_page, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 388\u001b[0m eo_products \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnormalize_results(provider_results, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n", "File \u001b[0;32m~/workspace/eodag/eodag/plugins/search/qssearch.py:582\u001b[0m, in \u001b[0;36mQueryStringSearch.collect_search_urls\u001b[0;34m(self, page, items_per_page, count, **kwargs)\u001b[0m\n\u001b[1;32m 580\u001b[0m urls \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m 581\u001b[0m total_results \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m count \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m--> 582\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m collection \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_collections\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m:\n\u001b[1;32m 583\u001b[0m \u001b[38;5;66;03m# skip empty collection if one is required in api_endpoint\u001b[39;00m\n\u001b[1;32m 584\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{collection}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig\u001b[38;5;241m.\u001b[39mapi_endpoint \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m collection:\n\u001b[1;32m 585\u001b[0m \u001b[38;5;28;01mcontinue\u001b[39;00m\n", "File \u001b[0;32m~/workspace/eodag/eodag/plugins/search/qssearch.py:812\u001b[0m, in \u001b[0;36mQueryStringSearch.get_collections\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 810\u001b[0m collections \u001b[38;5;241m=\u001b[39m (\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mS2\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mS2ST\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 811\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 812\u001b[0m match \u001b[38;5;241m=\u001b[39m \u001b[43mre\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmatch\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 813\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43mr\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m(?P\u001b[39;49m\u001b[38;5;124;43m\\\u001b[39;49m\u001b[38;5;124;43md\u001b[39;49m\u001b[38;5;132;43;01m{4}\u001b[39;49;00m\u001b[38;5;124;43m)-(?P\u001b[39;49m\u001b[38;5;124;43m\\\u001b[39;49m\u001b[38;5;124;43md\u001b[39;49m\u001b[38;5;132;43;01m{2}\u001b[39;49;00m\u001b[38;5;124;43m)-(?P\u001b[39;49m\u001b[38;5;124;43m\\\u001b[39;49m\u001b[38;5;124;43md\u001b[39;49m\u001b[38;5;132;43;01m{2}\u001b[39;49;00m\u001b[38;5;124;43m)\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdate\u001b[49m\n\u001b[1;32m 814\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgroupdict\u001b[49m()\n\u001b[1;32m 815\u001b[0m year, month, day \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m 816\u001b[0m \u001b[38;5;28mint\u001b[39m(match[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myear\u001b[39m\u001b[38;5;124m\"\u001b[39m]),\n\u001b[1;32m 817\u001b[0m \u001b[38;5;28mint\u001b[39m(match[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmonth\u001b[39m\u001b[38;5;124m\"\u001b[39m]),\n\u001b[1;32m 818\u001b[0m \u001b[38;5;28mint\u001b[39m(match[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mday\u001b[39m\u001b[38;5;124m\"\u001b[39m]),\n\u001b[1;32m 819\u001b[0m )\n\u001b[1;32m 820\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m year \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m2016\u001b[39m \u001b[38;5;129;01mor\u001b[39;00m (year \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m2016\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m month \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m12\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m day \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m5\u001b[39m):\n", "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'groupdict'" ] } ], "source": [ "products_first_page, estimated_total_number = dag.search(**bad_search_criteria, raise_errors=True)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### [search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "[search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all) takes the pain away from thinking about pagination. It returns a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) that contains **all** the products matching the search criteria. It does so by iterating over the pages of a search result (with [search_iter_page()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_iter_page)) and gathering products. Compared to [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search):\n", "\n", "* It does not return a tuple but directly a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult). The estimate of total number of products available isn't required here, since they all get collected anyway. This also spares some requests to be sent, since the estimate is usually obtained by sending an additional request.\n", "\n", "* It tries to optimize the number of items/products requested per page. The limit of most providers has been configured in `eodag`, it is used if available (e.g. 500 products per page). If not available, a default value of 50 is used. An arbitrary value can also be used.\n", "\n", "* It has no `raise_errors` parameter, errors are not caught. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:04:56,848 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:04:56,850 eodag.core [INFO ] Iterate search over multiple pages: page #1\n", "2023-01-13 11:04:56,853 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=500&page=1\n", "2023-01-13 11:04:58,343 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "all_products = dag.search_all(**default_search_criteria)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got a hand on a total number of 48 products.\n" ] } ], "source": [ "print(f\"Got a hand on a total number of {len(all_products)} products.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The parameter `items_per_page` controls the maximum number of products than can be retrieved at each iteration internally." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:05:03,535 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:05:03,537 eodag.core [INFO ] Iterate search over multiple pages: page #1\n", "2023-01-13 11:05:03,540 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=1\n", "2023-01-13 11:05:04,734 eodag.core [INFO ] Iterate search over multiple pages: page #2\n", "2023-01-13 11:05:04,736 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=2\n", "2023-01-13 11:05:05,550 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "all_products = dag.search_all(**default_search_criteria, items_per_page=30)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The logs show that two requests have been sent to gather all the products, while only one was required in the previous case where [search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all) used internally a limit higher than 48." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### [search_iter_page()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_iter_page)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "[search_iter_page()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_iter_page) is a generator that returns a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) page per page. Compared to [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) and [search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all), it is certainly dedicated to be used by advanced users for some particular application.\n", "\n", "As with [search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all), [search_iter_page()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_iter_page) doesn't have a `raise_errors` parameter, it doesn't catch errors. While [search_all()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_all) optimizes the number of items per page requested by iteration, [search_iter_page()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search_iter_page) uses a default value of 20, which can be set to any arbitrary value." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:05:15,177 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:05:15,180 eodag.core [INFO ] Iterate search over multiple pages: page #1\n", "2023-01-13 11:05:15,182 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=1\n", "2023-01-13 11:05:16,809 eodag.core [INFO ] Iterate search over multiple pages: page #2\n", "2023-01-13 11:05:16,810 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=30&page=2\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Got a hand on 30 products on page 1\n", "Got a hand on 18 products on page 2\n" ] } ], "source": [ "all_results = []\n", "for i, page_results in enumerate(dag.search_iter_page(**default_search_criteria, items_per_page=30)):\n", " print(f\"Got a hand on {len(page_results)} products on page {i+1}\")\n", " all_results.extend(page_results)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Got 48 products after iterating over 2 pages.\n" ] } ], "source": [ "print(f\"Got {len(all_products)} products after iterating over {i+1} pages.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) and [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Each search method returns an instance of the [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) class. This object is a sequence that stores a number of [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) instances. A [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) supports some of the capabilities of a classic Python `list` object." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SearchResult([EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=peps),\n", " EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCH_20210328T124650, provider=peps)])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "all_products[:2]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "An [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) is the representation of an EO product for `eodag`, it stores enough information about how it was obtained (search criteria, provider) and about how to download itself. Most importantly it stores all the metadata that have been acquired by `eodag` during the search made" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=peps)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product = all_products[0]\n", "one_product" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "An [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) has the following attributes:\n", "\n", "* `properties`: dictionary of the product's metadata\n", "* `geometry`: the product's geometry as a `shapely.geometry` object\n", "* `provider`: the provider used to obtain this product\n", "* `product_type`: `eodag`'s identifier of its product type\n", "* `search_kwargs`: a dictionary of the search criteria used to find the product.\n", "* `remote_location`: the URL to the product's download link\n", "* `location`: it is equal to `remote_location` before the product is downloaded. Once downloaded, it is updated to the absolute local path to the product." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product.geometry" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('peps', 'S2_MSI_L1C')" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product.provider, one_product.product_type" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'auth': GenericAuth(provider=peps, priority=1, topic=Authentication),\n", " 'productType': 'S2_MSI_L1C',\n", " 'startTimeFromAscendingNode': '2021-03-01',\n", " 'completionTimeFromAscendingNode': '2021-03-31',\n", " 'geometry': }" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product.search_kwargs" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('https://peps.cnes.fr/resto/collections/S2ST/387c7327-9a71-5a34-9163-0dfdeb024522/download',\n", " 'https://peps.cnes.fr/resto/collections/S2ST/387c7327-9a71-5a34-9163-0dfdeb024522/download')" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product.remote_location, one_product.location" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['abstract', 'instrument', 'platform', 'platformSerialIdentifier', 'processingLevel', 'keywords', 'sensorType', 'license', 'missionStartDate', 'title', 'productType', 'uid', 'keyword', 'resolution', 'organisationName', 'publicationDate', 'parentIdentifier', 'orbitNumber', 'orbitDirection', 'cloudCover', 'snowCover', 'creationDate', 'modificationDate', 'sensorMode', 'startTimeFromAscendingNode', 'completionTimeFromAscendingNode', 'id', 'quicklook', 'downloadLink', 'storageStatus', 'thumbnail', 'resourceSize', 'resourceChecksum', 'visible', 'newVersion', 'isNrt', 'realtime', 'relativeOrbitNumber', 'useDatalake', 's2TakeId', 'mgrs', 'bareSoil', 'highProbaClouds', 'mediumProbaClouds', 'lowProbaClouds', 'snowIce', 'vegetation', 'water', 'isRefined', 'nrtResource', 'services', 'links', 'storage'])" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product.properties.keys()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "An [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) has an [as_dict()](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct.as_dict) to convert it into a GeoJSON-like dictionary and a [from_geojson()](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct.from_geojson) method to create an [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) from a GeoJSON dictionary." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=peps)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from eodag import EOProduct\n", "product_geojson_structure = one_product.as_dict()\n", "recreated_product = EOProduct.from_geojson(product_geojson_structure)\n", "recreated_product" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Since [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct)s have a way to convert themselves to a GeoJSON dictionary, it is natural to be able to convert a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) object to a GeoJSON *FeatureCollection* ([as_geojson_object()](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult.as_geojson_object)). It is also possible to create a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) from a dictionary structured as a *FeatureCollection* with [from_geojson()](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult.from_geojson)." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SearchResult([EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=peps),\n", " EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCH_20210328T124650, provider=peps)])" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from eodag import SearchResult\n", "feature_collection = all_products.as_geojson_object()\n", "recreated_search_result = SearchResult.from_geojson(feature_collection)\n", "recreated_search_result[:2]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "You can also convert a [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) object to other formats, like `shapely.geometry.GeometryCollection` ([as_shapely_geometry_object()](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult.as_shapely_geometry_object)), and `WKT` ([as_wkt_object()](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult.as_wkt_object))." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Note\n", "\n", "[EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) and [SearchResult](../../api_reference/searchresult.rst#eodag.api.search_result.SearchResult) objects have more methods and attributes than the ones listed aboved.\n", "They are gradually introduced in the next pages of this user guide.\n", "\n", "
\n", "\n", "
\n", "\n", "Note\n", "\n", "When [eodag-cube](https://github.com/CS-SI/eodag-cube) is installed the [EOProduct](../../api_reference/eoproduct.rst#eodag.api.product._product.EOProduct) class used by `eodag` is the one provided by [eodag-cube](https://github.com/CS-SI/eodag-cube), which extends it with a few more capabilities, including a `get_data()` method to directly access the product's data.\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Search parameters" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The search methods have exposed in their signature a common set of search parameters:\n", "\n", "* `productType`\n", "* `start` and `end`\n", "* `geom`\n", "* `locations`\n", "\n", "More parameters can be passed through *kwargs*, they are also described below." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Product type" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The `productType` parameter allows to specify which kind of product is searched for. It should be one of `eodag`'s product type ID, such as `S2_MSI_L1C`. They are discoverable with the method [list_product_types()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.list_product_types) or can be viewed [here](../../getting_started_guide/product_types.rst)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Warning\n", "\n", "The feature described below is intended for advanced users.\n", "\n", "
\n", "\n", "The product types exposed by `eodag` for a given provider are hard-coded in a configuration file. This means that a provider can add a new product type to its catalog, which would then not be listed in `eodag`'s configuration until it is updated. To cover this case, it has been made possible to search for a product type not referenced by `eodag`.\n", "\n", "Internally, a generic product type has been created (`GENERIC_PRODUCT_TYPE`) to emulate a non-referenced product type. The parameters required to search for a non-referenced product type are saved in [eodag's internal provider configuration file](../../add_provider.rst#providers-pre-configuration). For example *creodias* is configured with this generic produc type:\n", "\n", "```\n", " GENERIC_PRODUCT_TYPE:\n", " productType: '{productType}'\n", " collection: '{collection}'\n", "```\n", "\n", "The above configuration indicates that for this provider a non-referenced product type can be searched for by passing the `productType` and `collection` parameters to a search method, e.g. `dag.search(collection=\"foo\", productType=\"bar\")`. **The values these parameters should take must be found by the user, eodag has just no clue about them**.\n", "\n", "To provide a more concrete example, `eodag` allows to search for `S2_MSI_L1C` product types with *creodias*. This product type configuration is the following:\n", "\n", "```\n", " S2_MSI_L1C:\n", " productType: L1C\n", " collection: Sentinel2\n", "```\n", "\n", "This means that `dag.search(productType=\"L1C\", collection=\"Sentinel2\")` should return the same products as `dag.search(productType=\"S2_MSI_L1C\")`." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Time period" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`start` and `end` parameters are optional. They are used to search for products that have been sensed between these two times. Dates or datetimes must be passed as `strings` in UTC ISO8601 format (e.g. `yyyy-MM-dd`, `yyyy-MM-ddThh:mm:ss.SSSZ`)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Area of interest" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`eodag` provides multiple ways to define the area over which products should be searched for. The products returned are those that **intersect** with the area of interest, which means that their geometry can sometimes only partially overlap with the search geometry." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Warning\n", "\n", "`eodag` transforms the area of interest passed by the user (e.g. a `shapely` polygon) to a geometry representation accepted by the targeted provider (e.g. a WKT string). Providers can set a limit to the size of its query parameters. In the case of a very large and detailed geometry passed by the user, its representation might exceed the provider's limit (e.g. a very long WKT string). `eodag` tries internally to detect too long WKT strings, and simplifies them if so, by reducing the resolution of the input geometry iteratively. However, this algorithm doesn't enforce the output WKT string to be lower than a certain limit, `eodag` doesn't want to change too much the geometry passed by the user. As a consequence, the request sent might still contain a WKT string that is above the provider's limit (a *WARNING* is emitted), which would then end up in an error. In that case, it's the user's responsability to pass to `eodag` a simplified geometry, either by generating a convex hull or by splitting it (and executing multiple searches).\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "[folium](https://python-visualization.github.io/folium/) is used in this section to display the search area and the extent of the products found on an interactive map." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "import folium" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### `geom` parameter" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The first way to define an area of interest is to define the optinal `geom` parameter which accepts the following different inputs:\n", "\n", "* a `Shapely` geometry object (any kind of geometry: point, line, polygon, multipolygon...)\n", "* a *Well-Known Text* (WKT) string (any kind of geometry: point, line, polygon, multipolygon...)\n", "* a bounding box as a dictionary with keys: `\"lonmin\"`, `\"latmin\"`, `\"lonmax\"`, `\"latmax\"`\n", "* a bounding box as a list with elements provided in the order [lonmin, latmin, lonmax, latmax]\n", "\n", "The coordinates must be provided in the **WGS84** projection (EPSG: 4326)." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10 products are going to be mapped.\n" ] } ], "source": [ "# Only a subset of the products is used not to overload the map\n", "prods_to_map = all_products[::5]\n", "print(f\"{len(prods_to_map)} products are going to be mapped.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "In the previous searches made `geom` was defined as a bounding box expressed as a dictionary." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'lonmin': 1, 'latmin': 43, 'lonmax': 2, 'latmax': 44}" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "geometry = default_search_criteria[\"geom\"]\n", "geometry" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "This is equivalent to:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "geom = [1, 43, 2, 44] # or geom = (1, 43, 1, 44)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "from shapely.geometry import Polygon\n", "geom = Polygon([[1, 43], [2, 43], [2, 44], [1, 44], [1, 43]])" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "geom = \"POLYGON ((1 43, 2 43, 2 44, 1 44, 1 43))\" # WKT string" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a map zoomed over the search area\n", "fmap = folium.Map([43.5, 1.5], zoom_start=7)\n", "# Create a layer that represents the search area in red\n", "folium.Rectangle(\n", " bounds=[[geometry[\"latmin\"], geometry[\"lonmin\"]], [geometry[\"latmax\"], geometry[\"lonmax\"]]],\n", " color=\"red\",\n", " tooltip=\"Search extent\"\n", ").add_to(fmap)\n", "# Create a layer that maps the products found\n", "folium.GeoJson(\n", " data=prods_to_map, # SearchResult has a __geo_interface__ interface used by folium to get its GeoJSON representation\n", " tooltip=folium.GeoJsonTooltip(fields=[\"title\"])\n", ").add_to(fmap)\n", "fmap" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Locations search" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Locations search is a powerful feature that greatly simplifies the setting of an area of interest.\n", "\n", "When the `EODataAcessGateway` instance was created the logs showed that a *locations configuration* was automatically loaded by `eodag` from its local configuration directory. A *locations configuration* is a YAML file that contains a shapefile list associated to a name and an attribute. A minimal example of such a file is provided below:\n", "\n", "```YAML\n", "shapefiles:\n", "\n", " - name: continent\n", " path: /path/to/continents.shp\n", " attr: fullname\n", "```\n", "\n", "Where:\n", "* `name` is the argument name that can be used in a [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) to refer to this specific location.\n", "* `path` is the absolute path to the shapefile\n", "* `attr` is the field of the shapefile that can be used to select features from it\n", "\n", "For example, a *continents.shp* shapefile is set as a location in this file. The *path* entry is set to its absolute filepath, the *name* entry is set to `continent`. The shapefile contains continent's areas (polygons) and a field *fullname* (it may have other fields, they just won't be of any use here). The following search uses the geometry of the features of *continents.shp* that have *fullname* equal to `Europe`:\n", "\n", "```python\n", "products, estimated_total_nbr_of_results = dag.search(\n", " productType=\"S2_MSI_L1C\",\n", " locations=dict(continent='Europe')\n", ")\n", "```\n", "\n", "The location query (`continent=\"Europe\"`) is passed as a dictionnary to the `locations` parameter. It accepts [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) which can come in handy when the query field has an underlying structure (e.g. see [this tutorial](../tutos/tuto_search_location_tile.ipynb) dedicated to search for products by tile(s)).\n", "\n", "The locations configuration is stored in the `locations_config` attribute of the `EODataAcessGateway` once instantiated. `eodag` provides a default location which is a [Natural Earth Countries](https://www.naturalearthdata.com/downloads/110m-cultural-vectors/110m-admin-0-countries/) shapefile whose *ADM0_A3_US* field can be used to query specific countries by a short code such as *FRA* for *France* or *JPN* for *Japan*." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'name': 'country',\n", " 'path': '/home/sylvain/.config/eodag/shp/ne_110m_admin_0_map_units.shp',\n", " 'attr': 'ADM0_A3_US'}]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.locations_config" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Values taken by `country` (ADM0_A3_US):\n", "\n", "AFG AGO ALB ARE ARG ARM ATA ATF AUS AUT AZE BDI BEL BEN BFA BGD BGR BHS BIH BLR BLZ BOL BRA BRN BTN BWA CAF CAN CHE CHL CHN CIV CMR COD COG COL CRI CUB CYP CZE DEU DJI DNK DOM DZA ECU EGY ERI ESP EST ETH FIN FJI FLK FRA GAB GBR GEO GHA GIN GMB GNB GNQ GRC GRL GTM GUY HND HRV HTI HUN IDN IND IRL IRN IRQ ISL ISR ITA JAM JOR JPN KAZ KEN KGZ KHM KOR KOS KWT LAO LBN LBR LBY LKA LSO LTU LUX LVA MAR MDA MDG MEX MKD MLI MMR MNE MNG MOZ MRT MWI MYS NAM NCL NER NGA NIC NLD NOR NPL NZL OMN PAK PAN PER PHL PNG POL PRI PRK PRT PRY PSX QAT ROU RUS RWA SAH SAU SDN SDS SEN SLB SLE SLV SOM SRB SUR SVK SVN SWE SWZ SYR TCD TGO THA TJK TKM TLS TTO TUN TUR TWN TZA UGA UKR URY USA UZB VEN VNM VUT YEM ZAF ZMB ZWE\n" ] } ], "source": [ "# Get the shapefile filepath and the field used as query parameter\n", "countries_shpfile = dag.locations_config[0][\"path\"]\n", "attr = dag.locations_config[0][\"attr\"]\n", "\n", "# pyshp is imported to read the shapefile and display the values taken\n", "# by the features for the field ADM0_A3_US.\n", "import shapefile\n", "with shapefile.Reader(countries_shpfile) as shp:\n", " shaperecs = shp.shapeRecords()\n", "countries_adm0 = sorted(set(shprec.record[attr] for shprec in shaperecs))\n", "print(f\"Values taken by `country` ({attr}):\\n\\n{' '.join(countries_adm0)}\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "A search will be made over Switzerland (CHE) and Belgium (BEL)." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'productType': 'S2_MSI_L1C',\n", " 'start': '2021-03-01',\n", " 'end': '2021-03-31',\n", " 'locations': {'country': 'BEL|CHE'}}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "location_search_criteria = default_search_criteria.copy()\n", "del location_search_criteria[\"geom\"]\n", "location_search_criteria[\"locations\"] = dict(country=\"BEL|CHE\") # This regex means: BEL or CHE\n", "location_search_criteria" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:06:53,694 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:06:53,696 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=MULTIPOLYGON (((6.1567 50.8037, 5.6070 51.0373, 4.9740 51.4750, 4.0471 51.2673, 3.3150 51.3458, 3.3150 51.3458, 3.3150 51.3458, 2.5136 51.1485, 2.6584 50.7968, 3.1233 50.7804, 3.5882 50.3790, 4.2860 49.9075, 4.7992 49.9854, 5.6741 49.5295, 5.7824 50.0903, 6.0431 50.1281, 6.1567 50.8037)), ((9.5942 47.5251, 8.5226 47.8308, 8.3173 47.6136, 7.4668 47.6206, 7.1922 47.4498, 6.7366 47.5418, 6.7687 47.2877, 6.0374 46.7258, 6.0226 46.2730, 6.5001 46.4297, 6.8436 45.9911, 7.2739 45.7769, 7.7560 45.8245, 8.3166 46.1636, 8.4900 46.0052, 8.9663 46.0369, 9.1829 46.4402, 9.9228 46.3149, 10.3634 46.4836, 10.4427 46.8935, 9.9324 46.9207, 9.4800 47.1028, 9.6329 47.3476, 9.5942 47.5251)))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:06:54,398 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=MULTIPOLYGON (((6.1567 50.8037, 5.6070 51.0373, 4.9740 51.4750, 4.0471 51.2673, 3.3150 51.3458, 3.3150 51.3458, 3.3150 51.3458, 2.5136 51.1485, 2.6584 50.7968, 3.1233 50.7804, 3.5882 50.3790, 4.2860 49.9075, 4.7992 49.9854, 5.6741 49.5295, 5.7824 50.0903, 6.0431 50.1281, 6.1567 50.8037)), ((9.5942 47.5251, 8.5226 47.8308, 8.3173 47.6136, 7.4668 47.6206, 7.1922 47.4498, 6.7366 47.5418, 6.7687 47.2877, 6.0374 46.7258, 6.0226 46.2730, 6.5001 46.4297, 6.8436 45.9911, 7.2739 45.7769, 7.7560 45.8245, 8.3166 46.1636, 8.4900 46.0052, 8.9663 46.0369, 9.1829 46.4402, 9.9228 46.3149, 10.3634 46.4836, 10.4427 46.8935, 9.9324 46.9207, 9.4800 47.1028, 9.6329 47.3476, 9.5942 47.5251)))&productType=S2MSI1C&maxRecords=20&page=1\n", "2023-01-13 11:06:56,590 eodag.core [INFO ] Found 248 result(s) on provider 'peps'\n" ] } ], "source": [ "locations_products, estimated_total_number = dag.search(**location_search_criteria)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The results obtained are displayed on a map in addition to the countries of the default location shapefile." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a map zoomed over the search area\n", "fmap = folium.Map([48, 1.5], zoom_start=5)\n", "# Create a layer that maps the countries in green\n", "folium.GeoJson(\n", " data=shaperecs,\n", " tooltip=folium.GeoJsonTooltip(fields=[\"ADM0_A3_US\"]),\n", " style_function=lambda x: {\"color\": \"green\"}\n", ").add_to(fmap)\n", "# Create a layer that maps the products found\n", "folium.GeoJson(\n", " data=locations_products,\n", " tooltip=folium.GeoJsonTooltip(fields=[\"title\"])\n", ").add_to(fmap)\n", "fmap" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Note\n", "\n", "\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Add locations search" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Users are obsviously able to modify the location configuration file to add their own *locations* or to create their own YAML location configuration file. [EODataAccessGateway](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway) accepts the optional `locations_conf_path` parameter that can set to the path to another location configuration file. Alternatively, the environment variable `EODAG_LOCS_CFG_FILE` can also be set to define the path to another location configuration file (this has priority over the parameter value if passed too)." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "# dag = EODataAccessGateway(locations_conf_path=\"path/to/my_own_location_config.yaml\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Combined `geom` and locations search" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`eodag` allows to combine a geometry search (e.g. a bounding box) with a location search. In that case the search area becomes the union of the two geometries found." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'productType': 'S2_MSI_L1C',\n", " 'start': '2021-03-01',\n", " 'end': '2021-03-31',\n", " 'geom': {'lonmin': 1, 'latmin': 43, 'lonmax': 2, 'latmax': 44},\n", " 'locations': {'country': 'BEL|CHE'}}" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "combined_search_criteria = default_search_criteria.copy()\n", "combined_search_criteria[\"locations\"] = location_search_criteria[\"locations\"]\n", "combined_search_criteria" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:07:18,062 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:07:18,064 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=MULTIPOLYGON (((6.1567 50.8037, 5.6070 51.0373, 4.9740 51.4750, 4.0471 51.2673, 3.3150 51.3458, 3.3150 51.3458, 3.3150 51.3458, 2.5136 51.1485, 2.6584 50.7968, 3.1233 50.7804, 3.5882 50.3790, 4.2860 49.9075, 4.7992 49.9854, 5.6741 49.5295, 5.7824 50.0903, 6.0431 50.1281, 6.1567 50.8037)), ((9.5942 47.5251, 8.5226 47.8308, 8.3173 47.6136, 7.4668 47.6206, 7.1922 47.4498, 6.7366 47.5418, 6.7687 47.2877, 6.0374 46.7258, 6.0226 46.2730, 6.5001 46.4297, 6.8436 45.9911, 7.2739 45.7769, 7.7560 45.8245, 8.3166 46.1636, 8.4900 46.0052, 8.9663 46.0369, 9.1829 46.4402, 9.9228 46.3149, 10.3634 46.4836, 10.4427 46.8935, 9.9324 46.9207, 9.4800 47.1028, 9.6329 47.3476, 9.5942 47.5251)), ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000)))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:07:18,515 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=MULTIPOLYGON (((6.1567 50.8037, 5.6070 51.0373, 4.9740 51.4750, 4.0471 51.2673, 3.3150 51.3458, 3.3150 51.3458, 3.3150 51.3458, 2.5136 51.1485, 2.6584 50.7968, 3.1233 50.7804, 3.5882 50.3790, 4.2860 49.9075, 4.7992 49.9854, 5.6741 49.5295, 5.7824 50.0903, 6.0431 50.1281, 6.1567 50.8037)), ((9.5942 47.5251, 8.5226 47.8308, 8.3173 47.6136, 7.4668 47.6206, 7.1922 47.4498, 6.7366 47.5418, 6.7687 47.2877, 6.0374 46.7258, 6.0226 46.2730, 6.5001 46.4297, 6.8436 45.9911, 7.2739 45.7769, 7.7560 45.8245, 8.3166 46.1636, 8.4900 46.0052, 8.9663 46.0369, 9.1829 46.4402, 9.9228 46.3149, 10.3634 46.4836, 10.4427 46.8935, 9.9324 46.9207, 9.4800 47.1028, 9.6329 47.3476, 9.5942 47.5251)), ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000)))&productType=S2MSI1C&maxRecords=50&page=1\n", "2023-01-13 11:07:20,210 eodag.core [INFO ] Found 425 result(s) on provider 'peps'\n" ] } ], "source": [ "combined_products, estimated_total_number = dag.search(**combined_search_criteria, items_per_page=50)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a map zoomed over the search area\n", "fmap = folium.Map([48, 1.5], zoom_start=5)\n", "# Create a layer that represents the 'geom' search area in red\n", "folium.Rectangle(\n", " bounds=[[geometry[\"latmin\"], geometry[\"lonmin\"]], [geometry[\"latmax\"], geometry[\"lonmax\"]]],\n", " color=\"red\",\n", " tooltip=\"Search extent\"\n", ").add_to(fmap)\n", "# Create a layer that maps the countries in green\n", "folium.GeoJson(\n", " data=shaperecs,\n", " tooltip=folium.GeoJsonTooltip(fields=[\"ADM0_A3_US\"]),\n", " style_function=lambda x: {\"color\": \"green\"}\n", ").add_to(fmap)\n", "# Create a layer that maps the products found\n", "folium.GeoJson(\n", " data=combined_products,\n", " tooltip=folium.GeoJsonTooltip(fields=[\"title\"])\n", ").add_to(fmap)\n", "fmap" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Supported kwargs" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### `id` and `provider`" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "EO products have in their metadata an `id` field." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650'" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "product_id = one_product.properties[\"id\"]\n", "product_id" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "This identifier, if known beforehand, can be used to search for this specific product by passing the `id` kwarg to the [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) method. The `provider` kwarg can also be passed to specify among which provider's catalog the product should be searched for. If `provider` is not provided, `eodag` will iterate over all the providers until it finds the product targeted." ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:09:17,102 eodag.core [INFO ] Searching product with id 'S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650' on provider: onda\n", "2023-01-13 11:09:17,104 eodag.plugins.search.qssearch [INFO ] Sending search request: https://catalogue.onda-dias.eu/dias-catalogue/Products?$format=json&$search=%22S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650%22\n", "2023-01-13 11:09:18,358 eodag.core [INFO ] Found 1 result(s) on provider 'onda'\n" ] }, { "data": { "text/plain": [ "(SearchResult([EOProduct(id=S2B_MSIL1C_20210328T103629_N0209_R008_T31TCJ_20210328T124650, provider=onda)]),\n", " 1)" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one_product_onda, _ = dag.search(id=product_id, provider=\"onda\")\n", "one_product_onda, _" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Guess a product type" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`eodag` has an internal product type catalog which stores a number of metadata for each product type (see [this page](../../getting_started_guide/product_types.rst) for more information). It is actually possible to search for products by using some of these metadata. In that case, `eodag` will query its own catalog and use **the product type that best matches the query** among the available product types (i.e. all product types offered by providers where the necessary credentials are provided). The supported kwargs are:\n", "\n", "* `instrument` (e.g. *MSI*)\n", "* `platform` (e.g. *SENTINEL2*)\n", "* `platformSerialIdentifier` (e.g. *S2A*)\n", "* `processingLevel` (e.g. *L1*)\n", "* `sensorType` (e.g. *OPTICAL*)\n", "* `keywords` (e.g. *SENTINEL2 L1C SAFE*), which is case insensitive and ignores `-` or `_` characters" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "For example if we call the method with platform=\"SENTINEL1\", all the product types offering products from this platform will be returned." ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_RAW', 'S1_SAR_SLC']" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.guess_product_type(platform=\"SENTINEL1\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "By passing the following argument we can get all collections that contain the keyword collection2 and a keyword that starts with \"LANDSAT\"." ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['LANDSAT_C2L1',\n", " 'LANDSAT_C2L2',\n", " 'LANDSAT_C2L2ALB_BT',\n", " 'LANDSAT_C2L2ALB_SR',\n", " 'LANDSAT_C2L2ALB_ST',\n", " 'LANDSAT_C2L2ALB_TA',\n", " 'LANDSAT_C2L2_SR',\n", " 'LANDSAT_C2L2_ST',\n", " 'LANDSAT_ETM_C2L1',\n", " 'LANDSAT_ETM_C2L2',\n", " 'LANDSAT_TM_C2L1',\n", " 'LANDSAT_TM_C2L2']" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.guess_product_type(keywords=\"LANDSAT* collection2\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "In the previous request we made use of the whoosh query language which can be used to do complex text search. It supports the boolean operators AND, OR and NOT to combine the search terms. If a space is given between two words as in the example above, this corresponds to the operator AND. Brackets '()' can also be used. The example above also shows the use of the wildcard operator '*' which can represent any numer of characters. The wildcard operator '?' always represents only one character. It is also possible to match a range of terms by using square brackets '[]' and TO, e.g. [A TO D] will match all words in the lexical range between A and D. Below you can find some examples for the different operators." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dag.guess_product_type(platform=\"LANDSAT OR SENTINEL1\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "returns all product types where the platform is either LANDSAT or SENTINEL1:\n", "\n", "['L57_REFLECTANCE', 'LANDSAT_C2L1', 'LANDSAT_C2L2', 'LANDSAT_C2L2ALB_BT', 'LANDSAT_C2L2ALB_SR', 'LANDSAT_C2L2ALB_ST', 'LANDSAT_C2L2ALB_TA', 'LANDSAT_C2L2_SR', 'LANDSAT_C2L2_ST', 'LANDSAT_ETM_C1', 'LANDSAT_ETM_C2L1', 'LANDSAT_ETM_C2L2', 'LANDSAT_TM_C1', 'LANDSAT_TM_C2L1', 'LANDSAT_TM_C2L2', 'S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_RAW', 'S1_SAR_SLC']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dag.guess_product_type(keywords=\"(LANDSAT AND collection2) OR SAR\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "returns all product types which contain either the keywords LANDSAT and collection2 or the keyword SAR:\n", "\n", "['LANDSAT_C2L1', 'LANDSAT_C2L2', 'LANDSAT_C2L2ALB_BT', 'LANDSAT_C2L2ALB_SR', 'LANDSAT_C2L2ALB_ST', 'LANDSAT_C2L2ALB_TA', 'LANDSAT_C2L2_SR', 'LANDSAT_C2L2_ST', 'LANDSAT_ETM_C2L1', 'LANDSAT_ETM_C2L2', 'LANDSAT_TM_C2L1', 'LANDSAT_TM_C2L2', 'S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_RAW', 'S1_SAR_SLC']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dag.guess_product_type(platformSerialIdentifier=\"L?\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "returns all product types where the platformSerialIdentifier is composed of 'L' and one other character:\n", "\n", "['L57_REFLECTANCE', 'L8_OLI_TIRS_C1L1', 'L8_REFLECTANCE', 'LANDSAT_C2L1', 'LANDSAT_C2L2', 'LANDSAT_C2L2ALB_BT', 'LANDSAT_C2L2ALB_SR', 'LANDSAT_C2L2ALB_ST', 'LANDSAT_C2L2ALB_TA', 'LANDSAT_C2L2_SR', 'LANDSAT_C2L2_ST', 'LANDSAT_ETM_C1', 'LANDSAT_ETM_C2L1', 'LANDSAT_ETM_C2L2', 'LANDSAT_TM_C1', 'LANDSAT_TM_C2L1', 'LANDSAT_TM_C2L2']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dag.guess_product_type(platform=\"[SENTINEL1 TO SENTINEL3]\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "returns all product types where the platform is SENTINEL1, SENTINEL2 or SENTINEL3:\n", "\n", "['S1_SAR_GRD', 'S1_SAR_OCN', 'S1_SAR_RAW', 'S1_SAR_SLC', 'S2_MSI_L1C', 'S2_MSI_L2A', 'S2_MSI_L2A_COG', 'S2_MSI_L2A_MAJA', 'S2_MSI_L2B_MAJA_SNOW', 'S2_MSI_L2B_MAJA_WATER', 'S2_MSI_L3A_WASP', 'S3_EFR', 'S3_ERR', 'S3_LAN', 'S3_OLCI_L2LFR', 'S3_OLCI_L2LRR', 'S3_OLCI_L2WFR', 'S3_OLCI_L2WRR', 'S3_RAC', 'S3_SLSTR_L1RBT', 'S3_SLSTR_L2AOD', 'S3_SLSTR_L2FRP', 'S3_SLSTR_L2LST', 'S3_SLSTR_L2WST', 'S3_SRA', 'S3_SRA_A', 'S3_SRA_BS', 'S3_SY_AOD', 'S3_SY_SYN', 'S3_SY_V10', 'S3_SY_VG1', 'S3_SY_VGP', 'S3_WAT']" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "We can also pass several parameters at the same time, e.g.:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dag.guess_product_type(platform=\"LANDSAT\", platformSerialIdentifier=\"L1\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "['LANDSAT_C2L1', \n", "'L57_REFLECTANCE', \n", "'LANDSAT_C2L2', \n", "'LANDSAT_C2L2ALB_BT', \n", "'LANDSAT_C2L2ALB_SR', \n", "'LANDSAT_C2L2ALB_ST', \n", "'LANDSAT_C2L2ALB_TA', \n", "'LANDSAT_C2L2_SR', \n", "'LANDSAT_C2L2_ST', \n", "'LANDSAT_ETM_C1', \n", "'LANDSAT_ETM_C2L1', \n", "'LANDSAT_ETM_C2L2', \n", "'LANDSAT_TM_C1', \n", "'LANDSAT_TM_C2L1', \n", "'LANDSAT_TM_C2L2']" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The product types in the result are ordered by how well they match the criteria. In the example above only the first product type (LANDSAT_C2L1) matches the second parameter (platformSerialIdentifier=\"L1\"), all other product types only match the first criterion. Therefore, it is usually best to use the first product type in the list as it will be the one that fits best." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "If a search is made without specifying `productType` but hints are passed with following supported kwargs, `guess_product_type()` will internally be used to get the best matching product type." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'start': '2021-03-01',\n", " 'end': '2021-03-31',\n", " 'geom': {'lonmin': 1, 'latmin': 43, 'lonmax': 2, 'latmax': 44},\n", " 'instrument': 'MSI',\n", " 'platform': 'SENTINEL2',\n", " 'platformSerialIdentifier': 'S2A',\n", " 'sensorType': 'OPTICAL'}" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "guess_search_criteria = default_search_criteria.copy()\n", "del guess_search_criteria[\"productType\"]\n", "guess_search_criteria.update(\n", " instrument=\"MSI\",\n", " platform=\"SENTINEL2\",\n", " platformSerialIdentifier=\"S2A\",\n", " sensorType=\"OPTICAL\",\n", ")\n", "guess_search_criteria" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'S2_MSI_L1C'" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.guess_product_type(**guess_search_criteria)[0]" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:09:49,233 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:09:49,235 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:09:49,743 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1\n", "2023-01-13 11:09:50,514 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "products, estimated_total_number = dag.search(**guess_search_criteria)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "`eodag` searches through its metadata catalog and checks which product type corresponds to the given parameters. There is a product type called `S2_MSI_L1C`, for the platform SENTINEL2 with the platformSerialIdentifiers S2A and S2B using the instrument MSI and the sensorType OPTICAL. Thus, all parameters of the product type `S2_MSI_L1C` are matching to the given search criteria and it is chosen to be internally used to search for products. If no product type is matching all the criteria, the one with the most matches is chosen, if several product types match all the given criteria, the first one is chosen." ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'S2_MSI_L1C'" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "products[0].product_type" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Extra query parameters" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The search methods accept any additional query parameter passed as kwargs. Some of them may already be known to `eodag` and be converted in a correct way." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Warning\n", "\n", "This feature is intended for advanced users.\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Providers generally accept many more request parameters than the basic ones provided by `eodag` by default (geometry and time period). Many of these parameters are actually known to `eodag` and configured internally (see the [providers pre-configuration section](../../add_provider.rst#providers-pre-configuration)). They are just not exposed in the signature of the search methods, not to overload it, and also because these parameters are not necessarily share among all the providers.\n", "\n", "A common parameter used when searching for EO products is the percentage of cloud cover. *PEPS* for instance accepts a `cloudCover` parameter. `eodag` knows how to transform it in a way that is adapted to *PEPS*." ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:10:05,660 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:10:05,664 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?cloudCover=[0,10]&startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:10:06,223 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?cloudCover=[0,10]&startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1\n", "2023-01-13 11:10:06,922 eodag.core [INFO ] Found 10 result(s) on provider 'peps'\n" ] } ], "source": [ "products, estimated_total_number = dag.search(\n", " cloudCover=10, # cloud cover Less than 10\n", " **default_search_criteria\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "It is possible to check that the search returned products than have a cloud cover less than 10%." ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0, 7.7926, 1.5388000000000002, 0.0185, 0, 0, 4.4443, 1.7617, 6.8998, 0]" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[p.properties[\"cloudCover\"] for p in products]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Having a closer look at the URL displayed in the logs above, it can be observed that `cloudCover=10` was actually converted by `eodag` to the query string `cloudCover=[0,10]`. This proves that `eodag` indeed knows about this parameter and how to adapt it for *PEPS*." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Query parameters unknown to `eodag` are also propagated until they are directly requested to the provider, albeit as is." ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-01-13 11:10:20,091 eodag.core [INFO ] Searching product type 'S2_MSI_L1C' on provider: peps\n", "2023-01-13 11:10:20,094 eodag.plugins.search.qssearch [INFO ] Sending count request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?random_parameter=random_value&startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=1&page=1\n", "2023-01-13 11:10:20,543 eodag.plugins.search.qssearch [INFO ] Sending search request: https://peps.cnes.fr/resto/api/collections/S2ST/search.json?random_parameter=random_value&startDate=2021-03-01&completionDate=2021-03-31&geometry=POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000))&productType=S2MSI1C&maxRecords=20&page=1\n", "2023-01-13 11:10:21,423 eodag.core [INFO ] Found 48 result(s) on provider 'peps'\n" ] } ], "source": [ "products, estimated_total_number = dag.search(\n", " random_parameter=\"random_value\",\n", " **default_search_criteria\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The logs above show that the random kwarg passed to the [search()](../../api_reference/core.rst#eodag.api.core.EODataAccessGateway.search) method was indeed propagated to the request made to *PEPS*. It happens that *PEPS* accept unsupported parameters and still returned products, another provider might just have sent an error." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Fallback in case of error\n", "\n", "Usually, search is performed on the provider with the highest priority. If the request to this provider fails, i.e. if an error is returned (for example because the provider is currently unavailable) or if no results are returned, a request to the the provider with the next highest priority offering the desired product type will be attempted. In this case a warning message will be shown in the logs. If the request fails for all providers offering the desired product type, an error message will be returned." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "2023-07-03 13:38:24,041 eodag.core [INFO] (core) No result from provider 'peps' due to an error during search. Raise verbosity of log messages for details\n", "2023-07-03 13:38:24,041 eodag.core [WARNING] (core) No result could be obtained from provider peps, we will try to get the data from another provider" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Queryables" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To know which query parameters can be used to filter the search result, the `list_queryables` method can be used. It has two optional arguments, `provider` and `productType`, additional keyword arguments to filter the queryable values can be added.\n", "\n", "If the function is called without any arguments, the basic queryables that are available for all providers and product types will be returned:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=True)],\n", " 'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)]}" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the function is called for a specific product type, the queryables available for this product type, i.e., the basic queryables and product type specific queryables, are shown. Only parameters available for all providers offering the product are returned." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'sky_type': typing.Annotated[typing.Literal['clear', 'observed_cloud'], FieldInfo(annotation=NoneType, required=False, default='clear')],\n", " 'time_step': typing.Annotated[typing.Literal['15minute', '1day', '1hour', '1minute', '1month'], FieldInfo(annotation=NoneType, required=False, default='1minute')],\n", " 'time_reference': typing.Annotated[typing.Literal['true_solar_time', 'universal_time'], FieldInfo(annotation=NoneType, required=False, default='true_solar_time')],\n", " 'location': typing.Annotated[dict, FieldInfo(annotation=NoneType, required=False, default={'latitude': 0, 'longitude': 0})],\n", " 'altitude': typing.Annotated[int, FieldInfo(annotation=NoneType, required=False, default=-999)],\n", " 'format': typing.Annotated[typing.Literal['csv', 'netcdf'], FieldInfo(annotation=NoneType, required=False, default='csv')],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)],\n", " 'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='CAMS_SOLAR_RADIATION')],\n", " 'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)]}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables(productType=\"CAMS_SOLAR_RADIATION\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The queryables can also be retrieved for a specific provider. These queryables are either taken from the queryables endpoint of the provider (if available), the constraints for the query parameter values defined by the provider (if available) or the provider configuration in EODAG. In the example below we use a provider offering a queryables endpoint." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-02-19 12:01:07,071 eodag.search.qssearch [INFO ] Fetching queryables: https://planetarycomputer.microsoft.com/api/stac/v1/search/../collections/sentinel-1-grd/queryables\n" ] }, { "data": { "text/plain": [ "{'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)],\n", " 'platform': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 's1:shape': typing.Annotated[list, FieldInfo(annotation=NoneType, required=False)],\n", " 'end_datetime': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='End datetime', description='End datetime', metadata=[PydanticGeneralMetadata(pattern='(\\\\+00:00|Z)$')])],\n", " 's1:resolution': typing.Annotated[typing.Literal['full', 'high', 'medium'], FieldInfo(annotation=NoneType, required=False, title='Resolution')],\n", " 's1:datatake_id': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='Datatake ID')],\n", " 'start_datetime': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='Start datetime', description='Start datetime', metadata=[PydanticGeneralMetadata(pattern='(\\\\+00:00|Z)$')])],\n", " 's1:orbit_source': typing.Annotated[typing.Literal['DOWNLINK', 'POEORB', 'PREORB', 'RESORB'], FieldInfo(annotation=NoneType, required=False, title='Orbit Source')],\n", " 's1:slice_number': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='Slice Number')],\n", " 's1:total_slices': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='Total Slices')],\n", " 'sar:looks_range': typing.Annotated[int, FieldInfo(annotation=NoneType, required=False, title='Looks range')],\n", " 'orbitDirection': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'sar:product_type': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, title='Product type')],\n", " 'sar:looks_azimuth': typing.Annotated[int, FieldInfo(annotation=NoneType, required=False, title='Looks azimuth')],\n", " 'polarizationChannels': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'dopplerFrequency': typing.Annotated[typing.Optional[float], FieldInfo(annotation=NoneType, required=False)],\n", " 'sat:absolute_orbit': typing.Annotated[NoneType, FieldInfo(annotation=NoneType, required=False, title='Absolute Orbit', json_schema_extra={'$ref': 'https://stac-extensions.github.io/sat/v1.0.0/schema.json#/definitions/fields/properties/sat:absolute_orbit'})],\n", " 'orbitNumber': typing.Annotated[typing.Optional[int], FieldInfo(annotation=NoneType, required=False)],\n", " 's1:processing_level': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False)],\n", " 'sensorMode': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'sar:center_frequency': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Center Frequency (GHz)')],\n", " 'sar:resolution_range': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Resolution range (m)')],\n", " 's1:product_timeliness': typing.Annotated[typing.Literal['Fast-24h', 'NRT-10m', 'NRT-1h', 'NRT-3h', 'Off-line', 'Reprocessing'], FieldInfo(annotation=NoneType, required=False, title='Product Timeliness')],\n", " 'sar:resolution_azimuth': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Resolution azimuth (m)')],\n", " 'sar:pixel_spacing_range': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Pixel spacing range (m)')],\n", " 'sar:observation_direction': typing.Annotated[typing.Literal['left', 'right'], FieldInfo(annotation=NoneType, required=False, title='Antenna pointing direction')],\n", " 'sar:pixel_spacing_azimuth': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Pixel spacing azimuth (m)')],\n", " 'sar:looks_equivalent_number': typing.Annotated[float, FieldInfo(annotation=NoneType, required=False, title='Equivalent number of looks (ENL)')],\n", " 's1:instrument_configuration_ID': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False)],\n", " 'sat:platform_international_designator': typing.Annotated[typing.Literal['0000-000A', '2014-016A', '2016-025A'], FieldInfo(annotation=NoneType, required=False, title='Platform Designation')],\n", " 'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='S1_SAR_GRD')],\n", " 'doi': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'platformSerialIdentifier': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'instrument': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'processingLevel': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'resolution': typing.Annotated[typing.Optional[int], FieldInfo(annotation=NoneType, required=False)],\n", " 'publicationDate': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'cloudCover': typing.Annotated[typing.Optional[typing.Annotated[int, Gt(gt=0), Lt(lt=100)]], FieldInfo(annotation=NoneType, required=False)],\n", " 'productVersion': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'creationDate': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'modificationDate': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'availabilityTime': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'acquisitionStation': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'acquisitionSubType': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'illuminationAzimuthAngle': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'illuminationElevationAngle': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)]}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables(productType=\"S1_SAR_GRD\", provider=\"planetary_computer\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For each queryable the possible values will be included in the result if they are available from either the provider queryables or the constraints. If constraints are available for a provider, the queryables can be filtered by the value of a specific parameter. E.g. if the queryables month and day are available, we can set the month to a specific value to get only the days available for that month." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'api_product_type': typing.Annotated[typing.Literal['reanalysis'], FieldInfo(annotation=NoneType, required=False, default='reanalysis')],\n", " 'time': typing.Annotated[typing.Literal['00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'], FieldInfo(annotation=NoneType, required=False, default='00:00')],\n", " 'format': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='grib')],\n", " 'month': typing.Annotated[typing.Literal['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12'], FieldInfo(annotation=NoneType, required=False, default='02')],\n", " 'day': typing.Annotated[typing.Literal['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29'], FieldInfo(annotation=NoneType, required=True)],\n", " 'variable': typing.Annotated[typing.Literal['100m_u_component_of_wind', '100m_v_component_of_wind', '10m_u_component_of_neutral_wind', '10m_u_component_of_wind', '10m_v_component_of_neutral_wind', '10m_v_component_of_wind', '10m_wind_gust_since_previous_post_processing', '2m_dewpoint_temperature', '2m_temperature', 'air_density_over_the_oceans', 'angle_of_sub_gridscale_orography', 'anisotropy_of_sub_gridscale_orography', 'benjamin_feir_index', 'boundary_layer_dissipation', 'boundary_layer_height', 'charnock', 'clear_sky_direct_solar_radiation_at_surface', 'cloud_base_height', 'coefficient_of_drag_with_waves', 'convective_available_potential_energy', 'convective_inhibition', 'convective_precipitation', 'convective_rain_rate', 'convective_snowfall', 'convective_snowfall_rate_water_equivalent', 'downward_uv_radiation_at_the_surface', 'duct_base_height', 'eastward_gravity_wave_surface_stress', 'eastward_turbulent_surface_stress', 'evaporation', 'forecast_albedo', 'forecast_logarithm_of_surface_roughness_for_heat', 'forecast_surface_roughness', 'free_convective_velocity_over_the_oceans', 'friction_velocity', 'geopotential', 'gravity_wave_dissipation', 'high_cloud_cover', 'high_vegetation_cover', 'ice_temperature_layer_1', 'ice_temperature_layer_2', 'ice_temperature_layer_3', 'ice_temperature_layer_4', 'instantaneous_10m_wind_gust', 'instantaneous_eastward_turbulent_surface_stress', 'instantaneous_large_scale_surface_precipitation_fraction', 'instantaneous_moisture_flux', 'instantaneous_northward_turbulent_surface_stress', 'instantaneous_surface_sensible_heat_flux', 'k_index', 'lake_bottom_temperature', 'lake_cover', 'lake_depth', 'lake_ice_depth', 'lake_ice_temperature', 'lake_mix_layer_depth', 'lake_mix_layer_temperature', 'lake_shape_factor', 'lake_total_layer_temperature', 'land_sea_mask', 'large_scale_precipitation', 'large_scale_precipitation_fraction', 'large_scale_rain_rate', 'large_scale_snowfall', 'large_scale_snowfall_rate_water_equivalent', 'leaf_area_index_high_vegetation', 'leaf_area_index_low_vegetation', 'low_cloud_cover', 'low_vegetation_cover', 'maximum_2m_temperature_since_previous_post_processing', 'maximum_individual_wave_height', 'maximum_total_precipitation_rate_since_previous_post_processing', 'mean_boundary_layer_dissipation', 'mean_convective_precipitation_rate', 'mean_convective_snowfall_rate', 'mean_direction_of_total_swell', 'mean_direction_of_wind_waves', 'mean_eastward_gravity_wave_surface_stress', 'mean_eastward_turbulent_surface_stress', 'mean_evaporation_rate', 'mean_gravity_wave_dissipation', 'mean_large_scale_precipitation_fraction', 'mean_large_scale_precipitation_rate', 'mean_large_scale_snowfall_rate', 'mean_northward_gravity_wave_surface_stress', 'mean_northward_turbulent_surface_stress', 'mean_period_of_total_swell', 'mean_period_of_wind_waves', 'mean_potential_evaporation_rate', 'mean_runoff_rate', 'mean_sea_level_pressure', 'mean_snow_evaporation_rate', 'mean_snowfall_rate', 'mean_snowmelt_rate', 'mean_square_slope_of_waves', 'mean_sub_surface_runoff_rate', 'mean_surface_direct_short_wave_radiation_flux', 'mean_surface_direct_short_wave_radiation_flux_clear_sky', 'mean_surface_downward_long_wave_radiation_flux', 'mean_surface_downward_long_wave_radiation_flux_clear_sky', 'mean_surface_downward_short_wave_radiation_flux', 'mean_surface_downward_short_wave_radiation_flux_clear_sky', 'mean_surface_downward_uv_radiation_flux', 'mean_surface_latent_heat_flux', 'mean_surface_net_long_wave_radiation_flux', 'mean_surface_net_long_wave_radiation_flux_clear_sky', 'mean_surface_net_short_wave_radiation_flux', 'mean_surface_net_short_wave_radiation_flux_clear_sky', 'mean_surface_runoff_rate', 'mean_surface_sensible_heat_flux', 'mean_top_downward_short_wave_radiation_flux', 'mean_top_net_long_wave_radiation_flux', 'mean_top_net_long_wave_radiation_flux_clear_sky', 'mean_top_net_short_wave_radiation_flux', 'mean_top_net_short_wave_radiation_flux_clear_sky', 'mean_total_precipitation_rate', 'mean_vertical_gradient_of_refractivity_inside_trapping_layer', 'mean_vertically_integrated_moisture_divergence', 'mean_wave_direction', 'mean_wave_direction_of_first_swell_partition', 'mean_wave_direction_of_second_swell_partition', 'mean_wave_direction_of_third_swell_partition', 'mean_wave_period', 'mean_wave_period_based_on_first_moment', 'mean_wave_period_based_on_first_moment_for_swell', 'mean_wave_period_based_on_first_moment_for_wind_waves', 'mean_wave_period_based_on_second_moment_for_swell', 'mean_wave_period_based_on_second_moment_for_wind_waves', 'mean_wave_period_of_first_swell_partition', 'mean_wave_period_of_second_swell_partition', 'mean_wave_period_of_third_swell_partition', 'mean_zero_crossing_wave_period', 'medium_cloud_cover', 'minimum_2m_temperature_since_previous_post_processing', 'minimum_total_precipitation_rate_since_previous_post_processing', 'minimum_vertical_gradient_of_refractivity_inside_trapping_layer', 'model_bathymetry', 'near_ir_albedo_for_diffuse_radiation', 'near_ir_albedo_for_direct_radiation', 'normalized_energy_flux_into_ocean', 'normalized_energy_flux_into_waves', 'normalized_stress_into_ocean', 'northward_gravity_wave_surface_stress', 'northward_turbulent_surface_stress', 'ocean_surface_stress_equivalent_10m_neutral_wind_direction', 'ocean_surface_stress_equivalent_10m_neutral_wind_speed', 'peak_wave_period', 'period_corresponding_to_maximum_individual_wave_height', 'potential_evaporation', 'precipitation_type', 'runoff', 'sea_ice_cover', 'sea_surface_temperature', 'significant_height_of_combined_wind_waves_and_swell', 'significant_height_of_total_swell', 'significant_height_of_wind_waves', 'significant_wave_height_of_first_swell_partition', 'significant_wave_height_of_second_swell_partition', 'significant_wave_height_of_third_swell_partition', 'skin_reservoir_content', 'skin_temperature', 'slope_of_sub_gridscale_orography', 'snow_albedo', 'snow_density', 'snow_depth', 'snow_evaporation', 'snowfall', 'snowmelt', 'soil_temperature_level_1', 'soil_temperature_level_2', 'soil_temperature_level_3', 'soil_temperature_level_4', 'soil_type', 'standard_deviation_of_filtered_subgrid_orography', 'standard_deviation_of_orography', 'sub_surface_runoff', 'surface_latent_heat_flux', 'surface_net_solar_radiation', 'surface_net_solar_radiation_clear_sky', 'surface_net_thermal_radiation', 'surface_net_thermal_radiation_clear_sky', 'surface_pressure', 'surface_runoff', 'surface_sensible_heat_flux', 'surface_solar_radiation_downward_clear_sky', 'surface_solar_radiation_downwards', 'surface_thermal_radiation_downward_clear_sky', 'surface_thermal_radiation_downwards', 'temperature_of_snow_layer', 'toa_incident_solar_radiation', 'top_net_solar_radiation', 'top_net_solar_radiation_clear_sky', 'top_net_thermal_radiation', 'top_net_thermal_radiation_clear_sky', 'total_cloud_cover', 'total_column_cloud_ice_water', 'total_column_cloud_liquid_water', 'total_column_ozone', 'total_column_rain_water', 'total_column_snow_water', 'total_column_supercooled_liquid_water', 'total_column_water', 'total_column_water_vapour', 'total_precipitation', 'total_sky_direct_solar_radiation_at_surface', 'total_totals_index', 'trapping_layer_base_height', 'trapping_layer_top_height', 'type_of_high_vegetation', 'type_of_low_vegetation', 'u_component_stokes_drift', 'uv_visible_albedo_for_diffuse_radiation', 'uv_visible_albedo_for_direct_radiation', 'v_component_stokes_drift', 'vertical_integral_of_divergence_of_cloud_frozen_water_flux', 'vertical_integral_of_divergence_of_cloud_liquid_water_flux', 'vertical_integral_of_divergence_of_geopotential_flux', 'vertical_integral_of_divergence_of_kinetic_energy_flux', 'vertical_integral_of_divergence_of_mass_flux', 'vertical_integral_of_divergence_of_moisture_flux', 'vertical_integral_of_divergence_of_ozone_flux', 'vertical_integral_of_divergence_of_thermal_energy_flux', 'vertical_integral_of_divergence_of_total_energy_flux', 'vertical_integral_of_eastward_cloud_frozen_water_flux', 'vertical_integral_of_eastward_cloud_liquid_water_flux', 'vertical_integral_of_eastward_geopotential_flux', 'vertical_integral_of_eastward_heat_flux', 'vertical_integral_of_eastward_kinetic_energy_flux', 'vertical_integral_of_eastward_mass_flux', 'vertical_integral_of_eastward_ozone_flux', 'vertical_integral_of_eastward_total_energy_flux', 'vertical_integral_of_eastward_water_vapour_flux', 'vertical_integral_of_energy_conversion', 'vertical_integral_of_kinetic_energy', 'vertical_integral_of_mass_of_atmosphere', 'vertical_integral_of_mass_tendency', 'vertical_integral_of_northward_cloud_frozen_water_flux', 'vertical_integral_of_northward_cloud_liquid_water_flux', 'vertical_integral_of_northward_geopotential_flux', 'vertical_integral_of_northward_heat_flux', 'vertical_integral_of_northward_kinetic_energy_flux', 'vertical_integral_of_northward_mass_flux', 'vertical_integral_of_northward_ozone_flux', 'vertical_integral_of_northward_total_energy_flux', 'vertical_integral_of_northward_water_vapour_flux', 'vertical_integral_of_potential_and_internal_energy', 'vertical_integral_of_potential_internal_and_latent_energy', 'vertical_integral_of_temperature', 'vertical_integral_of_thermal_energy', 'vertical_integral_of_total_energy', 'vertically_integrated_moisture_divergence', 'volumetric_soil_water_layer_1', 'volumetric_soil_water_layer_2', 'volumetric_soil_water_layer_3', 'volumetric_soil_water_layer_4', 'wave_spectral_directional_width', 'wave_spectral_directional_width_for_swell', 'wave_spectral_directional_width_for_wind_waves', 'wave_spectral_kurtosis', 'wave_spectral_peakedness', 'wave_spectral_skewness', 'zero_degree_level'], FieldInfo(annotation=NoneType, required=True)],\n", " 'year': typing.Annotated[typing.Literal['1940', '1941', '1942', '1943', '1944', '1945', '1946', '1947', '1948', '1949', '1950', '1951', '1952', '1953', '1954', '1955', '1956', '1957', '1958', '1959', '1960', '1961', '1962', '1963', '1964', '1965', '1966', '1967', '1968', '1969', '1970', '1971', '1972', '1973', '1974', '1975', '1976', '1977', '1978', '1979', '1980', '1981', '1982', '1983', '1984', '1985', '1986', '1987', '1988', '1989', '1990', '1991', '1992', '1993', '1994', '1995', '1996', '1997', '1998', '1999', '2000', '2001', '2002', '2003', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017', '2018', '2019', '2020', '2021', '2022', '2023', '2024'], FieldInfo(annotation=NoneType, required=True)],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)],\n", " 'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='ERA5_SL')],\n", " 'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)]}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables(\n", " productType=\"ERA5_SL\", \n", " provider=\"cop_cds\", \n", " month=\"02\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If default values are defined for a parameter in the configuration, the values of the other parameters will be filtered by this default value if constraints are available. If all possible configurations shall be shown, the default value can be removed by setting it to empty. The example below shows on request in which all default values are used and one where the default for api_product_type is removed." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'variable': typing.Annotated[typing.Literal['glacier_area'], FieldInfo(annotation=NoneType, required=False, default='glacier_area')],\n", " 'api_product_type': typing.Annotated[typing.Literal['gridded', 'hypsometry', 'vector'], FieldInfo(annotation=NoneType, required=False, default='gridded')],\n", " 'version': typing.Annotated[typing.Literal['6_0'], FieldInfo(annotation=NoneType, required=False, default='6_0')],\n", " 'format': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='zip')],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)],\n", " 'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='GLACIERS_DIST_RANDOLPH')],\n", " 'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)]}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables(\n", " productType=\"GLACIERS_DIST_RANDOLPH\", \n", " provider=\"cop_cds\"\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'variable': typing.Annotated[typing.Literal['glacier_area'], FieldInfo(annotation=NoneType, required=False, default='glacier_area')],\n", " 'api_product_type': typing.Annotated[typing.Literal['gridded', 'hypsometry', 'vector'], FieldInfo(annotation=NoneType, required=True)],\n", " 'version': typing.Annotated[typing.Literal['5_0', '6_0'], FieldInfo(annotation=NoneType, required=False, default='6_0')],\n", " 'format': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='zip')],\n", " 'end': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='completionTimeFromAscendingNode', alias_priority=2)],\n", " 'geom': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='geometry', alias_priority=2)],\n", " 'productType': typing.Annotated[str, FieldInfo(annotation=NoneType, required=False, default='GLACIERS_DIST_RANDOLPH')],\n", " 'id': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False)],\n", " 'start': typing.Annotated[typing.Optional[str], FieldInfo(annotation=NoneType, required=False, alias='startTimeFromAscendingNode', alias_priority=2)]}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dag.list_queryables(\n", " productType=\"GLACIERS_DIST_RANDOLPH\", \n", " provider=\"cop_cds\", \n", " api_product_type=None\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }