Merge pull request #42 from HeidiSteen/master

Edits to readme files and image-processing notebook

Author: HeidiSteen

Image-Processing/BFR_Sample_Rest.ipynb

@@ -7,7 +7,11 @@
     "# Azure Cognitive Search sample \n",
     "## Passing Images as Binary File References\n",
     "\n",
-    "Cognitive Search skillsets that need to pass images to custom skills use a binary file reference to serialize the images to pass them to and from skills. This sample demonstrates an example of how skills can be configured to accept an image as an input from the skillset and return images as outputs to the skillset. This example does nothing more than segment an image based on the layout from OCR. The sole purpose of this sample is to demonstrate how you pass images to skills and how skills can return images.\n",
+    "Skillsets that pass images to custom skills use a binary file reference to serialize the images before passing them to other skills. This sample demonstrates how skills can be configured to accept image inputs and return image outputs. \n",
+    "\n",
+    "While the other steps in this skillset, such as OCR and redaction, have relevance, the key takeaway is configuring and passing binary file references. The custom skill does the heavy lifting. Each input record contains an image that is serialized as a `Base64` encoded string. The input also contains the layout text of image, as returned from the OCR skill. Upon receiving the input, the custom skill segments the image into smaller images based on the coordinates of the layout text. It then returns a list of images, each `Base64` encoded, back to the skillset. While this is not a particularly realistic exercise, it demonstrates techniques that could be leverage in more interesting ways, such as in a [Custom Vision](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) skill that performs useful inferences on your images.\n",
+    "\n",
+    "For more information about the skills used in this example, see [OCR skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-ocr), [PII skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-pii-detection), and [custom skills](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api).\n",
     "\n"
    ]
   },
@@ -17,34 +21,37 @@
    "source": [
     "### Prerequisites \n",
     "\n",
-    "Provision the required services:\n",
-    "1. [Azure Cognitive Search](https://docs.microsoft.com/azure/search/search-create-service-portal)\n",
-    "2. [Azure Functions](https://docs.microsoft.com/azure/azure-functions/) used for hosting an API endpoint.\n",
-    "3. [Storage Account](https://docs.microsoft.com/azure/storage/blobs/)\n"
+    "+ [Azure subscription](https://Azure.Microsoft.com/subscription/free)\n",
+    "+ [Azure Cognitive Search service](https://docs.microsoft.com/azure/search/search-create-service-portal) (get the full service endpoint and an admin API key)\n",
+    "+ [Azure Blob storage service](https://docs.microsoft.com/azure/storage/common/storage-account-create) (get the connection string)\n",
+    "+ [Python 3.6+](https://www.python.org/downloads/)\n",
+    "+ [Jupyter Notebook](https://jupyter.org/install)\n",
+    "+ [Visual Studio Code](https://code.visualstudio.com/download) with the [Azure Functions extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions) and the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)\n",
+    "\n",
+    "If you adapt this exercise to include more image files, add [Azure Cognitive Services](https://docs.microsoft.com/azure/cognitive-services/cognitive-services-apis-create-account)."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Deploy the Azure functions app \n",
-    "The ```SplitImage``` folder contains an Azure function that will accept an input in the [custom skill format](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api#skill-inputs). \n",
-    "Each input record contains an image that is serialized as a ```Base64``` encoded string and the layout text returned from the OCR skill.\n",
-    "The skill then segments the image into smaller images based on the coordinates of the layout text. It then returns a list of images, each ```Base64``` encoded back to the skillset. While this is not very useful, you could build a [Custom Vision](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) skill to perform a useful inference on your images.\n",
+    "### Configure inputs\n",
     "\n",
-    "Follow the [Azure Functions tutorial](https://docs.microsoft.com/azure/developer/python/tutorial-vs-code-serverless-python-05) to deploy the function. Once the deployment completes, navigate to the function app in the portal, select the function (SplitImage) and click the Get Function Url button. Save the function url as we will use it in the next step."
+    "Follow the instructions in the [readme](https://github.com/Azure-Samples/azure-search-python-samples/blob/master/Image-Processing/README.md) to set up the inputs used by the indexer, data source, and skillset.\n",
+    "\n",
+    "Besides connection information, you will need a blob container for the sample JPEG file, and a function app that provides the code used in the custom skill. All the necessary files are provided. The `SplitImage` folder contains an Azure function that will accept an input in the [custom skill format](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api#skill-inputs). "
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Create the enrichment pipeline\n",
-    "In the next few steps we will configure the Cognitive Search enrichment pipeline with the following steps:\n",
-    "1. Create a blob storage data source. Ensure you have a blob storage container with at least one file containing images.\n",
-    "2. Create a skillset to enrich the documents in the data source\n",
-    "3. Create an index\n",
-    "4. Create an indexer to move documents from the data source to the index while invoking the skillset\n"
+    "In the next few steps, configure the Cognitive Search enrichment pipeline, creating these objects on your search service:\n",
+    "1. Create an indexer data source. The data source references a blob storage container with at least one image file.\n",
+    "2. Create a skillset that performs image analysis. The skillset references a Cognitive Services account, a custom function app, and a knowledge store.\n",
+    "3. Create a search index.\n",
+    "4. Create an indexer to move documents from the data source to the index while invoking the skillset.\n"
    ]
   },
   {
@@ -59,36 +66,39 @@
     "import json\n",
     "import requests\n",
     "\n",
-    "# Configure all required variables for Cognitive Search. Replace each with the credentials from your accounts.\n",
+    "# Configure all required variables for this exerences. Replace each with the credentials from your accounts.\n",
     "\n",
-    "# Replace with Search Service name, API key, and endpoint from the Azure portal.\n",
-    "search_service = \"\" # In the format \"https://searchservicename.search.windows.net\"\n",
-    "api_key = 'your search service API key'\n",
+    "# Replace with a full search service endpoint the format \"https://searchservicename.search.windows.net\"\n",
+    "# Paste in an admin API key. Both values can be obtained from the Azure portal.\n",
+    "search_service = \"https://<YOUR-SEARCH-SERVICE-NAME>.search.windows.net\"\n",
+    "api_key = '<YOUR-SEARCH-ADMIN-API-KEY>'\n",
     "\n",
     "# Leave the API version and content_type as they are listed here.\n",
     "api_version = '2020-06-30'\n",
     "content_type = 'application/json'\n",
     "\n",
-    "# Replace with a Cognitive Services all in one key.\n",
-    "cog_svcs_key = '' #Required only if processing more than 20 documents\n",
-    "cog_svcs_acct = 'your cog services account name'\n",
+    "# Replace with a Cognitive Services account name and all-in-one key.\n",
+    "# Required only if processing more than 20 documents\n",
+    "cog_svcs_key = '' \n",
+    "cog_svcs_acct = '' \n",
     "\n",
-    "#Connection string to the storage account. This will be used for the datasource, knowledge store and cache\n",
-    "STORAGECONNSTRING = \"DefaultEndpointsProtocol=https;AccountName=<Storage Acct>;AccountKey=<KEY>;EndpointSuffix=core.windows.net\"\n",
-    "# The container with your files containing images\n",
-    "datasource_container = 'bfrsample' # Replace with the container containging your files\n",
-    "# This sample assumes you will use the same storage account for the datasource, knowledge store and indexer cache. The knowledge store will contain the projected images\n",
-    "know_store_cache = STORAGECONNSTRING\n",
-    "# Container where the sliced images will be projected to\n",
+    "# Your Azure Storage account will be used for the datasource input and knowledge store output\n",
+    "# Replace with a connection string to your Azure Storage account. \n",
+    "STORAGECONNSTRING = \"DefaultEndpointsProtocol=https;AccountName=<YOUR-STORAGE-ACCOUNT>;AccountKey=<YOUR-ACCOUNT-KEY>;EndpointSuffix=core.windows.net\"\n",
+    "# Replace with the blob container containing your image file\n",
+    "datasource_container = 'bfr-sample' \n",
+    "# Container where the sliced images will be projected to. Use the value provided below.\n",
     "know_store_container = \"obfuscated\"\n",
-    "skill_uri = \"https://<skillname>.azurewebsites.net/api/SplitImage?code=CODE\""
+    "\n",
+    "# Replace with the Function HTTP URL of the app deployed to Azure Function\n",
+    "skill_uri = \"<YOUR-FUNCTION-APP-URL\""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Create a helper function to invoke the Cognitive Search REST API"
+    "Create a helper function to invoke the Cognitive Search REST APIs. "
    ]
   },
   {
@@ -153,7 +163,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the skillset"
+    "#### Create the skillset\n",
+    "\n",
+    "Binary image references are passed as inputs and outputs, starting with \"/document/normalized_images/*\" in the OCR skill. OCR output is text and layout. Only the text component is passed to PIIDectection for analysis and redactive formatting. In the custom skill, the image is sliced into component parts (text and layout from OCR, and PII entity created in the PIIDetection step).\n",
+    "\n",
+    "Besides skills, a skillset also specifies the knowledge store projections that shape the final output in Blob storage."
    ]
   },
   {
@@ -344,7 +358,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the index"
+    "#### Create the index\n",
+    "\n",
+    "A search index isn't used in this exercise, but because it's an indexer requirement, you'll create one anyway. You can use Search Explorer in the Azure portal to query the index on your own. It will contain text extracted from the image."
    ]
   },
   {
@@ -515,7 +531,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the indexer"
+    "#### Create the indexer\n",
+    "\n",
+    "This step creates the index (you'll run it in a separate step). At run time, the indexer connects to the data source, invokes the skillset, and outputs results. This indexer is scheduled to run every two hours. "
    ]
   },
   {
@@ -561,11 +579,7 @@
     "          \"sourceFieldName\": \"/document/normalized_images/*/text\",\n",
     "          \"targetFieldName\": \"image_text\"\n",
     "        }\n",
-    "    ],\n",
-    "    \"cache\": {\n",
-    "        \"enableReprocessing\": True,\n",
-    "        \"storageConnectionString\": f'{know_store_cache}'\n",
-    "    }\n",
+    "    ]\n",
     "}\n",
     "r = requests.post(construct_Url(search_service, \"indexers\", None, None, api_version), data=json.dumps(indexer_def), headers=headers)\n",
     "print(r)\n",
@@ -577,7 +591,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Run the indexer"
+    "#### Run the indexer\n",
+    "\n",
+    "This step executes the indexer you just created. It will take several minutes to process."
    ]
   },
   {
@@ -592,12 +608,33 @@
     "#print(json.dumps(res, indent=2))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Check status\n",
+    "\n",
+    "The final step in this exercise is to view results. Before doing so, make sure the lastResult status message indicates \"success\", which means that the indexer completed its work successfully, and the revised image now exists in blob storage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "r = requests.get(construct_Url(search_service, \"indexers\", indexername, \"status\", api_version), data=None,  headers=headers)\n",
+    "print(r)\n",
+    "res = r.json()\n",
+    "print(res[\"lastResult\"])"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### View Results\n",
-    "The following cell downloads the image so that you can verify skillset success."
+    "The following cell downloads the output image so that you can verify skillset success. If you get an error, check the indexer status to make sure the indexer is finished and that there were no errors."
    ]
   },
   {
@@ -622,16 +659,25 @@
     "    if(count == 3):\n",
     "        break\n",
     "\n",
-    "Image(filename='image2.jpg') "
+    "Image(filename='image0.jpg') "
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Next Steps\n",
-    "You now know how to pass images into skills and even return images to the skillset. As a next step, you can start from scratch and build a [custom AML Skill](https://docs.microsoft.com/en-us/azure/search/cognitive-search-aml-skill) to perform inferences on images or use the Custom Vision service to build a skill. The power skills github repository has a [sample custom vision skill](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) to help you get started."
+    "In this exercise, you learned how to pass images into skills and return the modified images to the skillset for further processing. \n",
+    "\n",
+    "As a next step, you can start from scratch and build a [custom AML Skill](https://docs.microsoft.com/azure/search/cognitive-search-aml-skill) to perform inferences on images, or use the Custom Vision service to build a skill. The power skills github repository has a [sample custom vision skill](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) to help you get started."
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -650,7 +696,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.7.3"
   }
  },
  "nbformat": 4,