Self-labeling of images in Sharepoint with Flow and Cognitive Services

Contents

You have 15 minutes? Good, You now have everything you need to be able to integrate Microsoft's cognitive image recognition service into your Sharepoint libraries and start auto-tagging your images according to their content for free and without the need for code..

How? Once again the key is in the FLOW

Currently the workflow creation and automation service Microsoft Flow Not only does it have a wide range of connections to manage items in Sharepoint libraries and lists, it also has multiple connectors for Microsoft's Cognitive Services, including service Computer Vision API.

dc65b677-21af-4e34-a0cc-eed6d9ea80c2-3603829

This service that has a free version up to 5000 requests / my, will allow us to extract the data from the image and among its applications we have the ability to know keywords or “labels” that provide us with relevant information about it, and then save them as metadata associated with said image in Sharepoint.

STEPS TO FOLLOW:

  1. Configure a Sharepoint library where to store the images
  2. Configure the Azure Computer Vision API resource
  3. Configure a flow that is launched when a new image is stored and sends its content to our image recognition API
  4. ADDITIONAL CHARACTERISTICS

FIRST STEP: SETTING UP OUR LIBRARY

In order to use the image recognition capabilities we will first have to establish where these will be stored and what will be the field where we add the labeling, for this we will use a Sharepoint library and create a column that we will call AutoTags.

ec8494a5-2e46-4bfd-be5b-73d37d3c0929-9340040

SECOND STEP: CONFIGURE OUR COGNITIVE VISION API SERVICE

To register the cognitive image recognition service, just go to the Azure portal and add a new resource indicating its name, the subscription where you will stay, the deployment location and the resource group to which it will belong.

55c6ce61-3c2f-4f78-a176-b9c7b95f6708-1206771

Before selecting where we will implement this service, it is necessary to verify that it is available in that region.

fe8bce8a-cb80-47d5-8a59-58d007ece2fc-5088363

Once the service is registered, we will need to point the name of the connection to the API, the Account Key and the end point to access this service.

eee1e6f7-457f-4752-abaa-0ca9ee6962e9-2428686

7a1662c0-f1aa-4702-86a3-adad63fea25b-9847858

THIRD STEP: SET THE FLOW

We access Flow and create a new flow that will be launched when a new file is created in a selected Sharepoint site and library:

4ab58a45-02d2-47fd-a199-8d4344121021-7598249

Once this is done, We will retrieve the data of the image that has just been stored in the Sharepoint to send it to our image recognition API.

ae79a779-e5ad-46b9-9528-457020a69019-7685792

At this point we will connect the Flow to our API with the capacity Label Image indicando los parameters que señalamos en el paso anterior.

4d0b713c-cfe7-45e3-91b4-b965ffa3fbe1-4419030

Then we will indicate that the shipment will be directly the content and not from the url and we will indicate that this is the output of the previous step.

f0f9807e-9bfc-46ba-9ad2-cd217c93eae9-2443172

Ahora solo será necesario adaptar el JSON del resultado a un arreglo de caracteres separados por coma, for this we will use the Select and Join functions.

86ecdf8e-4e2a-406e-97a2-009641603657-9841457

Finally, we will just have to add this label array to the column we created in step 1 for the image we are dealing with.

35ed9f91-0afe-46fd-81d8-098b876ef800-7488626

AND READY!

We save the Flow and proceed to run it to check its operation:

ca92a65f-7ce5-4058-b783-d09d20b1b719-3995324

Once the images are uploaded, we check that the Flow runs correctly.

81763a22-bbb7-4856-b22e-9fd89bd0b50f-1166178

We can see the JSON that our image recognition service returns.

b2e2eb3c-eb85-4cbd-9e53-fb8305863d0c-1424103

And finally the result in the AutoTags column

44a95179-0800-4a01-ae95-c2aaf6117b14-5670304

PASO 4: THIS IS NOT EVERYTHING, THERE IS MUCH MORE

As we have seen, the multiple abilities that this cognitive service has, added to the ease of connection with other applications thanks to Microsoft Flow, they allow us to play with multiple settings like adding tags only when they pass a specific success percentage, generating another stream that reads this tagged field and matches other metadata for more efficient tagging or even concatenates API requests to get more data from the photo, as the description.

We add the function to Flow describe the picture
(This flow is an example, in this case the Describe Image function also includes the labels as output, so it should not be concatenated with the Image Label function) pointing to a new multiline column that we have created in our library earlier.

e63048a2-61db-4417-b4ca-64f44fcc02d3-2167333

Now our images are self-labeled and also obtain a description in a fully automated way in our Sharepoint including the recognition of famous characters and brands, as in this case Keanu Reeves.

460fa498-3f8b-4a23-bc7a-81b9c2bfa369-4601056

554b60c8-bb6b-43cc-9440-68c3b026fea1-7391903

Disclaimer: For the elaboration of this tutorial, no animals related to Keanu Reeves or his alter ego John Wick have been abused.


Subscribe to our Newsletter

We will not send you SPAM mail. We hate it as much as you.