How to integrate Azure Cognitive Search with Kontent.ai
Visitors are looking for your content. How can you help them find exactly what they’re looking for?
Christopher JenningsPublished on Mar 30, 2020
One answer is by adding search. Let’s look at how you can integrate one of the more common ones: Azure Cognitive Search. In this article, we’ll cover the basics of creating, updating, and querying an index. The index will contain articles from the Dancing Goat sample project.
Getting started with Azure Cognitive Search
To get started, you first need to provision an Azure Cognitive Search service to use. You can do this in the Azure portal. Microsoft has a free tier you can use to get started at no cost. After provisioning the service, you’ll need to make a note of the service name and an admin API key. The screenshot below shows you where you can find the admin keys in the Azure Portal.
Creating the Initial Index
Spinning up the project
To build this simple demo, start by creating a new ASP.NET Core Web Application. The code samples in this tutorial assume you use “Kontent-Azure-Search-Demo” for the name. Choose the “Web Application (Model-View-Controller)” template as seen below:
Next, you’ll need to add some settings values. You can add them to your
appsettings.json. Some of the values are sensitive, so you might want to consider using “User Secrets” instead:
In either case, you need to add some key-value pairs to either of those JSON files. You’ll need to update them based on your values.
SearchServiceName is the name of your Azure Cognitive Search Service. The
SearchServiceAdminApiKey is the admin key mentioned earlier.
SearchServiceIndexName is the name you want to use for the index. The other two are for Kontent.ai.
KontentProjectID is the project ID for the project you’re searching in. The code you’ll write in the future expects the Dancing Goat sample project. The
KontentWebhookSecret is something we’ll use later. You can leave this either blank or with a dummy value for now.
Next, you need to add a couple of NuGet packages. Find and add both the Kontent.Ai.Delivery and Microsoft.Azure.Search packages. These add the Kontent.ai SDK and Azure Cognitive Search SDK to our project.
Adding a model
Key identifies the property that Azure search uses to identify each indexed item. It’s found in the
IsSearchable indicates if a property contains data that you want users to be able to search. It’s found in the
IsRetrievable indicates if a given property is returnable in a search result. It’s found in the Microsoft.Azure.Search namespace. This defaults to true, so you only need to add it if you don’t want to return the property with search results (for example, the body of an article).
Analyzer(AnalyzerName.AsString.EnMicrosoft) tells Azure which analyzer to use. It’s found in the Microsoft.Azure.Search.Models namespace. The
EnMicrosoft analyzer understands English words better than the default analyzer. As a result, it can provide better search results.
For the demo, you can add the following code to a new
Article.cs file in the “Models” folder:
Building the logic
Next, let’s create a couple of helpers to handle the main operations. We need to get our articles out of Kontent.ai, manage our search indexes in Azure, and perform searches.
First, let’s create a “Helpers” folder in our project root. In that folder, create a new class file called
KontentHelper.cs. This helper class will have the methods we need to get articles from Kontent.ai. In this new file, add the following:
The constructor accepts application configuration and uses it to create a delivery client. The methods use this delivery client to get articles from Kontent.
GetArticleByUrlPattern method gets the first article that matches the passed URL pattern. This is for our detail page.
GetArticlesForSearch method gets all articles from the delivery API. Its overload gets all the articles with IDs matching the provided IDs. In both cases, they transform the retrieved results into the article class.
GetArticle method maps an article content item to the article class. This is a private method used by the
GetArticlesForSearch method and its overload.
Next, create a new class file called
SearchHelper.cs in the “Helpers” folder. This helper class will have the methods we need to manage the search index in Azure as well as query it. In this new file, add the following:
The constructor prepares a few reusable resources. It saves the index name, creates a search service client, and saves a copy of the search index client.
AddToIndex<T> method takes care of adding articles to the Azure Search index. It takes the documents, creates a batch of upload actions for them, and sends that to Azure for processing. It expects a decorated document class like the
CreateIndex<T> method creates an index definition based on the passed in class. It expects a decorated class like the
Article class. It then creates the index in Azure.
DeleteIndexIfExists method determines if the configured index exists in Azure already. If it does exist, it deletes it. Otherwise, it does nothing.
QueryIndex<T> method expects search text. It passes this text along to Azure to perform a search and return the results. It returns only the list of documents from the results.
RemoveFromIndex<T> method handles removing documents from the Azure index. It takes the documents, creates a batch of delete actions for them, and sends that to Azure for processing. The documents only need to provide the property that acts as the document key to work. It expects a decorated document class like the
CreateSearchServiceClient method is for the constructor. It handles creating a search service client for Azure Cognitive Search. It reads the configured search service name and admin API key. It then creates a search service client using Azure’s SDK with them.
Creating a basic UI
Search Controller and View
Next, you need to add a controller to manage our search operations. Start by adding a new empty MVC controller named SearchController. Then add the following:
The constructor uses dependency injection to expose our configuration values. It also initializes an instance of our
SearchHelper class so we can use it in our controller actions.
The first action,
Create, is to create (or re-create) the index we’re working with. When invoked, this calls the SearchHelper class’s delete and create search index methods. This makes sure that the index is starting with a completely clean slate. It also writes a simple status message to the
ViewData so we can update the page. Finally, it returns the index view as the code will share a single view for all the search actions.
The second method,
Index, returns the basic index view for the search controller.
The last method,
Initialize, gets all the articles from Kontent.ai and adds them to the index. Its primary purpose is to load the initial data into the index.
Next up, let’s create the view for our controller. Create a new folder called “Search” In the “Views” folder and add a new
Index.cshtml file there with the following:
This will give us a simple page with buttons for the create and initialize actions and a simple search box. It also serves as our results page. Any returned results show their title, linked the article detail page, and the summary. The detail link won’t work yet since we haven’t added the controller and view for that yet. Let’s do that now.
Article Controller and View
To give the results view somewhere to link to, let’s add a bare-bones detail view for articles. Back in the “Controllers” folder, add a new empty MVC controller. Name this one
Article and add the following:
The constructor accepts the configuration and saves an initialized Kontent.ai helper.
Detail method takes the passed URL pattern, gets its content item, and passes it to the view. Let’s create this view now.
Create a new folder called “Article” In the “Views” folder. Add a new
Detail.cshtml file there with the following:
This takes the passed in content item and displays the teaser image (if there is one), the title, and the body copy. Since this is only a quick example, it doesn’t handle a lot. For example, it doesn’t resolve any item links, components, or items inside the rich text field.
Trying it out
That was a lot of code, but now we’ve got everything we need to give it a try. You can hit
F5 or start debugging the project to launch it in a browser. Once the welcome page has loaded, add
/search to the URL to visit the search page you built. It should look something like this:
The first thing we need to do is click the “Create index” button to create the index. The page will reload, and you should see the “Index deleted/created.” message. Next, click the “Initialize Index” button to load the index. After it loads the articles from your Dancing Goat into the index, the page will reload. After the reload, you’ll see an “Index initialized.” message. Finally, you’re ready to try an actual search. If you search for “coffee”, you should get something like:
If you click on one of the results, you’ll see a details page like this:
Keeping the Index Up to Date
Great, so it works, how do we keep it updated? Kontent.ai allows you to configure webhooks to react to certain events in a Kontent.ai project.
Getting a public URL for our local code
First, let’s get ourselves set up to receive webhooks from Kontent.ai on our local machine. If you don’t have a way to handle this already, we’ll cover using ngrok as an easy, free option. Follow their instructions to get started. When you’re ready to fire it up, run the command below to point it to your local development server. You’ll need to update localhost:44345 to wherever your project is running.
After running that command, you’ll see something like this:
You’ll want to use one of the URLs near at the end of the list as the target for your Kontent.ai webhook. In the screenshot above, I’d use “ https://a241e689.ngrok.io”.
Configuring a Webhook in Kontent.ai
Next, let’s configure the webhook in Kontent.ai. Go to “Settings”, then choose “Webhooks”. Click the “Create New Webhook” button and enter a webhook name. Next, make sure to enter the public URL to your endpoint and add “/webhook” to the end of it. Make sure to enable the “publish” and “unpublish” delivery API triggers. Also, clear the rest of the triggers. It should look like this:
Supporting Webhooks in our code
First, we need to revisit the
appsettings.json (or user
secrets.json). Copy the secret from the above webhook configuration screen. Paste it to the
KontentWebhookSecret setting value.
Models and Webhook Helper
In the “Models” folder, add a new file called
KontentWebhookModels.cs. In it, add the following:
This is a collection of models that match the JSON that Kontent.ai sends in a webhook.
Next, let’s add a new helper class in the “Helpers” folder named
KontentWebhookHelper.cs. This will contain the following:
The constructor accepts the application settings and saves the webhook secret.
ValidateAndProcessWebhook method takes the x-kc-signature header and the full webhook body. It validates that the request is from Kontent.ai. Then it makes sure there is something actionable for the search index. It checks several things to determine if there’s something actionable. First, it checks to make sure that the webhook fired for a content item variant. Next, it checks that the operation was either a publish or unpublish event. The final check determines if any of the event’s items are articles in the en-US language. If any of these checks fail, the method returns a false validation result with an error message. If the checks pass, a true validation result it returns the articles contained in the webhook.
GenerateHash method implements the hash algorithm documented in the documentation.
Search Controller Update
Switching over to our search controller, we need to add an action to respond to the “/webhook” URL we added to Kontent.ai.
We’ve added a new private field for the KontentWebhookHelper. The constructor initializes it. We’ve also added a new controller action.
ProcessWebhook action responds to an HTTP post to the “/webhook” URL. It parses the body manually because of the need to use the raw body for signature verification.
It passes the webhook body and signature to the
ValidateAndProcessWebhook method. If the result is not valid, it returns the error message passed from the method. If it’s valid, it will update or remove the items based on whether it was a “publish” or “unpublish” operation.
When it’s a publish operation, it calls the Kontent.ai helper’s
GetArticlesForSearch method. It passes it the IDs of the articles from the webhook to get their new data. Once it has the item data, it calls the search helper’s
AddToIndex method to update the index.
When it’s an unpublish operation, it creates an IEnumerable of the
Article class. It creates this list using only the IDs of the articles sent in the webhook. Finally, it calls the search helper’s
AddToIndex method to update the index.
That’s everything we need to respond to Kontent.ai webhooks and keep the index up to date.
Source Code and Next Steps
In this demo, we’ve seen all the basics. We initialized, searched, and updated an Azure search index with content from Kontent.ai.
This is far from a comprehensive implementation of Azure search. There are many features such as faceting and sorting that we haven’t touched on here. We also only covered a single content type in one language. Implementing any of these things will be great next steps if you’re looking for a little more.