How to integrate Azure Cognitive Search with Kontent.ai
Visitors are looking for your content. How can you help them find exactly what they’re looking for?
One answer is by adding search. Let’s look at how you can integrate one of the more common ones: Azure Cognitive Search. In this article, we’ll cover the basics of creating, updating, and querying an index. The index will contain articles from the Dancing Goat sample project.
Getting started with Azure Cognitive Search
To get started, you first need to provision an Azure Cognitive Search service to use. You can do this in the Azure portal. Microsoft has a free tier you can use to get started at no cost. After provisioning the service, you’ll need to make a note of the service name and an admin API key. The screenshot below shows you where you can find the admin keys in the Azure Portal.
Creating the Initial Index
Spinning up the project
To build this simple demo, start by creating a new ASP.NET Core Web Application. The code samples in this tutorial assume you use “Kontent-Azure-Search-Demo” for the name. Choose the “Web Application (Model-View-Controller)” template as seen below:
Next, you’ll need to add some settings values. You can add them to your appsettings.json. Some of the values are sensitive, so you might want to consider using “User Secrets” instead:
In either case, you need to add some key-value pairs to either of those JSON files. You’ll need to update them based on your values. SearchServiceName is the name of your Azure Cognitive Search Service. The SearchServiceAdminApiKey is the admin key mentioned earlier. SearchServiceIndexName is the name you want to use for the index. The other two are for Kontent.ai. KontentProjectID is the project ID for the project you’re searching in. The code you’ll write in the future expects the Dancing Goat sample project. The KontentWebhookSecret is something we’ll use later. You can leave this either blank or with a dummy value for now.
{
"SearchServiceName": "your-azure-search-service-name",
"SearchServiceAdminApiKey": "YourAdminApiKey",
"SearchIndexName": "desired-index-name",
"KontentProjectID": "kontent-project-id",
"KontentWebhookSecret": "WebhookSecretValue"
}Next, you need to add a couple of NuGet packages. Find and add both the Kontent.Ai.Delivery and Microsoft.Azure.Search packages. These add the Kontent.ai SDK and Azure Cognitive Search SDK to our project.
Adding a model
Key identifies the property that Azure search uses to identify each indexed item. It’s found in the System.ComponentModel.DataAnnotations namespace.
IsSearchable indicates if a property contains data that you want users to be able to search. It’s found in the Microsoft.Azure.Search namespace.
IsRetrievable indicates if a given property is returnable in a search result. It’s found in the Microsoft.Azure.Search namespace. This defaults to true, so you only need to add it if you don’t want to return the property with search results (for example, the body of an article).
Analyzer(AnalyzerName.AsString.EnMicrosoft) tells Azure which analyzer to use. It’s found in the Microsoft.Azure.Search.Models namespace. The EnMicrosoft analyzer understands English words better than the default analyzer. As a result, it can provide better search results.
For the demo, you can add the following code to a new Article.cs file in the “Models” folder:
using Microsoft.Azure.Search;
using Microsoft.Azure.Search.Models;
using System;
using System.ComponentModel.DataAnnotations;
namespace Kontent_Azure_Search_Demo.Models
{
public class Article
{
[IsSearchable]
[Analyzer(AnalyzerName.AsString.EnMicrosoft), IsRetrievable(false)]
public string Body { get; set; }
[Key]
public string ID { get; set; }
public DateTime? PostDate { get; set; }
[IsSearchable]
[Analyzer(AnalyzerName.AsString.EnMicrosoft)]
public string Summary { get; set; }
[IsSearchable]
[Analyzer(AnalyzerName.AsString.EnMicrosoft)]
public string Title { get; set; }
public string UrlPattern { get; set; }
}
}
Building the logic
Next, let’s create a couple of helpers to handle the main operations. We need to get our articles out of Kontent.ai, manage our search indexes in Azure, and perform searches.
Kontent.ai Helper
First, let’s create a “Helpers” folder in our project root. In that folder, create a new class file called KontentHelper.cs. This helper class will have the methods we need to get articles from Kontent.ai. In this new file, add the following:
using Kontent.Ai.Delivery;
using Kontent_Azure_Search_Demo.Models;
using Microsoft.Extensions.Configuration;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
namespace Kontent_Azure_Search_Demo.Helpers
{
public class KontentHelper
{
private readonly IDeliveryClient deliveryClient;
public KontentHelper(IConfiguration configuration)
{
var projectId = configuration["KontentProjectID"];
this.deliveryClient = DeliveryClientBuilder.WithProjectId(projectId).Build();
}
public async Task<ContentItem> GetArticleByUrlPattern(string urlPattern)
{
var parameters = new List<IQueryParameter>
{
new EqualsFilter("system.type", "article"),
new EqualsFilter("elements.url_pattern", urlPattern),
new LimitParameter(1)
};
var response = await deliveryClient.GetItemsAsync(parameters);
return response.Items.FirstOrDefault();
}
public async Task<IEnumerable<Article>> GetArticlesForSearch()
{
var parameters = new List<IQueryParameter>
{
new EqualsFilter("system.type", "article"),
new OrderParameter("elements.post_date", SortOrder.Descending)
};
var response = await deliveryClient.GetItemsAsync(parameters);
return response.Items.Select(GetArticle);
}
public async Task<IEnumerable<Article>> GetArticlesForSearch(IEnumerable<string> ids)
{
var parameters = new List<IQueryParameter>
{
new EqualsFilter("system.type", "article"),
new InFilter("system.id", string.Join(',',ids)),
new LimitParameter(1)
};
var response = await deliveryClient.GetItemsAsync(parameters);
return response.Items.Select(GetArticle);
}
private Article GetArticle(ContentItem item)
{
if (item == null)
{
return new Article();
}
return new Article
{
Body = item.GetString("body_copy"),
ID = item.System.Id,
PostDate = item.GetDateTime("post_date"),
Summary = item.GetString("summary"),
Title = item.GetString("title"),
UrlPattern = item.GetString("url_pattern"),
};
}
}
}
The constructor accepts application configuration and uses it to create a delivery client. The methods use this delivery client to get articles from Kontent.
The GetArticleByUrlPattern method gets the first article that matches the passed URL pattern. This is for our detail page.
The GetArticlesForSearch method gets all articles from the delivery API. Its overload gets all the articles with IDs matching the provided IDs. In both cases, they transform the retrieved results into the article class.
The GetArticle method maps an article content item to the article class. This is a private method used by the GetArticlesForSearch method and its overload.
Search Helper
Next, create a new class file called SearchHelper.cs in the “Helpers” folder. This helper class will have the methods we need to manage the search index in Azure as well as query it. In this new file, add the following:
using Microsoft.Azure.Search;
using Microsoft.Azure.Search.Models;
using Microsoft.Extensions.Configuration;
using System.Collections.Generic;
using System.Linq;
using Index = Microsoft.Azure.Search.Models.Index;
namespace Kontent_Azure_Search_Demo.Helpers
{
public class SearchHelper
{
private readonly string indexName;
private readonly ISearchIndexClient searchIndexClient;
private readonly SearchServiceClient searchServiceClient;
public SearchHelper(IConfiguration configuration)
{
indexName = configuration["SearchIndexName"];
searchServiceClient = CreateSearchServiceClient(configuration);
searchIndexClient = searchServiceClient.Indexes.GetClient(indexName);
}
public void AddToIndex<T>(IEnumerable<T> documents)
{
var actions = documents.Select(a => IndexAction.Upload(a));
var batch = IndexBatch.New(actions);
searchIndexClient.Documents.Index(batch);
}
public void CreateIndex<T>()
{
var definition = new Index()
{
Name = indexName,
Fields = FieldBuilder.BuildForType<T>()
};
searchServiceClient.Indexes.Create(definition);
}
public void DeleteIndexIfExists()
{
if (searchServiceClient.Indexes.Exists(indexName))
{
searchServiceClient.Indexes.Delete(indexName);
}
}
public IEnumerable<T> QueryIndex<T>(string searchText)
{
var parameters = new SearchParameters();
var results = searchIndexClient.Documents.Search<T>(searchText, parameters);
return results.Results.Select(r=>r.Document);
}
public void RemoveFromIndex<T>(IEnumerable<T> documents)
{
var actions = documents.Select(a => IndexAction.Delete(a));
var batch = IndexBatch.New(actions);
searchIndexClient.Documents.Index(batch);
}
private SearchServiceClient CreateSearchServiceClient(IConfiguration configuration)
{
string searchServiceName = configuration["SearchServiceName"];
string adminApiKey = configuration["SearchServiceAdminApiKey"];
return new SearchServiceClient(searchServiceName, new SearchCredentials(adminApiKey));
}
}
}
The constructor prepares a few reusable resources. It saves the index name, creates a search service client, and saves a copy of the search index client.
The AddToIndex<T> method takes care of adding articles to the Azure Search index. It takes the documents, creates a batch of upload actions for them, and sends that to Azure for processing. It expects a decorated document class like the Article class.
The CreateIndex<T> method creates an index definition based on the passed in class. It expects a decorated class like the Article class. It then creates the index in Azure.
The DeleteIndexIfExists method determines if the configured index exists in Azure already. If it does exist, it deletes it. Otherwise, it does nothing.
The QueryIndex<T> method expects search text. It passes this text along to Azure to perform a search and return the results. It returns only the list of documents from the results.
The RemoveFromIndex<T> method handles removing documents from the Azure index. It takes the documents, creates a batch of delete actions for them, and sends that to Azure for processing. The documents only need to provide the property that acts as the document key to work. It expects a decorated document class like the Article class.
The private CreateSearchServiceClient method is for the constructor. It handles creating a search service client for Azure Cognitive Search. It reads the configured search service name and admin API key. It then creates a search service client using Azure’s SDK with them.
Creating a basic UI
Search Controller and View
Next, you need to add a controller to manage our search operations. Start by adding a new empty MVC controller named SearchController. Then add the following:
using Kontent_Azure_Search_Demo.Helpers;
using Kontent_Azure_Search_Demo.Models;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
namespace Kontent_Azure_Search_Demo.Controllers
{
public class SearchController : Controller
{
private readonly KontentHelper kontentHelper;
private readonly SearchHelper searchHelper;
public SearchController(IConfiguration configuration)
{
kontentHelper = new KontentHelper(configuration);
searchHelper = new SearchHelper(configuration);
}
public IActionResult Create()
{
searchHelper.DeleteIndexIfExists();
searchHelper.CreateIndex<Article>();
ViewData["Message"] = "Index deleted/created.";
return View("Index");
}
public IActionResult Index(string searchText)
{
if (string.IsNullOrWhiteSpace(searchText)) {
return View();
}
var results = searchHelper.QueryIndex<Article>(searchText);
return View(results);
}
public IActionResult Initialize()
{
var articles = kontentHelper.GetArticlesForSearch().Result;
searchHelper.AddToIndex<Article>(articles);
ViewData["Message"] = "Index initialized.";
return View("Index");
}
}
}The constructor uses dependency injection to expose our configuration values. It also initializes an instance of our SearchHelper class so we can use it in our controller actions.
The first action, Create, is to create (or re-create) the index we’re working with. When invoked, this calls the SearchHelper class’s delete and create search index methods. This makes sure that the index is starting with a completely clean slate. It also writes a simple status message to the ViewData so we can update the page. Finally, it returns the index view as the code will share a single view for all the search actions.
The second method, Index, returns the basic index view for the search controller.
The last method, Initialize, gets all the articles from Kontent.ai and adds them to the index. Its primary purpose is to load the initial data into the index.
Next up, let’s create the view for our controller. Create a new folder called “Search” In the “Views” folder and add a new Index.cshtml file there with the following:
@model IEnumerable<Article>
@{
ViewData["Title"] = "Search Page";
}
<div>
<h1 class="display-4">Index actions</h1>
<a class="btn btn-danger" asp-controller="Search" asp-action="create">Create index</a>
<a class="btn btn-primary" asp-controller="Search" asp-action="initialize">Initialize index</a>
<br />
@ViewData["Message"]
<br />
<form class="form-inline flex-grow-1" asp-controller="Search" asp-action="Index" method="get">
<input class="form-control mr-sm-2" type="search" placeholder="Search" aria-label="Search" id="searchText" name="searchText">
<button class="btn btn-outline-success" type="submit">Search</button>
</form>
</div>
@{
if (Model != null)
{
<h2>@Model.Count() Results</h2>
foreach (var document in Model)
{
<div>
<h3>
<a asp-controller="article" asp-action="detail" asp-route-urlPattern="@document.UrlPattern">
@document.Title
</a>
</h3>
<p>
@document.Summary
</p>
</div>
}
}
}
This will give us a simple page with buttons for the create and initialize actions and a simple search box. It also serves as our results page. Any returned results show their title, linked the article detail page, and the summary. The detail link won’t work yet since we haven’t added the controller and view for that yet. Let’s do that now.
Article Controller and View
To give the results view somewhere to link to, let’s add a bare-bones detail view for articles. Back in the “Controllers” folder, add a new empty MVC controller. Name this one Article and add the following:
using Kontent_Azure_Search_Demo.Helpers;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using System.Threading.Tasks;
namespace Kontent_Azure_Search_Demo.Controllers
{
public class ArticleController : Controller
{
private readonly KontentHelper kontentHelper;
public ArticleController(IConfiguration configuration)
{
kontentHelper = new KontentHelper(configuration);
}
[Route("article/{urlPattern}")]
public async Task<IActionResult> Detail(string urlPattern)
{
var item = await kontentHelper.GetArticleByUrlPattern(urlPattern);
return View(item);
}
}
}The constructor accepts the configuration and saves an initialized Kontent.ai helper.
The Detail method takes the passed URL pattern, gets its content item, and passes it to the view. Let’s create this view now.
Create a new folder called “Article” In the “Views” folder. Add a new Detail.cshtml file there with the following:
@model Kontent.Ai.Delivery.ContentItem
@{
ViewData["Title"] = Model.GetString("title");
}
@if (Model.GetAssets("teaser_image").Count() > 0)
{
var image = Model.GetAssets("teaser_image").First();
<img src="@image.Url" alt="@image.Description" />
}
<h1>@Model.GetString("title")</h1>
@Html.Raw(Model.GetString("body_copy"))
This takes the passed in content item and displays the teaser image (if there is one), the title, and the body copy. Since this is only a quick example, it doesn’t handle a lot. For example, it doesn’t resolve any item links, components, or items inside the rich text field.
Trying it out
That was a lot of code, but now we’ve got everything we need to give it a try. You can hit F5 or start debugging the project to launch it in a browser. Once the welcome page has loaded, add /search to the URL to visit the search page you built. It should look something like this:
The first thing we need to do is click the “Create index” button to create the index. The page will reload, and you should see the “Index deleted/created.” message. Next, click the “Initialize Index” button to load the index. After it loads the articles from your Dancing Goat into the index, the page will reload. After the reload, you’ll see an “Index initialized.” message. Finally, you’re ready to try an actual search. If you search for “coffee”, you should get something like:
If you click on one of the results, you’ll see a details page like this:
Keeping the Index Up to Date
Great, so it works, how do we keep it updated? Kontent.ai allows you to configure webhooks to react to certain events in a Kontent.ai project.
Getting a public URL for our local code
First, let’s get ourselves set up to receive webhooks from Kontent.ai on our local machine. If you don’t have a way to handle this already, we’ll cover using ngrok as an easy, free option. Follow their instructions to get started. When you’re ready to fire it up, run the command below to point it to your local development server. You’ll need to update localhost:44345 to wherever your project is running.
./ngrok http https://localhost:44345 -host-header="localhost:44345"After running that command, you’ll see something like this:
You’ll want to use one of the URLs near at the end of the list as the target for your Kontent.ai webhook. In the screenshot above, I’d use “ https://a241e689.ngrok.io”.
Configuring a Webhook in Kontent.ai
Next, let’s configure the webhook in Kontent.ai. Go to “Settings”, then choose “Webhooks”. Click the “Create New Webhook” button and enter a webhook name. Next, make sure to enter the public URL to your endpoint and add “/webhook” to the end of it. Make sure to enable the “publish” and “unpublish” delivery API triggers. Also, clear the rest of the triggers. It should look like this:
Supporting Webhooks in our code
Application settings
First, we need to revisit the appsettings.json (or user secrets.json). Copy the secret from the above webhook configuration screen. Paste it to the KontentWebhookSecret setting value.
Models and Webhook Helper
In the “Models” folder, add a new file called KontentWebhookModels.cs. In it, add the following:
using Newtonsoft.Json;
using System;
namespace Kontent_Azure_Search_Demo.KontentWebhookModels
{
public class Data
{
[JsonProperty("items")]
public Item[] Items { get; set; }
[JsonProperty("taxonomies")]
public Taxonomy[] Taxonomies { get; set; }
}
public class Item
{
[JsonProperty("codename")]
public string Codename { get; set; }
[JsonProperty("id")]
public string Id { get; set; }
[JsonProperty("language")]
public string Language { get; set; }
[JsonProperty("type")]
public string Type { get; set; }
}
public class Message
{
[JsonProperty("api_name")]
public string ApiName { get; set; }
[JsonProperty("id")]
public Guid Id { get; set; }
[JsonProperty("operation")]
public string Operation { get; set; }
[JsonProperty("project_id")]
public Guid ProjectId { get; set; }
[JsonProperty("type")]
public string Type { get; set; }
}
public class Taxonomy
{
[JsonProperty("codename")]
public string Codename { get; set; }
[JsonProperty("id")]
public string Id { get; set; }
}
public class Webhook
{
[JsonProperty("data")]
public Data Data { get; set; }
[JsonProperty("message")]
public Message Message { get; set; }
}
}
This is a collection of models that match the JSON that Kontent.ai sends in a webhook.
Next, let’s add a new helper class in the “Helpers” folder named KontentWebhookHelper.cs. This will contain the following:
using Kontent_Azure_Search_Demo.KontentWebhookModels;
using Microsoft.Extensions.Configuration;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
namespace Kontent_Azure_Search_Demo.Helpers
{
public class KontentWebhookHelper
{
private readonly string webhookSecret;
public KontentWebhookHelper(IConfiguration configuration)
{
webhookSecret = configuration["KontentWebhookSecret"];
}
public ValidationResult ValidateAndProcessWebhook(string signature, string body)
{
var hash = GenerateHash(body, webhookSecret);
if (signature != hash)
{
return new ValidationResult
{
IsValid = false,
ErrorMessage = "Invalid signature"
};
}
// Manually deserialize request body because we needed to use the raw value previously to validate the request
var webhook = JsonConvert.DeserializeObject<Webhook>(body);
var isContentItemVariantEvent = webhook.Message.Type == "content_item_variant";
if (!isContentItemVariantEvent)
{
return new ValidationResult
{
IsValid = false,
ErrorMessage = "Not content item variant event"
};
}
var isPublishevent = webhook.Message.Operation == "publish";
var isUnpublishEvent = webhook.Message.Operation == "unpublish";
if (!isPublishevent && !isUnpublishEvent)
{
return new ValidationResult
{
IsValid = false,
ErrorMessage = "Not publish/unpublish event"
};
}
var articles = webhook.Data.Items.Where(i => i.Type == "article" && i.Language == "en-US");
if (articles.Count() == 0)
{
return new ValidationResult
{
IsValid = false,
ErrorMessage = "No 'en-US' articles effected by this event"
};
}
return new ValidationResult
{
IsValid = true,
Articles = articles
};
}
private string GenerateHash(string message, string secret)
{
secret ??= "";
UTF8Encoding SafeUTF8 = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true);
byte[] keyBytes = SafeUTF8.GetBytes(secret);
byte[] messageBytes = SafeUTF8.GetBytes(message);
using HMACSHA256 hmacsha256 = new HMACSHA256(keyBytes);
byte[] hashmessage = hmacsha256.ComputeHash(messageBytes);
return Convert.ToBase64String(hashmessage);
}
}
public class ValidationResult
{
public IEnumerable<Item> Articles { get; set; }
public string ErrorMessage { get; set; }
public bool IsValid { get; set; }
public string Operation { get; set; }
}
}
The constructor accepts the application settings and saves the webhook secret.
The ValidateAndProcessWebhook method takes the x-kc-signature header and the full webhook body. It validates that the request is from Kontent.ai. Then it makes sure there is something actionable for the search index. It checks several things to determine if there’s something actionable. First, it checks to make sure that the webhook fired for a content item variant. Next, it checks that the operation was either a publish or unpublish event. The final check determines if any of the event’s items are articles in the en-US language. If any of these checks fail, the method returns a false validation result with an error message. If the checks pass, a true validation result it returns the articles contained in the webhook.
The GenerateHash method implements the hash algorithm documented in the documentation.
Search Controller Update
Switching over to our search controller, we need to add an action to respond to the “/webhook” URL we added to Kontent.ai.
namespace Kontent_Azure_Search_Demo.Controllers
{
public class SearchController : Controller
{
// ...other private fields...
private readonly SearchHelper searchHelper;
public SearchController(IConfiguration configuration)
{
// ...other constructor code...
kontentWebhookHelper = new KontentWebhookHelper(configuration);
}
// ...other controller actions...
[HttpPost("webhook")]
public async Task<string> ProcessWebhook()
{
// Manually read request body so we can use it to validate the request. Body is gone if we use [FromBody] on a parameter.
string bodyRaw;
using (var reader = new StreamReader(Request.Body))
{
bodyRaw = await reader.ReadToEndAsync();
}
var processedWebhook = kontentWebhookHelper.ValidateAndProcessWebhook(Request.Headers["X-KC-Signature"], bodyRaw);
if (!processedWebhook.IsValid)
{
return processedWebhook.ErrorMessage;
}
if (processedWebhook.Operation == "publish")
{
var articlesToUpdate = kontentHelper.GetArticlesForSearch(processedWebhook.Articles.Select(a => a.Id)).Result;
searchHelper.AddToIndex(articlesToUpdate);
}
if (processedWebhook.Operation == "unpublish")
{
var articlesToRemove = processedWebhook.Articles.Select(a => new Article { ID = a.Id });
searchHelper.RemoveFromIndex(articlesToRemove);
}
return "Updated index";
}
}
}We’ve added a new private field for the KontentWebhookHelper. The constructor initializes it. We’ve also added a new controller action.
The new ProcessWebhook action responds to an HTTP post to the “/webhook” URL. It parses the body manually because of the need to use the raw body for signature verification.
It passes the webhook body and signature to the ValidateAndProcessWebhook method. If the result is not valid, it returns the error message passed from the method. If it’s valid, it will update or remove the items based on whether it was a “publish” or “unpublish” operation.
When it’s a publish operation, it calls the Kontent.ai helper’s GetArticlesForSearch method. It passes it the IDs of the articles from the webhook to get their new data. Once it has the item data, it calls the search helper’s AddToIndex method to update the index.
When it’s an unpublish operation, it creates an IEnumerable of the Article class. It creates this list using only the IDs of the articles sent in the webhook. Finally, it calls the search helper’s AddToIndex method to update the index.
That’s everything we need to respond to Kontent.ai webhooks and keep the index up to date.
Source Code and Next Steps
In this demo, we’ve seen all the basics. We initialized, searched, and updated an Azure search index with content from Kontent.ai.
This is far from a comprehensive implementation of Azure search. There are many features such as faceting and sorting that we haven’t touched on here. We also only covered a single content type in one language. Implementing any of these things will be great next steps if you’re looking for a little more.
Subscribe to Kontent.ai Newsletter
Stay in the loop. Get the hottest updates while they’re fresh!
