Searching headless content with Algolia, Rails, Kontent & Next.js
In this guest article, Tom Marshall from Kyan walks you through the process of delivering engaging and fast search experiences that return relevant results.
As anyone who has implemented search functionality before will know, search is a deceptively complex problem. Google set a high bar. Users naturally assume that search experiences will return exactly what they’re after in the first few results whilst forgiving any mistakes or inaccuracies in their input.
From the user’s perspective, search should ‘just work,’ but that’s easier said than done.
A headless CMS decouples the content from the presentation layer. That provides many benefits, which is why we love headless, but search functionality requires that presentational knowledge. Without this context of how the content comes together to form the site’s pages, headless CMSs cannot provide search solutions capable of meeting users’ expectations out of the box.
Thankfully integrations with Search-as-a-Service products like Algolia offer a solution here.
At Kyan, we deliver search experiences that are engaging, fast, and most importantly, return relevant results. In this article, I’ll show you how, with Algolia, Next.js, Ruby on Rails, and Kontent by Kentico.
Our system architecture
We have a Next.js project as our web front end, which fetches content from the Kontent headless CMS, speaks to a Rails API for additional functionality (e.g., comments), and integrates Algolia’s Instant Search to provide the search UI.
Kontent is the canonical content store, but Algolia cannot read from the Kontent Delivery API in real-time. That would be too slow. It needs a local copy. We could regularly copy the content in bulk, e.g., with a nightly cron job, but we want our search results to remain in sync as content changes are published.
We want changes to be pushed, not pulled.
Kontent emits webhook notifications as content changes are published, but we cannot connect the raw Kontent webhook notifications to Algolia directly. We need something in the middle to listen for those webhook notifications, extract the content relevant to the search, then push that to Algolia.
At Kyan, we love Rails for building backend services and APIs, so for us, that’s going to be the Rails app, but it could equally be a Next.js API endpoint or serverless function. It just needs to be something that speaks HTTP and ideally has open-source client libraries available for the Kontent Delivery and Algolia APIs, though that’s not essential.
Designing the index
Algolia does not need a complete copy of the content, only the fields relevant to the search functionality and UI. To determine which fields are required, we need to answer the following questions.
Which fields are required:
- For matching search terms against
- To filter the results with
- To order results by
- To display on result cards
- And, finally, any IDs needed for reference
Using blog articles as an example:
|Search Use||Fields Required|
|For matching search terms against||Title & Body|
|To filter the results with||Tags & Author|
|To order results by||Publish Date|
|For presenting results||Image, Estimated Reading Time & URL Slug|
|IDs for referencing||Kontent Item ID|
Some fields may fall into multiple categories, which is fine; we just need a complete set.
Now that we know which fields we need for the search functionality, we can set up our Rails application to maintain the index within Algolia.
Maintaining the index
As content editors make changes in Kontent, we need those propagated to Algolia via Webhook. The sequence for that is as follows:
Setting up Algolia
Having logged in to the Algolia dashboard, we need to create a new application. Don’t worry about creating an index or importing any records at this stage. The
algoliasearch-rails gem will take care of that. We just need to extract the API keys for the Rails and Next.js applications.
We will use the pre-generated 'Search-Only API Key' for Next.js and the 'Admin API Key' in the Rails application for demo purposes. For production, you should create specific API keys to secure access control, specify rate limits, HTTP referrers, etc.
Handling the webhook notifications
First, we need to create a Rails application and add the additional gems we’ll need:
We configure the Algolia gem by adding an initializer at
Next, let’s add the values for those Algolia environment variables to our
The Algolia gem works as an extension of Rails’ ActiveRecord ORM, so we need to create an
Article model and run the generated migration.
> Note: We’re using PostgreSQL, so we’ll tweak the
tags attribute to be an array (
t.string :tags, array: true, default: ) before running the migrations. Alternatively,
tags could be a reference to a
Tag model, but we only need the tag names to match and filter against currently.
Because the Algolia gem works as an extension of ActiveRecord, the
Article records will be automatically persisted locally in the PostgreSQL database. This database persistence isn’t strictly necessary, so you can skip it if you’re using a serverless function or Next.js API endpoint. However, this database persistence is how the
algoliasearch-rails gem operates by default, so it is the path of least resistance for us in Rails.
algoliasearch-rails methods, we can define how Algolia should use the
Article model attributes:
Webhook notifications from Kontent are POST’d to a URL of your choice. To handle the incoming webhook, we’ll need to add a route to
Next, we need a controller action to handle the requests for that route.
Here’s an example webhook request body from a change to an Article item in Kontent:
You’ll notice that the request body does not contain all of the fields of the Article item that we need for the search index. Instead, the webhook just enumerates which content items have changed. This is why we need to fetch the full content from Kontent after having received the notification.
To keep things manageable, we’re going to breakdown the Rails logic across 4 different files:
app/controllers/webhooks_controller.rb- for handling the incoming HTTP request
app/lib/kentico/webhook_notification.rb- for extracting any Article items within the webhook notification
app/lib/kentico/fetch_content_item.rb- for fetching the full Article item content from Kontent
app/lib/kentico/article_item_response.rb- for presenting the raw Kontent response as the Rails model attributes
> Note: In a production application, we’d use background jobs here to avoid overloading the main thread, but that’s beyond the scope of this demo.
Setting up Kontent
First, log in to Kontent and create a new, empty project.
Next, we’ll need to create a webhook.
To develop and test our Kontent webhook implementation locally, we’ll need the incoming webhooks from the public internet to reach our local development machine.
ngrok can provide a public URL for the webhook that will then tunnel to your local machine.
Create a webhook in the Kontent project settings and set the URL as the tunneled URL from ngrok, e.g.,
> Note: In Rails 6 or newer, you’ll need to add the ngrok host to development config or disable host restriction entirely by adding
config.hosts.clear to the
Add the Kontent project ID and webhook secret to the
.env alongside the Algolia environment variables:
Then start the local Rails server:
> Note: You’ll need to enable the Management API in the Kontent project settings to generate the API key, and the
--name parameter will automatically apply the
If everything is working correctly, you should see activity in the
rails terminals, as the backup restoration triggers webhook requests for the newly created content items.
Once complete, you should have the demo content items in the Kontent project and corresponding Article records in the Rails database.
As well as in the Algolia index via the Web UI (which might need a hard refresh):
If you run into issues, Kontent offers a debug log for each webhook to inspect any errors and copy the webhook request bodies, which allows you to test yourself locally with an HTTP client like Postman or
Finally, the Next.js UI
Algolia offers an InstantSearch component library that provides a customizable pre-built set of components for building a live search experience. As our front-end is a Next.js application, we’ll be using InstantSearch for React.
Algolia’s InstantSearch docs do an excellent job of explaining how to compose the InstantSearch components together into a search interface that works for your product, so we won’t cover that here. Instead, we’ll pull Algolia’s pre-built Next.js server-side rendering demo project and customize that for demonstration purposes.
yarn create next-app has finished, we need to customize two files to connect Algolia’s demo project to our Algolia application.
First, we need to update the Algolia search client configuration in the
pages/index.js to pull our Application ID, search API key from the environment config:
And then in the same file, the
indexName in the
> Note: With the
algoliasearch-rails gem, the index name will default to the model name, but if in doubt, the index names are visible within the Algolia web dashboard.
Second, we need to update the
HitComponent markup in the
components/app.js to use the attributes in our index, rather than the ones from the demo Algolia project:
And then in the same file update the
RefinementList menu with our faceted attributes:
Finally, Next.js and Rails both default to port 3000, so to run both at the same time, you’ll need to override the port for Next.js
dev script in the
With that, we have an end-to-end search solution. We can start the Next.js app locally with:
And view it in the browser at http://localhost:3001.
Any content updates published in Kontent are automatically propagated to Algolia via Rails, and users can filter the results in real-time as they type a search term and select from the faceted attributes.
You can find the demo project on GitHub if you want to try it out for yourself.
Hidden complexity—modular content
Currently, our Rails app will update the Algolia search index when users publish changes to Article content items within Kontent. However, we’re going to allow users to search and filter the blog articles by Tag and Author. What happens if these content items change? Currently, nothing.
To update the search index when Tags or Authors change, we need to update our webhook handling code so that instead of discarding webhook notifications for Tags and Authors, we update all of the Article items linked to that Tag or Author.
You find out more about indexing modular content with Kontent and Algolia on the Kontent blog.
Whenever you’re maintaining a copy of data from the canonical source, it’s prudent to have a mechanism in place for refreshing the complete data set. This mechanism can be used to load the initial data set, but also in the future if you need to update every record following a change in structure to the canonical data.
In our Rails app, we’d add a rake task for this, which would fetch and loop through all of the Article content items to update Algolia.
However, we need to keep in mind that the Kontent Delivery API is restricted to a maximum of 2000 items per response. That includes any linked items, so if you’re requesting data with any linked items included, paginating in chunks of 2000 will not prevent an error.
The simple solution is to loop through in page sizes small enough to be confident that you won’t hit a response with more than 2000 items, e.g., 10, though this value should be tuned based on your specific content.
A more sophisticated approach is to implement an incremental back-off so that if you encounter a maximum items error, the request is retried with a smaller page size until it succeeds. This approach ensures success and should also be more performant but is more complex to implement.
We are Kyan, a technology agency powered by people.