How to automatically translate content using webhooks

On multilingual sites, it has always been a challenge to keep the content translated to all languages. How does the recent update of webhooks help solve that challenge automatically and let you take a coffee break instead?

Eric Dugre

Published on Apr 2, 2020

Webhook 101

It’s nice when things just work automatically, isn’t it? Sure, you could translate that new Kontent.ai article into Spanish on your own, but why waste your time if it can be done for you? This is where our powerful webhook feature comes in—when you change a content item, the system sends a notification to the URL provided. This is generally a serverless function hosted in Azure or Amazon, or a third-party endpoint that performs the translation. In this example, we will be using Microsoft’s Translator Text Cognitive Service for translation.

Setting up the project

In this example, we will be using the Javascript SDK and ExpressJS to create an application that will consume the webhook. Let’s start by creating a basic Express application using the generator as described here:

mkdir webhook-app
cd webhook-app
npx express-generator
npm i

If you run the application with `npm start`, it should be accessible at "http://localhost:3000/". Now, we need to set up the endpoint that Kontent.ai will POST the webhook to. In app.js, add a new "/webhook" route and simply return a 200 response:

app.post('/webhook', (req, res) => {
  res.status(200).send('Success');
});

Note: You can also access all mentioned code files on this GitHub repository.

Accessing a local server from the Internet

We should test the /webhook route to ensure that Kontent.ai can reach your endpoint, before adding any more code. To make our local application publicly accessible, we can use ngrok. It’s free to register; then follow the steps here to get it running on your PC. In step 4, use the port your local application is running on:

./ngrok http 3000

If successful, you will see a message displaying information including your public URL:

We’ll use this URL in the webhook configuration in Kontent.ai

First, we need to set up our workflow. We want to automatically translate a language variant when it reaches a particular workflow step, so let’s add the Translation step:

Now go to Project settings > Webhooks to create a new webhook and call it “Automatic translation”. Delete the existing triggers and select your Translation step from the drop-down menu under “Workflow steps of content items to watch”. Copy the URL from ngrok and add it to the URL address field with our /webook route at the end:

We’re ready to test! Make sure both your Express application and ngrok are running and move a content item from the Review step to Translation. Within seconds, you should see ngrok report a successful POST to /webhook, and the dot next to your webhook in Kontent.ai should turn green, indicating that it’s working.

Validating the webhook

Now that we’re correctly receiving the POST in our application, it’s time to verify the signature and do something with it! All webhook notifications have a signature to verify that the request is coming from Kontent.ai. The signature is a hash of the webhook secret and body of the request. You can read more about the signatures here and view an example of verifying the signature in Javascript here.

As the sample code notes, we need to use body-parser to parse the raw JSON data. Install body-parser:

npm i body-parser

Then tell Express to use it when parsing JSON data in app.js:

const bodyParser = require('body-parser');
//app.use(express.json()); // original code added automatically to Express- comment or remove this line
app.use(bodyParser.raw({type:'application/json'})) // add this line

We can now get the signature from the header of the POST request and confirm whether the signature is valid. You will need to save the webhook secret found in Kontent.ai as a const in your app.js file, or use an environment variable. Create a new function to validate the signature:

const crypto = require('crypto');
const webhookSecret = 'EKAPGD8JrgjtqrAXqK2C1hTT0/JnHDf/5bziRnHtstM=';
const hasValidSignature = (body, signature) => {
  const computedSignature = crypto.createHmac('sha256', webhookSecret)
      .update(body)
      .digest();
  return crypto.timingSafeEqual(Buffer.from(signature, 'base64'), computedSignature);
}

In your /webhook route, call your hasValidSignature function by passing the body and signature of the req.headers object:

app.post('/webhook', (req, res) => {
  if(hasValidSignature(req.body, req.headers['x-kc-signature'])) {
    res.status(200).send('Success');
  }
  else {
    res.status(403).send('Invalid signature');
  }
});

Test your webhook again to ensure that you receive a successful 200 response. Once the webhook is properly validated, we can move on to reading the data that Kontent.ai sent in the body of the webhook notification.

Processing the webhook

We will be using the Javascript SDK’s Content Management API to retrieve data from Kontent.ai and update our translated language variants. Before you begin this section, follow the steps for installing the CM API client and instantiate the client in your app.js file:

const cmClient = new ContentManagementClient({
    projectId: 'your-project-ID',
    apiKey: 'your-CM-API-key'
});

Let’s take a look at the information given to us in the webhook notification and determine what we need to translate our variants. The body of the workflow webhook notification differs from previous non-workflow webhooks. It will look something like this:

"data": {
  "items": [
    {
      "item": {
        "id": "65f05e0f-40c3-436b-a641-e2d4cae16e46"
      },
      "language": {
        "id": "00000000-0000-0000-0000-000000000000"
      },
      "transition_from": {
        "id": "eee6db3b-545a-4785-8e86-e3772c8756f9"
      },
      "transition_to": {
        "id": "03b6ebd3-2f49-4621-92fd-4977b33681d1"
      }
    }
  ]
}

Item is the ID of the language variant that moved into the Translate workflow step. Since this webhook triggers only for that workflow step, we don’t need to worry about the transition_from or transition_to values. To completely translate this language variant into other languages, we need a few different pieces of information:

The elements of the content type that should be translated
The text of those elements from the updated language variant
The languages that we need to translate into

We can break this down into 4 steps:

Get the language variant using the ID in the webhook notification.
Get the content item of the language variant using the ID from step 1.
Get the content type of the content item using the ID from step 2.
Get the project languages to translate content into.

To begin this process, create a new function in app.js to contain the logic and call it from the /webhook route:

app.post('/webhook', (req, res) => {
  if(hasValidSignature(req.body, req.headers['x-kc-signature'])) {
    processWebhook(JSON.parse(req.body));
   //...

const processWebhook = (body) => {
}

We need the updated variant’s ID from the JSON data to use in the Content Management API call we will make in step 1 of the process. Let’s also get the language ID, since we only want to translate English variants (the default language in this example) into other languages:

const processWebhook = (body) => {
  const updatedVariantLangID = body.data.items[0].language.id
  const updatedVariantItemID = body.data.items[0].item.id;

  // Only translate variants when an English variant was updated
  if(updatedVariantLangID !== '00000000-0000-0000-0000-000000000000') return;
}

In the app.js file, declare some global variables that will store the results of our CM API calls:

let updatedVariant, contentItem, contentType;

Install rxjs and load the mergeMap() function, as we’re going to use this function to create a single Observable for our 4 requests, along with map() which we’ll use later:

npm i rxjs

const { mergeMap, map } = require('rxjs/operators');

Step 1 - Get the language variant using the ID in the webhook notification

Now we’re ready to get the data we need in our four-step process. The first step is to get the full language variant using viewLanguageVariant(), since we only have the ID at this point:

const getLanguageVariant = cmClient
                .viewLanguageVariant()
                .byItemId(updatedVariantItemID)
                .byLanguageId(updatedVariantLangID)
                .toObservable();

This will provide us with the variant’s element IDs and their values, and some system data about the variant itself. To determine which elements should be translated, we need the content item.

Step 2 - Get the content item of the language variant

We can use the result of the first Observable to get the content item’s ID and request the content item using viewContentItem():

const getContentItem = (result) => {
    updatedVariant = result.data;
    return cmClient
            .viewContentItem()
            .byItemId(updatedVariant.item.id)
            .toObservable();
  };

Step 3 - Get the content type of the content item

With the content item definition, we can now request the content type. This will allow us to identify which elements of the updatedVariant object are text or rich_text elements, and translate their values. We’ll save the results of this Observable, then call viewContentType():

const getContentType = (result) => {
    contentItem = result.data;
    return cmClient
            .viewContentType()
            .byTypeId(contentItem.type.id)
            .toObservable();
  };

Step 4 - Get the project languages to translate content into

We now have all of the data except the project languages. We can store the results of the above Observable and then use listLanguages() to get the languages:

const getLanguages = (result) => {
    contentType = result.data;
    return cmClient
            .listLanguages()
            .toObservable();
  }

Now, we can roll up these 4 Observables into one:

const obs = getLanguageVariant.pipe(mergeMap(getContentItem)).pipe(mergeMap(getContentType)).pipe(mergeMap(getLanguages));
const sub = obs.subscribe(result => {
  sub.unsubscribe();

The result of this final Observable will be the list of languages. It contains a lot of information about the language, but we only want the codename to send to our external service. If you are not using standard codenames like “en-us” in your project, go change them now! This allows us to easily send the codename to the external service, which is expecting the four-letter format. We’ll use rxjs’ map() function to create an array of codenames:

 const sub = obs.subscribe(result => {
  sub.unsubscribe();
  const projectLanguages = result.data.languages.map(l => l.codename);

With the results of the fourth CM API call, we now have all the information we need to translate the content!

Translating the content

Our next plan of action is to get all of the translatable elements in the content type, loop through the language code names, and send the values from the English language variant to an external service for translation. We’ll use filter() to find the text and rich_text elements, then create an array of their IDs using the map() function:

const sub = obs.subscribe(result => {
  sub.unsubscribe();
  const projectLanguages = result.data.languages.map(l => l.codename);
  const textElementIDs = type.elements.filter(e => e.type === 'text' || e.type === 'rich_text').map(e => e.id);

With this list of elements that need translating, we can loop through all of the language code names and call a function which we’ll create next to perform the actual translation and update the variant:

const sub = obs.subscribe(result => {
      sub.unsubscribe();
      const projectLanguages = result.data.languages.map(l => l.codename);
      const textElementIDs = type.elements.filter(e => e.type === 'text' || e.type === 'rich_text').map(e => e.id);
      projectLanguages.forEach(targetLangCode => {
        if(targetLangCode !== 'en-us') upsertLanguageVariant(targetLangCode, textElementIDs);
    });
  });

Our code is now calling upsertLanguageVariant() for each non-English culture and passing the array of element IDs that need to be translated. We’re going to use rxjs’ zip() to send multiple requests to Microsoft’s Translator Text API—one request per element. Create the new upsertLanguageVariant function and an array to hold our Observables, and register the zip() function:

const { zip } = require('rxjs');
const upsertLanguageVariant = (targetLangCode, textElementIDs) => {
  const translateObservables = [];
}

Now, we’ll loop through all of the elements in the updated English variant. If the element’s ID is in our list of translatable IDs, we create an Observable for translation and add it to our array:

const upsertLanguageVariant = (targetLangCode, textElementIDs) => {
  const translateObservables = [];
  updatedVariant.elements.forEach(e => {
        if(textElementIDs.includes(e.element.id)) {
            translateObservables.push(
                // Create Observable
          );
     }
  });

To create an Observable for a REST request to Microsoft, we’ll use axios-observable. Install it and then register it in app.js

const Axios = require('axios-observable').Axios;

You will also need a key from Microsoft to use their service. Create an Azure Cognitive Services account for the Translator Text service, and after that, save “Key 1” from the Keys tab as an environment variable or const in app.js.

const translationKey = '4e888811x03f4bd2732321683175d56b';

We’re ready to create the REST request to translate each individual element’s value. According to Microsoft’s API reference, we should provide the from parameter for the source language, the to parameter for the target language, and the textType as rich text elements store HTML in them. The key is sent in the “Ocp-Apim-Subscription-Key” header, and the body will conform to their model. Here is the function so far:

const upsertLanguageVariant = (targetLangCode, textElementIDs) => {
  const translateObservables = [];
  updatedVariant.elements.forEach(e => {
        if(textElementIDs.includes(e.element.id)) {
            translateObservables.push(
                Axios.request({
                  method: 'POST',
                  params: {
                      from: 'en-us',
                      to: targetLangCode,
                      textType: 'html'
                  },
                  url: 'https://api.cognitive.microsofttranslator.com/translate?api-version=3.0',
                  headers: {
                      'Ocp-Apim-Subscription-Key': translationKey,
                      'Content-type': 'application/json'
                  },
                  data: [{
                      'text': e.value
                  }]
              })
              .pipe(map(result => [e.element.id, result.data[0].translations[0].text]))
          );
     }
  });
}

You could technically translate all of the elements in one request, but there would be no way to tell which text belongs to which element due to the way Microsoft responds. This is what our pipe(map()) function is for after the request is created—the result of each individual request will be stored as an array with the element ID and translated text, like this: ['4e9acd7a-2db8-4c33-a13a-0c368ec2f108', 'Ahoj']. All of these results are then stored in a single object returned by our zipped Observable.

Let’s zip() this array of Observables using the spread operator and get the result. With the result, we want to loop through each element of the English variant and see if there is a matching element ID. If so, we can set the value in updatedVariant to the new translated value:

const sub = zip(...translateObservables).subscribe(result => {
      sub.unsubscribe();

      // Set new values
      updatedVariant.elements.forEach(e => {
          const match = result.filter(arr => arr[0] == e.element.id);
          if(match.length > 0) {
              let text = match[0][1];
              if(match.length > 0) e.value = text.replace(/<br>/g, '<br/>');
          }
      });
  });

You’ll notice we needed to do a small replace(). Microsoft’s service does not close line breaks, which the Kontent.ai API requires, so we need to fix that. The updatedVariants.data.elements array is now populated with the default English values and some translated values, and we can use these elements to upsert a new language variant.

To accomplish this, we’ll use createNewVersionOfLanguageVariant() to create a new version of the target variant. After that, we’ll call upsertLanguageVariant() and pass the content item’s ID, the target language, and the elements array which we’ve updated:

// Create new version then upsert data- only works for published variants!
cmClient.createNewVersionOfLanguageVariant()
        .byItemId(contentItem.id)
        .byLanguageCodename(targetLangCode)
        .toObservable()
        .subscribe(result => {
            cmClient.upsertLanguageVariant()
            .byItemId(contentItem.id)
            .byLanguageCodename(targetLangCode)
            .withElements(updatedVariant.elements)
            .toObservable()
            .subscribe(result => {
                console.log(`language ${result.data.language.id}: ${result.debug.response.status}`);
            });
});

Note that this automatic translation will only work for variants that are published. In order for this to work for all variants, you will need to check whether the variant is published, and move the variant to the Draft step if it is not. You can view the full source code for the app.js file on GitHub using the link at the end of this article.

What we’ve learned

Whew! It took several CM API calls to get the data we needed, but with the help of rxjs our code was organized and efficient. This is just one small example of the freedom that webhooks provide for your projects. They can be used to keep external integrations up to date, perform automatic functions with your content, and so much more!

If you’re interested in seeing other examples of using webhooks, you can check out this great article: Clearing Obsolete Cache Entries with Webhooks. You can view the full code for my implementation of automatic translation on GitHub, including instructions to get the sample application with automatic translation running on your machine!