Creating an LLM based content translation pipeline

In the previous post we setup internationalisation with Astro. Next up is using a large language model (LLM) as a content translator. The goal is to publish the sites content in every well LLM supported human language and to investigate how feasible this actually is.

Previously I’d used OpenAI’s models directly via their API but it’s not completely free and I’m intrigued to explore the process of downloading models to run inference locally.

It turns out that this is insanely easy thanks to the Ollama project. It goes pretty much like this:

$ brew install ollama
$ ollama serve
$ ollama pull llama3

The default llama3 model seemed perfectly sufficient to get started.

Ollama provides a CLI & REST api for interaction with downloaded models, but rather than work with that directly, we can use Vercel’s standard-defining ai library as well as it’s ollama adapter over in our Astro project:

yarn add ai ollama-ai-provider

Then I create a src-content folder and create a new translateContent.ts file inside it with the following content:

import { createOllama } from "ollama-ai-provider";
import { generateText } from "ai";

const ollama = createOllama({});

const contentResponse = await generateText({
  model: ollama("llama3"),
  prompt: "Please translate the following content: …",
});

console.log(content);

Over the years I’ve forgotten how to run a simply ts file with es module syntax in it simply more times than I can count, and after cycling through esbuild-register, and ts-node, I finally resettled on:

$ npx tsx ./translateContent

Just like that we’re programatically sending/receiving LLM output on the local machine for free. Perfect.

From here it’s just a bunch of node scripting to read in the English posts from src-content/posts/[slug].mdx, translate each and output them to each of the src/content/posts/[lang]/[slug].mdx files for each supported locale.

As this script will likely grow into a CLI for managing translations, I got started with commander to create a few options:

const program = new Command();

program
  .argument('[fileNameFilter]', 'Filter files to process by a filename')
  .option('-nc, --noclean', 'Don’t clean the content output directory. This is the default option when a filename is provided.')
  .parse(process.argv);

…

const [filenameArg] = program.args;
const { noclean } = program.opts();

How well is llama3 performing? One problem is we only want the prompt output to contain the translation itself without extra bits like “Certainly, here is the translation”. OpenAI supports functions that can guarantee a JSON shape output but for Ollama supported models we apparently need to resort to strongly encouraging the model to “respond with ONLY the exact translation WITHOUT commentary or prelude/notes”. This seems slightly unreliable.

One seemingly promising option was asking the model to output JSON as it seemingly forces it to understand there is only one place to put the translation value. Unfortunately the model struggled with escaping quotation marks in the content within the output JSON. For now we can resort to straight text output but it would be nice to tighten this up later as we learn more about prompt engineering or appropriate models.

As for the translations themselves I am fluent enough in french to verify the output is somewhat sane, but at this stage, I haven’t read side by side to investigate whether the more nuanced aspects of the content are being translated correctly. We can look at that in a future post.

In the spirit of MVP - that’s it E2E! Posts authored in English are automatically translated into a fully internationalised Astro site. It’s just that currently it has very few posts and very little functionality 😅.