On-device generative AI with Gemini Nano (Part 2)

In this demo, we want to enhance our previous On-device generative AI demo with 2 UX improvements.

  • Add a loading icon, to let the user know we are working on getting the result.
  • Use the streaming API to render the result via a text stream instead of waiting for the full response to be ready.

First, we change our code from the prompt() API to the promptStreaming() API.

const stream = session.promptStreaming(prompt);
for await (const chunk of stream) {
    this.outputTarget.textContent = chunk;
}

Next, we reset the output DOM with a loading icon every time the user triggers a generate.

const stream = session.promptStreaming(prompt);
this.outputTarget.classList.remove("hidden");
this.outputTarget.innerHTML = '<i class="fa-solid fa-pencil fa-fade"></i>';
for await (const chunk of stream) {
    this.outputTarget.textContent = chunk;
}

The outcome:


Since this is a case of summarizing technical articles, we want the API to be "less creative" and "more precise". We can further improve our code to set the temperature to a lower value, e.g. 0.1

    async createAiSession() {
        const defaults = await window.ai.defaultTextSessionOptions();
        return await window.ai.createTextSession(
            {
                temperature: 0.1,
                topK: defaults.topK
            }
        );
    }

Now we have a simple working AI feature.


AI Summary
gpt-4o-2024-05-13 2024-07-14 00:37:54
This blog post details enhancing an on-device generative AI demo by adding a loading icon and using a streaming API for incremental results. It includes code snippets showing the modifications and highlights setting a lower temperature value for more precise responses.
Chrome On-device AI 2024-07-20 04:56:42

Share Article