On-device generative AI with Gemini Nano (Part 2)

In this demo, we want to enhance our previous On-device generative AI demo with 2 UX improvements.

• Add a loading icon, to let the user know we are working on getting the result.
• Use the streaming API to render the result via a text stream instead of waiting for the full response to be ready.

First, we change our code from the prompt() API to the promptStreaming() API.

const stream = session.promptStreaming(prompt);
for await (const chunk of stream) {
    this.outputTarget.textContent = chunk;
}

Next, we reset the output DOM with a loading icon every time the user triggers a generate.

const stream = session.promptStreaming(prompt);
this.outputTarget.classList.remove("hidden");
this.outputTarget.innerHTML = '<i class="fa-solid fa-pencil fa-fade"></i>';
for await (const chunk of stream) {
    this.outputTarget.textContent = chunk;
}

The outcome:

Since this is a case of summarizing technical articles, we want the API to be "less creative" and "more precise". We can further improve our code to set the temperature to a lower value, e.g. 0.1

    async createAiSession() {
        const defaults = await window.ai.defaultTextSessionOptions();
        return await window.ai.createTextSession(
            {
                temperature: 0.1,
                topK: defaults.topK
            }
        );
    }

Now we have a simple working AI feature.