Build a simple app that describes an image and works offline

This article shows a mobile app that uses the Gemini Image Description API on Android.

Reference: https://developers.google.com/ml-kit/genai/image-description/android

Pre-requisites

1. Support Android SDK Level 29 and above
2. The feature availability is limited to certain flagship devices (Does not work on the emulator)
3. The description output is limited to English

Key Steps in the Implementation

1. Initialize the image descriptor

   import com.google.mlkit.genai.imagedescription.ImageDescriberOptions
   import com.google.mlkit.genai.imagedescription.ImageDescriber
   ...
   
       private var imageDescriber: ImageDescriber? = null
   
       fun initializeClient(context: Context) {
           if (imageDescriber == null) {
               val options = ImageDescriberOptions.builder(context).build()
               imageDescriber = ImageDescription.getClient(options)
           }
       }
   

2. To check if the device is supported and to download the model if it is not available

   import android.content.Context
   import com.google.mlkit.genai.common.FeatureStatus
   import kotlinx.coroutines.guava.await
   ...
   
       suspend fun checkFeatureStatus(context: Context): Int {
           initializeClient(context)
           val describer = imageDescriber ?: return FeatureStatus.UNAVAILABLE
           return describer.checkFeatureStatus().await()
       }
   

3. Similar to the Web API, the feature status has 4 possible values - Available, Unavailable, Downloading, Downloadable

   If the value is unavailable, you can't proceed further. Your device might not be supported.

4. If the value is downloadable, you need to trigger the model download. This usually happens when you use the API for the first time.

   suspendCoroutine { continuation ->
       try {
           val describer = imageDescriber
               ?: throw IllegalStateException("ImageDescriber not initialized")
   
           describer.downloadFeature(object : DownloadCallback {
               override fun onDownloadStarted(bytesToDownload: Long) {
                   onProgress(0)
               }
   
               override fun onDownloadProgress(totalBytesDownloaded: Long) {
                   onProgress(totalBytesDownloaded)
               }
   
               override fun onDownloadCompleted() {
                   continuation.resume(true)
               }
   
               override fun onDownloadFailed(error: GenAiException) {
                   continuation.resumeWithException(error)
               }
           })
       } catch (e: Exception) {
           continuation.resumeWithException(e)
       }
   }
   

5. Once the model is downloaded, you can now use the API on the device without an active internet connection.

6. We can now pass a Bitmap and ask for a description.

   return suspendCoroutine { continuation ->
       try {
           val describer = imageDescriber
               ?: throw IllegalStateException("ImageDescriber not initialized")
   
           val request = ImageDescriptionRequest.builder(bitmap).build()
           val result = StringBuilder()
           var lastUpdateTime = System.currentTimeMillis()
           var isResumed = false
   
           describer.runInference(request) { text ->
               result.append(text)
               lastUpdateTime = System.currentTimeMillis()
           }
   
           CoroutineScope(Dispatchers.Default).launch {
               while (!isResumed) {
                   delay(100)
                   val timeSinceLastUpdate = System.currentTimeMillis() - lastUpdateTime
                   if (timeSinceLastUpdate > 500 && result.isNotEmpty()) {
                       if (!isResumed) {
                           isResumed = true
                           continuation.resume(result.toString())
                       }
                   }
                   // Timeout after 30 seconds
                   if (timeSinceLastUpdate > 30000) {
                       if (!isResumed) {
                           isResumed = true
                           if (result.isEmpty()) {
                               continuation.resumeWithException(Exception("Timeout: No description generated"))
                           } else {
                               continuation.resume(result.toString())
                           }
                       }
                   }
               }
           }
       } catch (e: Exception) {
           continuation.resumeWithException(e)
       }
   }
   

7. Finally, we should call the close method to clean up the initialized resource.

   fun cleanup() {
       imageDescriber?.close()
       imageDescriber = null
   }
   

A sample screen flow illustration

1. Prompt the user to download the model.

   

   

2. Ask for storage permission to add a picker.

   

   

3. And we are ready to test.

The results