[FR]: Add Vertex AI Vision & Audio Sample Code for iOS

[1998code]

Description

@andrewheard

1. As we upgraded to 1.5 Flash ([Vertex AI] Update samples/docs to use gemini-1.5-flash-preview-0514 #12979), is it possible to achieve like Project Astra now?
2. Currently, the sample only provided text output.
3. The AI can do conversation with audio and vision.

Thank you :)

API Proposal

N/A

Firebase Product(s)

Vertex AI

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[andrewheard]

Thanks for the feature request, @1998code. It would currently be possible to add video and audio input to the sample apps but currently only text output is supported by the API.

I think we'd probably want to add this feature to the multi-modal sample but would need to refactor it a bit since it stores a list of PhotosPickerItem.

[1998code]

Awesome! Looking forward :)

[andrewheard]

1. is it possible to achieve like Project Astra now?

Note: This is now referred to publicly as the Multimodal Live API but it is not yet supported by the SDKs.

2. Currently, the sample only provided text output.

@1998code, PR #14545 adds image generation to the sample (using Imagen, not Gemini, though).

[1998code]

Cool! Thanks for the update 🤩