Add a Real-Time Voice AI Assistant to Your FlutterFlow App

Search for a FlutterFlow voice assistant today and you mostly find ElevenLabs library showcases, Siri-clone YouTube tutorials, and DIY guides built on the same pattern: speech_to_text to capture a transcript, one chat-completion call, then flutter_tts to read the answer aloud. It demos fine. In real users' hands it behaves like a walkie-talkie — talk, wait, listen, repeat — and it collapses the moment someone interrupts.

This post covers both routes: why the DIY pipeline lags and breaks, and how to ship a real speech-to-speech AI voice agent in a FlutterFlow app, with barge-in and live captions, by embedding the same Widget-Chat widget you'd use for text chat and switching voice on in the dashboard.

Why the DIY pipeline feels like a walkie-talkie

The classic three-package stack looks deceptively simple:

final speech = SpeechToText(); // package: speech_to_text
final tts = FlutterTts();      // package: flutter_tts

Future<void> askByVoice() async {
  await speech.initialize();
  await speech.listen(
    pauseFor: const Duration(seconds: 3),
    onResult: (result) async {
      if (result.finalResult) {
        final reply = await callChatApi(result.recognizedWords);
        await tts.speak(reply);
      }
    },
  );
}

Three problems show up the first time you test a real conversation.

1. Latency stacks in series

Every stage waits for the previous one to finish completely. speech_to_text waits out pauseFor seconds of silence before it commits a final transcript. Then your chat API generates the entire answer. Only then does flutter_tts start speaking from character zero. Stack the stages and a quick question routinely gets 5–10 seconds of dead air. Production voice agents overlap these stages — streaming recognition, streaming generation, incremental synthesis — which is exactly what three independent plugins glued together in Dart can't do.

2. Sessions die, especially on Android

speech_to_text wraps each platform's built-in recognizer, and its maintainers are explicit that continuous listening is not a supported use case: the OS recognizer times out and you must restart it, leaving a gap where the mic hears nothing. Android throws error_speech_timeout after silence, OEM builds vary, and every restart risks clipping the user's first word. The package's issue tracker has years of threads on "the correct way to restart the listener" — because there isn't a clean one.

3. Barge-in is the feature you can't build with these packages

Natural voice UX means the user can interrupt — "Actually, the blue one—" — and the assistant stops talking and listens. That's Flutter voice AI barge-in, and it needs two hard things at once:

• An open mic while TTS is playing. But then the mic hears the assistant's own voice, and your recognizer starts transcribing the bot talking to itself. Fixing that requires acoustic echo cancellation (AEC) tuned against the exact audio being played — telephony-grade signal processing, not a pub.dev afternoon.
• Instant playback cutoff when genuine human speech is detected, plus rewinding conversation state so the model knows it was cut off mid-sentence.

The common workaround — muting the mic while the assistant speaks — is precisely what creates the walkie-talkie feel: the user's interruption lands on a dead mic and is lost.

And you still have the boring problems

Your speech and LLM provider keys end up compiled into the app binary unless you build and host a proxy. iOS needs microphone and speech-recognition usage descriptions, web speech APIs behave differently again, and you now operate a real-time audio pipeline instead of shipping your product.

The shortcut: the chat widget you already embed can talk

Widget-Chat is a drop-in AI chat widget for Flutter and FlutterFlow apps that answers users from your own content. Its live voice chat turns that same widget into a voice call: the user taps the mic and the assistant listens and replies out loud, supports barge-in mid-sentence, and shows live captions during the call. It's the same widget and the same dashboard as text chat, on iOS, Android, and web. The model provider key never ships in your app.

So the integration is two steps: embed the chat widget once, then enable Voice.

Step 1: Embed WidgetChat in your FlutterFlow app

Widget-Chat comes as the widget_chat package. In FlutterFlow you add it as a custom pub dependency and use it in a custom widget (custom code needs a paid FlutterFlow plan). The FlutterFlow guide walks through it step by step.

For comparison, this is the hand-built text chat the widget replaces: a custom action that reads Server-Sent Events from your own streaming endpoint (POST /v1/chat/stream on your backend). In FlutterFlow, open Custom Code → Custom Actions, create an action, and add http under Pubspec Dependencies:

import 'dart:convert';
import 'package:http/http.dart' as http;

// FlutterFlow Custom Action.
// Pubspec dependency: http (latest from pub.dev)
Future<void> streamWidgetChatReply(
  String message,
  Future Function(String partialReply) onUpdate,
) async {
  final request = http.Request(
    'POST',
    Uri.parse('https://api.example.com/v1/chat/stream'),
  )
    ..headers['Content-Type'] = 'application/json'
    // Match the request fields to what
    // your own backend expects.
    ..body = jsonEncode({'message': message});

  final response = await http.Client().send(request);
  final buffer = StringBuffer();

  await response.stream
      .transform(utf8.decoder)
      .transform(const LineSplitter())
      .forEach((line) {
    if (line.startsWith('data: ')) {
      buffer.write(line.substring(6));
      onUpdate(buffer.toString()); // push into page state → UI streams in
    }
  });
}

Bind onUpdate to a page-state variable behind your chat UI and the assistant's reply streams in token by token. That gets you text only. To add voice chat to a FlutterFlow app on top of it you would still need the audio pipeline from the first half of this post, which is why the widget route is shorter: voice lives in the same widget as text.

One platform note: since voice uses the microphone, make sure your FlutterFlow project's iOS permission settings include a microphone usage description — iOS requires one for any mic access.

Step 2: Flip on Voice in the dashboard

In the Widget-Chat dashboard, open your project and switch voice on. Live voice is part of the paid plans and is toggled per project.

No app update, no new packages, no pipeline code. The mic entry point appears in the widget you already embedded, and your FlutterFlow AI agent's voice is managed from the dashboard from then on.

What the call feels like for users

Compared with the DIY loop, the differences users actually notice:

When DIY still makes sense

Be honest with yourself here. If you're building a voice product — your own wake words, custom audio models, offline recognition — you'll eventually need to own the pipeline, and packages like speech_to_text and flutter_tts are good building blocks for learning it. But if what you want is an AI voice agent in your FlutterFlow app that answers support and product questions out loud, building Flutter speech-to-speech AI infrastructure from scratch is months of audio engineering for a worse result than flipping one dashboard toggle.

Or skip the plumbing

Live voice is the hardest part of a hand-built assistant: series latency, recognizer timeouts and barge-in with echo cancellation all have to work at once. Widget-Chat is a drop-in AI chat widget for Flutter and FlutterFlow. Add the widget_chat package, pass your project key, and the whole chat screen is there: live voice chat with barge-in and live captions (paid plans, toggled per project in the dashboard), text answers from your own content in the same widget, and a free plan with no credit card to start. The FlutterFlow guide covers the custom widget setup.

ChatWidget(
  configuration: BotConfiguration(
    projectSecretKey: 'your-project-secret-key',
    userID: 'user-123',
  ),
)

Create a free Widget-Chat account and have it running in your app today.

flutter pub add widget_chat