Build a voice and vision agent workspace

An agent experience combines a realtime MediaSFU room with an application-owned speech, language, playback, and operator pipeline. The MediaSFU Agents repository shows this composition in React, Angular, React Native, Expo, and Flutter.

This is a blueprint, not a copy-and-paste tutorial. Use the headless room guide, your SDK guide, and the repository's complete starter when you build.

Build this: compare the runnable starter and its platform evidence. View source: MediaSFU Agents.

The user journey

1. Authenticate the person and ask your backend to create or join a room.
2. Connect a headless MediaSFU room without giving it ownership of your layout.
3. Let the participant choose microphone, camera, and supported screen sharing. Render your own media surface and keep remote audio mounted.
4. Send permitted audio or image input to the agent pipeline and show active, paused, completed, and failed states.
5. Stop the agent, leave the room, or request a human handoff.

MediaSFU carries room membership, transports, producers, consumers, and media events. Your application owns provider selection, transcripts, escalation rules, and business records.

Interface shapes

The reference repository offers four app-owned presentations:

• Workspace: media, transcript, response, room controls, metrics, and the live pipeline.
• Voice widget: compact start, mute, and end controls with an optional expanded view.
• Chat starters: support, sales, and meeting-copilot presets with optional voice input and a human-handoff callback.
• Classic: the earlier transcript, chat, media modal, and control bar.

A compact widget still needs the same permission, retry, audio, and cleanup behavior as a full workspace.

Package paths in the starters

These are the package declarations in the repository's current starter projects. Each starter stands on its own; use the package-specific guide for its API.

Starter	MediaSFU package declaration	Repository behavior
React	mediasfu-reactjs 4.3.0	Headless room, custom agent surfaces, and a two-participant room path
Angular	mediasfu-angular 2.3.1	Angular room handler and custom agent presentation
React Native	mediasfu-reactnative 2.4.0	Native room presentation and an Android API 36 acceptance path
Expo development build	mediasfu-reactnative-expo 2.5.0	Expo room presentation and web/development-build starter
Flutter	mediasfu_sdk 2.3.0	Flutter room handler, custom agent UI, and platform audio routing

The repository records a React two-participant room with decoded remote video and custom agent surfaces. It also records a release React Native APK, room connection, native camera rendering, and intentional end on an Android API 36 emulator. Those records do not measure remote audio playout or a live model response and do not certify every device, codec, or provider.

Keep room and agent state separate

The room controller owns MediaSFU state; the application owns agent state. The backend callbacks are the only place that knows reusable room credentials.

function AgentRoom({ roomId }: { roomId: string }) {
  const [agentState, setAgentState] =
    useState('idle');
  const sourceParameters = useRef({});

  return (
    <MediasfuGeneric
      returnUI={false}
      sourceParameters={sourceParameters.current}
      joinMediaSFURoom={(options) =>
        joinRoomThroughYourBackend(roomId, options)}
      onRoomReady={() => setAgentState('idle')}
    />
  );
}

Use the exact options and callback types exported by your installed package. Render local and remote media in your own component, mount every prepared remote-audio renderer, defer parameter publication until after rendering, and read the current snapshot when an action runs.

Human handoff

Treat escalation as an application workflow:

1. Mark the agent response as paused so a late response cannot speak over the operator.
2. Record the request in your authorized backend.
3. Admit or connect the human participant under room policy.
4. Switch the visible and audible source deliberately.

The operation concepts are: ai_agent.session_handoff, agent.operator_takeover, headless.operator_console, participant.list, media.consume, media.track.control, room.reconnect, and room.cleanup.

Backend authority and deployment

A restricted disposable development credential can be used for a private local experiment only when its exposure is understood and it is never committed. For shared builds, external testers, and production rooms, route create and join through an authenticated backend. Validate the caller and role, forward an Idempotency-Key for exact create/join retries, and return only room-scoped data. Keep provider and long-lived MediaSFU credentials server-side.

With MediaSFU Cloud, your backend calls the managed service. With MediaSFU Open, your team operates an already-running server and points the backend at its room endpoint. A localLink or server URL does not install or start MediaSFU Open.

This pattern depends on current-snapshot reads, post-render publication, safe late acknowledgements during teardown, a shared embedded layout boundary, screen/remote/local primary selection, and independent remote-audio mounting.

Recovery and cleanup

• Request microphone/camera permission before starting and show the actual browser or OS error.
• If response streaming fails, stop its playback source and leave the room usable for retry; do not replay cumulative audio.
• After reconnect, restore controls only when a current snapshot is available.
• On handoff, agent stop, participant leave, or host end, remove listeners, timers, queued audio, temporary files, and room media. Confirm backend cleanup before showing an ended state.

Before release

• Test the exact package version on every target device family.
• Exercise create, join, microphone, camera, remote audio, reconnect, agent stop, human handoff, and intentional cleanup with two independent users.
• Keep room and provider credentials out of bundles, URLs, logs, screenshots, and analytics.
• Define transcript, image, audio, consent, retention, and deletion policy in your product backend.