Commit Graph
117 Commits
Author SHA1 Message Date
Xin Wang 9e4dfbe596 feat: support image inputs in assistant tests 2026-08-23 23:46:18 +08:00
Xin Wang d003ee77a5 refactor: remove legacy assistant compatibility 2026-08-18 10:07:18 +08:00
Xin Wang abfcd4cd19 fix: support photo client tool in debug preview 2026-08-14 18:28:52 +08:00
Xin Wang 5be21f5ccc feat: organize tools and enable qwen realtime calls 2026-08-14 00:15:00 +08:00
Xin Wang eb111a3890 feat: unify debug runtime timelines 2026-08-13 09:30:43 +08:00
Xin Wang 94a02d7e2b fix: support vision prompt modes 2026-08-12 19:13:04 +08:00
Xin Wang 09b07751b6 feat: add configurable vision focus prompts 2026-08-12 17:29:30 +08:00
Xin Wang 7560f9a525 feat: add dynamic variables and startup result handling in batch run schemas
- Introduced dynamic_variables field in BatchRunCreate schema to support flexible input types.
- Added BatchStartupResult schema to encapsulate startup state, including greeting and auto-confirmed messages.
- Updated BatchRunCaseOut schema to include startup results, enhancing the structure for test run outputs.
- Enhanced test run orchestration to prepare and validate dynamic configurations, ensuring isolated variable management across test cases.
- Implemented changes in the frontend to accommodate new data structures for improved test case execution and result tracking.
2026-08-12 16:55:25 +08:00
Xin Wang 79a239489a feat: enhance call end coordination with state management
- Added support for tracking call end states (pending, speaking, cancelled) in CallEndCoordinator.
- Introduced a callback mechanism for state changes to allow external handling of call end states.
- Updated pipeline and tests to accommodate new state management features, ensuring proper handling of call end scenarios.
2026-08-12 14:25:24 +08:00
Xin Wang d0bb5b3868 Make end call tool can be cancelled 2026-08-12 09:18:40 +08:00
Xin Wang fc026e765e fix: record message confirmations in context 2026-08-11 14:15:10 +08:00
Xin Wang 86639692ba feat: implement OpenAI-compatible Realtime API with authentication and management features
- Added support for public Realtime API, including new routes for managing API keys and handling WebRTC connections.
- Introduced RealtimeApiKey model and associated CRUD operations for admin management of API keys.
- Implemented authentication mechanisms for API keys and client secrets.
- Enhanced environment configuration with new secrets for Realtime API.
- Created OpenAIRealtime session management and event processing for real-time interactions.
- Updated schemas and settings to accommodate new features and ensure compatibility with existing systems.
2026-08-11 10:05:55 +08:00
Xin Wang 19e8c8c108 feat: add persisted batch text testing 2026-08-10 13:49:24 +08:00
Xin Wang b2e99baa19 Implement webhook 2026-08-07 14:08:49 +08:00
Xin Wang 39ee5ef96e refactor(backend): change prompt field type from varchar(8192) to text
Update the 'prompt' field in the Assistant model to use a Text type for improved flexibility. This change is accompanied by a migration script to alter the database schema accordingly.
2026-08-07 11:30:58 +08:00
Xin Wang 56a32e81ca Add analysis feature 2026-08-07 10:59:47 +08:00
Xin Wang 4a07abd7fc fix(llm): adapt async tool results for compatible APIs 2026-08-06 10:22:05 +08:00
Xin Wang aaca128f82 fix(workflow): track fixed speech at transport output 2026-08-06 10:21:36 +08:00
Ubuntu 5cb13a3cac Merge branch 'main' of https://gitea.xiaowang.eu.org/wx44wx/ai-video-fullstack 2026-08-06 08:35:25 +08:00
Xin Wang 90bb27a377 fix: release input mute after async client tools 2026-08-06 00:12:34 +08:00
UbuntuandClaude 8bd781242a fix(workflow): prevent untracked BotStoppedSpeakingFrame from consuming end-node tracked-speech counter
The CallEndCoordinator.observe() decremented _tracked_speeches on every
BotStoppedSpeakingFrame, not just the one corresponding to tracked
(end-node) speech.  When a preceding LLM utterance finished between
track_speech() and the actual fixed-speech playback, the counter was
consumed prematurely, ending the call before the end-node message played.

Gate the decrement behind a _current_speech_is_tracked flag that is set
only when BotStartedSpeakingFrame follows a pending tracked-speech start.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 22:35:06 +08:00
Xin Wang 0c0a51da95 Improve image upload in debug drawer 2026-08-05 21:28:51 +08:00
Xin WangandCursor e36ca308b8 feat: support uploaded images in debug voice preview
Allow paste/drag temporary image assets so vision turns work without a live camera frame.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-05 19:52:37 +08:00
Xin Wang 317a0600bc feat: persist photos and enrich conversation history 2026-08-05 16:14:03 +08:00
Xin Wang 0e9a1bb857 fix: cancel pipeline on peer disconnect 2026-08-05 14:34:20 +08:00
Xin Wang 0456d2b4ae Configure self-hosted STUN for remote WebRTC 2026-08-05 14:05:22 +08:00
Xin Wang 16156028c9 fix: manage WebRTC pipeline shutdown 2026-08-05 13:48:29 +08:00
Xin Wang 10bea48d72 feat: add realtime workflow vision tools 2026-08-05 13:35:00 +08:00
Xin Wang 11374238f3 fix(realtime): complete workflow message transitions 2026-08-05 11:25:50 +08:00
Xin Wang 55381c5162 fix(realtime): unblock fixed speech after tool calls 2026-08-05 09:28:21 +08:00
Xin Wang 59588ba88d feat(workflow): add edge-tool routing and realtime runtime 2026-08-04 22:29:04 +08:00
Xin Wang 74a8be2357 refactor: unify system tools as resources 2026-08-04 17:05:26 +08:00
Xin Wang 731a372df9 feat: add system tools and state updates 2026-08-04 15:27:39 +08:00
Xin Wang 5bf5987fe4 feat: add prompt opening behavior modes 2026-08-04 13:29:21 +08:00
Xin Wang 78bba090a2 feat: separate prompt opening and entry behavior 2026-08-04 12:21:07 +08:00
Xin Wang 1af1cd7fed feat: trigger prompt reply after opening confirmation 2026-08-04 11:49:49 +08:00
Xin Wang 775e4058bc fix: relay confirmation interrupts past muted input 2026-08-04 11:12:34 +08:00
Xin Wang da828aca41 Try to fix confirm and interrupt 2026-08-04 10:19:59 +08:00
Xin Wang a16ecd8e01 fix: interrupt message output before tool result 2026-08-04 09:30:52 +08:00
Xin Wang c3cb410c17 fix: await message confirmation interruption 2026-08-03 17:11:43 +08:00
Xin Wang 67389fc5a1 fix: honor strict workflow agent entry mode 2026-08-03 16:34:26 +08:00
Xin Wang 06719e4801 fix: interrupt message speech on confirmation 2026-08-03 16:30:18 +08:00
Xin Wang 8064bb7152 refactor(workflow): simplify edge configuration 2026-08-03 15:17:01 +08:00
Xin Wang ef434c80b2 fix(workflow): prevent fixed speech continuation 2026-08-03 13:02:31 +08:00
Xin Wang 4c43e167db feat(workflow): enhance message stages and image routing 2026-08-03 12:38:02 +08:00
Xin Wang f3439b21d1 feat: route workflow image inputs natively 2026-08-03 10:55:57 +08:00
Xin Wang f5c36a62aa feat: make workflow messages resumable 2026-08-03 07:54:51 +08:00
Xin Wang 9daa46ed4d feat: add deterministic message interaction stages 2026-08-03 07:10:41 +08:00
Xin Wang 479a516546 fix: isolate startup tools and preview dialogs 2026-08-02 00:07:35 +08:00
Xin Wang 0331f8cd07 feat: add prompt startup actions and shared vision config 2026-08-01 23:31:14 +08:00