Commit Graph
100 Commits
Author SHA1 Message Date
Xin Wang 9a4d48a86e F3: connect Prompt tools and web task controls to native Jobs 2026-10-23 16:14:17 +08:00
Xin Wang 33fde56edd F2: persist native job projections and expose task controls 2026-10-23 16:03:40 +08:00
Xin Wang bae5ec3c58 F1: add correlated OpenCode HTTP and SSE task adapter 2026-10-23 15:57:33 +08:00
Xin Wang 82d479f9e1 F0: establish native Pipecat background job spine 2026-10-23 15:55:08 +08:00
Xin Wang d45be1e686 chore: upgrade Pipecat to 1.13.0 and adapt MCP 2.x 2026-10-23 14:03:42 +08:00
Xin Wang 81a6fdc30c feat: remove OpenCode assistant mode 2026-10-23 09:43:17 +08:00
Xin Wang 2649d64b55 feat: harden public OpenAI API contracts
Quality gate / frontend (push) Canceled after 0s
Quality gate / backend (push) Canceled after 0s
2026-09-07 14:25:48 +08:00
Xin Wang 8025ed8aa0 fix: harden realtime pipeline lifecycle 2026-09-07 08:27:52 +08:00
Xin Wang 43f2cc445e feat: add stateful responses API 2026-09-07 00:15:32 +08:00
Xin Wang 9e4dfbe596 feat: support image inputs in assistant tests 2026-08-23 23:46:18 +08:00
Xin Wang 0966f382b6 feat: implement auto-follow viewport for live test results
- Added AutoFollowViewport component to automatically scroll to the latest test results.
- Integrated auto-follow functionality into AutoTestLivePanel for improved user experience during test execution.
- Introduced logic to manage following state and handle scroll events effectively.
2026-08-18 10:32:35 +08:00
Xin Wang d003ee77a5 refactor: remove legacy assistant compatibility 2026-08-18 10:07:18 +08:00
Xin Wang abfcd4cd19 fix: support photo client tool in debug preview 2026-08-14 18:28:52 +08:00
Xin Wang 36253de864 feat: improve tool configuration UX 2026-08-14 08:48:03 +08:00
Xin Wang 5be21f5ccc feat: organize tools and enable qwen realtime calls 2026-08-14 00:15:00 +08:00
Xin Wang eb111a3890 feat: unify debug runtime timelines 2026-08-13 09:30:43 +08:00
Xin Wang 7c18cea16d refactor: use dropdown for vision prompt mode 2026-08-12 21:38:42 +08:00
Xin Wang 94a02d7e2b fix: support vision prompt modes 2026-08-12 19:13:04 +08:00
Xin Wang 09b07751b6 feat: add configurable vision focus prompts 2026-08-12 17:29:30 +08:00
Xin Wang 7560f9a525 feat: add dynamic variables and startup result handling in batch run schemas
- Introduced dynamic_variables field in BatchRunCreate schema to support flexible input types.
- Added BatchStartupResult schema to encapsulate startup state, including greeting and auto-confirmed messages.
- Updated BatchRunCaseOut schema to include startup results, enhancing the structure for test run outputs.
- Enhanced test run orchestration to prepare and validate dynamic configurations, ensuring isolated variable management across test cases.
- Implemented changes in the frontend to accommodate new data structures for improved test case execution and result tracking.
2026-08-12 16:55:25 +08:00
Xin Wang 79a239489a feat: enhance call end coordination with state management
- Added support for tracking call end states (pending, speaking, cancelled) in CallEndCoordinator.
- Introduced a callback mechanism for state changes to allow external handling of call end states.
- Updated pipeline and tests to accommodate new state management features, ensuring proper handling of call end scenarios.
2026-08-12 14:25:24 +08:00
Xin Wang d0bb5b3868 Make end call tool can be cancelled 2026-08-12 09:18:40 +08:00
Xin Wang fc026e765e fix: record message confirmations in context 2026-08-11 14:15:10 +08:00
Xin Wang 86639692ba feat: implement OpenAI-compatible Realtime API with authentication and management features
- Added support for public Realtime API, including new routes for managing API keys and handling WebRTC connections.
- Introduced RealtimeApiKey model and associated CRUD operations for admin management of API keys.
- Implemented authentication mechanisms for API keys and client secrets.
- Enhanced environment configuration with new secrets for Realtime API.
- Created OpenAIRealtime session management and event processing for real-time interactions.
- Updated schemas and settings to accommodate new features and ensure compatibility with existing systems.
2026-08-11 10:05:55 +08:00
Xin Wang cf32ea605d feat: mock continuous voice test editor 2026-08-10 14:46:29 +08:00
Xin Wang 19e8c8c108 feat: add persisted batch text testing 2026-08-10 13:49:24 +08:00
Xin Wang 5b8b9fd097 Remove .DS_Store from version control and update .gitignore. Enhance BatchCaseDetailDrawer and BatchRunCompletedView components to support error handling and improved UI for displaying test case statuses. Add functionality for editing and rerunning test cases. Update TestCasesPage to handle initial case selection. Refactor related components for better maintainability. 2026-08-10 09:33:29 +08:00
Xin WangandCursor ba6c7c6be3 Expand test case fixed-input editor and batch case detail split view.
Unify multi-turn behaviors (reply/tool), overall expectation, and input-mode picker while keeping non-text modes as coming soon; batch completed/running use a half-and-half detail panel below the top bar.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-09 17:08:53 +08:00
Xin WangandCursor 21a25874a9 feat(frontend): implement batch testing views and status components
Add components for batch testing, including BatchRunCompletedView and BatchRunRunningView, to display test results and progress. Introduce BatchCaseStatusBadge and BatchPhasePill for visual status representation. Implement mock data handling for batch run snapshots and case statuses, enhancing the user experience for managing batch tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-08 17:37:48 +08:00
Xin WangandCursor 398b20778d feat(frontend): add Test Suite management and Next Reply case editor
Replace the test-cases placeholder with suite list/detail flows, multi-select and drag reorder, and an edit-only single-step reply editor with keyword/LLM assertions.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-08 17:03:12 +08:00
Xin WangandCursor 2f755969d3 feat(frontend): add auto test functionality to debug drawer
Introduce a new auto test mode in the debug drawer, allowing users to run automated tests with mock data. This includes the addition of AutoTestPanel, AutoTestLivePanel, and AutoTestRunBar components for managing test cases and displaying results. The debug mode can now switch between manual and auto testing, enhancing the debugging experience.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-07 16:09:18 +08:00
Xin WangandCursor 36117b976e feat(frontend): split test assistant nav into cases and batch pages
Make 测试助手 a sidebar section like 监控观察, with placeholder routes for 测试用例 and 批量测试.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-07 16:00:50 +08:00
Xin Wang 39ee5ef96e refactor(backend): change prompt field type from varchar(8192) to text
Update the 'prompt' field in the Assistant model to use a Text type for improved flexibility. This change is accompanied by a migration script to alter the database schema accordingly.
2026-08-07 11:30:58 +08:00
Xin Wang a9d54e7c2b feat(frontend): enhance ProfilePage with language selection and scrollable sections
Add language selection functionality and implement scrollable sections for the ProfilePage. Introduce a new SectionAnchorTabs component for navigation between Webhook and Language sections, improving user experience and organization of settings.
2026-08-07 11:28:05 +08:00
Xin Wang 7575afaeee Splite docs to developer and user and bring profile link back 2026-08-07 11:13:45 +08:00
Xin WangandCursor 95e04d8c94 Split Mintlify docs into user and developer sections.
Organize product guides under docs/user and setup/integration under docs/developer, with tabbed navigation for clearer audience targeting.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-07 11:09:10 +08:00
Xin Wang 0c0a51da95 Improve image upload in debug drawer 2026-08-05 21:28:51 +08:00
Xin Wang 3a4cb9522e Merge branch 'main' of https://gitea.xiaowang.eu.org/wx44wx/ai-video-fullstack 2026-08-05 09:38:03 +08:00
Xin Wang 249dbe1fe3 Update workflow canvas 2026-08-05 09:37:43 +08:00
Xin Wang 206f732d5c Add effect on debug drawer start button 2026-08-05 09:27:17 +08:00
Xin WangandClaude bee1e97cdb feat(frontend): compact theme selector, adaptive prompt editor layout
- Theme dropdown: 亮色/暗色/随系统 as one row of three icons
- Prompt editor: sections centered (max-w-7xl) when debug drawer is
  closed, full-width aligned with tab edges when open
- Debug drawer opens by default
- Prompt textarea: 8 → 16 rows

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 08:54:24 +08:00
Xin Wang b88e37789a feat(frontend): refine app shell and assistant editors 2026-08-05 08:19:16 +08:00
Xin Wang 0ce5d9774e fix preview dialog and waveform behavior 2026-08-04 23:04:17 +08:00
Xin Wang 1902ffc240 Merge branch 'main' of https://gitea.xiaowang.eu.org/wx44wx/ai-video-fullstack 2026-08-04 20:19:01 +08:00
Xin Wang 74a8be2357 refactor: unify system tools as resources 2026-08-04 17:05:26 +08:00
Xin Wang 731a372df9 feat: add system tools and state updates 2026-08-04 15:27:39 +08:00
Xin Wang 5bf5987fe4 feat: add prompt opening behavior modes 2026-08-04 13:29:21 +08:00
Xin Wang 78bba090a2 feat: separate prompt opening and entry behavior 2026-08-04 12:21:07 +08:00
Xin Wang 1af1cd7fed feat: trigger prompt reply after opening confirmation 2026-08-04 11:49:49 +08:00
Xin Wang 775e4058bc fix: relay confirmation interrupts past muted input 2026-08-04 11:12:34 +08:00
Xin Wang da828aca41 Try to fix confirm and interrupt 2026-08-04 10:19:59 +08:00
Xin Wang caa2ff65fa feat(workflow): add drawer anchor navigation 2026-08-04 09:41:52 +08:00
Xin Wang a16ecd8e01 fix: interrupt message output before tool result 2026-08-04 09:30:52 +08:00
Xin Wang d068927b53 feat(workflow): add floating directed edges 2026-08-04 09:22:36 +08:00
Xin Wang c3cb410c17 fix: await message confirmation interruption 2026-08-03 17:11:43 +08:00
Xin Wang 67389fc5a1 fix: honor strict workflow agent entry mode 2026-08-03 16:34:26 +08:00
Xin Wang 06719e4801 fix: interrupt message speech on confirmation 2026-08-03 16:30:18 +08:00
Xin Wang 8064bb7152 refactor(workflow): simplify edge configuration 2026-08-03 15:17:01 +08:00
Xin Wang dd40bfb90a feat: show client tools in debug preview 2026-08-03 14:00:33 +08:00
Xin Wang ef434c80b2 fix(workflow): prevent fixed speech continuation 2026-08-03 13:02:31 +08:00
Xin Wang 4c43e167db feat(workflow): enhance message stages and image routing 2026-08-03 12:38:02 +08:00
Xin Wang b0991f239e fix(frontend): clean up voice preview listeners 2026-08-03 12:30:49 +08:00
Xin Wang 841d7b8fac Update AGENTS.md 2026-08-03 10:56:30 +08:00
Xin Wang f3439b21d1 feat: route workflow image inputs natively 2026-08-03 10:55:57 +08:00
Xin Wang 2e84de0798 feat(workflow): edit node names in drawer header 2026-08-03 10:52:25 +08:00
Xin Wang cff591aa13 feat(frontend): embed prompt dynamic variables section 2026-08-03 10:46:56 +08:00
Xin Wang b0f82142a5 feat(workflow): show variables in drawer 2026-08-03 09:58:01 +08:00
Xin Wang 807afb3099 feat(frontend): add top prompt anchor navigation 2026-08-03 09:56:03 +08:00
Xin Wang 02c88b36e6 feat(workflow): edit nodes on click 2026-08-03 09:52:03 +08:00
Xin Wang f5c36a62aa feat: make workflow messages resumable 2026-08-03 07:54:51 +08:00
Xin Wang e3035abcc2 feat(workflow): edit edges on click 2026-08-03 07:42:41 +08:00
Xin Wang c241e815cc feat(workflow): add node duplication 2026-08-03 07:32:43 +08:00
Xin Wang 9daa46ed4d feat: add deterministic message interaction stages 2026-08-03 07:10:41 +08:00
Xin Wang 479a516546 fix: isolate startup tools and preview dialogs 2026-08-02 00:07:35 +08:00
Xin Wang 0331f8cd07 feat: add prompt startup actions and shared vision config 2026-08-01 23:31:14 +08:00
Xin Wang b747144ff1 feat: add workflow action outcomes and tracing 2026-08-01 11:21:31 +08:00
Xin Wang ad5ff061bb feat: add session-scoped client tool waits 2026-07-31 23:53:47 +08:00
Xin Wang f155f98e6e feat: add workflow action runtime policies 2026-07-31 23:30:04 +08:00
Xin Wang c2f0f5eb04 feat: add session updates and message dialogs 2026-07-31 00:01:01 +08:00
Xin Wang 913435785e feat: add configurable client tools and photo input 2026-07-30 19:06:03 +08:00
Xin Wang 510a277b5a feat(qwen-audio): enhance user transcript handling in QwenAudioRealtimeService
- Added support for managing user transcripts, including pending states and timestamps.
- Implemented methods to handle user transcript completion and deferred assistant messages.
- Updated event handling to ensure user transcripts are emitted before assistant responses.
- Enhanced tests to verify the correct order of transcript and assistant message emissions during user interactions.
2026-07-29 17:01:02 +08:00
Xin Wang 5ef376b657 refactor(workflow): streamline workflow settings management
- Introduced new utility functions `settingsFromWorkflowGraph` and `workflowGraphWithSettings` to encapsulate settings extraction and graph updates.
- Updated `useWorkflowEditorState` to utilize the new utility functions for improved clarity and maintainability.
- Refactored `AssistantPage` and `WorkflowCanvas` components to leverage the new settings management functions, ensuring consistent handling of workflow settings across the application.
- Removed redundant settings handling code, enhancing overall code cleanliness.
2026-07-29 15:09:00 +08:00
Xin Wang f027ed99b7 feat(mcp): enhance MCP transport support and add server management UI
- Extend McpTransport to support "sse" in schemas.
- Refactor McpToolClient to handle both "streamable_http" and "sse" transports.
- Introduce McpServerDialog for managing MCP server configurations, including transport settings and tool synchronization.
- Replace McpServersSection with the new dialog component for improved server management.
- Add tests for MCP transport handling and server dialog functionality.
2026-07-19 12:04:11 +08:00
Xin Wang c54dac403b Refactor AssistantPage to streamline agent platform handling
- Remove the draftAgentPlatform state and related logic, simplifying the creation process for assistant types.
- Default the agent platform to "dify" when creating an assistant of type "智能体平台".
- Replace the platform selection buttons with a ResourceSelectField for improved user experience and maintainability.
- Update the confirmType function to reflect the changes in platform handling.
2026-07-19 10:28:35 +08:00
Xin Wang cd319d27b9 merge: add workflow vision configuration 2026-07-18 00:10:28 +08:00
Xin Wang 28c6380d8a feat(workflow): add per-agent vision configuration 2026-07-18 00:00:28 +08:00
Xin Wang e39bb48ba8 feat: add MCP tool integration 2026-07-18 00:00:06 +08:00
Xin Wang bdf3d3dd9c Refactor workflow agent and routing components for improved functionality
- Introduce WorkflowAgentStage to manage agent stage configurations and enhance interaction with the workflow engine.
- Implement WorkflowEdgeEvaluator for priority-aware edge evaluation, improving routing decisions based on conditions and user turns.
- Update WorkflowBrain to handle user turns and routing more effectively, ensuring agents cannot have only one default path.
- Enhance CallEndCoordinator to track speech events and manage call termination based on queued speech.
- Add new models and output handling for workflow interactions, improving clarity and maintainability.
- Update tests to validate the new routing logic and agent behavior under various scenarios.
2026-07-17 22:37:15 +08:00
Xin Wang 162a3d8bec Refactor workflow routing and greeting management in Brain classes
- Update WorkflowBrain to handle greeting playback more effectively, ensuring that the initial greeting completes before transitioning to the first node.
- Introduce new methods for managing greeting states and conditions, enhancing the interaction flow for user turns.
- Refactor WorkflowLLMRouter to improve routing logic and ensure proper handling of conditional paths.
- Enhance tests to verify the correct behavior of greeting management and routing under various scenarios, including waiting for audio playback to finish.
- Update frontend components to reflect changes in edge handling and improve user experience in workflow configurations.
2026-07-17 22:01:42 +08:00
Xin Wang 34c0d12d2a Refactor AssistantPage component to support agent platform integration
- Remove unused FastGpt and Dify form types, consolidating them into a single AgentPlatformForm.
- Update type definitions and state management to accommodate the new agent platform structure.
- Modify functions to handle agent platform forms and ensure proper integration with existing workflows.
- Enhance assistant type options and update related logic for improved clarity and maintainability.
2026-07-17 20:03:01 +08:00
Xin Wang e78dc4088a Implement response handling and speech tracking in CallEnd functionality
- Add begin_response and finish_after_current_speech methods to CallEndCoordinator for better management of speech events.
- Update PromptBrain to utilize new methods, ensuring proper handling of generated closing speech and tool-only calls.
- Enhance tests to verify the correct behavior of speech tracking and response handling in various scenarios, including waiting for audio to finish before ending calls.
- Introduce a new test suite for CallEndCoordinator to validate the interaction with speech frames.
2026-07-14 13:43:42 +08:00
Xin Wang d069e5282e Enhance greeting context management in Brain classes
- Introduce greeting context handling in BaseBrain and WorkflowBrain to manage assistant greetings effectively.
- Implement prepare_greeting_context method to add greeting messages to the local context while preserving playback order.
- Update pipeline event handling to ensure greeting timestamps are maintained until the client is ready.
- Enhance tests to verify the correct behavior of greeting context management in various scenarios.
2026-07-14 13:26:47 +08:00
Xin Wang 35cbee4786 Update GenericNode component styles for improved responsiveness and localization
- Adjust border radius calculation in GenericNode for better visual consistency.
- Update displayed text in GenericNode to enhance localization by removing the English label.
2026-07-14 13:02:38 +08:00
Xin Wang 6e8fc70c5a Refactor pipeline and assistant page components for improved structure and performance
- Remove unused imports and classes from pipeline.py to streamline the codebase.
- Consolidate dynamic variable handling and workflow management in AssistantPage, enhancing clarity and maintainability.
- Update WorkflowEditor to utilize a more modular approach, improving the overall architecture and reducing complexity.
- Enhance the import structure across components for better organization and readability.
2026-07-14 12:59:41 +08:00
Xin Wang 2d6ff5b7aa Add on_client_ready method to BaseBrain and WorkflowBrain
- Implement on_client_ready in BaseBrain to handle client-visible state after the app message channel is ready.
- Extend WorkflowBrain with on_client_ready to replay state that may have been emitted before WebRTC data was ready.
- Update pipeline to call on_client_ready when a client connects.
- Enhance tests to verify the correct behavior of on_client_ready in WorkflowBrain.
2026-07-14 11:20:18 +08:00
Xin Wang f74040adf3 Enhance conversation history and runtime variable management
- Update ConversationRecorder to include source and nodeId metadata in transcripts for better context tracking.
- Introduce optional variable handling in DynamicVariableStore, allowing for unset variables to be rendered as empty without raising errors.
- Refactor WorkflowBrain to apply turn configurations and manage interaction policies dynamically, improving agent responsiveness.
- Implement tests to ensure proper handling of updated session variables and workflow metadata in various scenarios.
2026-07-14 11:08:11 +08:00
Xin Wang 665f727796 Merge branch 'feature/workflow' 2026-07-14 09:39:00 +08:00
Xin Wang 72856bf3a7 Enhance workflow routing and agent configuration management
- Introduce WorkflowLLMRouter for pre-response LLM routing, allowing agents to determine the appropriate function to call based on user input.
- Implement UserTurnRoutingProcessor to manage user turns before reaching the LLM, ensuring proper routing and handling of user messages.
- Refactor WorkflowBrain to integrate new routing logic and enhance agent stage configuration, including entry modes and resource management.
- Update service factory to support dynamic LLM resource configuration based on workflow settings.
- Add tests for new routing functionality and ensure proper handling of user messages in various scenarios.
2026-07-14 09:36:28 +08:00
Xin Wang 32aef14ddb Add workflow support and enhance runtime configuration in models and services
- Introduce RuntimeModelResource and RuntimeKnowledgeBase classes to manage workflow resources.
- Update AssistantConfig to include workflow_model_resources and workflow_knowledge_bases for better integration.
- Refactor validation and processing logic in routes and services to accommodate workflow types.
- Implement dynamic variable support for workflow assistants and enhance graph normalization.
- Add ToolExecutor for reusable tool execution across different assistant types.
- Update various services to ensure compatibility with new workflow features and improve error handling.
2026-07-13 16:13:27 +08:00
Xin WangandCursor 6108b00007 Improve Element2 copilot WebRTC tooling and enlarge the audio probe panel.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-13 14:47:04 +08:00