- Added support for public Realtime API, including new routes for managing API keys and handling WebRTC connections.
- Introduced RealtimeApiKey model and associated CRUD operations for admin management of API keys.
- Implemented authentication mechanisms for API keys and client secrets.
- Enhanced environment configuration with new secrets for Realtime API.
- Created OpenAIRealtime session management and event processing for real-time interactions.
- Updated schemas and settings to accommodate new features and ensure compatibility with existing systems.
Unify multi-turn behaviors (reply/tool), overall expectation, and input-mode picker while keeping non-text modes as coming soon; batch completed/running use a half-and-half detail panel below the top bar.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add components for batch testing, including BatchRunCompletedView and BatchRunRunningView, to display test results and progress. Introduce BatchCaseStatusBadge and BatchPhasePill for visual status representation. Implement mock data handling for batch run snapshots and case statuses, enhancing the user experience for managing batch tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the test-cases placeholder with suite list/detail flows, multi-select and drag reorder, and an edit-only single-step reply editor with keyword/LLM assertions.
Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce a new auto test mode in the debug drawer, allowing users to run automated tests with mock data. This includes the addition of AutoTestPanel, AutoTestLivePanel, and AutoTestRunBar components for managing test cases and displaying results. The debug mode can now switch between manual and auto testing, enhancing the debugging experience.
Co-authored-by: Cursor <cursoragent@cursor.com>
Update the 'prompt' field in the Assistant model to use a Text type for improved flexibility. This change is accompanied by a migration script to alter the database schema accordingly.
Add language selection functionality and implement scrollable sections for the ProfilePage. Introduce a new SectionAnchorTabs component for navigation between Webhook and Language sections, improving user experience and organization of settings.
Organize product guides under docs/user and setup/integration under docs/developer, with tabbed navigation for clearer audience targeting.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Theme dropdown: 亮色/暗色/随系统 as one row of three icons
- Prompt editor: sections centered (max-w-7xl) when debug drawer is
closed, full-width aligned with tab edges when open
- Debug drawer opens by default
- Prompt textarea: 8 → 16 rows
Co-Authored-By: Claude <noreply@anthropic.com>
- Added support for managing user transcripts, including pending states and timestamps.
- Implemented methods to handle user transcript completion and deferred assistant messages.
- Updated event handling to ensure user transcripts are emitted before assistant responses.
- Enhanced tests to verify the correct order of transcript and assistant message emissions during user interactions.
- Introduced new utility functions `settingsFromWorkflowGraph` and `workflowGraphWithSettings` to encapsulate settings extraction and graph updates.
- Updated `useWorkflowEditorState` to utilize the new utility functions for improved clarity and maintainability.
- Refactored `AssistantPage` and `WorkflowCanvas` components to leverage the new settings management functions, ensuring consistent handling of workflow settings across the application.
- Removed redundant settings handling code, enhancing overall code cleanliness.
- Extend McpTransport to support "sse" in schemas.
- Refactor McpToolClient to handle both "streamable_http" and "sse" transports.
- Introduce McpServerDialog for managing MCP server configurations, including transport settings and tool synchronization.
- Replace McpServersSection with the new dialog component for improved server management.
- Add tests for MCP transport handling and server dialog functionality.
- Remove the draftAgentPlatform state and related logic, simplifying the creation process for assistant types.
- Default the agent platform to "dify" when creating an assistant of type "智能体平台".
- Replace the platform selection buttons with a ResourceSelectField for improved user experience and maintainability.
- Update the confirmType function to reflect the changes in platform handling.
- Introduce WorkflowAgentStage to manage agent stage configurations and enhance interaction with the workflow engine.
- Implement WorkflowEdgeEvaluator for priority-aware edge evaluation, improving routing decisions based on conditions and user turns.
- Update WorkflowBrain to handle user turns and routing more effectively, ensuring agents cannot have only one default path.
- Enhance CallEndCoordinator to track speech events and manage call termination based on queued speech.
- Add new models and output handling for workflow interactions, improving clarity and maintainability.
- Update tests to validate the new routing logic and agent behavior under various scenarios.
- Update WorkflowBrain to handle greeting playback more effectively, ensuring that the initial greeting completes before transitioning to the first node.
- Introduce new methods for managing greeting states and conditions, enhancing the interaction flow for user turns.
- Refactor WorkflowLLMRouter to improve routing logic and ensure proper handling of conditional paths.
- Enhance tests to verify the correct behavior of greeting management and routing under various scenarios, including waiting for audio playback to finish.
- Update frontend components to reflect changes in edge handling and improve user experience in workflow configurations.
- Remove unused FastGpt and Dify form types, consolidating them into a single AgentPlatformForm.
- Update type definitions and state management to accommodate the new agent platform structure.
- Modify functions to handle agent platform forms and ensure proper integration with existing workflows.
- Enhance assistant type options and update related logic for improved clarity and maintainability.
- Add begin_response and finish_after_current_speech methods to CallEndCoordinator for better management of speech events.
- Update PromptBrain to utilize new methods, ensuring proper handling of generated closing speech and tool-only calls.
- Enhance tests to verify the correct behavior of speech tracking and response handling in various scenarios, including waiting for audio to finish before ending calls.
- Introduce a new test suite for CallEndCoordinator to validate the interaction with speech frames.
- Introduce greeting context handling in BaseBrain and WorkflowBrain to manage assistant greetings effectively.
- Implement prepare_greeting_context method to add greeting messages to the local context while preserving playback order.
- Update pipeline event handling to ensure greeting timestamps are maintained until the client is ready.
- Enhance tests to verify the correct behavior of greeting context management in various scenarios.
- Adjust border radius calculation in GenericNode for better visual consistency.
- Update displayed text in GenericNode to enhance localization by removing the English label.
- Remove unused imports and classes from pipeline.py to streamline the codebase.
- Consolidate dynamic variable handling and workflow management in AssistantPage, enhancing clarity and maintainability.
- Update WorkflowEditor to utilize a more modular approach, improving the overall architecture and reducing complexity.
- Enhance the import structure across components for better organization and readability.
- Implement on_client_ready in BaseBrain to handle client-visible state after the app message channel is ready.
- Extend WorkflowBrain with on_client_ready to replay state that may have been emitted before WebRTC data was ready.
- Update pipeline to call on_client_ready when a client connects.
- Enhance tests to verify the correct behavior of on_client_ready in WorkflowBrain.
- Update ConversationRecorder to include source and nodeId metadata in transcripts for better context tracking.
- Introduce optional variable handling in DynamicVariableStore, allowing for unset variables to be rendered as empty without raising errors.
- Refactor WorkflowBrain to apply turn configurations and manage interaction policies dynamically, improving agent responsiveness.
- Implement tests to ensure proper handling of updated session variables and workflow metadata in various scenarios.
- Introduce WorkflowLLMRouter for pre-response LLM routing, allowing agents to determine the appropriate function to call based on user input.
- Implement UserTurnRoutingProcessor to manage user turns before reaching the LLM, ensuring proper routing and handling of user messages.
- Refactor WorkflowBrain to integrate new routing logic and enhance agent stage configuration, including entry modes and resource management.
- Update service factory to support dynamic LLM resource configuration based on workflow settings.
- Add tests for new routing functionality and ensure proper handling of user messages in various scenarios.
- Introduce RuntimeModelResource and RuntimeKnowledgeBase classes to manage workflow resources.
- Update AssistantConfig to include workflow_model_resources and workflow_knowledge_bases for better integration.
- Refactor validation and processing logic in routes and services to accommodate workflow types.
- Implement dynamic variable support for workflow assistants and enhance graph normalization.
- Add ToolExecutor for reusable tool execution across different assistant types.
- Update various services to ensure compatibility with new workflow features and improve error handling.
- Change the placeholder text in LoginPage from "admin" to "请输入用户名" for better localization.
- Adjust the styling of the floating video panel in MobileCallPage to improve responsiveness and maintain aspect ratio across different screen sizes.
- Update LoginPage to initialize username state as an empty string for improved user experience.
- Introduce floatingVideoRef in MobileCallVisualWorkspace to dynamically manage video panel positioning.
- Adjust position calculations for the floating video panel to account for its dimensions, enhancing drag-and-drop functionality and ensuring proper placement within the viewport.
- Update CallView type from "camera" to "video" for clarity.
- Introduce MobileVideoSurface component to handle video stream rendering.
- Implement MobileCallVisualWorkspace for managing chat and video views with drag-and-drop functionality for the video panel.
- Refactor state management to streamline view transitions between chat and video modes.
- Enhance network metrics handling in useVoicePreview for better performance monitoring during calls.
- Remove the SegmentedIconGroup for view selection and streamline view handling in DebugDrawer.
- Introduce DebugVisionWorkspace component to manage video and chat views more effectively.
- Update DebugVoicePanel to accept onViewChange prop for better view state management.
- Enhance video handling with dynamic positioning and drag functionality for the floating video panel.
- Remove direct camera start/stop calls and replace them with streamlined video stream handling.
- Update DebugVideoPanel to accept a streamOverride for better flexibility in video source management.
- Adjust MobileCallPage to utilize the new videoStream state, enhancing the call experience.
- Introduce videoStream state management in useVoicePreview for improved media handling during voice interactions.
- Add new HTTP endpoints for handling WebRTC offers and ICE candidates, enhancing the signaling process for voice interactions.
- Introduce dynamic variable decoding from request headers to support flexible offer payloads.
- Refactor existing WebSocket handling to accommodate new offer processing logic.
- Update frontend dependencies to include Pipecat client libraries for improved WebRTC transport management.
- Streamline voice preview functionality by integrating SmallWebRTCTransport for better media handling.
- Introduce dynamic variable definitions in AssistantConfig and Assistant models, allowing for flexible prompt customization.
- Implement validation for dynamic variable names and types in the schema.
- Update backend services and routes to handle dynamic variables in assistant configurations and runtime processing.
- Enhance frontend components to support dynamic variable definitions, including a new editor for managing variables.
- Add tests to ensure proper functionality and validation of dynamic variables in various scenarios.
- Introduce new fields for knowledge retrieval configuration in AssistantConfig and Assistant models, including mode, top_n, and score_threshold.
- Implement KnowledgeRetrievalConfig schema with validation for top_n.
- Update backend services and routes to handle knowledge retrieval settings.
- Enhance frontend components to support knowledge retrieval configuration, including a new dialog for advanced settings.
- Add tests for knowledge retrieval configuration validation and description generation.
- Add a guideline to reference frontend/DESIGN.md for interface design considerations.
- Maintain existing emphasis on writing clear thoughts before coding for better understanding and maintainability.
- Add DialogDescription components to provide context in the settings and search dialogs.
- Refactor layout for better spacing and organization of form elements in the settings dialog.
- Update search results display to improve user experience with clearer messaging and styling.
- Introduce footer buttons for closing dialogs, enhancing user interaction.
- Update the main knowledge page to support a list mode.
- Add new pages for creating and editing knowledge bases, utilizing the ComponentsKnowledgePage component with appropriate modes.
- Introduce a new API method for retrieving knowledge base details by ID.
- Enhance the ComponentsKnowledgePage to handle different modes (list, create, edit) and improve user interaction with knowledge bases.
- Add new models for `KnowledgeDocument` and `KnowledgeChunk` to manage document ingestion and chunking.
- Implement S3-compatible storage integration for knowledge documents, allowing for file uploads and retrieval.
- Introduce API endpoints for managing knowledge bases and documents, including creation, deletion, and searching.
- Update frontend components to support knowledge base configuration and document management, improving user interaction.
- Enhance backend services for knowledge processing and retrieval, ensuring robust handling of document statuses and errors.
- Introduce a new `turnConfig` field in `AssistantConfig` and `Assistant` models to manage user interaction settings.
- Implement `TurnConfig`, `BargeInConfig`, `VadConfig`, and `TurnDetectionConfig` schemas to define turn management strategies.
- Update the backend to handle turn configuration in the database and during assistant operations.
- Enhance frontend components with a `TurnConfigEditor` for configuring turn settings, including VAD and barge-in strategies.
- Modify existing pages to integrate turn configuration, improving user experience and interaction capabilities.
- Introduce new fields `dify_api_url` and `dify_api_key` in `AssistantConfig` for Dify API integration.
- Update `requirements.txt` to include `dify-client-python` for Dify SDK support.
- Modify `config_resolver` to handle Dify connection information.
- Add a new `globalNode` type in workflow specifications to provide unified settings across workflows.
- Enhance node specifications with additional constraints and default values for better configuration management.
- Update frontend components to support the new `globalNode` type and its properties, improving workflow editor functionality.
- Add a new guideline emphasizing the importance of articulating thoughts in simple language before coding.
- Maintain existing focus on building with MVP principles for better code maintainability and readability.
- Create a new JSON schema file for documentation settings.
- Add core features documentation detailing assistant configuration and model integration.
- Include an index page introducing the AI 视频助手 and its platform components.
- Document integration methods for model services and external agents.
- Provide a quickstart guide for setting up the local development environment using Docker.
- Add light and dark logo SVG files for branding purposes.
- Introduce a new state variable `callEnded` in `useVoicePreview` to manage call termination more effectively.
- Modify `MobileCallPage` to adjust the `inCall` condition based on `callEnded`, ensuring accurate UI updates during and after calls.
- Update rendering logic to display appropriate messages based on call status, enhancing user feedback during call transitions.
- Refactor button visibility to only show when in an active call, improving user interface clarity.
- Update the effect to reset the view to "camera" when the call status is "idle" or "failed", improving user experience during call interruptions.
- Modify the rendering condition for the MobileCallTranscript to only display when inCall is true, ensuring chat messages are shown appropriately during active calls.
- Implement a new API endpoint for duplicating tools in the backend.
- Add a duplicate_tool function in ComponentsToolsPage to handle tool duplication requests.
- Update the UI to include a dropdown menu option for duplicating tools, improving user experience.
- Refactor the remove function to accept a resource object instead of an ID for better clarity and maintainability.
- Implement a new API endpoint for deleting conversations in the backend.
- Enhance the HistoryPage component to include a dropdown menu for conversation actions, allowing users to delete conversations.
- Update the pagination logic to handle conversation removal and improve user experience during deletion.
- Adjust the loading state and error handling for the delete operation, ensuring smooth interaction.
- Introduce new database models for conversation sessions, messages, and artifacts to support conversation history tracking.
- Implement API routes for listing conversations and retrieving detailed conversation data, enhancing user interaction with historical records.
- Add a conversation recorder service to persist conversation messages in real-time without disrupting ongoing calls.
- Update the frontend to display conversation history, including filtering and sorting options, improving user experience.
- Enhance the pipeline to integrate conversation history recording seamlessly during interactions.
- Introduce a new `NetworkQualityIndicator` component to visually represent network quality status during calls.
- Integrate the `NetworkQualityIndicator` into `AssistantPage` and `MobileCallPage`, enhancing user awareness of connection quality.
- Update `use-voice-preview` hook to manage network quality metrics, improving overall call experience and performance monitoring.
Xin Wang