- Introduced dynamic_variables field in BatchRunCreate schema to support flexible input types.
- Added BatchStartupResult schema to encapsulate startup state, including greeting and auto-confirmed messages.
- Updated BatchRunCaseOut schema to include startup results, enhancing the structure for test run outputs.
- Enhanced test run orchestration to prepare and validate dynamic configurations, ensuring isolated variable management across test cases.
- Implemented changes in the frontend to accommodate new data structures for improved test case execution and result tracking.
- Added support for public Realtime API, including new routes for managing API keys and handling WebRTC connections.
- Introduced RealtimeApiKey model and associated CRUD operations for admin management of API keys.
- Implemented authentication mechanisms for API keys and client secrets.
- Enhanced environment configuration with new secrets for Realtime API.
- Created OpenAIRealtime session management and event processing for real-time interactions.
- Updated schemas and settings to accommodate new features and ensure compatibility with existing systems.
- Introduce WorkflowLLMRouter for pre-response LLM routing, allowing agents to determine the appropriate function to call based on user input.
- Implement UserTurnRoutingProcessor to manage user turns before reaching the LLM, ensuring proper routing and handling of user messages.
- Refactor WorkflowBrain to integrate new routing logic and enhance agent stage configuration, including entry modes and resource management.
- Update service factory to support dynamic LLM resource configuration based on workflow settings.
- Add tests for new routing functionality and ensure proper handling of user messages in various scenarios.
- Introduce RuntimeModelResource and RuntimeKnowledgeBase classes to manage workflow resources.
- Update AssistantConfig to include workflow_model_resources and workflow_knowledge_bases for better integration.
- Refactor validation and processing logic in routes and services to accommodate workflow types.
- Implement dynamic variable support for workflow assistants and enhance graph normalization.
- Add ToolExecutor for reusable tool execution across different assistant types.
- Update various services to ensure compatibility with new workflow features and improve error handling.
- Add new HTTP endpoints for handling WebRTC offers and ICE candidates, enhancing the signaling process for voice interactions.
- Introduce dynamic variable decoding from request headers to support flexible offer payloads.
- Refactor existing WebSocket handling to accommodate new offer processing logic.
- Update frontend dependencies to include Pipecat client libraries for improved WebRTC transport management.
- Streamline voice preview functionality by integrating SmallWebRTCTransport for better media handling.
- Introduce dynamic variable definitions in AssistantConfig and Assistant models, allowing for flexible prompt customization.
- Implement validation for dynamic variable names and types in the schema.
- Update backend services and routes to handle dynamic variables in assistant configurations and runtime processing.
- Enhance frontend components to support dynamic variable definitions, including a new editor for managing variables.
- Add tests to ensure proper functionality and validation of dynamic variables in various scenarios.
- Introduce new fields for knowledge retrieval configuration in AssistantConfig and Assistant models, including mode, top_n, and score_threshold.
- Implement KnowledgeRetrievalConfig schema with validation for top_n.
- Update backend services and routes to handle knowledge retrieval settings.
- Enhance frontend components to support knowledge retrieval configuration, including a new dialog for advanced settings.
- Add tests for knowledge retrieval configuration validation and description generation.
- Add new models for `KnowledgeDocument` and `KnowledgeChunk` to manage document ingestion and chunking.
- Implement S3-compatible storage integration for knowledge documents, allowing for file uploads and retrieval.
- Introduce API endpoints for managing knowledge bases and documents, including creation, deletion, and searching.
- Update frontend components to support knowledge base configuration and document management, improving user interaction.
- Enhance backend services for knowledge processing and retrieval, ensuring robust handling of document statuses and errors.
- Introduce a new `turnConfig` field in `AssistantConfig` and `Assistant` models to manage user interaction settings.
- Implement `TurnConfig`, `BargeInConfig`, `VadConfig`, and `TurnDetectionConfig` schemas to define turn management strategies.
- Update the backend to handle turn configuration in the database and during assistant operations.
- Enhance frontend components with a `TurnConfigEditor` for configuring turn settings, including VAD and barge-in strategies.
- Modify existing pages to integrate turn configuration, improving user experience and interaction capabilities.
- Implement a new API endpoint for duplicating tools in the backend.
- Add a duplicate_tool function in ComponentsToolsPage to handle tool duplication requests.
- Update the UI to include a dropdown menu option for duplicating tools, improving user experience.
- Refactor the remove function to accept a resource object instead of an ID for better clarity and maintainability.
- Implement a new API endpoint for deleting conversations in the backend.
- Enhance the HistoryPage component to include a dropdown menu for conversation actions, allowing users to delete conversations.
- Update the pagination logic to handle conversation removal and improve user experience during deletion.
- Adjust the loading state and error handling for the delete operation, ensuring smooth interaction.
- Introduce new database models for conversation sessions, messages, and artifacts to support conversation history tracking.
- Implement API routes for listing conversations and retrieving detailed conversation data, enhancing user interaction with historical records.
- Add a conversation recorder service to persist conversation messages in real-time without disrupting ongoing calls.
- Update the frontend to display conversation history, including filtering and sorting options, improving user experience.
- Enhance the pipeline to integrate conversation history recording seamlessly during interactions.
- Introduce a new `RuntimeTool` model to encapsulate tool data for runtime sessions, including attributes like `id`, `name`, `function_name`, `type`, and `description`.
- Update the `AssistantConfig` model to include a list of reusable tools, allowing for better management of tools within assistant configurations.
- Modify the `config_resolver` service to fetch and resolve tools associated with assistants, ensuring they are available during runtime.
- Refactor tool-related CRUD operations in the `tools` route to support the new runtime execution model, enhancing the overall tool management system.
- Update documentation and comments to reflect changes in tool execution and configuration handling, improving clarity for future development.
- Introduce a new `Tool` model and `AssistantToolBinding` for managing reusable tools within the application.
- Implement CRUD operations for tools in the new `tools` route, allowing for the creation, retrieval, updating, and deletion of tools.
- Update the `Assistant` model to include a list of tool IDs, enabling assistants to utilize these tools.
- Enhance the backend routes to synchronize tool bindings with assistants, ensuring proper management of tool associations.
- Add frontend components for tool management, including a tool picker in the assistant configuration, improving user experience in tool selection.
- Create a mobile call page to facilitate video calls, integrating camera and microphone selection for enhanced communication capabilities.
- Update API definitions to include tool-related types and operations, ensuring consistency across the application.
- Add a migration script to create the necessary database tables for tools and bindings, supporting the new functionality.
- Introduce a new `auth` module with login, logout, and user verification endpoints for a single admin user.
- Update backend routes to require admin authentication for sensitive operations, enhancing security.
- Modify frontend components to include an authentication provider and gate, ensuring only authorized users can access the application.
- Implement a login page for admin access, improving user experience and security management.
- Update API request handling to redirect unauthorized users to the login page, ensuring proper access control.
- Extend the AssistantConfig model to include agent_interface_type, agent_values, and agent_secrets for enhanced agent management.
- Update ModelType to include "Agent" for broader capability support.
- Seed new agent models and configurations in the database for Dify, FastGPT, and OpenCode applications.
- Modify the Assistant routes to validate and handle agent capabilities.
- Implement agent resource resolution in the config resolver for runtime configuration.
- Enhance the interface catalog with agent-specific fields and definitions.
- Update frontend components to manage agent configurations, including form handling and state management for agent-related inputs.
- Update vision model authorization logic to rely solely on server-side configuration, removing client offer dependency for vision capabilities.
- Enhance clarity in comments regarding the role of server configurations in enabling vision understanding.
- Update `requirements.txt` to include Pillow for image handling.
- Refactor vision model validation logic in `voice_webrtc.py` to improve error handling for unsupported image input.
- Introduce new functions in `pipeline.py` for image data processing and analysis using vision models.
- Implement `VisionCaptureProcessor` to manage video frame requests for auxiliary vision model analysis.
- Enhance the pipeline to support image input requests and integrate vision model responses into the processing flow.
- Introduce new fields in AssistantConfig, schemas, and database models to support vision capabilities, including `vision_enabled` and `vision_model_resource_id`.
- Enhance validation logic in routes to ensure proper handling of vision models and their requirements.
- Update the AssistantPage and related frontend components to include options for enabling vision understanding and selecting appropriate vision models.
- Modify database seed scripts to include vision-related data for assistants, ensuring consistent setup.
- Refactor related functions to integrate vision model handling in the audio-visual processing pipeline.
- Introduce a new `support_image_input` field in model resources, allowing models to indicate support for image input.
- Update the backend models, schemas, and database seed scripts to accommodate the new field.
- Enhance the AssistantConfig and related routes to handle image input capabilities, ensuring proper validation and error handling.
- Modify the frontend components to include toggles for enabling visual understanding and filtering models based on image input support.
- Implement necessary adjustments in the voice preview and pipeline to integrate video stream handling alongside audio functionalities.
- Introduce a new `.env.example` file for environment variable setup, including `PUBLIC_IP`, `TURN_SECRET`, and `TURN_URLS` for WebRTC TURN server configuration.
- Update `docker-compose.yaml` to support TURN server deployment with necessary environment variables and commands.
- Enhance backend configuration and routes to include WebRTC ICE server settings, allowing for STUN/TURN server integration.
- Implement a new service for managing WebRTC ICE server configurations, providing credentials for TURN when configured.
- Modify frontend API to fetch ICE server configurations dynamically, improving support for cross-network voice preview.
Xin Wang