Model Provider Plugin

Build a model provider plugin, from project setup and provider configuration to model implementation, debugging, and publishing

Prerequisites

• Dify CLI
• Basic Python programming skills and understanding of object-oriented programming
• Familiarity with the API documentation of the model provider you want to integrate

Step 1: Create and Configure a New Plugin Project

Initialize the Project

dify plugin init

Choose Model Plugin Template

Select the LLM type plugin template from the available options. This template provides a complete code structure for model integration.

Configure Plugin Permissions

For a model provider plugin, configure the following essential permissions:

• Models: Base permission for model operations.
• LLM: Permission for large language model functionality.
• Storage: Permission for file operations (if needed).

Directory Structure Overview

After initialization, your plugin project has a directory structure similar to this (assuming a provider named my_provider that supports LLM and Embedding):

models/my_provider/
├── models                # Model implementation and configuration directory
│   ├── llm               # LLM type
│   │   ├── _position.yaml  (Optional, controls sorting)
│   │   ├── model1.yaml     # Configuration for specific model
│   │   └── llm.py          # LLM implementation logic
│   └── text_embedding    # Embedding type
│       ├── _position.yaml
│       ├── embedding-model.yaml
│       └── text_embedding.py
├── provider              # Provider-level code directory
│   └── my_provider.py    # Provider credential validation
└── manifest.yaml         # Plugin manifest file

Step 2: Understand Model Configuration Methods

Dify supports two model configuration methods that determine how users interact with your provider’s models:

Predefined Models (predefined-model)

Predefined models require only unified provider credentials. Once a user configures their API key or other authentication details for the provider, they can immediately access all predefined models.

Example: The OpenAI provider offers predefined models like gpt-3.5-turbo-0125 and gpt-4o-2024-05-13. A user only needs to configure their OpenAI API key once to access all these models.

Custom Models (customizable-model)

Custom models require additional configuration for each model instance. This approach is useful when models need individual parameters beyond the provider-level credentials.

Example: Xinference supports both LLM and Text Embedding, but each model has a unique model_uid. Users must configure this model_uid separately for each model they want to use.

The two configuration methods can coexist within a single provider. For instance, a provider might offer some predefined models while also allowing users to add custom models with specific configurations.

Step 3: Create Model Provider Files

Creating a new model provider involves two main components:

• Provider configuration YAML file: Defines the provider’s basic information, supported model types, and credential requirements.
• Provider class implementation: Implements authentication validation and other provider-level functionality.

3.1 Create the Model Provider Configuration File

The provider configuration is a YAML file that declares the provider’s basic information, supported model types, configuration methods, and credential rules. Place it in the root directory of your plugin project.

Here’s an annotated example of the anthropic.yaml configuration file:

# Basic provider identification
provider: anthropic                # Provider ID (must be unique)
label:
  en_US: Anthropic                 # Display name in UI
description:
  en_US: Anthropic's powerful models, such as Claude 3.
  zh_Hans: Anthropic 的强大模型，例如 Claude 3。
icon_small:
  en_US: icon_s_en.svg            # Small icon for provider (displayed in selection UI)
icon_large:
  en_US: icon_l_en.svg            # Large icon (displayed in detail views)
background: "#F0F0EB"             # Background color for provider in UI

# Help information for users
help:
  title:
    en_US: Get your API Key from Anthropic
    zh_Hans: 从 Anthropic 获取 API Key
  url:
    en_US: https://console.anthropic.com/account/keys

# Supported model types and configuration approach
supported_model_types:
  - llm                           # This provider offers LLM models
configurate_methods:
  - predefined-model              # Uses predefined models approach

# Provider-level credential form definition
provider_credential_schema:
  credential_form_schemas:
    - variable: anthropic_api_key  # Variable name for API key
      label:
        en_US: API Key
      type: secret-input           # Secure input for sensitive data
      required: true
      placeholder:
        zh_Hans: 在此输入你的 API Key
        en_US: Enter your API Key
    - variable: anthropic_api_url
      label:
        en_US: API URL
      type: text-input             # Regular text input
      required: false
      placeholder:
        zh_Hans: 在此输入你的 API URL
        en_US: Enter your API URL

# Model configuration
models:
  llm:                            # Configuration for LLM type models
    predefined:
      - "models/llm/*.yaml"       # Pattern to locate model configuration files
    position: "models/llm/_position.yaml"  # File defining display order

# Implementation file locations
extra:
  python:
    provider_source: provider/anthropic.py  # Provider class implementation
    model_sources:
      - "models/llm/llm.py"                 # Model implementation file

Custom Model Configuration

If your provider supports custom models, add a model_credential_schema section defining the additional fields users must configure for each model. This is typical for providers that support fine-tuned models or require model-specific parameters.

Here’s an example from the OpenAI provider:

model_credential_schema:
  model: # Fine-tuned model name field
    label:
      en_US: Model Name
      zh_Hans: 模型名称
    placeholder:
      en_US: Enter your model name
      zh_Hans: 输入模型名称
  credential_form_schemas:
  - variable: openai_api_key
    label:
      en_US: API Key
    type: secret-input
    required: true
    placeholder:
      zh_Hans: 在此输入你的 API Key
      en_US: Enter your API Key
  - variable: openai_organization
    label:
        zh_Hans: 组织 ID
        en_US: Organization
    type: text-input
    required: false
    placeholder:
      zh_Hans: 在此输入你的组织 ID
      en_US: Enter your Organization ID
  # Additional fields as needed...

For the complete model provider YAML specification, see Model Schema.

3.2 Write the Model Provider Code

Next, create a Python file for your provider class in the /provider directory, named after your provider (e.g., anthropic.py).

The provider class must inherit from ModelProvider and implement at least the validate_provider_credentials method:

import logging
from dify_plugin.entities.model import ModelType
from dify_plugin.errors.model import CredentialsValidateFailedError
from dify_plugin import ModelProvider

logger = logging.getLogger(__name__)


class AnthropicProvider(ModelProvider):
    def validate_provider_credentials(self, credentials: dict) -> None:
        """
        Validate provider credentials by testing them against the API.
        
        This method should attempt to make a simple API call to verify
        that the credentials are valid.
        
        :param credentials: Provider credentials as defined in the YAML schema
        :raises CredentialsValidateFailedError: If validation fails
        """
        try:
            # Get an instance of the LLM model type and use it to validate credentials
            model_instance = self.get_model_instance(ModelType.LLM)
            model_instance.validate_credentials(
                model="claude-3-opus-20240229", 
                credentials=credentials
            )
        except CredentialsValidateFailedError as ex:
            # Pass through credential validation errors
            raise ex
        except Exception as ex:
            # Log and re-raise other exceptions
            logger.exception(f"{self.get_provider_schema().provider} credentials validate failed")
            raise ex

Dify calls validate_provider_credentials whenever a user saves provider credentials, so it should:

1. Attempt to validate the credentials by making a simple API call.
2. Return silently if validation succeeds.
3. Raise CredentialsValidateFailedError with a helpful message if validation fails.

For Custom Model Providers

For providers that exclusively use custom models (where each model requires its own configuration), you can implement a simpler provider class. For example, with Xinference:

from dify_plugin import ModelProvider

class XinferenceProvider(ModelProvider):
    def validate_provider_credentials(self, credentials: dict) -> None:
        """
        For custom-only model providers, validation happens at the model level.
        This method exists to satisfy the abstract base class requirement.
        """
        pass

Step 4: Implement Model-Specific Code

After setting up your provider, implement the model-specific code that handles API calls for each model type you support. This involves:

1. Creating model configuration YAML files for each specific model.
2. Implementing the model type classes that handle API communication.

For detailed instructions, see:

• Model Design Rules: Standards for integrating predefined models.
• Model Schema: Standards for model configuration files.

4.1 Define Model Configuration (YAML)

For each specific model, create a YAML file in the appropriate model type directory (e.g., models/llm/) to define its properties, parameters, and features.

Example (claude-3-5-sonnet-20240620.yaml):

model: claude-3-5-sonnet-20240620   # API identifier for the model
label:
  en_US: claude-3-5-sonnet-20240620 # Display name in UI
model_type: llm                     # Must match directory type
features:                           # Special capabilities
  - agent-thought
  - vision
  - tool-call
  - stream-tool-call
  - document
model_properties:                   # Inherent model properties
  mode: chat                        # "chat" or "completion"
  context_size: 200000              # Maximum context window
parameter_rules:                    # User-adjustable parameters
  - name: temperature
    use_template: temperature       # Reference predefined template
  - name: top_p
    use_template: top_p
  - name: max_tokens
    use_template: max_tokens
    required: true
    default: 8192
    min: 1
    max: 8192
pricing:                           # Optional pricing information
  input: '3.00'
  output: '15.00'
  unit: '0.000001'                 # Per million tokens
  currency: USD

4.2 Implement Model Calling Code (Python)

Create a Python file for each model type you support (e.g., llm.py in the models/llm/ directory). This class handles API communication, parameter transformation, and result formatting.

Here’s an example implementation structure for an LLM:

import logging
from typing import Union, Generator, Optional, List
from dify_plugin.provider_kits.llm import LargeLanguageModel # Base class
from dify_plugin.provider_kits.llm import LLMResult, LLMResultChunk, LLMUsage # Result classes
from dify_plugin.provider_kits.llm import PromptMessage, PromptMessageTool # Message classes
from dify_plugin.errors.provider_error import InvokeError, InvokeAuthorizationError # Error classes

logger = logging.getLogger(__name__)

class MyProviderLargeLanguageModel(LargeLanguageModel):
    def _invoke(self, model: str, credentials: dict, prompt_messages: List[PromptMessage],
                model_parameters: dict, tools: Optional[List[PromptMessageTool]] = None,
                stop: Optional[List[str]] = None, stream: bool = True,
                user: Optional[str] = None) -> Union[LLMResult, Generator[LLMResultChunk, None, None]]:
        """
        Core method for invoking the model API.
        
        Parameters:
            model: The model identifier to call
            credentials: Authentication credentials
            prompt_messages: List of messages to send
            model_parameters: Parameters like temperature, max_tokens
            tools: Optional tool definitions for function calling
            stop: Optional list of stop sequences
            stream: Whether to stream responses (True) or return complete response (False)
            user: Optional user identifier for API tracking
            
        Returns:
            If stream=True: Generator yielding LLMResultChunk objects
            If stream=False: Complete LLMResult object
        """
        # Prepare API request parameters
        api_params = self._prepare_api_params(
            credentials, model_parameters, prompt_messages, tools, stop
        )
        
        try:
            # Call appropriate helper method based on streaming preference
            if stream:
                return self._invoke_stream(model, api_params, user)
            else:
                return self._invoke_sync(model, api_params, user)
        except Exception as e:
            # Handle and map errors
            self._handle_api_error(e)
    
    def _invoke_stream(self, model: str, api_params: dict, user: Optional[str]) -> Generator[LLMResultChunk, None, None]:
        """Helper method for streaming API calls"""
        # Implementation details for streaming calls
        pass
        
    def _invoke_sync(self, model: str, api_params: dict, user: Optional[str]) -> LLMResult:
        """Helper method for synchronous API calls"""
        # Implementation details for synchronous calls
        pass
        
    def validate_credentials(self, model: str, credentials: dict) -> None:
        """
        Validate that the credentials work for this specific model.
        Called when a user tries to add or modify credentials.
        """
        # Implementation for credential validation
        pass
        
    def get_num_tokens(self, model: str, credentials: dict, 
                       prompt_messages: List[PromptMessage],
                       tools: Optional[List[PromptMessageTool]] = None) -> int:
        """
        Estimate the number of tokens for given input.
        Optional but recommended for accurate cost estimation.
        """
        # Implementation for token counting
        pass
        
    @property
    def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:
        """
        Define mapping from vendor-specific exceptions to Dify standard exceptions.
        This helps standardize error handling across different providers.
        """
        return {
            InvokeAuthorizationError: [
                # List vendor-specific auth errors here
            ],
            # Other error mappings
        }

The most important method to implement is _invoke, which handles the core API communication. This method should:

1. Transform Dify’s standardized inputs into the format required by the provider’s API.
2. Make the API call with proper error handling.
3. Transform the API response into Dify’s standardized output format.
4. Handle both streaming and non-streaming modes.

Step 5: Debug and Test Your Plugin

Dify supports remote debugging, so you can test your plugin during development:

1. In your Dify instance, go to Plugin Management and click Debug Plugin to get your debug key and server address.

2. Configure your local environment with these values in a .env file:

   INSTALL_METHOD=remote
   REMOTE_INSTALL_URL=<your-dify-host>:5003
   REMOTE_INSTALL_KEY=****-****-****-****-****
   

3. Run your plugin locally with python -m main and test it in Dify.

Step 6: Package and Publish

When your plugin is ready:

1. Package it using the scaffolding tool:

   dify plugin package models/<provider_name>
   

2. Test the packaged plugin locally before submitting.

3. Submit a pull request to the Dify official plugins repository.

For more details on the publishing process, see the Publishing Overview.

Reference Resources

• Quick Integration of a New Model: How to add new models to existing providers.
• Basic Concepts of Plugin Development: Return to the plugin development getting started guide.
• Model Schema: Detailed model configuration specifications.
• General Specifications: Plugin manifest file configuration.
• Dify Plugin SDK Reference: Base classes, data structures, and error types.

---

Edit this page | Report an issue