OpenAI Compatible Client

OpenAI Compatible Client enables seamless integration with AI models from multiple providers that are compatible with OpenAI’s offerings. The AI providers offering models that are compatible with the OpenAI platform can be easily connected and configured within Gathr.

You can access and leverage a diverse set of powerful AI capabilities, such as natural language processing, image generation, and machine learning from platforms like, SambaNova, Hugging Face, Cohere, Mistral, and more.

Perform a wide range of data-related tasks, all through intuitive and natural language instructions.

Processor Configuration

Configure the processor parameters as explained below.

Connection Name

A connection name can be selected from the list if you have created and saved connection details for OpenAI earlier. Or create one as explained in the topic - OpenAI Compatible Client Connection →

Prompt

A prompt is a concise instruction or query in natural language provided to the OpenAI processor to guide its actions or responses.

In the Prompts section, you have the flexibility to:

  • Choose Predefined Sample Prompts: Discover a set of ready-to-use sample prompts that can kickstart your interactions with the OpenAI processor.

    openai_prompt_01

  • Configuration Options: Customize prompts to suit your specific needs.

  • Save Prompts: Store your preferred prompts for future use.

  • Delete Prompts: Remove prompts that are no longer necessary.

  • Prompt Reset: To reset the prompt, clear the details in the prompt field, restoring it to its default state.

    openai_save_prompt_01

System

Provide high-level instructions and context for the AI model, guiding its behavior and setting the overall tone or role it should play in generating responses.

Note: <|endoftext|> is a document separator that the model sees during training, so if a prompt is not specified, the model will generate a response from the beginning of a new document.

The placeholder {some_key} represents a variable that can be replaced with specific column data. You can map this key to a column in the next section using “EXTRACT INPUTS FROM PROMPT”.

User

The user prompt is a specific instruction or question provided by the user to the AI model, directing it to perform a particular task or provide a response based on the user’s request.

Note: <|endoftext|> is a document separator that the model sees during training, so if a prompt is not specified, the model will generate a response from the beginning of a new document.

The placeholder {some_key} represents a variable that can be replaced with specific column data. You can map this key to a column in the next section using “EXTRACT INPUTS FROM PROMPT”.

Input

The placeholders {__} provided in the prompt can be mapped to columns to replace its value with the placeholder keys.

openai_input_from_prompt_01

Input from prompt

All the placeholders {__} provided in the fields above are extracted here to map them with the column.

Input column

Select the column name to replace its value with the placeholder keys.

Output

The output can be configured to emit data received from input. This configuration includes utilizing Prompts, which can be interpreted as input columns via placeholders, and allows for emitting output either by specifying a column name or parsing it as JSON.

Process Response

Choose how to handle the response. Options:

  • Assign to Column: Store the entire response in a single column.

  • Parse as JSON: Map JSON keys to specific output columns.

Json Key in Response

Enter the JSON keys expected in response and map them to the output column names.

Output Column as JSON

For each JSON key, select or enter the corresponding column name for the data.

Output Column

Specify the column where the entire response will be stored.

Assign JSON Response to Column

JSON Response can be stored as String in the provided column.

Example: Below JSON response can be further transformed using a JSON Parser to extract details like ‘choices’, ‘usage’, and so on.

{
  "choices": [
    {
      "finish_reason": "stop",
      "message": {
        "content": "What would you like to talk about?",
        "role": "assistant"
      }
    }
  ],
  "id": "d35112a5-b872-4363-b6b3-3973bd6784e2",
  "model": "Meta-Llama-3.1-8B-Instruct",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "end_time": 1741854978.7057998,
    "total_tokens": 56
  }
}

RATE CONTROL

Choose how to utilize the OpenAI’s services:

Make Concurrent Requests with Token Limit: You can specify the number of simultaneous requests to be made to OpenAI, and each request can use up to the number of tokens you provide.

This option is suitable for scenarios where you need larger text input for fewer simultaneous requests.

OR

Rate-Limited Requests: Alternatively, you can make a total of specified number of requests within a 60-second window.

This option is useful when you require a high volume of requests within a specified time frame, each potentially processing smaller amounts of text.

Handle Incomplete Responses

Handle Incomplete Responses by automatically retrying up to a specified number of times to ensure complete and accurate output.

No. of Retries

Retry if generated response is incomplete.

Example: For 3 retries, the model would retry generating response up to 3 times of the specified max token limit.

Retry requests

Enabling retry requests allows for automatic resubmission of failed or incomplete requests, enhancing data processing reliability.

No. of Retries

The number of times a request will be automatically retried if the response from OpenAI is not in JSON format or does not have all the entities.

When

Please select the criteria for retry. Select “Any output key is missing” if all keys are mandatory. Else, select the mandatory keys.

Include previous response

Please mark if a previous incorrect response should be added to the retry request messages prompt. Else, leave it unchecked.

Additional User Prompt

Please type prompt text to be considered while retrying the request.

OpenAI Compatible Client Parameters

The parameters described below are configuration settings that govern the behavior and performance of the AI models, influencing how they respond to prompts and generate outputs.

Model Name

Specify the model name to be used.

Max Token

The maximum number of tokens to generate in the chat completion.

The total length of input tokens and generated tokens is limited by the model’s context length.

Please note that it also affects the total tokens consumed per minute, potentially limiting the number of requests you can make. For example, if you set Max Tokens per request to 1500 and the model supports 90,000 tokens per minute, you can only make approximately 40 requests per minute.

Temperature

The sampling temperature to be used between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

It is generally recommended to alter this or top_p but not both.

Add Extra Headers

Allows adding custom headers as key-value pairs.

Add Extra Query

Enables inclusion of extra query parameters as key-value pairs.

Example, { "temperature": 0.7 } to control model’s creativity in responses.

Add Extra Body

Additional data to the request body as key-value pairs.

Example, { "max_tokens": 1000 } to set the response length limit.

Advanced Configuration

Provides more granular control over request parameters as key-value pairs.

Example, { "stop": ["\n"] } to define stop sequences or customize other settings.

If you have any feedback on Gathr documentation, please email us!

← OpenAI Compatible Client Connection
OpenAI Compatible Client Validation →