NVIDIA Triton Connection

NVIDIA Triton Integration Prerequisites #

• NVIDIA Triton model should be hosted in user’s server.

• Obtain the hosted model’s connection URL.

• Ensure the model services are up and running before using them for inferences within Gathr.

To access the Connections page, simply navigate through the Connections (main-menu) > Create Connection.

To set up a connection with NVIDIA Triton tracking server from Gathr, you need to configure these fields.

Component Type #

Select NVIDIA Triton as the ‘Component Type’ to create the connection for NVIDIA Triton.

Connection Name #

Provide a unique name for your NVIDIA Triton connection. The saved connection can be identified in Gathr with the name you provide.

Scope #

Define the connection scope to customize their accessibility.

• Organization: Accessible to organization users across all Gathr projects for usage in applications.

• Project: Accessible to organization users limited to projects that are specified by the connection owner for usage in applications.

• Private: Accessible only to the connection owner for usage in applications.

Connection URL #

NVIDIA Triton server URL.

Connection Type #

Option to either connect securely using SSH tunnel or with a direct connection.

Upon selecting SSH tunnel option the below fields need to be provided: SSH Key, SSH Host, SSH User, SSH Port.

SSH Key #

The SSH key is to be given to authenticate the associated SSH host and SSH user.

SSH Host #

Public IP address of the SSH host.

SSH User #

Default SSH username.

SSH Port #

Port number used for SSH connections. The default SSH port is 22 unless otherwise specified.

Authentication Type #

Select the method used by your environment to authenticate access to the Triton server.

• None: No authentication method is used.

• Basic: Authenticate via a username and password.

• Token: Authenticate through a token-based system.

For basic authentication type:

Username #

Enter the username required for basic authentication to access the Triton server.

Provide a username/password combination that grants access. Check with your NVIDIA Triton server administrator for the authentication method in place and obtain the necessary credentials.

Password #

Enter the password associated with the provided username.

For token-based authentication type:

Token Id #

Enter the unique identifier for the token.

Token #

Enter the authentication token associated with the provided Token Id.

Enable SSL #

It is set to False by default.

Set this option to True, if the resource that is to be requested is SSL-enabled.

Keystore select option #

If SSL is set to True, choose how the SSL-enabled resource should be verified.

Either a keystore file or a certificate file needs to be uploaded based on the chosen verification method.

The Keystore Password or Certificate Alias should then be provided as per the type of file uploaded for verification.

Model Repository #

Specify the model repository where model’s artifacts are stored.

• S3: Provide the S3 connection details where model’s artifacts are stored.

• Fetch from server: Automatically fetch the model repository details from the Triton server.

S3 Connection #

An S3 connection can be selected from the list if you have created and saved connection details earlier. Or create one as explained in the topic - Amazon S3 Connection →

Bucket Name #

Provide the bucket name where the model artifacts are stored.

Path #

Directory path for the model repository.

Allow Gathr to Load Model #

Enable this option to load models from the selected model repository. To load a model version please navigate to the Models page.

Test Connection #

After entering all the necessary details, it’s important to verify that the connection to the specified resources is correctly configured.

• Click on the Test Connection button.

• If the provided details are correct and all required services are operational, you will receive a confirmation message indicating that the connection to the specified resources is successfully established.

• If any of the provided credentials are incorrect, or if any required services are not running or accessible, you will receive an appropriate message indicating an issue with the connection setup. You will need to review and correct the configuration details accordingly.