Skip to content

Bring your own CLIP model

Most customers can get great performance from publicly available CLIP models. However, some use cases will benefit even more from a model fine-tuned for their domain specific task. In this circumstance, you should use your own model with fine-tuned weights and parameters. It is very convenient to incorporate your own model in Marqo as long as your model belongs to one of the following frameworks:

To use your fine-tuned model, here are the detailed steps:

1. Fine-tune your model

The first step is to fine-tune your model using the frameworks mentioned above. Here we use Open CLIP framework as an example. You should follow the guide to fine-tune your own model and store the trained model (checkpoint) as a *.pt file.

2. Upload your model to a server

You need to upload your model (the *.pt file) to a cloud storage (e.g., Amazon S3, GitHub) and use the downloading address to reference it in Marqo.

3. Use your model in Marqo

You need to create an index in Marqo with you custom model. For Open CLIP, the code is:

settings = {
    "index_defaults": {
        "treat_urls_and_pointers_as_images": True,
        "model": 'generic-clip-test-model-1',
            "model_properties": {
            "name": "ViT-B-32-quickgelu",
                "dimensions": 512,
                "url": "",
                "type": "open_clip",
        "normalize_embeddings": True,
response = mq.create_index("my-own-clip", settings_dict=settings)
You can check Generic CLIP for more detailed information and settings in "model_properties".