Bring your own models

If the models in our registry do not meet your requirements, or you have a custom model that you want to use, you can bring your own model to Marqo. In this section, we will show you how to use your own OpenCLIP models and sentence transformers models in Marqo.

Bring your own OpenCLIP model

Marqo supports you to use your own OpenCLIP models fine-tune under the OpenCLIP framework. To load a custom OpenCLIP model, you need to provide the model properties in the index settings. A full details of the settings in modelProperties are listed below:

Field Name	Type	Default Value	Description
`name`	String	No Default	The name of the model. It can be the architecture (e.g., `"ViT-B-32"`) of the model or the HuggingFace model card starting with `"hf-hub:"`.
`dimensions`	Integer	No Default	The dimension of the embeddings generated by the model.
`type`	String	No Default	The type of the model. It should be `"open_clip"` since we are loading an OpenCLIP model here.
`url`	String (Optional)	`None`	The URL of the model checkpoint. Cannot be provided together with `"modelLocation"`.
`modelLocation`	Dict (Optional)	`None`	The location of the model in S3 or HuggingFace. Cannot be provided together with `"url"`.
`jit`	Boolean (Optional)	`False`	A boolean indicating whether the model is JIT compiled.
`precision`	String (Optional)	`"fp32"`	The precision of the model. It should be either `"fp32"` or `"fp16"`.
`tokenizer`	String (Optional)	`None`	The HuggingFace tokenizer to be loaded. Provide this if you want to overwrite the tokenizer inferred from `name`.
`imagePreprocessor`	String (Optional)	`"OpenCLIP"`	The image preprocess configuration. Must be one of `"SigLIP"`, `"OpenAI"`, `"OpenCLIP"`, `"MobileCLIP"`, or `"CLIPA"`.
`mean`	List[float] (Optional)	`None`	The mean of the image preprocessor. If provided, it will overwrite the loaded configuration.
`std`	List[float] (Optional)	`None`	The standard deviation of the image preprocessor. If provided, it will overwrite the loaded configuration.
`size`	Integer (Optional)	`None`	The size of the image preprocessor. If provided, it will overwrite the loaded configuration.
`note`	String (Optional)	`None`	A place to add notes to your model. This does not affect your model loading process.
`pretrained`	String (Optional)	`None`	A place to indicate the pretrained dataset of your model. This does not affect your model loading process.

Most of the fields are optional and have default values. You can provide the fields you want to customize in the modelProperties. However, you need to provide at least the name, dimensions, and type fields to load a custom OpenCLIP model. There are two ways to load a custom OpenCLIP model in Marqo:

Load from a Hugging Face model card

To load a custom OpenCLIP model from a Hugging Face model card, you need to provide the model card name with the "hf-hub:" in the name, the dimensions of the model in dimensions, and the type of the model in type as "open_clip". Other fields are neglected in this case. This suits the case where you want to load a public model card from Hugging Face.

For example, instead of using loading the Marqo FashionCLIP model from the registry, you can load it from the Hugging Face with the following code:

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "marqo-fashion-clip-custom-load",
    "modelProperties": {
        "name": "hf-hub:Marqo/marqo-fashionCLIP",
        "dimensions": 512,
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("marqo-fashion-clip-custom-load-index", settings_dict=settings)

Load from a checkpoint file

This is the case where you have a custom OpenCLIP model checkpoint file and you want to load it in Marqo. This has the highest flexibility as you can load any custom model you have fine-tuned, from any source, and with any configurations, as long as the architecture is supported by OpenCLIP.

You need to provide the model name in name which is the architecture of the model (e.g., "ViT-B-32", "ViT-L-16-SigLIP"), the dimensions of the model in dimensions, and the type of the model in type as "open_clip".

You have two options to provide the checkpoint file: - 1. Provide the URL of the checkpoint file in url. The url should be accessible by Marqo and link to the checkpoint file with the format of *.pt.

1. Provide the location of the checkpoint file in S3 or Hugging Face in modelLocation. The modelLocation has the following fields:

Field Name	Type	Default Value	Description
`s3`	Dict	No Default	A dictionary with `"Bucket"` and `"Key"` fields to locate the `*.pt` checkpoint
`hf`	Dict	No Default	A dictionary with `"repoId"` and `"filename"` fields to locate the `*.pt` checkpoint
`authRequired`	Bool	`False`	A boolean indicating whether the authentication is required.

If authentication is required, you need to provide the authentication information in when you search or add documents to the index.

You can provide other fields like jit, precision, tokenizer, imagePreprocessor, mean, std, size, note, in the modelProperties to configure your model.

Examples

Here are some examples to load a custom OpenCLIP model in Marqo. Note that if your name has the "hf-hub:" prefix, we will try to load it from Hugging Face and ignore the url and modelLocation fields. Otherwise, if you provide the url or modelLocation, we will load the model from the provided location and treat the name as the model architecture.

Example 1: Load a custom OpenCLIP model from a public URL without configurations

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-own-clip-model",
    "modelProperties": {
        "name": "ViT-B-32",
        "dimensions": 512,
        "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_e32-46683a32.pt",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)

The above code loads a custom OpenCLIP model from a public URL. Note it is the same as loading the model open_clip/ViT-B-32/lainon400m_e32 from the registry. We use the public URL of the model checkpoint as an example.

Example 2: Load a custom OpenCLIP model from a public URL with custom configurations

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-own-clip-model",
    "modelProperties": {
        "name": "ViT-B-16-SigLIP",
        "dimensions": 768,
        "url": "https://huggingface.co/Marqo/marqo-fashionSigLIP/resolve/main/open_clip_pytorch_model.bin",
        "imagePreprocessor": "SigLIP",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)

The above code loads a custom OpenCLIP model from a public URL with custom configurations for the image preprocessor. It is very important to provide the correct imagePreprocessor configuration to match the model architecture as Marqo can not infer the correct configuration from the model name when you load a checkpoint file and will use the default configuration("OpenCLIP"). The imagePreprocessor is set to "SigLIP" in this example to match the model architecture ViT-B-16-SigLIP.

Note this is the same as loading the Marqo FashionSigLIP model from the registry. We use the public URL of the model checkpoint as an example.

Example 3: Load a custom OpenCLIP model from a private S3 bucket with authentication

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-private-clip-model",
    "modelProperties": {
        "name": "ViT-B-32",
        "dimensions": 512,
        "modelLocation": {
            "s3": {
                "Bucket": "my-prive-bucket",
                "Key": "my-private-model-checkpoint.pt",
        },
        "authRequired": True,
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)

model_auth = {
    "s3": {
        "aws_secret_access_key": "my-secret-access-key",
        "aws_access_key_id": "my-access-key-id"
    }
}

mq.index("my-own-clip-model").search("test", model_auth=model_auth)

The above code loads a custom OpenCLIP model from a private S3 bucket with authentication. The authRequired is set to True and you need to provide the authentication information when you search or add documents to the index.

Preloading your model

There may be cases wherein you want to preload (or prewarm, in other terms) your model before using it to index. This can be done by adding your model (with model and modelProperties) to the list of models on startup in your marqo configuration.

The syntax for this can be found in Configuring preloaded models

Bring your own Hugging Face Sentence Transformers models

Marqo supports you to use your own Hugging Face Sentence Transformers models. You can use your own model with fine-tuned weights and parameters. It is very convenient to incorporate your own model in Marqo as long as your Sentence Transformers model follows the Hugging Face Sentence Transformers model format.

To use your fine-tuned model, here are the detailed steps:

1. Fine-tune your model

The first step is to fine-tune your model using the sentence-transformers framework. The fine-tuning guide can be found here.

2. Upload your model to a cloud storage

For the Sentence Transformers model, you should include all your files in a directory including the model weights, tokenzier, configurations etc. You should compress the directory into a single file and upload the file to a cloud storage (e.g., Amazon S3, Hugging Face) and use the downloading address to reference it in Marqo. In addition, you can also create a model card in Hugging Face for your fine-tuned and Marqo can use the model card to download your model.

3. Use your model in Marqo

We provide different entries load your own Sentence Transformers model in Marqo, either from a compressed file or from a Hugging Face model card, with or without authentication.

3.1 Load from a compressed file

You can load your own Sentence Transformers model from a compressed file by specifying modelProperties in your index settings.

# load from a public url
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "url": "https://path/to/your/sbert/model.zip",
        "type": "hf",
    },
}
# load from a s3 bucket
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "s3": {
                "Bucket": s3_bucket,
                "Key": s3_object_key,  # a zip file
            },
            "auth_required": True,
        },
    },
}
# load from a Hugging Face zip file
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "hf": {
                "repo_id": hf_repo_name,
                "filename": hf_object,  # a zip file
            },
            "auth_required": True,  # can be True or False
        },
    },
}
response = mq.create_index(
    "your-own-sentence-transformers-model", settings_dict=settings
)

3.2 Loading from a Hugging Face model card (recommended)

# Loading from a Hugging Face model card with or without authentication using `repo_id` (recommended)
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "dimensions": 384,
        "type": "hf",
        "model_location": {
            "hf": {
                "repo_id": hf_repo_name,
            },
            "auth_required": True,  # can be True or False
        },
    },
}
# Loading from a Hugging Face model card without authentication using `name`
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "name": public_repo_name,
        "dimensions": 384,
        "type": "hf",
    },
}
response = mq.create_index(
    "your-own-sentence-transformers-model", settings_dict=settings
)

Please check authentication-in-search and authentication-in-add-documents for the ways to authenticate your model safely in Marqo.

4. Preloading your model

Check here for how to preload your Sentence Transformers models in Marqo.