Skip to content

Eject a Loaded Model

Note: This is for Open Source only.

Eject a model from a specific device.

The /models endpoint provides necessary information relating to models loaded in your devices (cpu or cuda). You can check the currently loaded models, and eject a loaded model to free memory.

DELETE '/models?model_name={model_name}&model_device={model_device}'

Path parameters

Name Type Description
model_name String name of the name
model_device String ("cuda" or "cpu") device of the model

Example

Models can be loaded into devices cpu and cuda, respectively.

Note: The best practice is to check the device memory usage using \device api, and check the currently loaded model \models first.

Supposing the results from mq.get_loaded_models() are

{
    "models": [
        {"model_name": "hf/e5-base-v2", "model_device": "cpu"},
        {"model_name": "hf/e5-base-v2", "model_device": "cuda"},
        {"model_name": "open_clip/ViT-B-32/laion2b_s34b_b79k", "model_device": "cpu"},
        {"model_name": "open_clip/ViT-B-32/laion2b_s34b_b79k", "model_device": "cuda"},
        {"model_name": "ViT-B/16", "model_device": "cpu"}
    ]
}
We would like to eject the model "open_clip/ViT-B-32/laion2b_s34b_b79k" from device "cuda".

curl -X DELETE 'http://localhost:8882/models?model_name=open_clip/ViT-B-32/laion2b_s34b_b79k&model_device=cuda'

mq.index("my-index").eject_model(model_name = "open_clip/ViT-B-32/laion2b_s34b_b79k", model_device = "cuda")

Response: 200 OK

{
    "result": "success",
    "message": "successfully eject model_name `open_clip/ViT-B-32/laion2b_s34b_b79k` from device `cuda`"
}

If you try to eject a model that is not loaded into cache, you will get

Response: 404 Not Found

{
    "message":"The model_name `open_clip/ViT-B-32/laion2b_s34b_b79k` device `cuda` is not cached or found",
    "code":"model_not_in_cache", 
    "type":"invalid_request", 
    "link":null
}