Skip to content

Transferring Marqo's state

Version compatibility note

Please be aware of the Marqo versions of the old and new Marqo instances. It is safest to transfer state between Marqo containers of the same version.

Marqo versions 0.x.x, 1.x.x, and 2.x.x are not compatible. Attempting to transfer state between these Marqo versions will result in unexpected behavior.

Minor version updates are backward compatible. However, it is still recommended to upgrade just a few minor versions (ideally no more than 2) at a time to reduce the risk of encountering unexpected issues during state transfer.

Change in version 2.9

We changed the volume mounting points of the Marqo image in version 2.9. Prior to version 2.9, we exposed the entire /opt/vespa folder as a volume. Starting with version 2.9, we expose two separate volumes: /opt/vespa/var, which contains data and configs, and /opt/vespa/logs, which contains Vespa logs. This change addresses an issue that can cause Vespa to crash due to missing dependencies when an older version volume is mounted.

Guide

Please follow the steps below to transfer the state to a new Marqo docker container (with embedded Vespa)

From Marqo 2.9 release

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. You can create named volumes to persist data and transfer state across different Marqo versions.

docker volume create --name opt_vespa_var
docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var marqoai/marqo:<version>

If you forget to specify the volume mapping when you run marqo the first time, you can find the name of the anonymous volume using following command if the marqo container is not removed from docker.

export VESPA_VAR_VOLUME=$(docker inspect marqo | jq -r '.[0].Mounts.[] | select (.Destination == "/opt/vespa/var") | .Name')

# To upgrade to a newer version of Marqo
docker rm marqo
docker run --name marqo -it -p 8882:8882 -v $VESPA_VAR_VOLUME:/opt/vespa/var marqoai/marqo:<version>

Optionally, you can also bind-mount a local folder to expose vespa logs for debugging. 

docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var -v $(pwd)/logs:/opt/vespa/logs marqoai/marqo:<version>

Prior to 2.9 release

In order to mount a volume or a local folder created prior to Marqo 2.9 to be used by any newer version, you will need to follow these steps:

If you use bind mounts, you can simply run this: 

docker run --name marqo -it -p 8882:8882 -v <your local folder>/var:/opt/vespa/var marqoai/marqo:<version>

If you use volumes, you need to copy the var folder into a new volume to be used in newer versions of Marqo. 

docker volume create --name opt_vespa_var
# copy var folder in the old volume to the new volume using marqo image. 
docker run --rm -it --entrypoint='' \
           -v <old volume name>:/opt/vespa_old \
           -v opt_vespa_var:/opt/vespa/var \
           marqoai/marqo:<version> sh -c "cd /opt/vespa_old/var ; cp -a . /opt/vespa/var"
docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var marqoai/marqo:<version>