Content Moderation

Multimodal vector search is a powerful tool for content moderation. Large datasets often contain undersirable content. This can take many forms such as harmful images, copyrighted content, NSFW images, spam, or other forms of potentially inappropriate, undersired, or low quality content.

Why Content Moderation Matters

We encourage anyone exposing vector search to the public, especially where the dataset is large and may contain unknown or user generated content, to use the techniques discussed here to explore the data and identify any potential issues.

Traditional keyword or text-only vector search can often fail to discover much of this content, especially with user generated content. This is because often these items are tagged with text that purposefully obfuscates the true nature of the content. Image however tend to be non-obfuscated as without multimodal vector search they are not searchable.

Techniques for Content Moderation

The semantic nature of multimodal vector search means that the task of identifying problematic content comes down to searching the index with examples of the content you wish to identify and/or phrases which describe the content you wish to identify.

For example, NSFW content can be identified effectively with a combinations of keywords and reference images. In the below example we also add a negative weight to terms which are not necessarily NSFW but may exist nearby to NSFW content in the vector space (this reduces false positives).

import marqo

mq = marqo.Client()
index_name = "my-first-index"

results = mq.index(index_name).search(
    {
        "NSFW, nude": 1.0,  # keywords
        "https://url/for/NSFW_image_1.png": 1.0,  # reference image 1
        "https://url/for/NSFW_image_2.png": 1.0,  # reference image 2
        "short shorts, underwear": -0.4,  # reduce false positives by negating terms
    },
)

Likewise copyrighted content (which is often obfuscated in the text) can be easily identified with simple keywords.

import marqo

mq = marqo.Client()
index_name = "my-first-index"

results = mq.index(index_name).search(
    {"the office": 1.0, "michael scott, the office": 0.4},
)

Outputs from these sorts of queries can be used to visually screen content or be used as a candidate selection for other downstream systems which are not tractable over the entire dataset.