Recency Scoring Guide

Overview

Recency scoring allows you to boost search results based on how recent a document is, using a timestamp field. This is particularly useful for e-commerce (promoting new products), news (prioritizing recent articles), or any application where freshness matters.

Marqo's recency scoring is Elasticsearch-compatible and provides four decay functions with configurable parameters for fine-grained control over how document scores decay over time.

---

Table of Contents

1. Decay Functions
2. Parameters
3. API Usage
4. Applying Recency to Scores

---

Decay Functions

Recency scoring uses decay functions to calculate a score multiplier based on document age. All functions support:

• Grace period via offset parameter
• Score floor via decay_to parameter
• Elasticsearch-compatible formulas

Mathematical Formulas

Effective Age

Before applying any decay function, the effective age is calculated:

\[ \text{effective_age} = \max(0, \text{age} - \text{offset}) \]

• During the grace period (age < offset), documents get perfect score (1.0)
• After the grace period, decay begins

---

1. Exponential Decay

Best for: News, social media, time-sensitive content

Characteristics: Fast initial decay that slows down over time (logarithmic curve)

Formula:

\[ \lambda = \frac{\ln(\text{decay_to})}{\text{scale}} \]

\[ \text{recency_score} = \begin{cases} 1.0 & \text{if effective_age} = 0 \\ \max(\text{decay_to}, \, e^{\lambda \times \text{effective_age}}) & \text{otherwise} \end{cases} \]

Key Points:

• At age = offset: score = 1.0
• At age = offset + scale: score = decay_to
• Rapid decay for recently published content
• Gradual tapering for older content

---

2. Linear Decay

Best for: General purpose, predictable decay

Characteristics: Constant rate of decay (straight line)

Formula:

\[ \text{recency_score} = \begin{cases} 1.0 & \text{if effective_age} = 0 \\ \max\left(\text{decay_to}, \, \frac{\text{scale} - \text{effective_age} \times (1 - \text{decay_to})}{\text{scale}}\right) & \text{otherwise} \end{cases} \]

Key Points:

• At age = offset: score = 1.0
• At age = offset + scale: score = decay_to
• Uniform, predictable decay rate
• Easy to reason about

---

3. Gaussian Decay

Best for: Content with relevance peak, smooth transitions

Characteristics: S-curve - slow decay near offset, steep drop in middle, then tapering

Formula:

\[ \sigma^2 = -\frac{\text{scale}^2}{2 \times \ln(\text{decay_to})} \]

\[ \text{recency_score} = \begin{cases} 1.0 & \text{if effective_age} = 0 \\ \max\left(\text{decay_to}, \, e^{\frac{\text{effective_age}^2 \times \ln(\text{decay_to})}{\text{scale}^2}}\right) & \text{otherwise} \end{cases} \]

Key Points:

• At age = offset: score = 1.0
• At age = offset + scale: score = decay_to
• Bell curve shape (actually half of it)
• Smooth, natural-looking decay

---

4. Binary Decay

Best for: Hard cutoffs, promotional periods

Characteristics: Step function - no decay until threshold, then immediate drop

Formula:

\[ \text{recency_score} = \begin{cases} 1.0 & \text{if effective_age} < \text{scale} \\ \text{decay_to} & \text{otherwise} \end{cases} \]

Key Points:

• Perfect score until age = offset + scale
• Immediate drop to decay_to after threshold
• No gradual decay
• Useful for "new arrivals" sections

---

Parameters

Core Parameters

Parameter	Type	Default	Description
recencyField	string	required	Name of the timestamp field (Unix timestamp in seconds)
scale	string	"7d"	Time scale duration string. At offset + scale, score reaches decay_to. Format: {number}{unit} where unit is d (days) or h (hours)
offset	string	"0d"	Grace period duration string. Documents within this age get perfect score (1.0). Format: {number}{unit} where unit is d (days) or h (hours)
decayFunction	string	"exponential"	Decay function type: "exponential", "linear", "gaussian", "binary"
decayTo	float	0.5	Target score at offset + scale, also acts as floor. Must be in range (0.0, 1.0]
growFrom	float	null	Starting score for documents with timestamps far in the future. Must be in range (0.0, 1.0]
growFunction	string	null	Growth function type: "exponential", "linear", "gaussian", "binary"
growScale	string	null	Time scale for growth function. Same format as scale
growOffset	string	null	Grace period for future timestamps. Documents within now + growOffset get score 1.0
applyInRankingPhase	string	"all"	Controls which ranking phases recency is applied in. See Ranking Phase Options
addToScoreWeight	float	null	If provided, applies recency as additive instead of multiplicative. See Additive Scoring Mode

Duration String Format

Both scale and offset parameters accept duration strings in the format {number}{unit}:

• Supported units:

  • d - days
  • h - hours

• Examples:

  • "7d" - 7 days
  • "168h" - 168 hours (equivalent to 7 days)
  • "0.5d" - 0.5 days (12 hours)
  • "12h" - 12 hours
  • "1.5h" - 1.5 hours (90 minutes)

• Rules:

  • Decimal values are supported (e.g., "1.5d", "0.5h")
  • Units are case-sensitive (must be lowercase)
  • No spaces allowed between number and unit
  • Different units can be mixed (e.g., offset="48h", scale="7d")

---

API Usage

Basic Recency Scoring

import marqo
import time

# Create client
mq = marqo.Client(url="http://localhost:8882")
index_name = "products"

# Feed some documents with various release_date timestamp
mq.create_index(index_name)
now = int(time.time())
mq.index(index_name).add_documents(
    documents=[
        {
            "_id": "1",
            "title": "a fancy dress",
            "color": "red",
            "price": 24.5,
            "release_date": now + 100_000,
        },  # release in about 1 day
        {
            "_id": "2",
            "title": "blue dress",
            "color": "blue",
            "price": 124.5,
            "release_date": now - 500_000,
        },  # released 6 days ago
        {
            "_id": "3",
            "title": "yellow skirt",
            "color": "yellow",
            "price": 50,
            "release_date": now - 100_000,
        },  # released about 1 day ago
        {
            "_id": "4",
            "title": "a picture of a red dress",
            "color": "red",
            "price": 500.5,
            "release_date": now - 1000,
        },  # just released
    ],
    tensor_fields=["title"],
)

# Search with recency scoring
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "scale": "14d",
            "decayFunction": "exponential",
            "decayTo": 0.5,
        },
    },
)

# Results will include _recency_score field
for hit in results["hits"]:
    print(f"{hit['_id']}: score={hit['_score']}, recency={hit['_recency_score']}")

With Grace Period (Offset)

# Promote products released in last 7 days, then gradual decay over 30 days
index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "offset": "7d",  # 0-7 days: perfect score
            "scale": "30d",  # 7-37 days: decay period
            "decayFunction": "linear",
            "decayTo": 0.3,  # 37+ days: floor at 0.3
        },
    },
)

Binary Boost for New Releases

This setup uses binary decay with additive scoring to give a large, fixed score boost to recently released products. Products released within the last 7 days and up to 1 day into the future receive a recency score of 1.0, so 100 is added to their final score. Products outside this window receive a recency score of 0.01, so only 1 is added to the final score. This effectively creates a "new releases" boost that sharply separates recent products from older ones.

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            # Decay for past documents
            "scale": "7d",  # Binary cutoff at 7 days
            "decayFunction": "binary",
            "decayTo": 0.01,  # Floor for older products
            # Growth for future documents
            "growFrom": 0.01,  # Floor for far-future products
            "growFunction": "binary",
            "growScale": "1d",  # Binary cutoff at 1 day into future
            "growOffset": "0d",
            # Additive scoring
            "addToScoreWeight": 100,  # recency_score × 100 added to final score
        },
    },
)

# Scoring behavior:
# - Within last 7 days or up to 1 day in future: recency = 1.0 → +100 to score
# - Older than 7 days or more than 1 day in future: recency = 0.01 → +1 to score

---

Future Timestamp Handling

By default, documents with future timestamps (timestamps after the current time) receive a recency score of 1.0. This is often undesirable for pre-release products, scheduled content, or data quality issues. The grow parameters allow you to penalize future-dated documents, mirroring how decay works for past documents. Important: All four grow parameters must be provided together or all omitted.

Use Cases

1. E-commerce pre-orders: Penalize products with release dates far in the future
2. Scheduled content: Reduce visibility of content scheduled for future publication
3. Data quality: Handle documents with incorrect future timestamps gracefully
4. Event listings: Balance visibility between upcoming and current events

Example: Mixed Decay and Growth Functions

This example uses different functions for past and future documents. Past documents decay with a gaussian curve over 14 days, providing a smooth, natural decay down to a 0.01 floor. Future documents use linear growth over 7 days — documents far in the future start at a floor of 0.05 and linearly increase toward 1.0 as they approach the current time.

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            # Gaussian decay for past documents
            "decayFunction": "gaussian",
            "scale": "14d",  # Smooth decay over 14 days
            "decayTo": 0.01,  # Near-zero floor for old documents
            # Linear growth for future documents
            "growFunction": "linear",
            "growScale": "7d",  # Linear growth over 7 days
            "growFrom": 0.05,  # Floor for far-future documents
            "growOffset": "0d",
        },
    },
)

# Scoring behavior:
# - Now: score = 1.0
# - Past (0–14d ago): gaussian decay from 1.0 toward 0.01
# - Past (14d+ ago): score = 0.01 (floor)
# - Future (0–7d ahead): linear growth from 0.05 up to 1.0
# - Future (7d+ ahead): score = 0.05 (floor)

---

Visualization

---

Applying Recency to Scores

This section explains how recency scores are applied to document scores, including scoring modes, ranking phases, and integration with score modifiers.

Scoring Modes

Recency can be applied in two modes:

Mode	Formula	Use Case
Multiplicative (default)	final = base_score × recency_score	Recency scales relevance proportionally; older docs are penalized more if they have high base scores
Additive	final = base_score + (recency_score × weight)	Fixed recency boost regardless of base score; gentler on older documents

When to use additive mode (addToScoreWeight):

• You want a consistent recency boost regardless of relevance score
• Multiplicative mode is too aggressive for your use case
• Recency is one of several independent scoring factors

---

Ranking Pipeline Overview

Hybrid search uses a multi-phase ranking pipeline. Recency can be applied at different phases via the applyInRankingPhase parameter:

┌─────────────────────────────────────┐
│  Phase 1: Vespa Ranking (Parallel)  │
│  - Lexical search (BM25)            │
│  - Tensor search (embeddings)       │
│  - Lexical/Tensor score modifiers   │
│  - [Optional] Recency applied here  │
└─────────────────────────────────────┘
             ↓
┌─────────────────────────────────────┐
│  Phase 2: RRF Fusion                │
│  - Combines lexical + tensor        │
│  - Reciprocal rank fusion           │
└─────────────────────────────────────┘
             ↓
┌─────────────────────────────────────┐
│  Phase 3: Global Reranking          │
│  - Global score modifiers           │
│  - [Optional] Recency applied here  │
└─────────────────────────────────────┘

---

Ranking Phase Options

The applyInRankingPhase parameter controls when recency is applied:

Value	Recency Applied In	Use Case
"all" (default)	Vespa phases + Global phase	Maximum recency impact
"only-global"	Global phase only	Fair ranking before fusion, then apply recency to final scores
"exclude-global"	Vespa phases only	Recency affects retrieval but not final fused scores

---

Score Modifiers

Marqo supports three types of score modifiers that work alongside recency:

Type	Applied In	Purpose
Lexical (scoreModifiersLexical)	Vespa phase	Modify BM25 scores
Tensor (scoreModifiersTensor)	Vespa phase	Modify embedding similarity scores
Global (scoreModifiers)	Global phase	Modify RRF-fused scores

---

Order of Operations

The formula depends on both applyInRankingPhase and scoring mode (multiplicative vs additive).

Multiplicative Mode (Default)

With applyInRankingPhase = "all":

Vespa:  score = (relevance × mult_modifier + add_modifier) × recency_score
Global: final = (rrf_score × global_mult + global_add) × recency_score

With applyInRankingPhase = "only-global":

Vespa:  score = (relevance × mult_modifier + add_modifier)  # No recency
Global: final = (rrf_score × global_mult + global_add) × recency_score

With applyInRankingPhase = "exclude-global":

Vespa:  score = (relevance × mult_modifier + add_modifier) × recency_score
Global: final = (rrf_score × global_mult + global_add)  # No recency

Additive Mode (with addToScoreWeight)

final = (base_score × mult_modifier + add_modifier) + (recency_score × addToScoreWeight)

The additive formula applies in whichever phases are enabled by applyInRankingPhase.

---

Examples

Basic Recency (Default Settings)

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "scale": "14d",
            "decayTo": 0.5,
            # applyInRankingPhase: "all" (default)
        },
    },
)

Additive Recency Scoring

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "scale": "7d",
            "decayFunction": "exponential",
            "decayTo": 0.1,
            "addToScoreWeight": 0.5,  # Adds recency_score × 0.5 to base score
        },
    },
)

# Result:
# - Recent doc (recency_score = 1.0): final = base + 0.5
# - Week-old doc (recency_score = 0.1): final = base + 0.05

Recency Applied Only After Fusion

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "scale": "7d",
            "decayTo": 0.3,
            "applyInRankingPhase": "only-global",  # Fair ranking first, then recency
        },
    },
)

Recency + All Score Modifiers

index_name = "products"
results = mq.index(index_name).http.post(
    path=f"indexes/{index_name}/search",
    body={
        "q": "dress",
        "searchMethod": "HYBRID",
        "recencyParameters": {
            "recencyField": "release_date",
            "scale": "14d",
            "offset": "2d",
            "decayFunction": "exponential",
            "decayTo": 0.4,
            "applyInRankingPhase": "all",
        },
        # Global score modifiers
        "scoreModifiers": {
            "add_to_score": [{"field_name": "price", "weight": 10.0}],
        },
        # Hybrid parameters with lexical/tensor modifiers
        "hybridParameters": {
            "retrievalMethod": "disjunction",
            "rankingMethod": "rrf",
            "scoreModifiersLexical": {
                "multiply_score_by": [{"field_name": "price", "weight": 1.5}]
            },
            "scoreModifiersTensor": {
                "multiply_score_by": [{"field_name": "price", "weight": 1.3}]
            },
        },
    },
)

for hit in results["hits"]:
    print(f"{hit['_id']}: score={hit['_score']}, recency={hit['_recency_score']}")

---