Search Indexes¶

A SearchIndex is a Synapse entity whose content is defined by a Synapse SQL query. Synapse builds an OpenSearch index from the rows that query returns, which gives you full-text search, relevance ranking, faceting, and autocomplete over a table or view.

This is a different way of asking questions than a Table or a Materialized View. A table is queried with Synapse SQL and answers "which rows match these exact conditions?". A search index is queried with the OpenSearch Query DSL and answers "which rows are most relevant to this text?" — matching word stems, ignoring punctuation and case, ranking the best matches first, and counting how many rows fall into each category. It is what you would put behind a search box.

This tutorial will walk you through creating a search index and querying it with the Synapse Python client.

Tutorial Purpose¶

In this tutorial, you will:

Log in, get your project, and create a table to index
Create a SearchIndex
Run a full-text search
Highlight where the match happened
Combine scored clauses with unscored filters, and sort the results
Count facets with aggregations
Power a type-ahead box with autocomplete
Paginated results
Tune matching with synonyms and analyzers

Prerequisites¶

This tutorial assumes that you have a Synapse project.
Pandas must also be installed as shown in the installation documentation.

1. Log in, get your project, and create a table to index¶

A search index is always defined over an existing table-like entity, so we first create a small table of study summaries to search over.

You will want to replace "My uniquely named project about Alzheimer's Disease" with the name of your project.

import json

import pandas as pd

from synapseclient import Synapse
from synapseclient.models import (
    Column,
    ColumnType,
    Project,
    SearchIndex,
    SearchQuery,
    SearchQueryPart,
    Table,
)
from synapseclient.models.search_dsl import (
    Aggregation,
    AvgAggregation,
    BoolQuery,
    Highlight,
    HighlightField,
    MatchBoolPrefixFieldOptions,
    MatchFieldOptions,
    MatchPhraseFieldOptions,
    MultiMatchQuery,
    Query,
    RangeFieldOptions,
    SourceFilter,
    TermsAggregation,
)

# Initialize Synapse client
syn = Synapse()
syn.login()

# Get the project where we want to create the search index
project = Project(name="My uniquely named project about Alzheimer's Disease").get()
project_id = project.id
print(f"Got project with ID: {project_id}")

# Create the table that will be indexed
table = Table(
    name="Study Summaries",
    parent_id=project_id,
    columns=[
        Column(name="study_name", column_type=ColumnType.STRING),
        Column(name="abstract", column_type=ColumnType.LARGETEXT),
        Column(name="diagnosis", column_type=ColumnType.STRING),
        Column(name="assay", column_type=ColumnType.STRING),
        Column(name="participant_count", column_type=ColumnType.INTEGER),
    ],
).store()
print(f"Created table with ID: {table.id}")

# Add the rows we are going to search over
studies = pd.DataFrame(
    [
        {
            "study_name": "ROSMAP Cortex Proteomics",
            "abstract": "Quantitative proteomics of dorsolateral prefrontal cortex "
            "from donors with Alzheimer's disease and cognitively normal controls.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "TMT quantitation",
            "participant_count": 400,
        },
        {
            "study_name": "MSBB RNA Sequencing",
            "abstract": "Bulk RNA sequencing across four brain regions in a cohort "
            "spanning the full range of Alzheimer's disease neuropathology.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "rnaSeq",
            "participant_count": 300,
        },
        {
            "study_name": "Mayo Clinic Whole Genome",
            "abstract": "Whole genome sequencing of temporal cortex samples from "
            "donors with Alzheimer's disease, progressive supranuclear palsy, "
            "and controls.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "wholeGenomeSeq",
            "participant_count": 350,
        },
        {
            "study_name": "Healthy Aging Single Cell Atlas",
            "abstract": "Single nucleus RNA sequencing of hippocampus from "
            "cognitively normal aged donors, establishing a baseline atlas.",
            "diagnosis": "Cognitively Normal",
            "assay": "snrnaSeq",
            "participant_count": 120,
        },
        {
            "study_name": "MCI Plasma Biomarkers",
            "abstract": "Plasma biomarker panel measuring phosphorylated tau and "
            "neurofilament light chain in mild cognitive impairment.",
            "diagnosis": "Mild Cognitive Impairment",
            "assay": "immunoassay",
            "participant_count": 220,
        },
        {
            "study_name": "Parkinson Comparative Cohort",
            "abstract": "Comparative transcriptomic profiling of substantia nigra "
            "in Parkinson disease versus age-matched controls.",
            "diagnosis": "Parkinson's Disease",
            "assay": "rnaSeq",
            "participant_count": 180,
        },
    ]
)
table.upsert_rows(values=studies, primary_keys=["study_name"])
print(f"Stored {len(studies)} rows in {table.id}")

2. Create a SearchIndex Entity¶

The defining_sql decides which rows and columns are indexed. It must reference exactly one table-like entity — unlike a Materialized View, JOIN and UNION across several entities are not supported.

If you need to search across several tables, build a Materialized View first and index that.

Any of these can be the source, whichever one the SQL selects from:

The SQL can pin a specific version of the source (SELECT * FROM syn12345.7), select a subset of columns, and carry WHERE, ORDER BY, and LIMIT clauses — only those rows and columns end up in the index.

Storing the search index entity returns as soon as Synapse has accepted it, but the OpenSearch index behind it is built in the background.

# Create a SearchIndex over a single table and wait for it to build.
index = SearchIndex(
    name="Study Summaries Search Index",
    description="Full text search over the study summary table",
    parent_id=project_id,
    # The defining SQL must reference exactly one table-like entity
    defining_sql=f"SELECT * FROM {table.id}",
)
index = index.store()
print(f"Created SearchIndex with ID: {index.id}")

Creating the index should look like:

Created SearchIndex with ID: syn68123456
Index syn68123456 is queryable with 6 rows

Note: The index tracks its source. When rows in the underlying table change, the index is updated in the background — you do not need to re-store the SearchIndex.

3. Run a full-text search¶

A match clause is the workhorse of full-text search: the text you pass is analyzed the same way the column was analyzed, so "alzheimer" matches "Alzheimer's disease". Every clause kind Synapse accepts is listed on Query.

By default a hit carries every indexed column. source narrows that down, and response_parts asks for extras beyond the hits themselves — here the total hit count and the columns each hit carries.

Adding fuzziness to a match clause buys typo tolerance: the term someone typed will still match a term in the index that is a few single-character edits away. "AUTO" scales the allowance with term length, and prefix_length pins the first few characters so unrelated short words don't start matching each other. Both options are available on match, match_bool_prefix, and multi_match.

# Find every study whose abstract mentions Alzheimer's disease, then
# search across several columns at once.
results = index.query(
    search_query=SearchQuery(
        query=Query(match={"abstract": MatchFieldOptions(query="alzheimer")}),
        # Every indexed column comes back on each hit unless a source filter
        # narrows them down, and the abstracts are long
        source=SourceFilter(includes=["study_name", "diagnosis"]),
        size=10,
    ),
    response_parts=[
        SearchQueryPart.HITS,
        SearchQueryPart.TOTAL_HITS,
        SearchQueryPart.SELECT_COLUMNS,
    ],
)
print("Abstracts mentioning Alzheimer's:")
print(f"columns: {[column.name for column in results.select_columns]}")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

# A multi_match clause runs the same text across several columns, so the
# person searching does not need to know which column holds the term.
results = index.query(
    search_query=SearchQuery(
        query=Query(
            multi_match=MultiMatchQuery(
                query="tau",
                # ^2 boosts a match in the study name over one in the abstract
                fields=["study_name^2", "abstract"],
            )
        ),
        source=SourceFilter(includes=["study_name"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("\nAnything mentioning tau:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

# People misspell things. `fuzziness` tolerates a number of single-character
# edits -- insert, delete, substitute, or transpose -- between what was typed
# and what is in the index.
results = index.query(
    search_query=SearchQuery(
        query=Query(
            match={
                "abstract": MatchFieldOptions(
                    # "sequencing" with a missing "i"
                    query="sequencng",
                    # "AUTO" scales the allowance with term length: 0 edits for
                    # very short terms, 1 for medium, 2 for long ones
                    fuzziness="AUTO",
                    # The first 3 characters still have to be exact. Without
                    # this, short unrelated words start matching each other
                    prefix_length=3,
                )
            }
        ),
        source=SourceFilter(includes=["study_name"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("\nMisspelling 'sequencng' still finds:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

The results of your searches should look like:

Abstracts mentioning Alzheimer's:
columns: ['study_name', 'diagnosis']
total_hits=3, returned=3
  {'study_name': 'ROSMAP Cortex Proteomics', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'MSBB RNA Sequencing', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'Mayo Clinic Whole Genome', 'diagnosis': "Alzheimer's Disease"}

Anything mentioning tau:
total_hits=1, returned=1
  {'study_name': 'MCI Plasma Biomarkers'}

Misspelling 'sequencng' still finds:
total_hits=3, returned=3
  {'study_name': 'MSBB RNA Sequencing'}
  {'study_name': 'Mayo Clinic Whole Genome'}
  {'study_name': 'Healthy Aging Single Cell Atlas'}

Hits come back ranked by relevance, and each one carries its score on hit.score along with the row_id and row_version of the source row.

4. Highlight where the match happened¶

A result list is much easier to read when it shows the matching text in context. highlight returns short fragments of the matched columns with the matching terms wrapped in <em> tags.

# Ask for a snippet of the matching text alongside each hit, so a
# result list can show why the row matched.
results = index.query(
    search_query=SearchQuery(
        query=Query(match={"abstract": MatchFieldOptions(query="sequencing")}),
        source=SourceFilter(includes=["study_name", "assay"]),
        highlight=Highlight(fields={"abstract": HighlightField(number_of_fragments=1)}),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Studies that sequenced something:")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")
    for highlight in hit.highlights:
        print(f"    {highlight.name}: {highlight.snippets}")

The result of your highlighted search should look like:

Studies that sequenced something:
  {'study_name': 'MSBB RNA Sequencing', 'assay': 'rnaSeq'}
    abstract: ['Bulk RNA <em>sequencing</em> across four brain regions in a cohort']
  {'study_name': 'Mayo Clinic Whole Genome', 'assay': 'wholeGenomeSeq'}
    abstract: ['Whole genome <em>sequencing</em> of temporal cortex samples from']
  {'study_name': 'Healthy Aging Single Cell Atlas', 'assay': 'snrnaSeq'}
    abstract: ['Single nucleus RNA <em>sequencing</em> of hippocampus from']

Note: Highlighting, like relevance scoring, depends on the column being indexed as analyzed text. Step 9 covers how to control that with a SearchConfiguration.

5. Combine scored clauses with unscored filters, and sort the results¶

A bool clause is how you build a real search request out of several conditions:

must clauses have to match and do contribute to the relevance score
filter and must_not clauses have to match (or not match) but do not affect the score — use these for hard constraints like a numeric cutoff
should clauses boost the rows that match them without excluding the rows that don't

Passing sort replaces relevance ranking with an ordering of your choosing. Only column and _score sorts are accepted.

# Combine a scored clause with unscored filters using a bool query,
# then order the results by a numeric column instead of by relevance.

results = index.query(
    search_query=SearchQuery(
        query=Query(
            bool=BoolQuery(
                # Scored: how well the abstract matches drives relevance
                must=[Query(match={"abstract": MatchFieldOptions(query="sequencing")})],
                # Unscored: a hard cutoff on cohort size
                filter=[Query(range={"participant_count": RangeFieldOptions(gte=200)})],
                # Unscored: drop a diagnosis we are not interested in
                must_not=[
                    Query(
                        match_phrase={
                            "diagnosis": MatchPhraseFieldOptions(
                                query="Parkinson's Disease"
                            )
                        }
                    )
                ],
            )
        ),
        source=SourceFilter(includes=["study_name", "participant_count"]),
        sort=[{"participant_count": "desc"}],
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Sequencing studies with at least 200 participants, largest first:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

The result of your filtered search should look like:

Sequencing studies with at least 200 participants, largest first:
total_hits=2, returned=2
  {'study_name': 'Mayo Clinic Whole Genome', 'participant_count': '350'}
  {'study_name': 'MSBB RNA Sequencing', 'participant_count': '300'}

Aggregations answer "how many rows are there of each kind?" — the counts you see next to the checkboxes in a faceted search UI. A terms aggregation produces one bucket per distinct value of a column; metric aggregations like avg and stats summarize a numeric column. Results come back on aggregation_results as the raw OpenSearch response, with field references rewritten back to your column names.

post_filter is what keeps a facet list usable: it narrows the hits after the aggregations have been computed, so selecting one diagnosis does not make the other diagnosis counts disappear.

# Count how many studies fall under each diagnosis and average their
# cohort sizes, while the hit list itself shows only one diagnosis.

results = index.query(
    search_query=SearchQuery(
        query=Query(match_all={}),
        aggregations={
            "by_diagnosis": Aggregation(
                terms=TermsAggregation(field="diagnosis", size=10)
            ),
            "mean_cohort_size": Aggregation(
                avg=AvgAggregation(field="participant_count")
            ),
        },
        # post_filter narrows the hits but not the aggregations, so the facet
        # counts still show every option a person could pick next
        post_filter=Query(
            match_phrase={
                "diagnosis": MatchPhraseFieldOptions(query="Alzheimer's Disease")
            }
        ),
        source=SourceFilter(includes=["study_name", "diagnosis"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Hits after the post filter:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")
print("\nFacet counts across all studies:")
print(json.dumps(results.aggregation_results, indent=2))

The result of your faceted search should look like:

Hits after the post filter:
total_hits=3, returned=3
  {'study_name': 'ROSMAP Cortex Proteomics', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'MSBB RNA Sequencing', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'Mayo Clinic Whole Genome', 'diagnosis': "Alzheimer's Disease"}

Facet counts across all studies:
{
  "by_diagnosis": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": "Alzheimer's Disease",
        "doc_count": 3
      },
      {
        "key": "Cognitively Normal",
        "doc_count": 1
      },
      {
        "key": "Mild Cognitive Impairment",
        "doc_count": 1
      },
      {
        "key": "Parkinson's Disease",
        "doc_count": 1
      }
    ]
  },
  "mean_cohort_size": {
    "value": 261.6666666666667
  }
}

7. Power a type-ahead box with autocomplete¶

autocomplete() is a separate, synchronous endpoint meant for search-as-you-type: it returns its hits directly instead of going through the asynchronous job service, so it is fast enough to call on every keystroke. In exchange, it only accepts prefix-style clauses — prefix, match_phrase_prefix, or match_bool_prefix — and returns at most 8 hits.

# Back a type-ahead box with the autocomplete endpoint, which returns
# its results directly instead of running as an asynchronous job.
hits = index.autocomplete(
    query=Query(
        match_bool_prefix={"study_name": MatchBoolPrefixFieldOptions(query="Mayo Cl")}
    ),
    source=SourceFilter(includes=["study_name"]),
)
print("Suggestions for 'Mayo Cl':")
for hit in hits:
    print(f"  {[field.value for field in hit.fields]}")

The result of your autocomplete request should look like:

Suggestions for 'Mayo Cl':
  ['Mayo Clinic Whole Genome']

8. Paginated results¶

A query returns at most 100 hits at a time (25 by default), so anything larger requires special attention. There are two ways to do it.

Specifying the from_ and size arguments on SearchQuery to an offset the way page numbers do.
search_after picks up from where the last page ended. Each response has next_search_after; pass it back unchanged on the next request and leave from_ unset.

Offset paging with `from_` and `size`¶

Simple, and extracts the results in pages. The cost grows with depth and the server collects and discards every hit before the offset — so it is the wrong tool for sweeping a large index.

# Walk every row in the index two hits at a time with a growing offset.
page_size = 2
offset = 0
while True:
    results = index.query(
        search_query=SearchQuery(
            query=Query(match_all={}),
            source=SourceFilter(includes=["study_name", "participant_count"]),
            sort=[{"participant_count": "desc"}],
            from_=offset,
            size=page_size,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print(f"Page starting at offset {offset}:")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")

    offset += page_size
    if offset >= results.total_hits:
        break

The result of paging through your index should look like:

Page starting at offset 0:
total_hits=6, returned=2
  {'study_name': 'ROSMAP Cortex Proteomics', 'participant_count': '400'}
  {'study_name': 'Mayo Clinic Whole Genome', 'participant_count': '350'}
Page starting at offset 2:
total_hits=6, returned=2
  {'study_name': 'MSBB RNA Sequencing', 'participant_count': '300'}
  {'study_name': 'MCI Plasma Biomarkers', 'participant_count': '220'}
Page starting at offset 4:
total_hits=6, returned=2
  {'study_name': 'Parkinson Comparative Cohort', 'participant_count': '180'}
  {'study_name': 'Healthy Aging Single Cell Atlas', 'participant_count': '120'}

Cursor paging with `search_after`¶

This is the solution if you need every row. The catch is that search_after is a position in a sort order, so the sort has to place every row unambiguously. If two rows tie on every sort column, a page boundary landing between them can skip or repeat rows. Sort on something unique, or append a unique column as a final tie-breaker.

# The same walk, using the search_after cursor the server hands back instead of
# a growing offset.
page_size = 2
search_after = None
page = 0
while True:
    results = index.query(
        search_query=SearchQuery(
            query=Query(match_all={}),
            source=SourceFilter(includes=["study_name", "participant_count"]),
            # search_after walks a sort order, so the sort has to put every row
            # in a definite position. participant_count is unique in this table;
            # on real data append a unique column to break ties, or a page
            # boundary can skip or repeat rows.
            sort=[{"participant_count": "desc"}],
            # None on the first request, then the cursor from the previous one
            search_after=search_after,
            size=page_size,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print(f"Page {page}:")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")

    # The cursor is opaque -- pass it back unchanged. It goes None on the last
    # page, which is what ends the walk.
    search_after = results.next_search_after
    if not search_after or not results.hits:
        break
    page += 1

The result of walking your index should look like:

Page 0:
total_hits=6, returned=2
  {'study_name': 'ROSMAP Cortex Proteomics', 'participant_count': '400'}
  {'study_name': 'Mayo Clinic Whole Genome', 'participant_count': '350'}
Page 1:
total_hits=6, returned=2
  {'study_name': 'MSBB RNA Sequencing', 'participant_count': '300'}
  {'study_name': 'MCI Plasma Biomarkers', 'participant_count': '220'}
Page 2:
total_hits=6, returned=2
  {'study_name': 'Parkinson Comparative Cohort', 'participant_count': '180'}
  {'study_name': 'Healthy Aging Single Cell Atlas', 'participant_count': '120'}

Note: Each page is a separate asynchronous job either way, not a cheap follow-up GET, so ask for the largest size you can use rather than walking a big index in small pages.

Advanced: Tune matching with synonyms and analyzers¶

Restricted and permanent

Creating and updating the following resources is restricted to Sage Bionetworks employees, and the REST API has no delete endpoint for any of them. Once created, a SynonymSet, TextAnalyzer, ColumnAnalyzerOverride, or SearchConfiguration cannot be removed, and its owning Organization can no longer be deleted either. Choose names deliberately.

Everything in this tutorial relies on how each column was analyzed when the index was built: how text is split into tokens, which tokens are dropped, and how they are normalized. There are four Organization-scoped resources that let you control that:

SynonymSet — terms that should be treated as equivalent, so someone searching AD finds abstracts that say "Alzheimer's disease"
TextAnalyzer — a named OpenSearch analyzer: a tokenizer plus a chain of token filters, which may reference a SynonymSet
ColumnAnalyzerOverride — a reusable bundle assigning specific analyzers to specific columns
SearchConfiguration — bundles a default analyzer with any column overrides; this is what a SearchIndex actually points at

Each resource belongs to an Organization and is referenced from another resource by its qualified name, {organization_name}-{name}, written as {"$ref": "my.org-my_analyzer"}.

Note where the synonym filter goes below. The analyzer declares both a default chain, used when rows are indexed, and a default_search chain, used when a query is analyzed. Putting the synonyms only in default_search expands the incoming query instead of storing every synonym for every row.

def create_search_configuration() -> str:
    """
    Example: Teach the index that "AD" means "Alzheimer's disease" by building a
    SynonymSet, wrapping it in a TextAnalyzer, and bundling that analyzer into a
    SearchConfiguration.

    These resources belong to an Organization, and creating them is restricted
    to Sage Bionetworks employees. None of them can be deleted once created.
    """
    from synapseclient.models import (
        ColumnAnalyzerOverride,
        ColumnAnalyzerOverrideEntry,
        Organization,
        SearchConfiguration,
        SynonymSet,
        TextAnalyzer,
    )

    organization_name = "my.uniquely.named.organization"
    organization = Organization(name=organization_name).store()
    print(f"Using organization: {organization.id} ({organization.name})")

    # Comma-separated entries are interchangeable in both directions; entries
    # written with "=>" expand the left side to the right side only.
    synonyms = SynonymSet(
        organization_name=organization_name,
        name="ad_synonyms",
        description="Abbreviations used across Alzheimer's disease studies",
        definition={
            "type": "synonym_graph",
            "synonyms": [
                "rna sequencing, rna-seq, rnaseq",
                "ad => alzheimer's disease, alzheimers disease",
                "mci => mild cognitive impairment",
            ],
        },
    ).store()
    print(f"Created SynonymSet: {synonyms.id} ({synonyms.qualified_name})")

    # The synonym filter is applied in `default_search` only, so synonyms expand
    # the incoming query rather than bloating the stored index.
    analyzer = TextAnalyzer(
        organization_name=organization_name,
        name="ad_synonym_analyzer",
        description="English analyzer that expands AD abbreviations at search time",
        settings={
            "filter": {
                "english_stop": {"type": "stop", "stopwords": "_english_"},
                "english_stemmer": {"type": "stemmer", "language": "english"},
                # A $ref resolves to the SynonymSet by its qualified name
                "ad_synonyms": {"$ref": synonyms.qualified_name},
            },
            "analyzer": {
                "default": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "english_stop", "english_stemmer"],
                },
                "default_search": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": [
                        "lowercase",
                        "ad_synonyms",
                        "english_stop",
                        "english_stemmer",
                    ],
                },
            },
        },
    ).store()
    print(f"Created TextAnalyzer: {analyzer.id} ({analyzer.qualified_name})")

    # Columns not named here fall back to the configuration's default analyzer
    overrides = ColumnAnalyzerOverride(
        organization_name=organization_name,
        name="study_column_overrides",
        description="Treat the diagnosis column as a single exact value",
        overrides=[
            ColumnAnalyzerOverrideEntry(
                column_name="diagnosis",
                analyzer={"analyzer": {"default": {"type": "keyword"}}},
            ),
        ],
    ).store()
    print(f"Created ColumnAnalyzerOverride: {overrides.id}")

    configuration = SearchConfiguration(
        organization_name=organization_name,
        name="study_search_config",
        description="Analyzer settings for the study summary search index",
        default_analyzer={"$ref": analyzer.qualified_name},
        column_analyzer_overrides=[{"$ref": overrides.qualified_name}],
    ).store()
    print(f"Created SearchConfiguration: {configuration.id}")
    return configuration.id

A SearchIndex resolves its configuration when the index is built, so set it up front. Either point the index straight at a configuration with search_configuration_id, or bind a configuration to the parent folder or project — an index with no search_configuration_id of its own walks up the entity hierarchy and uses the first SearchConfigBinding it finds, falling back to the platform defaults.

def create_index_with_configuration(search_configuration_id: str) -> SearchIndex:
    """
    Example: Build an index that uses a specific SearchConfiguration, and bind
    the same configuration to the project so later indexes inherit it.
    """
    from synapseclient.models import SearchConfigBinding

    index = SearchIndex(
        name="Study Summaries Search Index With Synonyms",
        parent_id=project_id,
        defining_sql=f"SELECT * FROM {table.id}",
        search_configuration_id=search_configuration_id,
    ).store()
    print(f"Created SearchIndex {index.id} using config {search_configuration_id}")

    # Any index created under this project without its own
    # search_configuration_id now inherits this configuration
    binding = SearchConfigBinding(
        object_id=project_id,
        search_configuration_id=search_configuration_id,
    ).store()
    print(f"Bound configuration {binding.search_configuration_id} to {project_id}")

    # "AD" now matches the abstracts that spell out "Alzheimer's disease"
    results = index.query(
        search_query=SearchQuery(
            query=Query(match={"abstract": MatchFieldOptions(query="AD")}),
            source=SourceFilter(includes=["study_name", "diagnosis"]),
            size=10,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print("Abstracts matching the abbreviation 'AD':")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")
    return index

Searching the abbreviation against the new index should look like:

Created SearchIndex syn68123457 using config 4321
Index syn68123457 is queryable with 6 rows
Bound configuration 4321 to syn12345678
Abstracts matching the abbreviation 'AD':
total_hits=3, returned=3
  {'study_name': 'ROSMAP Cortex Proteomics', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'MSBB RNA Sequencing', 'diagnosis': "Alzheimer's Disease"}
  {'study_name': 'Mayo Clinic Whole Genome', 'diagnosis': "Alzheimer's Disease"}

Source Code for this Tutorial¶

Click to show me

"""Here is where you'll find the code for the SearchIndex tutorial."""

import json

import pandas as pd

from synapseclient import Synapse
from synapseclient.models import (
    Column,
    ColumnType,
    Project,
    SearchIndex,
    SearchQuery,
    SearchQueryPart,
    Table,
)
from synapseclient.models.search_dsl import (
    Aggregation,
    AvgAggregation,
    BoolQuery,
    Highlight,
    HighlightField,
    MatchBoolPrefixFieldOptions,
    MatchFieldOptions,
    MatchPhraseFieldOptions,
    MultiMatchQuery,
    Query,
    RangeFieldOptions,
    SourceFilter,
    TermsAggregation,
)

# Initialize Synapse client
syn = Synapse()
syn.login()

# Get the project where we want to create the search index
project = Project(name="My uniquely named project about Alzheimer's Disease").get()
project_id = project.id
print(f"Got project with ID: {project_id}")

# Create the table that will be indexed
table = Table(
    name="Study Summaries",
    parent_id=project_id,
    columns=[
        Column(name="study_name", column_type=ColumnType.STRING),
        Column(name="abstract", column_type=ColumnType.LARGETEXT),
        Column(name="diagnosis", column_type=ColumnType.STRING),
        Column(name="assay", column_type=ColumnType.STRING),
        Column(name="participant_count", column_type=ColumnType.INTEGER),
    ],
).store()
print(f"Created table with ID: {table.id}")

# Add the rows we are going to search over
studies = pd.DataFrame(
    [
        {
            "study_name": "ROSMAP Cortex Proteomics",
            "abstract": "Quantitative proteomics of dorsolateral prefrontal cortex "
            "from donors with Alzheimer's disease and cognitively normal controls.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "TMT quantitation",
            "participant_count": 400,
        },
        {
            "study_name": "MSBB RNA Sequencing",
            "abstract": "Bulk RNA sequencing across four brain regions in a cohort "
            "spanning the full range of Alzheimer's disease neuropathology.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "rnaSeq",
            "participant_count": 300,
        },
        {
            "study_name": "Mayo Clinic Whole Genome",
            "abstract": "Whole genome sequencing of temporal cortex samples from "
            "donors with Alzheimer's disease, progressive supranuclear palsy, "
            "and controls.",
            "diagnosis": "Alzheimer's Disease",
            "assay": "wholeGenomeSeq",
            "participant_count": 350,
        },
        {
            "study_name": "Healthy Aging Single Cell Atlas",
            "abstract": "Single nucleus RNA sequencing of hippocampus from "
            "cognitively normal aged donors, establishing a baseline atlas.",
            "diagnosis": "Cognitively Normal",
            "assay": "snrnaSeq",
            "participant_count": 120,
        },
        {
            "study_name": "MCI Plasma Biomarkers",
            "abstract": "Plasma biomarker panel measuring phosphorylated tau and "
            "neurofilament light chain in mild cognitive impairment.",
            "diagnosis": "Mild Cognitive Impairment",
            "assay": "immunoassay",
            "participant_count": 220,
        },
        {
            "study_name": "Parkinson Comparative Cohort",
            "abstract": "Comparative transcriptomic profiling of substantia nigra "
            "in Parkinson disease versus age-matched controls.",
            "diagnosis": "Parkinson's Disease",
            "assay": "rnaSeq",
            "participant_count": 180,
        },
    ]
)
table.upsert_rows(values=studies, primary_keys=["study_name"])
print(f"Stored {len(studies)} rows in {table.id}")

# Create a SearchIndex over a single table and wait for it to build.
index = SearchIndex(
    name="Study Summaries Search Index",
    description="Full text search over the study summary table",
    parent_id=project_id,
    # The defining SQL must reference exactly one table-like entity
    defining_sql=f"SELECT * FROM {table.id}",
)
index = index.store()
print(f"Created SearchIndex with ID: {index.id}")



# Find every study whose abstract mentions Alzheimer's disease, then
# search across several columns at once.
results = index.query(
    search_query=SearchQuery(
        query=Query(match={"abstract": MatchFieldOptions(query="alzheimer")}),
        # Every indexed column comes back on each hit unless a source filter
        # narrows them down, and the abstracts are long
        source=SourceFilter(includes=["study_name", "diagnosis"]),
        size=10,
    ),
    response_parts=[
        SearchQueryPart.HITS,
        SearchQueryPart.TOTAL_HITS,
        SearchQueryPart.SELECT_COLUMNS,
    ],
)
print("Abstracts mentioning Alzheimer's:")
print(f"columns: {[column.name for column in results.select_columns]}")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

# A multi_match clause runs the same text across several columns, so the
# person searching does not need to know which column holds the term.
results = index.query(
    search_query=SearchQuery(
        query=Query(
            multi_match=MultiMatchQuery(
                query="tau",
                # ^2 boosts a match in the study name over one in the abstract
                fields=["study_name^2", "abstract"],
            )
        ),
        source=SourceFilter(includes=["study_name"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("\nAnything mentioning tau:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")

# People misspell things. `fuzziness` tolerates a number of single-character
# edits -- insert, delete, substitute, or transpose -- between what was typed
# and what is in the index.
results = index.query(
    search_query=SearchQuery(
        query=Query(
            match={
                "abstract": MatchFieldOptions(
                    # "sequencing" with a missing "i"
                    query="sequencng",
                    # "AUTO" scales the allowance with term length: 0 edits for
                    # very short terms, 1 for medium, 2 for long ones
                    fuzziness="AUTO",
                    # The first 3 characters still have to be exact. Without
                    # this, short unrelated words start matching each other
                    prefix_length=3,
                )
            }
        ),
        source=SourceFilter(includes=["study_name"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("\nMisspelling 'sequencng' still finds:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")



# Ask for a snippet of the matching text alongside each hit, so a
# result list can show why the row matched.
results = index.query(
    search_query=SearchQuery(
        query=Query(match={"abstract": MatchFieldOptions(query="sequencing")}),
        source=SourceFilter(includes=["study_name", "assay"]),
        highlight=Highlight(fields={"abstract": HighlightField(number_of_fragments=1)}),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Studies that sequenced something:")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")
    for highlight in hit.highlights:
        print(f"    {highlight.name}: {highlight.snippets}")




# Combine a scored clause with unscored filters using a bool query,
# then order the results by a numeric column instead of by relevance.

results = index.query(
    search_query=SearchQuery(
        query=Query(
            bool=BoolQuery(
                # Scored: how well the abstract matches drives relevance
                must=[Query(match={"abstract": MatchFieldOptions(query="sequencing")})],
                # Unscored: a hard cutoff on cohort size
                filter=[Query(range={"participant_count": RangeFieldOptions(gte=200)})],
                # Unscored: drop a diagnosis we are not interested in
                must_not=[
                    Query(
                        match_phrase={
                            "diagnosis": MatchPhraseFieldOptions(
                                query="Parkinson's Disease"
                            )
                        }
                    )
                ],
            )
        ),
        source=SourceFilter(includes=["study_name", "participant_count"]),
        sort=[{"participant_count": "desc"}],
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Sequencing studies with at least 200 participants, largest first:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")



# Count how many studies fall under each diagnosis and average their
# cohort sizes, while the hit list itself shows only one diagnosis.

results = index.query(
    search_query=SearchQuery(
        query=Query(match_all={}),
        aggregations={
            "by_diagnosis": Aggregation(
                terms=TermsAggregation(field="diagnosis", size=10)
            ),
            "mean_cohort_size": Aggregation(
                avg=AvgAggregation(field="participant_count")
            ),
        },
        # post_filter narrows the hits but not the aggregations, so the facet
        # counts still show every option a person could pick next
        post_filter=Query(
            match_phrase={
                "diagnosis": MatchPhraseFieldOptions(query="Alzheimer's Disease")
            }
        ),
        source=SourceFilter(includes=["study_name", "diagnosis"]),
        size=10,
    ),
    response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
)
print("Hits after the post filter:")
print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
for hit in results.hits:
    fields = {field.name: field.value for field in hit.fields}
    print(f"  {fields}")
print("\nFacet counts across all studies:")
print(json.dumps(results.aggregation_results, indent=2))



# Back a type-ahead box with the autocomplete endpoint, which returns
# its results directly instead of running as an asynchronous job.
hits = index.autocomplete(
    query=Query(
        match_bool_prefix={"study_name": MatchBoolPrefixFieldOptions(query="Mayo Cl")}
    ),
    source=SourceFilter(includes=["study_name"]),
)
print("Suggestions for 'Mayo Cl':")
for hit in hits:
    print(f"  {[field.value for field in hit.fields]}")




# Walk every row in the index two hits at a time with a growing offset.
page_size = 2
offset = 0
while True:
    results = index.query(
        search_query=SearchQuery(
            query=Query(match_all={}),
            source=SourceFilter(includes=["study_name", "participant_count"]),
            sort=[{"participant_count": "desc"}],
            from_=offset,
            size=page_size,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print(f"Page starting at offset {offset}:")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")

    offset += page_size
    if offset >= results.total_hits:
        break



# The same walk, using the search_after cursor the server hands back instead of
# a growing offset.
page_size = 2
search_after = None
page = 0
while True:
    results = index.query(
        search_query=SearchQuery(
            query=Query(match_all={}),
            source=SourceFilter(includes=["study_name", "participant_count"]),
            # search_after walks a sort order, so the sort has to put every row
            # in a definite position. participant_count is unique in this table;
            # on real data append a unique column to break ties, or a page
            # boundary can skip or repeat rows.
            sort=[{"participant_count": "desc"}],
            # None on the first request, then the cursor from the previous one
            search_after=search_after,
            size=page_size,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print(f"Page {page}:")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")

    # The cursor is opaque -- pass it back unchanged. It goes None on the last
    # page, which is what ends the walk.
    search_after = results.next_search_after
    if not search_after or not results.hits:
        break
    page += 1



def create_search_configuration() -> str:
    """
    Example: Teach the index that "AD" means "Alzheimer's disease" by building a
    SynonymSet, wrapping it in a TextAnalyzer, and bundling that analyzer into a
    SearchConfiguration.

    These resources belong to an Organization, and creating them is restricted
    to Sage Bionetworks employees. None of them can be deleted once created.
    """
    from synapseclient.models import (
        ColumnAnalyzerOverride,
        ColumnAnalyzerOverrideEntry,
        Organization,
        SearchConfiguration,
        SynonymSet,
        TextAnalyzer,
    )

    organization_name = "my.uniquely.named.organization"
    organization = Organization(name=organization_name).store()
    print(f"Using organization: {organization.id} ({organization.name})")

    # Comma-separated entries are interchangeable in both directions; entries
    # written with "=>" expand the left side to the right side only.
    synonyms = SynonymSet(
        organization_name=organization_name,
        name="ad_synonyms",
        description="Abbreviations used across Alzheimer's disease studies",
        definition={
            "type": "synonym_graph",
            "synonyms": [
                "rna sequencing, rna-seq, rnaseq",
                "ad => alzheimer's disease, alzheimers disease",
                "mci => mild cognitive impairment",
            ],
        },
    ).store()
    print(f"Created SynonymSet: {synonyms.id} ({synonyms.qualified_name})")

    # The synonym filter is applied in `default_search` only, so synonyms expand
    # the incoming query rather than bloating the stored index.
    analyzer = TextAnalyzer(
        organization_name=organization_name,
        name="ad_synonym_analyzer",
        description="English analyzer that expands AD abbreviations at search time",
        settings={
            "filter": {
                "english_stop": {"type": "stop", "stopwords": "_english_"},
                "english_stemmer": {"type": "stemmer", "language": "english"},
                # A $ref resolves to the SynonymSet by its qualified name
                "ad_synonyms": {"$ref": synonyms.qualified_name},
            },
            "analyzer": {
                "default": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": ["lowercase", "english_stop", "english_stemmer"],
                },
                "default_search": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": [
                        "lowercase",
                        "ad_synonyms",
                        "english_stop",
                        "english_stemmer",
                    ],
                },
            },
        },
    ).store()
    print(f"Created TextAnalyzer: {analyzer.id} ({analyzer.qualified_name})")

    # Columns not named here fall back to the configuration's default analyzer
    overrides = ColumnAnalyzerOverride(
        organization_name=organization_name,
        name="study_column_overrides",
        description="Treat the diagnosis column as a single exact value",
        overrides=[
            ColumnAnalyzerOverrideEntry(
                column_name="diagnosis",
                analyzer={"analyzer": {"default": {"type": "keyword"}}},
            ),
        ],
    ).store()
    print(f"Created ColumnAnalyzerOverride: {overrides.id}")

    configuration = SearchConfiguration(
        organization_name=organization_name,
        name="study_search_config",
        description="Analyzer settings for the study summary search index",
        default_analyzer={"$ref": analyzer.qualified_name},
        column_analyzer_overrides=[{"$ref": overrides.qualified_name}],
    ).store()
    print(f"Created SearchConfiguration: {configuration.id}")
    return configuration.id




def create_index_with_configuration(search_configuration_id: str) -> SearchIndex:
    """
    Example: Build an index that uses a specific SearchConfiguration, and bind
    the same configuration to the project so later indexes inherit it.
    """
    from synapseclient.models import SearchConfigBinding

    index = SearchIndex(
        name="Study Summaries Search Index With Synonyms",
        parent_id=project_id,
        defining_sql=f"SELECT * FROM {table.id}",
        search_configuration_id=search_configuration_id,
    ).store()
    print(f"Created SearchIndex {index.id} using config {search_configuration_id}")

    # Any index created under this project without its own
    # search_configuration_id now inherits this configuration
    binding = SearchConfigBinding(
        object_id=project_id,
        search_configuration_id=search_configuration_id,
    ).store()
    print(f"Bound configuration {binding.search_configuration_id} to {project_id}")

    # "AD" now matches the abstracts that spell out "Alzheimer's disease"
    results = index.query(
        search_query=SearchQuery(
            query=Query(match={"abstract": MatchFieldOptions(query="AD")}),
            source=SourceFilter(includes=["study_name", "diagnosis"]),
            size=10,
        ),
        response_parts=[SearchQueryPart.HITS, SearchQueryPart.TOTAL_HITS],
    )
    print("Abstracts matching the abbreviation 'AD':")
    print(f"total_hits={results.total_hits}, returned={len(results.hits)}")
    for hit in results.hits:
        fields = {field.name: field.value for field in hit.fields}
        print(f"  {fields}")
    return index

Search Indexes¶

Tutorial Purpose¶

Prerequisites¶

1. Log in, get your project, and create a table to index¶

2. Create a SearchIndex Entity¶

3. Run a full-text search¶

4. Highlight where the match happened¶

5. Combine scored clauses with unscored filters, and sort the results¶

6. Count facets with aggregations¶

7. Power a type-ahead box with autocomplete¶

8. Paginated results¶

Offset paging with `from_` and `size`¶

Cursor paging with `search_after`¶

Advanced: Tune matching with synonyms and analyzers¶

Source Code for this Tutorial¶

References¶

Search Indexes¶

Tutorial Purpose¶

Prerequisites¶

1. Log in, get your project, and create a table to index¶

2. Create a SearchIndex Entity¶

3. Run a full-text search¶

4. Highlight where the match happened¶

5. Combine scored clauses with unscored filters, and sort the results¶

6. Count facets with aggregations¶

7. Power a type-ahead box with autocomplete¶

8. Paginated results¶

Offset paging with from_ and size¶

Cursor paging with search_after¶

Advanced: Tune matching with synonyms and analyzers¶

Source Code for this Tutorial¶

References¶

Offset paging with `from_` and `size`¶

Cursor paging with `search_after`¶