Extensions

Engine extensions

Restrict every search engine query, declare server-only index attributes and lists of strings, and provide the profile of a Scout engine.

Three kinds act on the searches that run on a search engine through ScoutEngine: RestrictsEngineQueries, DeclaresIndexAttributes and ProvidesEngineProfiles. They throw Lomkit\Rest\Exceptions\InvalidEngineExtensionException, which names the extension by its name and its class. See Search engines for the engine searches themselves.

Restricting engine searches

RestrictsEngineQueries::restrictEngine(EngineRestrictionContext $context) restricts each engine query of a resource. Here, the tenant extension of Query extensions also restricts the engine, on an attribute that it declares:

Vendor/Tenancy/TenantExtension.php
use Lomkit\Rest\Extensions\Contexts\EngineRestrictionContext;
use Lomkit\Rest\Extensions\Contexts\IndexAttributesContext;
use Lomkit\Rest\Extensions\Contexts\RestrictionContext;
use Lomkit\Rest\Extensions\Extension;
use Lomkit\Rest\Extensions\Kinds\DeclaresIndexAttributes;
use Lomkit\Rest\Extensions\Kinds\RestrictsEngineQueries;
use Lomkit\Rest\Extensions\Kinds\RestrictsQueries;
use Lomkit\Rest\Fields\Integer;

final class TenantExtension extends Extension implements DeclaresIndexAttributes, RestrictsEngineQueries, RestrictsQueries
{
    public function __construct(private readonly CurrentTenant $tenant) {}

    #[\Override]
    public static function name(): string
    {
        return 'tenant';
    }

    #[\Override]
    public function restrict(RestrictionContext $context): void
    {
        // the SQL restriction, see Query extensions
    }

    #[\Override]
    public function indexAttributes(IndexAttributesContext $context): array
    {
        return [Integer::make('tenant_id')];
    }

    #[\Override]
    public function restrictEngine(EngineRestrictionContext $context): void
    {
        if ($context->rowsRestrictedInSql) {
            return;
        }

        if ($context->user === null) {
            $context->query->matchNothing();

            return;
        }

        $context->query->where('tenant_id', '=', $this->tenant->id());
    }
}
  • It runs once per engine query, after ScoutEngine::perimeter() and before the filters, sorts and instructions of the client: searches with a text, engine lists sent with "source": "engine", Rest::search() and actions on their results, also behind withoutPerimeter().
  • Its conditions are AND-ed with the perimeter, the client and the other extensions, in any extension order. On a single-value attribute, whereIn() lists intersect and whereNotIn() lists add up, so a client filter cannot bring back a value it excludes.
  • matchNothing() gives an empty page without calling the engine, and an action then runs on no row. Every method of $context->query returns it, matchNothing() included.
  • $context->user is the user that the policies run for, also under Rest::search()->as($user), null for a guest.
  • perimeter() or withoutPerimeter() is still required: an extension does not stand for it.

The context

PropertyValue
$context->queryThe restriction: where(), whereIn(), whereNotIn(), whereAny() and matchNothing()
$context->resource, $context->userThe searched resource and the user
$context->rowsRestrictedInSqltrue when the RestrictsQueries restriction of the same extension already applies in the SQL of the engine query: the Scout database, collection and null drivers, and any profile that implements AppliesRowRestrictions. false for an extension that does not implement RestrictsQueries
$context->comparesColumnstrue when Scout compares each condition with a column of the model table: name only columns there
$context->matchesStringListsWhether the engine matches lists of strings
$context->filtersWithGroupsWhether the engine accepts whereAny() groups
$context->engineThe Scout engine instance that runs the query: test it with instanceof to write conditions for one engine

Groups

whereAny() OR-s a group of conditions:

use Lomkit\Rest\Text\Engine\EngineConditionGroup;

$context->query->whereAny(fn (EngineConditionGroup $group) => $group
    ->where('owner_id', '=', $id)
    ->whereAll(fn (EngineConditionGroup $all) => $all->where('public', '=', true)->whereIn('team_id', $teams)));
  • EngineConditionGroup offers where(), whereIn(), whereNotIn(), whereAll() for an AND group and whereAny() for an OR group.
  • The whole group is one condition of the extension, AND-ed with the perimeter, the other restrictions, the instructions, the lists of strings and the client filters: it widens only within itself.
  • A group without any condition matches nothing. whereIn() with an empty list is false and whereNotIn() with an empty list is true, so a true member of a whereAny() group removes the whole condition of the extension: refuse or replace an empty list read from the database before you group it.
  • A condition added to a group after its closure returned throws InvalidEngineQueryException.
  • Check $context->filtersWithGroups first. The built-in Typesense, Meilisearch, Turbopuffer, database, collection and null profiles accept groups. On Algolia, and on a profile that an extension provides, whereAny() throws InvalidEngineExtensionException, which names the extension, its class, the resource and the engine.

What it refuses

  • Anything but where(), whereIn(), whereNotIn(), whereAny() and matchNothing(): no client-style or, no sort, no engine option.
  • An attribute that is not in the filterable(), indexAttributes() or perimeter() list of the resource, or declared by an extension, and any condition that the engine query refuses. InvalidEngineExtensionException quotes the refusal and keeps it as its previous exception.

Pages, actions and the index

  • The rows of an engine page load by key without the RestrictsQueries restrictions and without perimeterQuery(): the engine is the proof, so a row that a stale index still matches is shown until the index is updated. Make both restrictions say the same, and keep the index in sync.
  • A condition that restrictEngine() reads live, such as a whereIn() on keys read from the database, applies at once. A change carried by an indexed attribute applies once the row is indexed again.
  • Actions on search results, queued or not, read their keys through the RestrictsQueries restrictions: a row that the index still matches but the restriction now excludes is not handed to the action.
  • A RestrictsQueries extension without this kind makes the text searches of the resource throw InvalidTextSearchException on an engine that does not apply SQL restrictions. The message offers to leave the resource out with appliesTo(), or to add the class to rest.disabled_extensions, and rest:check reports it under rows.
  • The Scout null driver returns no row, and a RestrictsQueries extension is never refused on it. Like collection and database, it does not compare lists of strings: a whereIn() or whereNotIn() on a list throws there.

Declaring index attributes

DeclaresIndexAttributes::indexAttributes(IndexAttributesContext $context) declares index attributes that the engine restrictions, ScoutEngine::perimeter() and the engine instructions compare. Here, a list of the keys that may read each row:

Vendor/Sharing/ReadersExtension.php
use Lomkit\Rest\Extensions\Contexts\EngineRestrictionContext;
use Lomkit\Rest\Extensions\Contexts\IndexAttributesContext;
use Lomkit\Rest\Extensions\Extension;
use Lomkit\Rest\Extensions\Kinds\DeclaresIndexAttributes;
use Lomkit\Rest\Extensions\Kinds\RestrictsEngineQueries;
use Lomkit\Rest\Text\Engine\StringListAttribute;

final class ReadersExtension extends Extension implements DeclaresIndexAttributes, RestrictsEngineQueries
{
    #[\Override]
    public static function name(): string
    {
        return 'readers';
    }

    #[\Override]
    public function indexAttributes(IndexAttributesContext $context): array
    {
        return $context->matchesStringLists ? [StringListAttribute::make('readers')] : [];
    }

    #[\Override]
    public function restrictEngine(EngineRestrictionContext $context): void
    {
        $reader = $context->user?->getAuthIdentifier();

        if (! $context->matchesStringLists || $reader === null) {
            $context->query->matchNothing();

            return;
        }

        $context->query->whereIn('readers', [(string) $reader]);
    }
}

The model indexes the list: toSearchableArray() returns 'readers' => ['12', 'group-3']. On an engine without lists, this extension shows no row.

  • It receives $context->resource, a class name, and $context->matchesStringLists, no user. It runs once per request and resource.
  • Return fields (ID, Integer, Number, Boolean, Text or Enum) or lists of strings with StringListAttribute::make('readers'). The attributes keep their real index name, never prefixed, and follow the rules of filterable().
  • Clients never use them: they stay out of the client grammar (a filter on one is the 422 of an unknown name), /details and the OpenAPI document.
  • The package cannot check that toSearchableArray() and the index settings hold these attributes: index each one yourself, or check it with ChecksResources. A resource that alone needs an attribute declares it with ScoutEngine::indexAttributes().

Owners

  • A name that filterable() declares throws InvalidEngineExtensionException, and so does a name that ScoutEngine::indexAttributes(), perimeter() or another extension declares with another field type or list marker. The message names the list of the resource as the indexAttributes() list or the perimeter() list.
  • The same declaration is accepted: the one of the resource is used, else the one of the first extension by name.
  • rest:check reports such a conflict as a rows problem.

Lists of strings

  • whereIn('readers', [...]) keeps the rows that share at least one value with the list, and whereNotIn() the rows that share none. where() is refused on a list.
  • Two or more whereIn() lists on one list attribute keep the rows that share at least one value with each list, whichever source sets them: perimeter(), the restrictions or the instructions. No list widens another.
  • When one list holds the other, only the narrower one is sent: the same rows, with a shorter filter. Otherwise each further list joins the native filter as its own condition, AND-ed at the top level, on the built-in Typesense, Meilisearch and Turbopuffer profiles.
  • On Algolia, and on a profile that an extension provides, two lists that neither holds throw Lomkit\Rest\Exceptions\ConflictingListsException, which names the sources, the attribute and the engine. Set by an instruction, such a list answers 422 on search.instructions.N: The search.instructions.0 field restricts a list with values that neither hold nor are held by the list that the search already keeps, and the search engine cannot require both lists. On such an engine, an instruction must not take client values for a list that the perimeter or a restriction also restricts.
  • A list can be declared on every engine. Comparing it on an engine whose profile does not implement MatchesStringLists, such as the Scout database, collection and null drivers, throws InvalidEngineExtensionException: check $context->matchesStringLists first.

The index holds each list as:

EngineIndex schema
AlgoliafilterOnly(readers) in attributesForFaceting
Meilisearchreaders in filterableAttributes
Typesensea collection field ['name' => 'readers', 'type' => 'string[]']
Turbopuffera []string attribute, matched with ContainsAny and excluded with NotContainsAny

Providing engine profiles

A profile tells the package what a Scout engine runs: its operators, sorts, page sizes, owned options and how it pages. ProvidesEngineProfiles::engineProfile(EngineProfileContext $context) returns the EngineProfile of an engine, or null:

Vendor/OpenSearch/OpenSearchExtension.php
use Lomkit\Rest\Extensions\Contexts\EngineProfileContext;
use Lomkit\Rest\Extensions\Extension;
use Lomkit\Rest\Extensions\Kinds\ProvidesEngineProfiles;
use Lomkit\Rest\Text\Engine\Profiles\EngineProfile;

final class OpenSearchExtension extends Extension implements ProvidesEngineProfiles
{
    #[\Override]
    public static function name(): string
    {
        return 'opensearch';
    }

    #[\Override]
    public function engineProfile(EngineProfileContext $context): ?EngineProfile
    {
        return $context->engine instanceof OpenSearchEngine ? new OpenSearchProfile($context->engine) : null;
    }
}
  • It receives $context->resource and $context->engine only, no user. Answer without side effects, and null for an engine that the extension does not know.
  • The package asks the extensions before its own profiles, so an extension can also replace the profile of a built-in engine. A provided profile goes through the same checks as a built-in one.
  • Two extensions that provide a profile for one engine throw InvalidEngineExtensionException, which names both and the engine class: return null from one of them, or disable one.

Writing a profile

A profile implements Lomkit\Rest\Text\Engine\Profiles\EngineProfile:

MethodReturns
name()The name of the engine in messages
operators(Field $attribute)The FilterOperator cases that the engine compares on the attribute. EngineOperators::for($attribute) gives the usual set
sortKeys()The most sorts of a search, 0 to refuse sortable(), PHP_INT_MAX for no bound
ownedOptions()The Scout options that the profile sets itself, which options() cannot set
exactTotal()Whether the total of a page is exact
maxPageSize(), window()The largest page and the deepest position the engine pages to, null when unbounded
ensureUsableBy(TextTarget $target)Nothing: throws when the model cannot be searched there
page(Builder $builder, ScoutModel $model, int $perPage, int $page)An EnginePage: the keys of the page in engine order and the total, EnginePage::empty() for none
reachable(ScoutModel $model)The number of hits that an action may read, null when unknown
literal()The EngineLiteral that writes a value into an engine filter
  • Every key that page() returns must match the filter that the builder carries: pages trust the engine. Turn off the features that add hits outside the filter, such as pinned, promoted or curated hits, in page(), and list their options in ownedOptions().
  • A profile written for 3.5 still works.

Optional interfaces of Lomkit\Rest\Text\Engine\Profiles add capabilities, checked with instanceof:

InterfaceCapability
MatchesStringListsThe engine matches lists of strings. A provided profile gets its lists through Scout's whereIn() and whereNotIn(): a profile that replaces the built-in Turbopuffer one sends them as In and NotIn, no longer as ContainsAny and NotContainsAny
ComparesColumnsScout compares each condition with a column of the model table
AppliesRowRestrictionsThe profile runs SQL: restrictedPage(Builder $builder, ScoutModel $model, int $perPage, int $page, RowRestriction $rows) applies the RestrictsQueries restrictions, so they work without RestrictsEngineQueries
ListsWithoutTextThe engine serves engine lists: matchAll() returns the query that matches every document, listSortKeys() the most sorts of a search without text
  • restrictedPage() is called only when a RestrictsQueries restriction applies, and page() serves every other search. It must call $rows->apply() on the query it runs: one that returns rows without calling it makes the search throw InvalidEngineQueryException. A page takes the keys that it returns as they are.
  • servesLists() on an engine whose profile does not implement ListsWithoutText throws InvalidTextSearchException at the first search, and rest:check reports it.
  • The interface that writes or and nested filters is internal to the built-in profiles: a provided profile answers 422 for them, as Algolia does, and refuses whereAny().
  • A provided profile computes no totals: search.totals answers The search.totals field is refused: the search engine of this resource computes no totals.