Search engines

ScoutEngine runs the text search of a resource on the search engine of its Laravel Scout model. The engine matches the text, applies the filters and the sorts, pages the results and counts them. The package then loads the rows of the page in one SQL query, with the includes, aggregates, selects and gates of the search.

use Lomkit\Rest\Fields\Integer;
use Lomkit\Rest\Fields\Number;
use Lomkit\Rest\Text\Engine\EngineQuery;
use Lomkit\Rest\Text\ScoutEngine;
use Lomkit\Rest\Text\TextSearch;

public function text(): ?TextSearch
{
    return ScoutEngine::make()
        ->filterable([Integer::make('category_id'), Number::make('price'), 'status'])
        ->sortable(['price'])
        ->perimeter(
            fn (EngineQuery $query) => $query->where('team_id', '=', $this->request()->user()->team_id),
            [Integer::make('team_id')],
        );
}

{
  "search": {
    "text": {"value": "desk lamp"},
    "filters": [{"field": "category_id", "operator": "in", "value": [1, 2]}],
    "sorts": [{"field": "price"}],
    "includes": [{"relation": "category"}]
  }
}

A blank text, or a text without a letter or a digit, runs the normal SQL search, with its whole grammar.

Declaration

• The model uses Laravel\Scout\Searchable, and its engine is a Scout engine or a class that extends one: collection, database, meilisearch, typesense, algolia, turbopuffer or null. Another engine is refused, because the package cannot know which filters and sorts it runs.
• filterable() lists the index attributes that clients may filter on: fields, such as Integer::make('category_id'), or the names of fields of the resource, such as 'status', which take the type and the operators of that field.
• An attribute is an ID, Integer, Number, Boolean, Text or Enum field, without filterUsing() or sortUsing(). Date, DateTime, Json and app types that extend a field type are refused: index a date as a timestamp and declare it as an Integer.
• A name starts with a letter or _ and holds only letters, digits and _. __soft_deleted is reserved.
• A name that points to a field declared selectable(false) needs that field to be declared filterable().
• sortable() names the filterable() attributes that clients may sort on.
• searchable() does not apply to ScoutEngine.
• The declaration is checked when the resource is first searched in a request. A missing perimeter, a model without Searchable, another engine, a refused type, a refused name, an attribute declared twice or a sort on an attribute that is not filterable throws InvalidTextSearchException, which names the resource.

The index perimeter

The index perimeter is required: declare perimeter() or withoutPerimeter(), not both.

• The closure of perimeter() receives the EngineQuery of each search and reads the request with $this->request(). Its conditions narrow every search of the resource, its total and its pages, and clients cannot widen them.
• Its second argument lists the attributes that only the perimeter uses: clients cannot filter or sort on them.
• The rows of a page are loaded by key without perimeterQuery(): the index and its perimeter decide the rows. Declare in perimeter() every condition that keeps rows from some users, on attributes that the index holds.
• Keep these attributes in sync with the database: set scout.after_commit to true, and leave scout.queue off for a model whose perimeter keeps rows from some users, so that a change reaches the index before the next search.
• search.text matches the attributes that the index searches (with the database and collection engines, every key of toSearchableArray()). Index only attributes that every user who can list the resource may read.
• Use withoutPerimeter() only when every such user may see every indexed row.

Filters and sorts

Next to a filled text, the grammar follows the declaration of the engine:

• filters take the filterable() attributes, combined with and. The or type, nested, relation paths and relation groups give a 422.
• =, !=, in and not in work on every attribute; >, >=, < and <= on Integer and Number attributes, among the operators of the field.
• in and not in need a list of at least one value. null gives a 422. A number that is not finite gives a 422.
• Every engine receives filter values as literals. A string that holds a backtick or a colon followed by ( or [ (, ends with *, is wrapped in double quotes, or is a number written with + gives The search.filters.0.value field holds a value the search engine cannot compare.
• sorts take the sortable() attributes. @relevance keeps the order of the engine: it is accepted alone, without direction or with desc. asc gives a 422 under search.sorts.0.direction, and @relevance next to another sort gives a 422.
• text.mode accepts words only. The engine receives the letters and digits of the text, separated by single spaces: the query syntax of the engine has no effect.
• scopes gives a 422. instructions takes the instructions that implement AppliesToEngine, see below.
• trashed needs a model that uses SoftDeletes and scout.soft_delete set to true, otherwise it gives a 422 next to a text.
• includes, aggregates, selects and gates are validated and applied as in a SQL search, on the rows of the page. Each include and aggregate applies the perimeterQuery() of its own resource.
• The update, delete, restore and forceDelete gates of a page row answer false when perimeterQuery() leaves the row out.

Pages and totals

• The engine pages the results with limit and page. total and last_page are the engine's, and the rows keep the order of the engine.
• The rows of the page load in one SQL query, by their scout key (getScoutKeyName()), with the global scopes and the soft deletes of the model. A row that the index returns and that the database no longer holds, or that a global scope leaves out, is left out of data, and total still counts it.
• Keys follow getScoutKeyType(): int or integer gives integer keys, any other type string keys. A model whose scout key is not its primary key declares getScoutKeyType().
• The package builds the Scout query itself: an override of the model's search(), getScoutModelsByIds() or queryScoutModelsByIds() is not used. Resource::paginate() serves the SQL searches only.
• $maxTotal does not apply: the page keeps the total of the engine and leaves total_capped out.

Engines

Meilisearch and Typesense compare filter strings without regard to case: prefer Integer, ID, Boolean or Enum perimeter attributes. On Meilisearch, Typesense, Algolia and Turbopuffer, a row that a global scope of the model leaves out can be in the index: keep it out with shouldBeSearchable(), or add the condition to the perimeter.

Engine queries

The perimeter and the instructions receive an EngineQuery:

• where($attribute, $operator, $value) takes =, !=, <, >, <= or >=. whereIn($attribute, $values) and whereNotIn($attribute, $values) take a list.
• The attribute must be declared in filterable() or in the perimeter, with this operator. The value is converted to the type of the attribute, a backed enum to its value. Anything else throws InvalidEngineQueryException, a LogicException.
• The conditions add up and never replace each other: two whereIn() on one attribute keep the values that both lists hold. An empty whereIn() returns an empty page without calling the engine.
• orderBy($attribute, $direction) sorts on a sortable() attribute.
• options([...]) adds options for the engine. An option set again with another value throws, and so do the options that the package sets itself.

Instructions

An instruction runs next to an engine text when it implements AppliesToEngine:

use Illuminate\Database\Eloquent\Builder;
use Lomkit\Rest\Fields\Number;
use Lomkit\Rest\Instructions\AppliesToEngine;
use Lomkit\Rest\Instructions\Instruction;
use Lomkit\Rest\Instructions\InstructionFields;
use Lomkit\Rest\Text\Engine\EngineQuery;

class PricedUpTo extends Instruction implements AppliesToEngine
{
    public function fields(): array
    {
        return [Number::make('max')->rules('required')];
    }

    public function handle(InstructionFields $fields, Builder $query): void
    {
        $query->where('price', '<=', $fields->max);
    }

    public function handleEngine(InstructionFields $fields, EngineQuery $query): void
    {
        $query->where('price', '<=', $fields->max);
    }
}

• Another instruction gives a 422 next to an engine text.
• An AppliesToEngine instruction that does not override handle() only applies to engine text searches: it gives a 422 in a SQL search, such as a search without text, an include or the search of a Contains resource.
• In the details, each instruction says under text whether it applies to a search with text on this resource.

Actions

An action whose search has an engine text reads every key from the engine first, in pages of chunkCount, then loads each chunk in SQL by key, before it runs, is queued or starts a batch.

• A search that matches more than maxResources resources gives This action can be run on at most 1000 resources; narrow the search. under search. With search.limit, the action acts on the first limit results.
• Without search.limit, the action reads pagination.maxTotalHits on Meilisearch or paginationLimitedTo on Algolia: the application key must be allowed to read the settings of the index. Set these limits above the maxResources of the actions.
• When the results change while the action reads them, it gives The search results changed while the action was resolving them; run the action again.

See Actions.