Introduce batched query execution and data-node side reduce #121885
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Shortest version I could think of. Still WIP, have to make some test adjustments and polish rough edges, but it shouldn't get longer than this.
original-brownbear
added
WIP
:Search Foundations/Search
|
trivial dependency: #121887 |
|
Another trivial dependency #121922 to avoid some duplication here and remove existing dead code. |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Feb 10, 2025
An easy change we can split out of elastic#121885 to make that shorter.
|
Alrighty, flag is in place, serverless setting reference is removed :) Thanks Luca + Jim! I'll open a back port PR to 8.19 after some benchmarking! :) |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Apr 4, 2025
…astic#122188) An easy change we can split out of elastic#121885 to make that shorter.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Apr 4, 2025
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Apr 9, 2025
…121885) This change moves the query phase a single roundtrip per node just like can_match or field_caps work already. A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node. As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node! Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards.
original-brownbear
added a commit
that referenced
this pull request
Apr 10, 2025
…#126563) * Introduce batched query execution and data-node side reduce (#121885) This change moves the query phase a single roundtrip per node just like can_match or field_caps work already. A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node. As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node! Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards. * Filter out empty top docs results before merging (#126385) `Lucene.EMPTY_TOP_DOCS` to identify empty to docs results. These were previously null results, but did not need to be send over transport as incremental reduction was performed only on the data node. Now it can happen that the coord node received a merge result with empty top docs, which has nothing interesting for merging, but that can lead to an exception because the type of the empty array does not match the type of other shards results, for instance if the query was sorted by field. To resolve this, we filter out empty top docs results before merging. Closes #126118 --------- Co-authored-by: Luca Cavanna <javanna@apache.org>
This was referenced Sep 8, 2025
benchaplin
added a commit
that referenced
this pull request
Nov 5, 2025
#121885 attempted to shortcut a phase failure caused by a reduction failure on the data node by failing the query phase in the batched query action response listener. Before batching the query phase, we did not fail the phase immediately upon a reduction failure. We held on to the failure and continued querying all shards, only failing during final reduction at the beginning of the fetch phase. I can't think of anything inherently wrong with this approach, besides the fact that the phase cannot be failed multiple times (#134151). However certain cleanup aspects of the code (specifically releasing reader contexts and query search results, see: #130821, #122707) rely on the assumption that all shards are queried before failing the phase. This commit reworks batched requests to fail in the same way: only after all shards are queried. To do this, we must include results in transport response even when a reduction failure occurred.
afoucret
pushed a commit
to afoucret/elasticsearch
that referenced
this pull request
Nov 6, 2025
elastic#121885 attempted to shortcut a phase failure caused by a reduction failure on the data node by failing the query phase in the batched query action response listener. Before batching the query phase, we did not fail the phase immediately upon a reduction failure. We held on to the failure and continued querying all shards, only failing during final reduction at the beginning of the fetch phase. I can't think of anything inherently wrong with this approach, besides the fact that the phase cannot be failed multiple times (elastic#134151). However certain cleanup aspects of the code (specifically releasing reader contexts and query search results, see: elastic#130821, elastic#122707) rely on the assumption that all shards are queried before failing the phase. This commit reworks batched requests to fail in the same way: only after all shards are queried. To do this, we must include results in transport response even when a reduction failure occurred.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Shortest version I could think of for this from where we are now.
This change moves the query phase a single roundtrip per node just like can_match or field_caps work already.
A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node.
As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node!
Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards.