mahara-contributors team mailing list archive

[Robert Lyon <robertl@xxxxxxxxxxxxxxx>]

Public bug reported:

It throws errors like:


[2018-07-25T11:53:45,582][DEBUG][o.e.a.s.TransportSearchAction] *****  Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[websearch-mahara-prod], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"from":11460,"size":10,"query":{"bool":{"must":[{"match":{"catch_all":{"query":"mahara","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[{"bool":{"should":[{"term":{"access.general":{"value":"public","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"term":{"secfacetterm":{"value":"Forumpost","boost":1.0}}},{"term":{"mainfacetterm":{"value":"Text","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"_score":{"order":"desc"}},{"_score":{"order":"desc"}}],"highlight":{"pre_tags":["<span class=\"search-highlight\">"],"post_tags":["</span>"],"fragment_size":100,"number_of_fragments":2,"require_field_match":false,"fields":{"description":{}}}}}]

Exception: Result window is too large, from + size must be less than or
equal to: [10000] but was [11470]. See the scroll api for a more
efficient way to request large data sets. This limit can be set by
changing the [index.max_result_window] index level setting.

This is due to a search like 'mahara' returning more than 10000 rows and
so if one paginates to the last results it throws error

What would be better to do is alert the user that there are more than
10000 records and ask them to refine their search and only have
pagination for 10000

** Affects: mahara
     Importance: Medium
         Status: Confirmed

-- 
You received this bug notification because you are a member of Mahara
Contributors, which is subscribed to Mahara.
Matching subscriptions: Subscription for all Mahara Contributors -- please ask on #mahara-dev or mahara.org forum before editing or unsubscribing it!
https://bugs.launchpad.net/bugs/1783447

Title:
  ElasticSearch struggles with pagination on results over 10000

Status in Mahara:
  Confirmed

Bug description:
  It throws errors like:

  
  [2018-07-25T11:53:45,582][DEBUG][o.e.a.s.TransportSearchAction] *****  Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[websearch-mahara-prod], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"from":11460,"size":10,"query":{"bool":{"must":[{"match":{"catch_all":{"query":"mahara","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[{"bool":{"should":[{"term":{"access.general":{"value":"public","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"term":{"secfacetterm":{"value":"Forumpost","boost":1.0}}},{"term":{"mainfacetterm":{"value":"Text","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"_score":{"order":"desc"}},{"_score":{"order":"desc"}}],"highlight":{"pre_tags":["<span class=\"search-highlight\">"],"post_tags":["</span>"],"fragment_size":100,"number_of_fragments":2,"require_field_match":false,"fields":{"description":{}}}}}]

  Exception: Result window is too large, from + size must be less than
  or equal to: [10000] but was [11470]. See the scroll api for a more
  efficient way to request large data sets. This limit can be set by
  changing the [index.max_result_window] index level setting.

  This is due to a search like 'mahara' returning more than 10000 rows
  and so if one paginates to the last results it throws error

  What would be better to do is alert the user that there are more than
  10000 records and ask them to refine their search and only have
  pagination for 10000

To manage notifications about this bug go to:
https://bugs.launchpad.net/mahara/+bug/1783447/+subscriptions