mahara-contributors team mailing list archive

[Mahara Bot <1457712@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://reviews.mahara.org/4831
Committed: https://git.nzoss.org.nz/mahara/mahara/commit/f7194ce1c0c47c9da1b69026e61b71f9024b1a86
Submitter: Robert Lyon (robertl@xxxxxxxxxxxxxxx)
Branch:    master

commit f7194ce1c0c47c9da1b69026e61b71f9024b1a86
Author: Aaron Wells <aaronw@xxxxxxxxxxxxxxx>
Date:   Tue Jun 9 18:00:47 2015 +1200

More robust handling of Elasticsearch bulk operations

Bug 1457712. This patch accomplishes four main things:

1. Sets a limit on the number on the number of documents
per Elasticsearch bulk request.

2. Doesn't delete records from the queue table until
after they have been successfully sent

3. If a bulk request fails, later retries the records
individually

4. Performs deletion in bulk

Change-Id: I9ac5e3a33b473e256fdf331800dc60101c126dcc

-- 
You received this bug notification because you are a member of Mahara
Contributors, which is subscribed to Mahara.
Matching subscriptions: Subscription for all Mahara Contributors -- please ask on #mahara-dev or mahara.org forum before editing or unsubscribing it!
https://bugs.launchpad.net/bugs/1457712

Title:
  More robust Elasticsearch data uploading

Status in Mahara ePortfolio:
  Fix Committed

Bug description:
  The search/elasticsearch plugin has a cron job that loads data into
  the Elasticsearch server. Currently, this cron job has a couple of
  problems:

  1. It sends all the data from search_elasticsearch_queue in just one
  /_bulk operation. This means that if you've got a proxy server in
  front of your Elasticsearch server, it's quite likely to reject the
  request because it'll be larger than the allowed request size.

  2. It deletes the records from the queue table *before* attempting to
  upload them to Elasticsearch. If the upload fails, all of the records
  that were part of that bulk operation are basically lost and never get
  indexed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mahara/+bug/1457712/+subscriptions