getdeb-collaboration team mailing list archive

[Joao Pinto <joao.pinto@xxxxxxxxxx>]

Hello,
I have implemented and played the APT mirror: method and the a mirror
scanning tool, found a limitation which as far as I can see is unavoidable
and turns it inefficient for our use case.

The mirror: method depends on having a list of the updated mirrors, if a
mirror is not updated even if it has the specific package the user is
needing it will not be selected, during the mirror selection there is no way
to determine which files will be required.
With the current changes to mirroring, on the worst case, a change to the
repository takes 2h, this means that every time a package is updated we
would have all mirrors "invalid" during this time, all package requested
during this time would need to be server from the master server. This is not
acceptable.

Unless there is a way to workaround this issue we will need to keep on the
per file basis http redirection method.
Still our current http selector script is highly inefficient and needs to be
improved, as it is now, for each file request a request will be sent to a
remote mirror to check the file size. If there are 100 requests for the same
file, still there will be 100 remote requests generated. To improve this a
per mirror file availability info cache must be created and shared across
requests, this is not trivial because it involves shared resources across
multiple web requests, delays from remote requests and caching.

An alternative to use our own custom solution is to use mirrorbrain,
http://mirrorbrain.org/, I have tried to set it up in the paste failed. I
did not have much time and there were no packages for Ubuntu/Debian.

Best regards,

-- 
João Luís Marques Pinto
GetDeb Team Leader
http://www.getdeb.net
http://blog.getdeb.net