debcrafters-packages team mailing list archive

[Ponnuvel Palaniyappan <2121543@xxxxxxxxxxxxxxxxxx>]

Attaching the debdiff for Questing.

** Description changed:

  libdnss-db opens and closes the DB file each time. This is normally not
  an issue as the DB files are small.
  
  However, in a user environment, this takes considerably longer because
  the DB files have entries to the tune of 20k which severely impacts the
  performance of lookups.
+ 
+ ----
+ 
+ [ Impact ]
+ 
+ The libnss-db library, which processes Berkeley DB files, closes
+ and re-opens the DB file for each entry in the file that it currently scans.
+ This results in very poor performance and noticiable botteneck on the
+ application side. In a user environment that has 20k entries in the DB,
+ this results in about 40 times worse performance (compared to keeping
+ the file open).
+ 
+ [ Test Plan ]
+ 
+ The test plan sets up some DB files and queries them to see if the
+ open(2) and close(2) calls are repeatedly called.
+ 
+ 1. update /etc/default/libnss-db
+ DBS = passwd group shadow
+ VAR_DB = /var/lib/misc
+ ETC = ${VAR_DB}/etc
+ AWK = awk
+ MAKEDB = makedb --quiet
+  
+ 2. update /etc/nsswitch.conf
+ passwd:         files systemd db
+ group:          files systemd db
+ shadow:         files systemd db
+  
+ 3. create required files
+ mkdir -p /var/lib/misc/etc
+ touch /var/lib/misc/etc/{passwd,group,shadow}
+ cp /etc/login.defs /var/lib/misc/etc
+  
+ 4. update /var/lib/misc/etc/login.defs so that UIDs are less likely to conflict
+ UID_MIN                  5000
+  
+ 5. create test users in berkelydb file
+ cd /var/lib/misc
+ for i in $(seq 0001 1000); do useradd -P /var/lib/misc test$i; done
+ make
+  
+ 6. verify users in berkeleydb is recognized
+ id test1000
+  
+ 7. verify that 'getent passwd' and 'getenv group' calls correctly retrieve the entr fom 
+ verify that there's not as many open/close calls as there are entries in the DB files
+ 
+ [ Where problems could occur ]
+ 
+ To my understanding the historical reason for doing close-open for each line is to
+ minimise possible resource usage and reduce the possibility of DB files getting stale
+ if DB files get updated externally. Potential risks revolve around those two.
+ 
+ In the former case, this could result in higher load on the system (modern hardware should be capable of handling that).
+ In the latter case, lookups could fail and likely result in re-scans from the application.
+ 
+ [ Other Info ]
+ 
+ Debian bug and patch(not merged): https://bugs.debian.org/cgi-
+ bin/bugreport.cgi?bug=1101371

** Changed in: libnss-db (Ubuntu)
       Status: New => In Progress

** Also affects: libnss-db (Ubuntu Noble)
   Importance: Undecided
       Status: New

** Also affects: libnss-db (Ubuntu Plucky)
   Importance: Undecided
       Status: New

** Also affects: libnss-db (Ubuntu Questing)
   Importance: Wishlist
       Status: In Progress

** Attachment added: "questing-debdiff.txt"
   https://bugs.launchpad.net/ubuntu/+source/libnss-db/+bug/2121543/+attachment/5904175/+files/questing-debdiff.txt

** Changed in: libnss-db (Ubuntu Questing)
     Assignee: (unassigned) => Ponnuvel Palaniyappan (pponnuvel)

-- 
You received this bug notification because you are a member of
Debcrafters packages, which is subscribed to libnss-db in Ubuntu.
https://bugs.launchpad.net/bugs/2121543

Title:
  [SRU] Poor performance of libnss-db on large db files

Status in libnss-db package in Ubuntu:
  In Progress
Status in libnss-db source package in Noble:
  New
Status in libnss-db source package in Plucky:
  New
Status in libnss-db source package in Questing:
  In Progress

Bug description:
  libdnss-db opens and closes the DB file each time. This is normally
  not an issue as the DB files are small.

  However, in a user environment, this takes considerably longer because
  the DB files have entries to the tune of 20k which severely impacts
  the performance of lookups.

  ----

  [ Impact ]

  The libnss-db library, which processes Berkeley DB files, closes
  and re-opens the DB file for each entry in the file that it currently scans.
  This results in very poor performance and noticiable botteneck on the
  application side. In a user environment that has 20k entries in the DB,
  this results in about 40 times worse performance (compared to keeping
  the file open).

  [ Test Plan ]

  The test plan sets up some DB files and queries them to see if the
  open(2) and close(2) calls are repeatedly called.

  1. update /etc/default/libnss-db
  DBS = passwd group shadow
  VAR_DB = /var/lib/misc
  ETC = ${VAR_DB}/etc
  AWK = awk
  MAKEDB = makedb --quiet
   
  2. update /etc/nsswitch.conf
  passwd:         files systemd db
  group:          files systemd db
  shadow:         files systemd db
   
  3. create required files
  mkdir -p /var/lib/misc/etc
  touch /var/lib/misc/etc/{passwd,group,shadow}
  cp /etc/login.defs /var/lib/misc/etc
   
  4. update /var/lib/misc/etc/login.defs so that UIDs are less likely to conflict
  UID_MIN                  5000
   
  5. create test users in berkelydb file
  cd /var/lib/misc
  for i in $(seq 0001 1000); do useradd -P /var/lib/misc test$i; done
  make
   
  6. verify users in berkeleydb is recognized
  id test1000
   
  7. verify that 'getent passwd' and 'getenv group' calls correctly retrieve the entr fom 
  verify that there's not as many open/close calls as there are entries in the DB files

  [ Where problems could occur ]

  To my understanding the historical reason for doing close-open for each line is to
  minimise possible resource usage and reduce the possibility of DB files getting stale
  if DB files get updated externally. Potential risks revolve around those two.

  In the former case, this could result in higher load on the system (modern hardware should be capable of handling that).
  In the latter case, lookups could fail and likely result in re-scans from the application.

  [ Other Info ]

  Debian bug and patch(not merged): https://bugs.debian.org/cgi-
  bin/bugreport.cgi?bug=1101371

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libnss-db/+bug/2121543/+subscriptions