launchpad-reviewers team mailing list archive

[Enrique Sánchez <mp+492534@xxxxxxxxxxxxxxxxxx>]

Enrique Sánchez has proposed merging ~enriqueesanchz/launchpad:importvulnerabilityjob-urlfetch-timeout into launchpad:master.

Commit message:
Add override getBlob request timeout
    
Catch RequestExceptionWrapper, a non-requests exception that can occur
during the git hosting API call.


Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~enriqueesanchz/launchpad/+git/launchpad/+merge/492534

During the import job execution we can face this exception:
lp.code.model.githosting.RequestExceptionWrapper: Attempted to set connect timeout to 0, but the timeout cannot be set to a value less than or equal to 0.

This change aims to set a custom value to that timeout instead of using the default one. Also we handle the `RequestExceptionWrapper` so we stop making API calls if they are failing.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~enriqueesanchz/launchpad:importvulnerabilityjob-urlfetch-timeout into launchpad:master.

diff --git a/lib/lp/bugs/model/importvulnerabilityjob.py b/lib/lp/bugs/model/importvulnerabilityjob.py
index db7b2f0..0b758bd 100644
--- a/lib/lp/bugs/model/importvulnerabilityjob.py
+++ b/lib/lp/bugs/model/importvulnerabilityjob.py
@@ -30,10 +30,12 @@ from lp.bugs.scripts.soss.sossimport import SOSSImporter
 from lp.code.errors import GitRepositoryBlobNotFound, GitRepositoryScanFault
 from lp.code.interfaces.githosting import IGitHostingClient
 from lp.code.interfaces.gitlookup import IGitLookup
+from lp.code.model.githosting import RequestExceptionWrapper
 from lp.services.config import config
 from lp.services.database.interfaces import IPrimaryStore, IStore
 from lp.services.job.interfaces.job import JobStatus
 from lp.services.job.model.job import Job
+from lp.services.timeout import override_timeout
 
 CVE_PATTERN = re.compile(r"^CVE-\d{4}-\d+$")
 logger = logging.getLogger(__name__)
@@ -250,7 +252,9 @@ class ImportVulnerabilityJob(VulnerabilityJobDerived):
             success = False
             try:
                 logger.debug(f"[ImportVulnerabilityJob] Getting {file}")
-                blob = ref.getBlob(file)
+
+                with override_timeout(self.config.getblob_timeout):
+                    blob = ref.getBlob(file)
 
                 logger.debug(
                     f"[ImportVulnerabilityJob] Parsing {cve_sequence}"
@@ -262,11 +266,17 @@ class ImportVulnerabilityJob(VulnerabilityJobDerived):
                 if bug and vulnerability:
                     success = True
                     self.metadata["result"]["succeeded"].append(cve_sequence)
-            except (GitRepositoryScanFault, GitRepositoryBlobNotFound):
-                # If there is a problem with git repository, we stop as we
-                # won't be able to fetch more files
+            except (
+                GitRepositoryScanFault,
+                GitRepositoryBlobNotFound,
+                RequestExceptionWrapper,
+            ) as e:
+                # If there is a problem with git repository API call, we stop
+                # as we won't be able to fetch more files
+                self.notifyUserError(f"{cve_sequence}: {e}")
                 raise VulnerabilityJobException(
-                    f"Failed to get file from Git repository: {file}"
+                    f"Failed to get file from Git repository: {file}."
+                    "Aborting."
                 )
             except Exception as e:
                 self.notifyUserError(f"{cve_sequence}: {e}")
diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf
index d3c2892..7dbf78a 100644
--- a/lib/lp/services/config/schema-lazr.conf
+++ b/lib/lp/services/config/schema-lazr.conf
@@ -2253,6 +2253,7 @@ link: IBranchMergeProposalJobSource
 [IImportVulnerabilityJobSource]
 module: lp.bugs.interfaces.vulnerabilityjob
 dbuser: launchpad_main
+getblob_timeout: 10
 
 [IExportVulnerabilityJobSource]
 module: lp.bugs.interfaces.vulnerabilityjob