canonical-ubuntu-qa team mailing list archive

[Tim Andersson <mp+471513@xxxxxxxxxxxxxxxxxx>]

Tim Andersson has proposed merging ~andersson123/qa-jenkins-jobs:no-britney-runs-alerts into qa-jenkins-jobs:master.

Requested reviews:
  Canonical Platform QA Team (canonical-platform-qa)

For more details, see:
https://code.launchpad.net/~andersson123/qa-jenkins-jobs/+git/qa-jenkins-jobs/+merge/471513

Alert the QA team when britney is potentially stuck or not running at all
-- 
Your team Canonical Platform QA Team is requested to review the proposed merge of ~andersson123/qa-jenkins-jobs:no-britney-runs-alerts into qa-jenkins-jobs:master.

diff --git a/jobs/release-management/jobs.yaml b/jobs/release-management/jobs.yaml
new file mode 100644
index 0000000..c455790
--- /dev/null
+++ b/jobs/release-management/jobs.yaml
@@ -0,0 +1,51 @@
+---
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The project stanza describes all jobs and parameters
+
+- job:
+    name: check-britney-running-periodically
+    description: |
+        Runs a check on update_excuses to see when it was last updated.
+        If it hasn't been updated in the last X hours, then we can assume
+        britney is either stuck or not running for some reason.
+    node: 'venonat'
+    parameters:
+      - string:
+          name: QA_JENKINS_JOBS_BRANCH
+          default: lp:qa-jenkins-jobs
+          description: "Launchpad branch of qa-jenkins-jobs from which to pick the scripts from."
+    triggers:
+        - timed: '@hourly'
+    wrappers:
+        - timestamps
+        - timeout:
+            timeout: 1
+            fail: true
+    builders:
+        - clear-artifacts:
+        - git-shallow-clone:
+            project-url: '$QA_JENKINS_JOBS_BRANCH'
+            project-name: 'qa-jenkins-jobs'
+        - shell: |
+            #!/bin/bash
+            cd qa-jenkins-jobs/
+            ./scripts/release-management/check-britney-running-periodically
+    publishers:
+      - email:
+          recipients: florent.jacquet@xxxxxxxxxxxxx, tim.andersson@xxxxxxxxxxxxx, paride.legovini@xxxxxxxxxxxxx, ural.tunaboyu@xxxxxxxxxxxxx
\ No newline at end of file
diff --git a/scripts/release-management/check-britney-running-periodically b/scripts/release-management/check-britney-running-periodically
new file mode 100755
index 0000000..1d7c350
--- /dev/null
+++ b/scripts/release-management/check-britney-running-periodically
@@ -0,0 +1,62 @@
+#!/usr/bin/python3
+
+import requests
+import re
+import datetime
+import sys
+
+
+MAX_AGE_THRESHOLD = 6  # hours
+
+
+def check_security_britney():
+    security_britney_url = "http://security-britney.internal/";
+    content = requests.get(security_britney_url).content.decode("utf-8")
+    filter_re = re.compile(r'run.log<\/a><\/td><td align="right">(.*?)<')
+    match = filter_re.search(content).group(1).rstrip()
+    latest_security_britney_run = datetime.datetime.strptime(match, "%Y-%m-%d %H:%M")
+    now = datetime.datetime.now()
+    if (now - latest_security_britney_run) > datetime.timedelta(
+        hours=MAX_AGE_THRESHOLD
+    ):
+        print(
+            f"Latest security britney run completed more than {MAX_AGE_THRESHOLD} hours ago, alerting the Release Management Team."
+        )
+        return False
+    else:
+        print(
+            f"Latest security britney run completed within the last {MAX_AGE_THRESHOLD} hours, all is well"
+        )
+        return True
+
+
+def check_britney():
+    update_excuses_url = (
+        "https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_excuses.html";
+    )
+    filter_re = re.compile(r"Generated: (.*) \+")
+    content = requests.get(update_excuses_url).content.decode("utf-8")
+    match = filter_re.search(content).group(1)
+    latest_update_excuses = datetime.datetime.strptime(match, "%Y.%m.%d %H:%M:%S")
+    now = datetime.datetime.now()
+    if (now - latest_update_excuses) > datetime.timedelta(hours=MAX_AGE_THRESHOLD):
+        print(
+            f"Latest britney run completed more than {MAX_AGE_THRESHOLD} hours ago, alerting the Release Management Team."
+        )
+        return False
+    else:
+        print(
+            f"Latest britney run completed within the last {MAX_AGE_THRESHOLD} hours, all is well"
+        )
+        return True
+
+
+def main():
+    britney_result = check_britney()
+    security_britney_result = check_security_britney()
+    if not britney_result or not security_britney_result:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()