canonical-hw-cert team mailing list archive

[Jonathan Cave <mp+436237@xxxxxxxxxxxxxxxxxx>]

Jonathan Cave has proposed merging ~jocave/hwcert-jenkins-jobs:trello-flock-everywhere into hwcert-jenkins-jobs:master.

Requested reviews:
  hwcert-jenkins (hwcert-jenkins)

For more details, see:
https://code.launchpad.net/~jocave/hwcert-jenkins-jobs/+git/hwcert-jenkins-jobs/+merge/436237

There seems to have been an increasing number of cases of hitting rate limit errors from the trello API during jenkins jobs, this looks like: 

trello.exceptions.ResourceUnavailable: Too Many Requests at https://api.trello.com/1/boards/599e8531d3ed724a1d312791/cards/open (HTTP status: 429)

The docs suggest the limit is per key and not per board so I think we need all jobs to use the same file lock and also to ensure all interactions are mutexed.

https://support.atlassian.com/trello/docs/api-rate-limits/
-- 
Your team hwcert-jenkins is requested to review the proposed merge of ~jocave/hwcert-jenkins-jobs:trello-flock-everywhere into hwcert-jenkins-jobs:master.

diff --git a/jobs/image-testing/run-image-testing-server.sh b/jobs/image-testing/run-image-testing-server.sh
index f0e04b2..547f6a9 100755
--- a/jobs/image-testing/run-image-testing-server.sh
+++ b/jobs/image-testing/run-image-testing-server.sh
@@ -197,6 +197,6 @@ if [ -f artifacts/submission.json ]; then
     cat artifacts/submission.json |jq -r '. | {{results: .results}}' > results.json
     curl -s "https://testflinger-revcache.canonical.com/v1/results/$JOB_NAME"; -X POST --header "Content-Type: application/json" --data @results.json
 fi
-flock /tmp/TRELLO-IMAGE-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello imageupdater -n {name} -i {image_file} -c {image_channel} -v $BUILD_ID --summaryfile artifacts/summary
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello imageupdater -n {name} -i {image_file} -c {image_channel} -v $BUILD_ID --summaryfile artifacts/summary
 echo "Test exit status: $TEST_STATUS"
 exit $EXITCODE
diff --git a/jobs/image-testing/run-image-testing.sh b/jobs/image-testing/run-image-testing.sh
index e6bf093..c329e50 100755
--- a/jobs/image-testing/run-image-testing.sh
+++ b/jobs/image-testing/run-image-testing.sh
@@ -241,6 +241,6 @@ if [ -f artifacts/submission.json ]; then
     cat artifacts/submission.json |jq -r '. | {{results: .results}}' > results.json
     curl -s "https://testflinger-revcache.canonical.com/v1/results/$JOB_NAME"; -X POST --header "Content-Type: application/json" --data @results.json
 fi
-flock /tmp/TRELLO-IMAGE-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello imageupdater -n {device}-{image-type} -i {image_file} -c {image_channel} -v $BUILD_ID --summaryfile artifacts/summary
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello imageupdater -n {device}-{image-type} -i {image_file} -c {image_channel} -v $BUILD_ID --summaryfile artifacts/summary
 echo "Test exit status: $TEST_STATUS"
 exit $EXITCODE
diff --git a/jobs/infrastructure/trello/deb-board-manager.sh b/jobs/infrastructure/trello/deb-board-manager.sh
index 39c979a..13889e6 100755
--- a/jobs/infrastructure/trello/deb-board-manager.sh
+++ b/jobs/infrastructure/trello/deb-board-manager.sh
@@ -19,11 +19,11 @@ set -x
 git -C hwcert-data pull || git clone git+ssh://ce-certification-qa@xxxxxxxxxxxxxxxxx/~canonical-hw-cert/hwcert-jenkins-jobs/+git/hwcert-data
 
 # This marks kernel sru bugs complete for kernel debs that are marked complete in trello
-certification-dashboard-manager.dashboard-manager --app trello kernelsrusync --project kernel-sru-workflow --deb
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello kernelsrusync --project kernel-sru-workflow --deb
 BUG_MANAGER_RESULT="$?"
 
 # This migrates trello cards to the appropriate lane based on kernel deb status
-certification-dashboard-manager.dashboard-manager --app trello --board "$APP_BOARD" debmanager --config hwcert-data/debs.yaml
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello --board "$APP_BOARD" debmanager --config hwcert-data/debs.yaml
 BOARD_MANAGER_RESULT="$?"
 
 if [ "$BUG_MANAGER_RESULT" -ne 0 ] || [ "$BOARD_MANAGER_RESULT" -ne 0 ]; then
diff --git a/jobs/infrastructure/trello/snap-board-manager.sh b/jobs/infrastructure/trello/snap-board-manager.sh
index bc250e2..e0b8ad4 100755
--- a/jobs/infrastructure/trello/snap-board-manager.sh
+++ b/jobs/infrastructure/trello/snap-board-manager.sh
@@ -19,11 +19,11 @@ set -x
 git -C hwcert-data pull || git clone git+ssh://ce-certification-qa@xxxxxxxxxxxxxxxxx/~canonical-hw-cert/hwcert-jenkins-jobs/+git/hwcert-data
 
 # This marks kernel sru bugs complete for snaps that are marked complete in trello
-certification-dashboard-manager.dashboard-manager --app trello kernelsrusync --project kernel-sru-workflow
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello kernelsrusync --project kernel-sru-workflow
 BUG_MANAGER_RESULT="$?"
 
 # This migrates trello cards to the appropriate lane based on snap status
-certification-dashboard-manager.dashboard-manager --app trello --board "$APP_BOARD" snapmanager --config hwcert-data/snaps.yaml
+flock /tmp/TRELLO-UPDATE.LOCK certification-dashboard-manager.dashboard-manager --app trello --board "$APP_BOARD" snapmanager --config hwcert-data/snaps.yaml
 BOARD_MANAGER_RESULT="$?"
 
 if [ "$BUG_MANAGER_RESULT" -ne 0 ] || [ "$BOARD_MANAGER_RESULT" -ne 0 ]; then