We saw it failing sometimes due to dying executors:
[2019-08-06T08:32:52.159Z] Cannot contact optimize-ci-build-es-6.7.0-elasticsearch-compatibility-31-h0mk8: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelExceptio
And sometimes due to read timeouts on the AWS IT stage (higher latency and poor performance of the AWS instance).
- we should retry stages as we do on branch builds to handle dying executors like we do in branch builds
- for the AWS IT we should increase the timeouts to cover up hiccups