Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-7361

Investigate Query Performance Timeouts

    Details

      Description

      In the last days multiple timeouts occurred in the query performance tests. For example:
      https://hq2.camunda.com/jenkins/ci/job/7.7-platform-query-performance-oracle-12/

      AT:

      • check locally if the query performance downgrade in the last weeks
      • if possible identify reason for this downgrade

        Activity

        Hide
        christopher.zell Christopher Zell added a comment -

        The main reason lies in the data generation, which takes sometimes longer than usual. The timeouts are really short. If something will take same seconds longer it will fail.
        For example see this jobs: this fails and this succeeds , the difference is only a few seconds.

        Also here you can see that the data generation is the reason why the job timeouts https://hq2.camunda.com/jenkins/ci/job/7.7-platform-query-performance-db2-97/40/console.
        Data generation needs more than ~ 11 min. Next job succeeds because data generation takes only ~ 4 min https://hq2.camunda.com/jenkins/ci/job/7.7-platform-query-performance-db2-97/41/console

        There exist some approaches to fix this issue:

        • increase the timeouts -> The problem is, that in this case some performance issues are not spotted.
        • Generate data and create dumps to reuse the data -> We have to create new Jobs which are triggered if the SQL-Scripts are changed. In this case we generate for each database
          the new data and create a dump. These dumps are used in the performance tests.
        • Adjust the data generation -> Generate data for a given amount of time only. So the parameter will be the running time instead of the data count.
          This means sometimes we have less data sometimes more, but the data generation will take only the time which is defined.

        Sebastian Menski perhaps we can discuss this?

        Show
        christopher.zell Christopher Zell added a comment - The main reason lies in the data generation, which takes sometimes longer than usual. The timeouts are really short. If something will take same seconds longer it will fail. For example see this jobs: this fails and this succeeds , the difference is only a few seconds. Also here you can see that the data generation is the reason why the job timeouts https://hq2.camunda.com/jenkins/ci/job/7.7-platform-query-performance-db2-97/40/console . Data generation needs more than ~ 11 min. Next job succeeds because data generation takes only ~ 4 min https://hq2.camunda.com/jenkins/ci/job/7.7-platform-query-performance-db2-97/41/console There exist some approaches to fix this issue: increase the timeouts -> The problem is, that in this case some performance issues are not spotted. Generate data and create dumps to reuse the data -> We have to create new Jobs which are triggered if the SQL-Scripts are changed. In this case we generate for each database the new data and create a dump. These dumps are used in the performance tests. Adjust the data generation -> Generate data for a given amount of time only. So the parameter will be the running time instead of the data count. This means sometimes we have less data sometimes more, but the data generation will take only the time which is defined. Sebastian Menski perhaps we can discuss this?
        Hide
        christopher.zell Christopher Zell added a comment -

        In coordination with Christian Lipphardt we decided to use an timeout only for the maven step, which executes the query performance test. This can be done with the help of the 'Run with Timeout plugin'.

        That means the global timeout is replaced with an timeout which is set only for the second maven step. This maven step runs the query performance test.
        We had to add a new functionality to the BuildSteps.groovy to create a maven step with timeout.

        For each database an individual timeout is used for the query performance tests. These timeout can be adjusted, if they are not suitable.

        Show
        christopher.zell Christopher Zell added a comment - In coordination with Christian Lipphardt we decided to use an timeout only for the maven step, which executes the query performance test. This can be done with the help of the 'Run with Timeout plugin'. That means the global timeout is replaced with an timeout which is set only for the second maven step. This maven step runs the query performance test. We had to add a new functionality to the BuildSteps.groovy to create a maven step with timeout. For each database an individual timeout is used for the query performance tests. These timeout can be adjusted, if they are not suitable.
        Hide
        sebastian.menski Sebastian Menski added a comment -

        Christian Lipphardt could you please do the review of this? I would like to focus on the kawago project during the week. Thanks.

        Show
        sebastian.menski Sebastian Menski added a comment - Christian Lipphardt could you please do the review of this? I would like to focus on the kawago project during the week. Thanks.
        Hide
        lipphardt Christian Lipphardt added a comment -

        Christopher Zell:
        I added a Plugin version check to the build step with timeout so it fails during generation, when the necessary plugin version isn't available.
        I also updated the build-timeout plugin in Jenkins docker image to the required version.

        Show
        lipphardt Christian Lipphardt added a comment - Christopher Zell : I added a Plugin version check to the build step with timeout so it fails during generation, when the necessary plugin version isn't available. I also updated the build-timeout plugin in Jenkins docker image to the required version .
        Hide
        christopher.zell Christopher Zell added a comment -

        Christian Lipphardt a very nice thanks

        Show
        christopher.zell Christopher Zell added a comment - Christian Lipphardt a very nice thanks

          People

          • Assignee:
            lipphardt Christian Lipphardt
            Reporter:
            sebastian.menski Sebastian Menski
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: