[CAM-9612] BulkFetch complex/object-Variable values in context of historic detail data Created: 19/Dec/18  Updated: 01/Feb/19  Resolved: 21/Dec/18

Status: Closed
Project: camunda BPM
Component/s: engine
Affects Version/s: None
Fix Version/s: 7.11.0, 7.10.2, 7.9.9, 7.8.14, 7.11.0-alpha1

Type: Task Priority: L3 - Default
Reporter: Sebastian Bathke Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0 minutes
Time Spent: Not Specified
Original Estimate: 0 minutes


 Description   

AT:

  • given:
    • I have more than 100 million variables in the Camunda engine
    • at least 10 % of the variables are complex variables (e.g. JSON or XML)
  • when:
    • I fetch 10 000 historic variable updates from the historic detail table using the Optimize rest api or the historic detail endpoint
  • then:
    • it does not take more than 2 seconds
  • such that:
    • Even if the user has a lot of complex variables, Optimize can import the data in a very fast manner

Hints:
In the context of bulk fetching historic variable updates from the historic detail table the current implementation of e.g. HistoricDetailQueryImpl#executeList and OptimizeHistoricVariableUpdateQueryCmd#fetchVariableValues sequentially calls getTypedValue for each variable entry which will perform one additional query per complex/object variable in AbstractSerializableValueSerializer#readValue to resolve the actual value.

This doesn't scale with maxResults values in the magnitude of several thousand as used by optimize in the context of importing data e.g.
maxResults -> response time
100 -> 400ms
200 -> 640ms
500 -> 1s
1000 -> 2s
10_000 -> 20s

We need a better scaling implementation of fetching variables for this usecase that e.g. fetches all byteArray entries in one bulk.


Generated at Thu Nov 14 05:05:40 CET 2019 using JIRA 6.4.6#64021-sha1:33e5b454af4594f54560ac233c30a6e00459507e.