Performance tuning through iterations
-
Upload
gert-franz -
Category
Internet
-
view
151 -
download
2
Transcript of Performance tuning through iterations
![Page 1: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/1.jpg)
Performance Tuning a CFML script
Gert FranzRasia GmbH
![Page 2: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/2.jpg)
![Page 3: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/3.jpg)
WHO AM I?
• Gert Franz
– Involved with Lucee and Railo since day 1
– Into CFML for 10 11 12 13 14 15 16 17 years
– DBA, System architect
– CTO Rasia Ltd, CIO Rasia CH
• Rasia is founding Member of the Lucee association
![Page 4: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/4.jpg)
MY EVOLUTION
Assembler Basic Clipper Pascal Delphi Java CFML Lucee
![Page 5: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/5.jpg)
WHAT IS THIS ABOUT
• Having some fun with CFML
• Discrediting long known approaches
• Looking for new ways of solving a given problem
![Page 6: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/6.jpg)
TASK AT HAND
• The client reported that there was
– An import job that imports an .xlsx file
– Into a database table
– With some special rules
![Page 7: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/7.jpg)
INITIAL CODE
• The code is around 2 years old
• Takes around 20 minutes to execute
• Simply imports a total of 25000 recordsinto a database with some specific checks
• So nothing really fancy
![Page 8: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/8.jpg)
STARTING POINT
• Let's have a look at the source code that was used to fulfil the task
![Page 9: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/9.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
![Page 10: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/10.jpg)
MAIN POINTS VERSION 1
• Uses a default datasource• Uses lists for lookups• Creates an array to hold fieldnames• Creates a query and imports all into the
query• Uses two QoQ to delete unnecessary
records• Uses a QoQ to add only needed records to
an array• Insert the array into the database table
![Page 11: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/11.jpg)
IDEAS? PROPOSALS?
![Page 12: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/12.jpg)
FIRST ITERATION
• Convert everything to CFScript
WHY???
![Page 13: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/13.jpg)
ITERATION 1
• Use a function to load the XLSX content
• Optimize filling the fields
• Exit early in the cleanUpNumber() function
![Page 14: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/14.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript, reduced the field building effort
![Page 15: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/15.jpg)
ITERATION 2
• Replace the main QoQ with a nested struct
![Page 16: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/16.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
![Page 17: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/17.jpg)
QUERY OF QUERY EVIL
![Page 18: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/18.jpg)
QUERY OF QUERY
• Lucee is no database server
• So nothing of the following
– Query analyzer
– Indexes
– Statistics
– Query Optimizations
– etc!
![Page 19: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/19.jpg)
QUERY OF QUERY
• Avoid if possible at all times!
![Page 20: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/20.jpg)
ITERATION 3
• Eliminated the evaluate approach
![Page 21: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/21.jpg)
EVALUATE – SMALLER EVIL
![Page 22: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/22.jpg)
EVALUATE
• Evaluate sometimes does a lot of parsing
• I mostly use it only on serialized complex objects
• VERY OFTEN SEEN:
<cfset b = evaluate("form.#fieldName#")>
Instead use:
<cfset b = form[fieldname]>
![Page 23: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/23.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
![Page 24: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/24.jpg)
ITERATION 4
• Eliminated the SQL statements from getMarketID()
• Avoid too many unnecessary SQL statements
![Page 25: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/25.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
![Page 26: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/26.jpg)
ITERATION 5
• Eliminated all QoQ
• Only added stuff when model > 0
![Page 27: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/27.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
![Page 28: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/28.jpg)
ITERATION 6
• Why create a query in the first place?
• Why not create the resulting struct right away?
![Page 29: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/29.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminated creating a query and filling the struct
![Page 30: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/30.jpg)
ITERATION 7
• What is always the thing that consumes the most time?
• Queries
• Let's get rid of many of them
![Page 31: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/31.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
![Page 32: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/32.jpg)
ITERATION 8
• We only fill the struct in cases where it makes sense
• So why filter afterwards, if we can avoid filling these values in, in the first place
![Page 33: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/33.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Exiting early from filling the struct
![Page 34: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/34.jpg)
ITERATION 9
• Create a bulk insert, instead of lots of individual inserts
![Page 35: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/35.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
![Page 36: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/36.jpg)
ITERATION 10
• Instead of nested structs, use composite keys
• Nested structs use lots of recursions
![Page 37: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/37.jpg)
INTERMEDIATE RESULT
# T (ms) Δ (ms) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key
![Page 38: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/38.jpg)
ITERATION 11
• Replaced lists with arrays
• Generally speaking: Lists are to be avoided if possible
• Used len() instead of isEmpty() and added an exit early strategy
![Page 39: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/39.jpg)
INTERMEDIATE RESULT
# T (s) Δ (s) Imp. Total Factor Size Remarks
0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version
1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript
2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ
3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate
4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID
5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ
6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct
7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records
8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine
9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File
10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key
11 0.32 0.09 21.90% 0.05% 2050.31 4'035 B Replaced isEmpty() with len() and eq 0, eliminated Trim
![Page 40: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/40.jpg)
FINAL RESULT CHART
0
100
200
300
400
500
600
700
1 2 3 4 5 6 7 8 9 10 11 12
![Page 41: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/41.jpg)
FINAL RESULT CHART
0
100
200
300
400
500
600
700
1 2 3 4 5 6 7 8 9 10 11 12
![Page 42: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/42.jpg)
FINAL RESULT CHART
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12
![Page 43: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/43.jpg)
FINAL RESULT CHART
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12
![Page 44: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/44.jpg)
FINAL RESULT CHART
0
2
4
6
8
10
12
14
16
1 2 3 4 5 6 7 8 9 10 11 12
![Page 45: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/45.jpg)
FINAL RESULT CHART
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12
![Page 46: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/46.jpg)
FINAL RESULT CHART
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12
![Page 47: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/47.jpg)
FINAL RESULT CHART
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 2 3 4 5 6 7 8 9 10 11 12
![Page 48: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/48.jpg)
FINAL RESULT CHART
0
0.5
1
1.5
2
2.5
3
3.5
4
1 2 3 4 5 6 7 8 9 10 11 12
![Page 49: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/49.jpg)
FINAL RESULT CHART
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 3 4 5 6 7 8 9 10 11 12
![Page 50: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/50.jpg)
FINAL RESULT CHART
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1 2 3 4 5 6 7 8 9 10 11 12
![Page 51: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/51.jpg)
AS A PIE CHART
3.6%
91.7%
0.9%1.5% 0.4%0.2%1.0%0.1%0.5%0.0%0.0%
3.6%
0.9%
1.5%
0.4%
0.2%
1.0%
0.1% 0.5%
0.0%0.0%
![Page 52: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/52.jpg)
![Page 53: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/53.jpg)
CONCLUSIONS
• DON'T USE QueryOfQuery
• Don't use heavily nested structs
• Avoid lists, use structs or arrays instead
• Avoid too many database calls
![Page 54: Performance tuning through iterations](https://reader033.fdocuments.us/reader033/viewer/2022051521/5a64b0e27f8b9a2c5d8b45f7/html5/thumbnails/54.jpg)
CONCLUSIONS
• If you know the data you're handling, performance tuning is a lot easier
• First generalize in order to understand the code
• Then specialize in order to tune it