Addons/math/mt/Benchmarking

From J Wiki
Jump to: navigation, search
User Guide | Installation | Development | Categories | Git | Build Log

Benchmarking process uses matrices of size(s) and generator given, and produces scalar result for each size. Benchmarking result is a sum of estimated execution durations (in seconds) for some verbs set. Verbs set is defined by self-test verb test_mt_ . An execution duration for each verb is estimated as proposed in [1]: "the minimum run-time of 3-5 executions of the program when the machine is lightly loaded.".

Benchmarking process outputs some useful information while progress. This output is directed into:

  • console
  • log file: ~temp/mt.benchmark.*.log
  • raw measurements data file: ~temp/mt.benchmark.*.raw
  • formatted result file: ~temp/mt.benchmark.*.result

Typical workflow to compare different hardware environments performance for the same J interpreter version

  # on one host:
  user@i5-3230M:~/j64-805> ./jconsole.sh
     load 'math/mt'
     mkmat=. _1 1 0 3 _6 4&gemat_mt_
     bmkrun=. mkmat benchmark_mt_
     sizes=. 100 150 200 300 400 500
     ] out1=. 'i5-3230M_j64-805' bmkrun sizes
     NB. a lot of output, very long execution...
  75.305 283.243 1040.378 9666.966 26840.712 66151.747
     exit ''

  # on another host:
  user@i7-4790:~/j64-805> ./jconsole.sh
     load 'math/mt'
     mkmat=. _1 1 0 3 _6 4&gemat_mt_
     bmkrun=. mkmat benchmark_mt_
     sizes=. 100 150 200 300 400 500
     ] out2=. 'i7-4790_j64-805' bmkrun sizes
     NB. a lot of output, very long execution...
  42.323 145.205 395.336 1782.877 8592.176 21163.120
     NB. now we can compare a speed-up of 2nd host against 1st one:
     sizes ,. 75.305 283.243 1040.378 9666.966 26840.712 66151.747 % out2
  100 1.77929
  150 1.95064
  200 2.63163
  300 5.42212
  400 3.12386
  500  3.1258
     exit ''

Typical workflow to compare different J interpreter versions performance for the same hardware environment

  # in one interpreter:
  user@i7-4790:~/j64-805> ./jconsole.sh
     load 'math/mt'
     mkmat=. _1 1 0 3 _6 4&gemat_mt_
     bmkrun=. mkmat benchmark_mt_
     sizes=. 100 150 200 300 400 500
     ] out1=. 'i7-4790_j64-805' bmkrun sizes
     NB. a lot of output, very long execution...
  42.323 145.205 395.336 1782.877 8592.176 21163.120
     exit ''

  # in another interpreter:
  user@i7-4790:~/j64-806> ./jconsole.sh
     load 'math/mt'
     mkmat=. _1 1 0 3 _6 4&gemat_mt_
     bmkrun=. mkmat benchmark_mt_
     sizes=. 100 150 200 300 400 500
     ] out2=. 'i7-4790_j64-806' bmkrun sizes
     NB. a lot of output, very long execution...
  43.179 150.100 400.637 1814.688 8729.023 20626.823
     NB. now we can compare a speed-up of 2nd interpreter against 1st one:
     sizes ,. 42.323 145.205 395.336 1782.877 8592.176 21163.120 % out2
  100 0.980176
  150 0.967388
  200 0.986769
  300  0.98247
  400 0.984323
  500    1.026
     exit ''

References

  1. Magne Haveraaen, Hogne Hundvebakke. Some Statistical Performance Estimation Techniques for Dynamic Machines. Appeared in Weihai Yu & al. (eds.): Norsk Informatikk-konferanse 2001, Tapir, Trondheim Norway 2001, pp. 176-185. URL: https://www.ii.uib.no/saga/papers/perfor-5d.pdf