Vocabulary/JCPUBenchmarks

This page is currently for discussion only. After there is consensus about the benchmarks it will be opened for user updates

Overview

This page reports timing measurements of a standard script on users' machines. To add info for your machine, run the script and edit this page to add your results. To see what each benchmark is measuring, look at the script.

Example of using the table: the sentence

a =. a + 1.

is made up of

 Sentence start/end + private name lookup + parse & exec 1-atom IP + private IP assignment

The timings are taken by running the same small code snippets millions of times. This will underestimate the number of mispredicted branches that would happen in a live system, because the test system will predict perfectly after so much repetition. For example if you had the sequence

   a =. a + 5
   b =. a * 2

the total time for the two sentences would be longer than the sum of the reported times for each sentence individually. Why? Because executing a verb requires an indirect branch to the routine executing the verb. Current CPUs can only predict that an indirect branch will go to the same address as it did the previous time the branch was executed; thus each verb (+ and *) will mispredict in the live system, with a penalty of ~12 clock cycles. When each sentence is run individually many times, each verb execution predicts perfectly.

Results

Date/ rev	Your name (optional)	Machine Make Model	CPU mfgr.	CPU Model Codename	OS	Year	Cost (USD)	Clock (GHz)	Benchmark times (IP=operation in place; sizes: D1$ [682-1365], D2$ [10922-21845], D3$ [174762-349525] atoms)
									Interpretive Overheads (ns)															Selected Array Operations
									Sentence start/end		Name lookup			Parse + exec monad/dyad (ns)						Assignment		Named verb exec	Empty {{ }}	x + y on lists (ns/#y)						D2$ lists (ns/#x or y)
														null		x + y short args				=. IP	=: IP			D1$		D2$		D3$		+/ y	+/\ y	Integer		Float		i.-family
									1 wd	>1 wd	priv.	base	z	]	0]	atom IP	atom IP	2=$y IP	+"0 1					IP	IP	IP	IP	IP	IP	D2$	D2$	/: y	/:~ y	/: y	/:~ y	x i. y (ns/#x,y)	x&i. y (ns/#y)	i.~ y (ns/#y)
									1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30
20241229/1	Henry Rich	MSI Summit A12Mt	Intel	i7-1280P Alder Lake	Win 11	2022	700	2.0	2.4	4.7	1.0	12	11	8.2	11	16	28	51	69	9.1	31	34	28	0.12	0.20	0.14	0.18	0.36	1.9	0.053	0.28	3.9	7.7	11	11	0.68	1.0	1.4

How To Add Your Results

1. Shut down other time-consuming processes on your PC. No browser! No email!
2. Start a clean J session under Jqt
3. Copy the script below onto your clipboard.
4. Go to Jqt and press F8. It will run the script and copy your result info onto the clipboard
   • If you don't have Jqt, execute the components of the test and put the result onto the clipboard
5. Edit this Wiki page by clicking on 'Edit' after the heading for the Results
6. Paste your info at the beginning of the last line of the table (i. e. at the start of the line |})
7. Edit the fields enclosed in << >> to add your personal info

Benchmark Script

sigd2 =: (((0 j. 0 >. 1 - [: <. 10&^.)@[ ": ] *~ [: <. 0.5 + %) (_1+<.)&.(10&^.))`('0'"_)@.(0&>:)"0  NB. convert, to 2 sig digits
NB. time sentence m, after replacing each @n by n{::y
etimem =: {{ 'setup timed' =. (}: ;  {:) ((,.~ ('@',":)&.>@i.@#) ":&.> y)&stringreplace&.> boxopen m   NB. (<setup) ; timed
3 : (setup (,<) 'y 6!:2 ' , 5!:5 <'timed') x
}}
estimate =: {{
'Expect the full run with y=',(":y),' to take about ',(": sigd2 y * 240 * (5e7 (6!:2) '1. + 0.') % 2.8e_8),' seconds'
}}
benchmark =: {{
stime =. 6!:1''  NB. start time
mplr=.y   NB. Set as high as you can stand to wait

midcache =. 32768 524288 8388608  NB. byte lengths that fit into D1$ D2$ D3$ but not the next-smaller cache.  On the low side, to cater for old CPUs
arglens =. <. (midcache % 3*8) */ 2 ^ <: (%~ i.@>:) 10  NB. 3x11, lens [0.5,1] * midcache

text=.'|-',LF,'|| ',(6!:0 'YYYYMMDD'),'/1',LF,'|| <<Your name (optional)>> ',LF,'|| <<Machine Make<br>Model>> ',LF,'|| <<CPU mfgr>> ',LF,'|| <<CPU Model<br>Code Name>> ',LF,'|| <<OS>> ',LF,'|| <<Year>> ',LF,'|| <<Cost(USD)>> ',LF,'|| <<Clock(GHz)>> ',LF

NB. smoutput 'probe time ' , (": 1e8 (6!:2) '1. + 0.') , 'seconds'

NB. Test 1 - single self-defining term (1-word sentences bypass most parse overhead)
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * minsdttime =. (<. mplr * 1e9) ('';'1') etimem '' 

NB. Test 2 - full parse, but no memory allocation or parse/exec of words
parse1 =. (<. mplr * 1e9) ('';']0') etimem ''  NB. 1 exec - Run long for accuracy
parse2 =. (<. mplr * 1e8) ('';'0]0]0]0]0]0]0]0]0]0]]0') etimem ''  NB. 2 execs monad then dyad Run long for accuracy 10 times
parse3 =. (<. mplr * 1e8) ('';']]]]]]]]]]]0') etimem ''  NB. 2 execs monad then monad Run long for accuracy 10 times
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * parse0 =. parse1 - 0.1 * (parse2-parse1)  NB. Extrapolate to time reqd for 0 execs, i. e. parser setup only

NB. Test 3 - lookup private name
text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =. 01';']a') etimem '') -  ((<. mplr * 5e8) ('a =. 01';']0') etimem '')

NB. Test 4 - single public name in base
text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =: 01';']a') etimem '') -  ((<. mplr * 5e8) ('a =: 01';']0') etimem '')
4!:55 <'a'

NB. Test 5 - single public name in z.  We hope this misses the Bloom filter in base, and hits an empty chain in base
text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('xyzzy_z_ =: 01';']xyzzy') etimem '') -  ((<. mplr * 5e8) ('xyzzy_z_ =: 01';']0') etimem '')
4!:55 <'xyzzy_z_'

NB. Test 6 - parse & execute ]
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * 0.1 * (parse3-parse1)  NB. Extrapolate to time reqd for ]

NB. Test 7 - parse & execute 0]
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * 0.1 * (parse2-parse1)  NB. Extrapolate to time reqd for 0] 

NB. Test 8 - add singletons, inplace (a+b+c net of time for b+c) - includes ONLY parse/exec time, no parser startup overhead
singletonnonip =. (<. mplr * 1e8) ('';'1.+]0.') etimem ''
singletonip =. (<. mplr * 1e8) ('';'0.+0.+0.+0.+0.+0.+0.+0.+0.+0.+1.+]0.') etimem ''
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * 0.1 * singletonip - singletonnonip 

NB. Test 9 - add singletons, not inplace (includes parsing a+b, allocating & freeing result area)
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * singletonnonip - parse0

NB. Test 10 - add 2-atom lists (not inplace) - parse a+b, arg analysis, allo/deallo, operation
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * parse0 -~ listsetup =. (<. mplr * 5e7) ('';'1. 2+]0. 3') etimem ''

NB. Test 11 - add with rank (creates 2x2 result not inplace) - parse a+b, arg analysis, allo/deallo, operation
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * parse0 -~ (<. mplr * 5e7) ('';'1. 2 (+"0 1) ]0. 3') etimem ''

NB. Test 12 - modify private name in place (assignment only)
text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =. 01';'a =. a {:: ((00;01))') etimem '') -  ((<. mplr * 5e8) ('a =. 01';'a {:: ((00;01))') etimem '')

NB. not inplace, abandoned value, a little slower  text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =. 01';'a =. 0. + 1.') etimem '') -  ((<. mplr * 5e8) ('a =. 01';'0. + 1.') etimem '')

NB. not inplace, non-abandoned value, a little slower  text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =. 01';'a =. a {:: ((01;00))') etimem '') -  ((<. mplr * 5e8) ('a =. 01';'a {:: ((01;00))') etimem '')

NB. public name inplace   text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =: 01';'a =: a {:: ((00;01))') etimem '') -  ((<. mplr * 5e8) ('a =: 01';'a {:: ((00;01))') etimem '')
NB. 4!:55 <'a'

NB. public name not inplace, abandoned value, a little slower  text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 5e8) ('a =: 01';'a =: 0. + 1.') etimem '') -  ((<. mplr * 5e8) ('a =: 01';'0. + 1.') etimem '')
NB. 4!:55 <'a'

NB. Test 13 - modify public name not in place with non-abandoned value (assignment only)
text =. ' ' ,~ text  , '|| ' , sigd2 1e9 * ((<. mplr * 1e8) ('a =: 01';'a =: a {:: ((01;00))') etimem '') -  ((<. mplr * 1e8) ('a =: 01';'a {:: ((01;00))') etimem '')
4!:55 <'a'

NB. Test 14 - Execute named verb.  Report net of (minimum parse) - do not deduct name lookup because the name is short-cached.  Also count the lookup of the name (to create the reference) because nouns don't need references
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * cachednametime =. (parse0) -~ (<. mplr * 1e8) ('vb =. ]';'vb 0') etimem ''
4!:55 <'vb'

NB. Test 15 - start & end explicit definition.  We have to use a name because pppp does not allow n : m.  Report time net of (named verb+time to exec 0) - do not deduct name lookup because the name is short-cached
text =. ' ' ,~ text , '|| ' , sigd2 1e9 * (cachednametime+minsdttime) -~ (<. mplr * 1e8) ('vb =. {{ 0 }}';'vb 0') etimem ''
4!:55 <'vb'

NB. Tests 16-21: add lists in various caches, inplace and not
NB. Although the caches have very different characteristics, different parts of the arguments are taken in different caches which gives
NB. strong nonlinear components in any regression.  This is misleading because it can't be extrapolated beyond the sizes of the user's
NB. actual caches.  We spurn the regression and just report average time, which tends to give weight to the longer arguments
withip =. ('a =. ';'') ,"0/ <"0 arglens  NB. 2x3x11, ip then cache# then lens
times =. listsetup -.~ mplr {{ (<. x * 1e9 % len) (('a =. _0.5 + ($b) ?@$ 0 [ b =. _0.5 + @1 ?@$ 0';'@0a + b') etimem) 'ip len' =. y }}"0 1 withip
NB. regret =. ((|: (_1 _2 _3 * {:$arglens) {."0 1 arglens) %.~ ,)"2 times  NB. (*** not used ***) Find linear model for each cachesize, 2x3 (no constant term)
regret =. times %"1&:(+/"1) arglens
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret  NB. change axis order to ip then cache

NB. Test 22 - +/ D2$
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e9 % >y) (('b =. _0.5 + @0 ?@$ 0';'(+/) b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 23 - +/\ D2$
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e8 % >y) (('b =. _0.5 + @0 ?@$ 0';'(+/\) b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 24 - /: D2$  integer
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e7 % >y) (('b =.  @0 ?@$ 2 * @0';'/: b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 25 - /:~ D2$ integer
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e7 % >y) (('b =.  @0 ?@$ 2 * @0';'(/:~) b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 26 - /: D2$  float
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e7 % >y) (('b =.  @0 ?@$ 0';'/: b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 27 - /:~ D2$ float
currarglens =. (,1) { arglens  NB. run this only on D2$
times =. mplr {{ (<. x * 1e7 % >y) (('b =.  @0 ?@$ 0';'(/:~) b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 28 - x i. y D2$ integer not smallrange (using hash).  200 lookups, many more hashes
currarglens =. <. 0.4 * (,1) { arglens  NB. run this only on D2$ - smaller x because of hashtable
times =. mplr {{ (<. x * 1e9 % >y) (('b =. ({~ ?~@#) ((-:200) ?@$ 1e6) , (-:200) ((? #) { ]) a [ a =.  @0 ?@$ 1e6';'a i. b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens + 200    NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 29 - x&i. y D2$ integer not smallrange  2000 lookups
currarglens =. <. 0.4 * (,1) { arglens  NB. run this only on D2$ - smaller x because of hashtable
times =. mplr {{ (<. x * 5e8 % 2000) (('b =. ({~ ?~@#) ((-:2000) ?@$ 1e6) , (-:2000) ((? #) { ]) a [ a =.  @0 ?@$ 1e6';'xiy =. a&i.';'xiy b') etimem) y }}"0 <"0 currarglens NB. 50% mix of found/notfound, random order
regret =. times %"1&:(+/"1) currarglens ]"0 (2000)  NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

NB. Test 30 - i.~ y self-classify   D2$ integer not smallrange
currarglens =. <. 0.4 * (,1) { arglens  NB. run this only on D2$ - smaller x because of hashtable
times =. mplr {{ (<. x * 5e8 % >y) (('b =.  @0 ?@$ 1e6';'i.~ b') etimem) y }}"0 <"0 currarglens
regret =. times %"1&:(+/"1) currarglens    NB. Average time (ignoring setup)
text =. text , ; ' || '&,&.> |: sigd2&.> 1e9 * regret

text =. text,LF
smoutput 'elapsed time ' ,": stime -~ 6!:1''
text
}}  NB. argument controls run time

estimate 1.0  NB. predict the total running time

'result put onto clipboard - paste it into Wiki' (([: wd 'clipcopy *' , ]) [ smoutput@[) benchmark 1.0  NB. run the benchmark