NYCJUG/2006-03-08
Meeting Overview
We had a good, meandering meeting that somehow managed to touch on all the agenda items. I'll present them in the order of the agenda though that's not necessarily the order in which we touched on them.
For the "Beginner's regatta", I presented an update of a topic from last month's meeting: different ways of waiting for an external command to finish. As is often the case in these matters, there are routines in the standard J libraries superior to my hand-crafted solutions.
In this case, the functions in "task.ijs", fork and shell, do a fine job of invoking a command and either returning immediately or waiting for it to finish. Using the "wait" command I've been invoking to test for any lag in waiting for a command to finish, one could enter
load 'task' _1 fork 'cmd /c wait 5'
where the "_1" left argument tells fork to wait for the command to finish. The small disadvantage of invoking the command processor "cmd" (necessary because "wait" is an internal DOS command), is that it brings up a DOS window. I haven't figured out a way around this. For example tests, see "cmdWaitTestingFork.txt".
Regular Expression Classification Table
We explored a technique I've been using with good success: table-driven file classification using regular expressions. This is a table consisting most essentially of a column of regular expressions and another column of executable expressions. Files sent to a central server during the day are processed by a routine that classifies files according to which regular expression they match, then running the corresponding expression on the files thus selected. See "RegExTableTests.txt" for an example.
A most important restriction is that each file be classified by only one of the regular expressions; also, each expression should pick up a likely file. This quickly highlights the necessity of building the test suite alongside the code, a common practice in "Extreme Programming". However, as the test suite grows in complexity, it, too, needs to be tested. This raises the question of how we test the tests. For instance, we need to be sure we'll pick up a duplicate classification but we don't want to pollute our good table with deliberate errors. Dan had some good suggestions on dynamically modifying the table to accomplish this.
Learning/Teaching Array Thinking
We discussed a problem from the "TopCoder" website involving finding the longest prime-free intervals for a given range of positive integers. See "topcoderfarfromprimeEG.txt" for a fuller explanation.
We compared the looping solutions presented with more array-oriented approaches but were stymied to come up with any sort of general scalar-to-array method. Much of array-oriented thinking comes from a different initial approach that might just require lots of practice.
- Beginner's regatta: update to "wait for external command": standard functions in "task.ijs": fork and shell; see also "cmdWaitTestingFork.txt".
- Show-and-tell: can Tom tell us about openGL? Look at "findEEparametric.txt" and the points generated by the brute force solution in the "EqnEquality" files, e.g. "findEqnEqualityPrecis.txt".
- Advanced topics: dropOffs.ijs - using regular expressions (see "testingTheRegExpTable.txt", "regExChars.txt") to classify files and testing the classifications ->"addNewRegExp&Test.txt", "RegExpTable.xls" and "RegExTableTests.txt"
- Learning and teaching J: quotesWin example or "topcoderfarfromprime.ijx"? See "topcoderfarfromprimeEG.txt".
The three cavities of the body are
the head cavity, the tooth cavity,
and the abominable cavity.
- The Revenge of Anguished English by Richard Lederer
Beginner's regatta
Testing "Fork" with the "Wait" Command
I tried using J's fork command with the system wait command to test if the fork returns immediately or waits for the underlying command to finish.
6!:2 '_1 fork ''cmd /c wait 5''' NB. Sometimes initial invocation is slow.
9.6700656
6!:2 '_1 fork ''cmd /c wait 5'''
5.4788475
6!:2 '_1 fork ''cmd /c wait 8'''
8.0502921
NB. Works fine except that it pops up DOS window: how to get rid of it?
_1 fork 'wait 5'
_1 fork 'cmd /c wait 5'
_1 fork 'cmd /c wait 5 | NUL'
NB. These didn't work. What's the difference between a desktop shortcut that runs
NB. with a normal window versus a minimized one?
wmin=. fread '\amisc\Wait3WinMin.lnk'
wnorm=. fread '\amisc\Wait3WinNorm.lnk'
wmin-:wnorm
0
NB. The difference is this one byte in position 60.
I. wmin~:wnorm
60
a. i. 60{&>wmin;<wnorm
7 1
NB. Values for minimized versus normal. But can we invoke a .LNK from command
NB. line?
_1 fork 'cmd /c C:\amisc\Wait3WinMin.lnk'
_1 fork 'C:\amisc\Wait3WinMin.lnk'
_1 fork 'cmd /c C:\amisc\Wait3WinNorm.lnk'
NB. These don't work.
NB. Run test suite used on other command waiters:
load '\amisc\j\nycjug\200603\commandWait.ijs'
6!:2 'tmtsk=. testWaitGapTask 10'
55.895464
tmtsk
1 1.9605981
2 1.9954598
3 2.9473634
4 4.0659215
5 5.0212745
6 5.9370571
7 7.0856086
8 7.9757402
9 8.9272701
10 9.9783364
tmtsk=. tmtsk,testWaitGapTask 10 [ tmtsk=. tmtsk,testWaitGapTask 10 [ ...
$tmtsk
4 10 2
tmtsk=. tmtsk,testWaitGapTask 10 [ tmtsk=. tmtsk,testWaitGapTask 10 [ ...
dd=. 'C:\amisc\J\NYCJUG\200603\'
(dd) fileVar 'tmtsk' NB. Save these timings and get the others.
+-+---------+
|1|TMTSK.DAT|
+-+---------+
(<'C:\amisc\J\NYCJUG\200602\') unfileVar_WS_&.> 'tmwts';'tmwe'
+-----------------------------------------------+------------------------...
|+-+-------------------------------------------+|+-+---------------------...
||1|+-------------------------+---------+-----+|||1|+--------------------...
|| ||C:\amisc\J\NYCJUG\200602\|TMWTS.DAT|tmwts|||| ||C:\amisc\J\NYCJUG\20...
|| |+-------------------------+---------+-----+||| |+--------------------...
|+-+-------------------------------------------+|+-+---------------------...
+-----------------------------------------------+------------------------...
$tmwts
10 10 2
$tmtsk
11 10 2
>,&.>}."1&.>mean &.>tmwe;tmwts;tmtsk
1.59798 2.25568 3.22976 3.77794 5.78405 6.77765 6.83266 7.66218 8.58564 9.55089
1.02312 2.02381 3.01202 3.97675 4.99607 6.01084 7.05268 8.00437 8.96521 9.99008
1.06253 2.00606 2.99934 4.0352 4.98803 6.00605 6.99741 8.01578 8.97279 10.0006
>,&.>}."1&.>stddev &.>tmwe;tmwts;tmtsk
0.584813 0.678296 1.17039 1.31878 0.634437 0.467846 2.14851 2.6231 2.11798 2.86598
0.401127 0.0330933 0.0299177 0.0394623 0.0152714 0.0276781 0.0526376 0.0477404 0.0262925 0.032303
0.314534 0.0604375 0.0418279 0.0269598 0.0278296 0.0496729 0.0616933 0.0539134 0.0450852 0.0332828
plot >,&.>}."1&.>mean &.>tmwe;tmwts;tmtsk
plot >,&.>}."1&.>stddev &.>tmwe;tmwts;tmtsk
NB. the latter 2 are much more congruent...
6!:2 'tmtsk60=. testWaitGapTask 60'
1770.5921
Code to Test Fork Overhead
[This is ancient code which no longer has a chance of working (except for testWaitGapTask, at the end) because it requires tools from another age - like oleautomation and win32api - and for other reasons as well.]
NB.* commandWait.ijs: test how well we can wait for an external command.
load '~user/code/cmdtool.ijs'
coinsert 'fldir' [ load 'filefns'
NB. +.-------- Either use "cmdtool" (requires WinSh) ------------+.
testWaitGap=: 3 : 0
aa=: '' conew 'cmdtool'
tmwt=. (y.,2)$0
for_ii. >:i.y. do.
cmd=. 'wait ',":ii
tm=. 6!:2 'runCmd__aa ''',cmd,''''
tmwt=. (ii,tm) (<:ii)}tmwt
end.
tmwt
)
NB. +.-------- or use "winexec" (requires Windows API) ------------+.
winexec=: 'WinExec' win32api NB. Run external command
NB.* runCmdWait: run external command and wait for it to finish.
runCmdWait=: 3 : 0
pid0=. cmdShellNums '' NB. Get process ids of all command shells running.
winexec y.;1 NB. This starts another command shell.
pidn=. pid0-.~cmdShellNums '' NB. New processes since 1st check may include the
while. +./(cmdShellNums '')e. pidn do. wait 1 end. NB. .bat file we just started.
)
NB.* cmdShellNums: get process numbers of all DOS command shells running.
cmdShellNums=: 3 : 0
if. 0=#y. do. y.=. 'CMD.EXE' end.
;n2j&.>(+./&>(boxopen y.) E.&.>toupper&.>1{pss)#0{pss=. getPS ''
)
getPS=: 3 : 0
NB.* getPS: get information of running processes: col 0 is process ids,
NB. col 1 is class and text name.
alph=. '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
rnd=. (?6##alph){alph
tmpfl=. (endSlash getTempDir ''),'PS',rnd,'.txt'
delFile tmpfl
sink=. winexec ('cmd /C ps > ',tmpfl);0
wait 0.1 NB. Wait 0.1 second for command to finish
max=. 9 NB. and try 9 times just for sake of bogged system.
ctr=. 1
while. ctr<:max do.
if. fexist tmpfl do. break.
else. wait (ctr%10) [ ctr=. >:ctr end. NB. Wait longer each time since
end. NB. must be slow.
ps=. dlsp&.>f2v tmpfl
ps=. (2+(3{.&.>ps)i.<'PID')}.ps NB. Drop header
delFile tmpfl
ps=. (wh{.&.>ps),:(>:&.>wh=. ps i.&.>' ')}.&.>ps
)
testWaitGapWE=: 3 : 0
tmwt=. (y.,2)$0
for_ii. >:i.y. do.
cmd=. 'cmd /c wait ',":ii
tm=. 6!:2 'runCmdWait ''',cmd,''''
tmwt=. (ii,tm) (<:ii)}tmwt
end.
tmwt
)
difTest=: 0 : 0
fread batfl=. (ad=. '\amisc\'),'testWait.bat'
del foo.tmp
wait %1
echo > foo.tmp
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 1'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 2'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 3'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 4'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 5'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 6'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 7'
fexist ad,'foo.tmp' [ runCmdWait 'cmd /c ',batfl,' 8'
)
The following should work assuming a Windows "cmd" shell and a "wait" command that takes a number of seconds as an argument.
NB. +.-------- or use "task.ijs" ------------+.
testWaitGapTask=: 3 : 0
tmwt=. (y.,2)$0
for_ii. >:i.y. do.
cmd=. 'cmd /c wait ',":ii
tm=. 6!:2 '_1 fork cmd'
tmwt=. (ii,tm) (<:ii)}tmwt
end.
tmwt
)
Show-and-tell
We looked at a problem posed by Mike Daly on the J Forum: he wanted to know what J syntax to use to find solutions to a^2 + b^2 + c^2 = (a - b)(b - c)(c - a). He is aware that there are infinitely many solutions.
There were a few replies to this, after which Mike clarified that he was looking for solutions over integers. See "findEqnEqualityPrecis.txt" for a solution. However, I had a method for real numbers that I liked but became interested in the shape of the solution-space. To see this, I needed a way to plot 3-dimensional points.
Fortunately, Oleg Kobchenko provided a way to do this with his "plot3d.ijs" contribution to the forum. The plots are somewhat interesting and there are a number of useful lessons to be learned from different aspects of solving the system of equations, but I intend to write these up later. See "ptsWtw*.jpg" for example plots.
Initial Attempts
Here is how we set out to find solutions to this equation. Please note that this code was written in an earlier version of J that used "y." for "y" and "x." for "x".
eqn0=: 3 : '+/*:y.' NB. Left-hand side of equation
eqn0 1 2 3
14
eqn1=: 3 : '*/2-/\y.,{.y.' NB. Right-hand side of equation
eqn1 1 2 3
2
rndn=. ?1000$1e3 NB. 1000 random integers
$3 eqn0\rndn
998
$3 eqn1\rndn
998
I.(3 eqn0\rndn)=3 eqn1\rndn NB. Any equalities?
rndn=. _1000+?100000$2001 NB. Try again
I.(3 eqn0\rndn)=3 eqn1\rndn
rndn=. _10000+?100000$20001 NB. and again
I.(3 eqn0\rndn)=3 eqn1\rndn
rndn=. (_10000+?100000$20001)%>:?100000$1000 NB. and again
I.(3 eqn0\rndn)=3 eqn1\rndn
diffeqns=: 3 : '(eqn0 y.)-eqn1 y.' NB. Consolidate the tests
diffs=. 3 diffeqns\rndn
(<./,>./)diffs
_2.0998913e10 2.9516792e10
(<./,>./)|diffs NB. We care about absolute differences
0.01071377 2.9516792e10
Clearly this is very unlikely to give us a solution. However, if we ignore the integer constraint, we should be able to iterate toward a solution.
Iterating Toward a Solution
Let's start with some of the closest solutions from our random set and iterate around those values.
eqn0=: 3 : '+/*:y.' NB. Set up equations
eqn1=: 3 : '*/2-/\y.,{.y.'
diffeqns=: 3 : '(eqn0 y.)-eqn1 y.' NB. and differencer.
We start by picking wide range of random arguments and looking at the smallest and largest values.
(<./,>./)rndn=. (_1e4+?1e5$20001)%>:?1e5$1e3 _9879 9658
Check how close these points are: the best (closest to zero) and the worst.
(<./,>./)|diffs=. 3 diffeqns\rndn 0.010086928 5.5282426e10
Let's start with the ten best points.
close1s=. 10{./:|diffs NB. Pick 10 closest results as
seeds=. close1s{3[\rndn NB. starting points...
_0.63838384 0.37931034 _17.451852
_6.4783599 5.4468339 6.1741868
. . .
_0.10789474 0.10091743 0.3027933
diffeqns"1 seeds
0.010086928 0.011770021 _0.023991408 ... 0.096197183
Start with 1st seed and look a little bit around it in 3-space.
$&.>nn=. (0{seeds) +&.>/ <steps _1 1 11
+--+--+--+
|11|11|11|
+--+--+--+
We are using steps to choose evenly-spaced points.
steps=: 3 : 0 NB.* steps: vector of numbers from num, to num, in numsteps steps. 'from to numsteps'=. y. from+(to-from)*(numsteps-1)%~i.numsteps )
We make an 11x11x11 neighborhood around a seed. We use an odd number of points to include original point so we can do no worse.
$>>,&.>/&.>/ nn 11 11 11 3 <./,|dd=. diffeqns&> >,&.>/&.>/ nn 0.006545809
This is an improvement. Checking the index of the point on which it is based:
>ix=. <($dd)#:(,|dd)i.<./,|dd 5 5 0
This new point and how close it is:
>ix{>,&.>/&.>/ nn
_0.63838384 0.37931034 _17.461852
diffeqns >ix{>,&.>/&.>/ nn NB. Check it...
0.0065475394
Now we can use this point as new centroid for tighter neighborhood constrained to plus or minus 0.1 from it.
nn=. (>ix{>,&.>/&.>/ nn) +&.>/ <steps _0.1 0.1 11
<./,|dd=. diffeqns&> >,&.>/&.>/ nn
0.00054704513
This shows further improvement. Which point is that?
>ix=. <($dd)#:(,|dd)i.<./,|dd
5 5 4
>ix{>,&.>/&.>/ nn NB. New point...
_0.63838384 0.37931034 _17.481852
We can keep on doing this until we see no significant improvement, like this where we start with another seed:
nn=. (1{seeds) +&.>/ <steps _1 1 11
<./,|dd=. diffeqns&> >,&.>/&.>/ nn
0.011770021
diffeqns 1{seeds
0.011770021
This shows no improvement, so we tighten the search neighborhood and try again.
nn=. (1{seeds) +&.>/ <steps _0.1 0.1 13
<./,|dd=. diffeqns&> >,&.>/&.>/ nn
0.0029753954
>ix=. <($dd)#:(,|dd)i.<./,|dd
8 4 4
>ix{>,&.>/&.>/ nn
_6.4450266 5.4135006 6.1408534
Make it tighter around this better point and continue to find better values.
nn=. (>ix{>,&.>/&.>/ nn) +&.>/ <steps _0.001 0.001 13
<./,|dd=. diffeqns&> >,&.>/&.>/ nn
9.0611054e_5
>ix=. <($dd)#:(,|dd)i.<./,|dd
3 12 12
>ix{>,&.>/&.>/ nn
_6.4455266 5.4145006 6.1418534
Picturing a Solution
Thanks to some 3-D graphing help from Oleg Kobchenko, I was able to get an idea of what the solution space to this equality looks like.
Advanced topics
We look at using regular expressions to classify files and at how we test the classifications.
Using Regular Expressions
Testing the Regular Expression Table
0!:101 wdclipread ''
createNewDatedDir ymd NB. New day, new sub-directory
'TSNOW FLSNOW'=: listFiles DODIR NB. Current files' info
noChange=. ONCETHRU*.FLSNOW-:FLSOLD NB. Change or 1st invocation
noChange
0
0!:101 wdclipread ''
rc=. 0
checkDeletions '' [ ASSERR=: 'Error in "checkDeletions"'
0
ASSERR=: 'Error in "checkAdditions"'
0!:101 wdclipread ''
0<checkAdditions ''
1
0!:101 wdclipread ''
renameUnderscoresToBlanks DODIR NB. E.G. "inner_space.txt"->"inner space.txt"
1684 1616
wait 2 NB. Let first set of renames finish.
2
wait 2 [ renameDatedFls DODIR NB. E.G. "dif 1-3-06.xls"->"dif 01-03-06
2
0!:101 wdclipread ''
'TSNOW FLSNOW'=: listFiles DODIR NB. Update current file info for name ch
FLSNOW
+-----------------------------------+-------------------+------+---+------+
|27280 03-10-2006 CASH.xls |2006 3 10 9 28 29 |37888 |rw-|-----a|
+-----------------------------------+-------------------+------+---+------+
|xxx80 02-15-2006 CASH.xls |2006 2 15 9 46 47 |51200 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|AAFUtils.ijs |2006 2 27 18 30 21 |35972 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|Copy of Rpt Somefund VP NAV (2).xls|2006 2 14 19 49 44 |24576 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|Copy_of_Rpt_Somefund_VP_NAV_(2).xls|2006 2 23 18 20 46 |25088 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|Sometg Manager 03-06-06.xls |2006 3 7 11 34 29 |104960|rw-|------|
+-----------------------------------+-------------------+------+---+------+
|dropOffs.ijs |2006 2 23 16 3 21 |17320 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|fileFns.ijs |2006 2 24 10 18 48 |20547 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|fof.ijs |2006 2 27 17 54 34 |67019 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
|stopDropOffs.ijs |2005 12 21 13 37 41|1263 |rw-|------|
+-----------------------------------+-------------------+------+---+------+
0!:101 wdclipread ''
fls=. jfi dir DODIR,'*.*' [ ASSERR=: 'Error in "checkFiles"' NB. Get file names
reconfls=. (1{"1 FT) rxmatch&.>/toupper&.>{."1 fls NB. and classify
reconfls=. _1~:({.@:,)&>reconfls NB. them.
0!:101 wdclipread ''
wfr=. +./reconfls NB. Which files recognized
wfr
1 0 0 0 0 0 0 0 0 0
0{reconfls
0 0 0 0 0 0 0 0 0 0
reconfls
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
$reconfls
16 10
$FT
16 3
$FLSNOW
10 5
I. {."1 reconfls
5
5{FT
+--------------+--------------------------------------------------------------------+-----------------------+
|AAF daily cash|^(27280|16250|38660|27350) [0-9]{1,2}-[0-9]{1,2}-[0-9]{2,4} CASH.XLS|moveCashFilesToDatedDir|
+--------------+--------------------------------------------------------------------+-----------------------+
0!:101 wdclipread ''
pl=. 's'#~1~:nrf=. +/wfr
0!:101 wdclipread ''
logMsg_logger_ 'Recognized ',(":nrf),' file',pl,':'
New file at 2006/03/10 10:09:28.391:
"27280 3-10-2006 CASH.xls"
Recognized 1 file:
msg=. punclist wfr#_1|.&.>({."1 fls),&.><'""'
logMsg_logger_ ' ',msg,'.'
New file at 2006/03/10 10:09:28.391:
"27280 3-10-2006 CASH.xls"
Recognized 1 file:
"27280 03-10-2006 CASH.xls".
0!:101 wdclipread ''
I.+./|:reconfls
5
fltype=. 5
0!:101 wdclipread ''
recogfls=. (fltype{reconfls)#{."1 fls NB. get the file names and run
0!:101 wdclipread ''
' recogfls',~,>(<fltype,FTHDR i. <'PGM'){FT
moveCashFilesToDatedDir recogfls
recogfls
+-------------------------+
|27280 03-10-2006 CASH.xls|
+-------------------------+
moveCashFilesToDatedDir recogfls
+-+
|1|
+-+
qts''
2006 3 10 10 13 48.446
regExChars.txt
There are two different sets of metacharacters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized in square brackets. Outside square brackets, the metacharacters are as follows:
\ general escape character with several uses
^ assert start of string (or line, in multiline mode)
$ assert end of string (or line, in multiline mode)
. match any character except newline (by default)
[ start character class definition
| start of alternative branch
( start subpattern
) end subpattern
? extends the meaning of (
also 0 or 1 quantifier
also quantifier minimizer
* 0 or more quantifier
+ 1 or more quantifier
also "possessive quantifier"
{ start min/max quantifier
Part of a pattern that is in square brackets is called a "character class". In a character class the only metacharacters are:
\ general escape character ^ negate the class, but only if the first character - indicates character range [ POSIX character class (only if followed by POSIX syntax) ] terminates the character class
The following sections describe the use of each of the metacharacters....
Introduction
Professor Massimo Di Pierro (DePaul University) in his article "What is Blockchain?", Computing in Engineering and Science Vol. 19 No. 5 2017 https://www.computer.org/csdl/magazine/cs/2017/05/mcs2017050092/13rRUyv53Jl (Reprinted in Computing Edge April 2018)[1] provides a quick review of Blockchain technology and a small implementation example in Python. The example uses JSON library to serialize Python data into a string that can then be run through a hash to complete the block chain. It also takes advantage of Python's built in list management features. There is nothing Python specific in the implementation and making some reasonable design choices a J language implementation is straight forward.
- ↑ M. Pierro, "What Is the Blockchain?" in Computing in Science & Engineering, vol. 19, no. 05, pp. 92-95, 2017. doi: 10.1109/MCSE.2017.3421554 keywords: {bitcoin;contracts;peer-to-peer computing;digital signatures;authentication} url: https://doi.ieeecomputersociety.org/10.1109/MCSE.2017.3421554