User:Devon McCormick/ParallelizedJCodeExamples

From J Wiki
Jump to: navigation, search

Examples of Parallel Code in J

This contains complete examples of code presented at the APL2010 conference in Berlin, Germany, in September, 2010. There are two examples of parallel code tasks as well as an implementation of mutex (mutual exclusion), all written in the J programming language (version 6.02). A copy of the J programming environment is freely available from jsoftware.com. When attempting to run this code, be aware that some longer lines may have “wrapped”.

Immediately below is the first task, the original “photo flipping” code. The customizations for my site should be contained in the first few lines of globals’ assignment near the top. There are a number of utility functions here for ancillary uses but the core function is regflipphotos.

Examples of the parallelizing extensions to this code are contained below as well. When this script is loaded, an anonymous noun is displayed by the section of code beginning “smoutput 0 : 0” and ending with the bare “)” at the end of that short paragraph. This displays examples of some of the most commonly used phrases in our J session when we load this script so that frequently-used invocations can simply be selected and run in the session.

So, as typically used, I would select the line

   info,<6!:2 'info=: getFrom1Drive ''F:\'''

to get all the photos from directories on the camera memory chip available as the “F:” drive, assuming a standard set of directory names that may vary by camera model. The “info” and “6!:2” parts assign information about the photos to a variable for later use and time the transfer, respectively. Currently these names are set for a Fuji FinePix which creates numbered sub-directories with names starting with “DSCF” under a directory “DCIM”. A Nikon camera might create numbered directories starting with “DSCN”.

After this function moves the pictures from the camera chip to a sub-directory named by the date range of pictures obtained, I would run some variant of the line

   4 parcelOutFlipping dd=. endSlash '"'-.~>0{info   NB.  4 parallel flippers

to spin off four independent processes to flip the photos in the recently-created directory. Typically I have been using the left argument “2” instead of “4” because my current machine has two cores and this seems logical. However, I’ve been running with four parallel processes lately to gather realistic timings to evaluate the efficiency of running more processes than there are cores.

[More recently, I've reverted to invoking only two processes because the four-process version seemed to occasionally lock up my PC.]

Parallel Photo Flipping

NB.* savePhotoDirInfo.ijs: save dir info of photos on camera/chip, copy pix.

load 'csv task filefns images'
coinsert 'fldir'
PHOTOP=: 'c:\amisc\pix\Photos\'  NB. Top dir where I keep photos.
jpeg_quality_ima3_=: 99          NB. Save .jpgs as 99% lossless.

smoutput 0 : 0           NB. Remind us of usual use.
info,<6!:2 'info=: getFrom1Drive ''F:\'''
info,<6!:2 'info=: getFrom1Drive ''C:\amisc\'''   NB. This or preceding...
6!:2 'regflipphotos ''"''-.~>0{info'              NB. Single-thread or
4 parcelOutFlipping dd=. endSlash '"'-.~>0{info   NB.  4 parallel flippers
fsize dd,'time.out'
ratioKept >0{info
)

NB.* jpgSzs: give size and number of .jpg files in dir y.
jpgSzs=: 13 : '(+/,#);2{"1 dir ''*.jpg'',~endSlash y'

NB.* szRatio: give files sizes, ratio of photos discarded and kept.
szRatio=: (jpgSzs,ratioKept)
szRatio_eguse_=: 0 : 0
   TD=. 'C:\amisc\pix\Photos\2010Q2\'
   kr=. (3 : 'y(],.[:szRatio&.>],&.>~[:<[) jd dir ''*'',~y=. endSlash y') TD
   ((0{"1 kr),.":&.>&>1{"1 kr) writecsv 'KeptRatio2010Q2.csv'
)

NB.* evenlyPartition: evenly partition y into x pieces w/smaller at end.
evenlyPartition=: 4 : '(x(([:i.]) e. ([:i.[) * [:>.%~)#y)<;.1 y'

NB.* evenlyPartitionBy: evenly partition y into (>0{x) pieces by >1{x amts.
evenlyPartitionBy=: 4 : '(+/ (+/\>1{x) >/~ (}.i.>0{x)*(>0{x)%~+/>1{x) </. y'

NB.* delFlIfExist: delete file if it exists.
delFlIfExist=: 3 : 'if. fexist y do. ferase y end.'

NB.* parcelOutFlipping: run x separate processes to flip photos in dir y.
parcelOutFlipping=: 4 : 0
   'fls szs'=. |:0 2{"1 dir '*.jpg',~y=. endSlash y
   szs=. ;szs [ fls=. <"1 fls
NB.   fls=. x evenlyPartition fls  NB. Put filenames into evenly-divided lists.
   fls=. (x;szs) evenlyPartitionBy fls  NB. % filenames evenly by file size.
   scrfls=. (<'.ijs'),~&.>(<'FlipScript'),&.>":&.>i.x
   delFlIfExist&.>scrfls
   1!:44 y [ svdir=. 1!:43 ''      NB. Move to target dir to flip photos.
   (<fread jpath '~Code/sampFlipPhotos.ijs') fwrite&.>scrfls
   (<y) appendArgsToScript&.>scrfls;&.><"0 fls
   exe=. (<'"',(jpath '~bin'),'\j.exe" -jijx "'),&.><endSlash 1!:43 ''
   fork&.>exe,&.>scrfls,&.>'"'
   1!:44 svdir
NB.EG parcelOutFlipping '"'-.~>0{info
)

NB.* appendArgsToScript: for photo directory x, add filename info to scripts.
appendArgsToScript=: 4 : 0
   'scrflnm flnms'=. y   NB. Script file names, photo file names
   (LF,~LF,~'IPD=: <''',x,'''') fappend scrflnm   NB. Photo dir as global
   (')',~;LF,~&.>'PHFLS=: <;._2]0 : 0';dltb&.>flnms) fappend scrflnm
   (LF,LF,~'onlyRuntime ''''') fappend scrflnm    NB. Run code if standalone.
)

NB.EG Append code like this, with different "PHFLS", to multiple script files.
NB. IPD=: 'c:\amisc\pix\Photos\2010Q1\20100311\'
NB. PHFLS=: 0 : 0
NB. DSCF3837.jpg
NB. DSCF3838.jpg
NB. DSCF3839.jpg
NB. )

NB.* regflipphoto: flip a .jpg pic 1/4 counterclockwise as is most common.
regflipphoto=: 3 : 0
   (1 0 2|:|."2 a.{~read_image y) write_image y
NB.EG
)

NB.* regflipphotos: flip all .jpg in dir 1/4 counterclockwise.
regflipphotos=: 3 : 0
   y=. endSlash y
   regflipphoto&.>(<y),&.>{."1 dir y,'*.jpg'
NB.EG 6!:2 'regflipphotos ''c:\pix\'''
)

NB.* getFrom1Drive: get photos from drive y -> \amisc\pix\photos\[yyyyQq] year and quarter
getFrom1Drive=: 3 : 0
   phd=. endSlash y
   topdirs=. dir phd,'*.'
NB. +----+------------------+-+---+------+
NB. |DCIM|2008 6 22 13 28 44|0|rw-|----d-|   NB. Example top-level photo dir
NB. +----+------------------+-+---+------+
   if. (<'DCIM') e. {."1 topdirs do. getPixDir phd,'DCIM\' else. 0 end.
NB.EG getFrom1Drive 'F:\'
NB.EG info;]6!:2 'info=: getFrom1Drive ''E:\'''
)

NB.* getPixDir: get pictures from directory on camera assuming standard names.
getPixDir=: 3 : 0
NB.   subds=. {."1 dir phd,'DCIM\*.'
   subds=. {."1 dir '*.',~y=. endSlash y
NB. +--------+--------+
NB. |102_FUJI|101_FUJI|                      NB. Example photo sub-dirs
NB. +--------+--------+
   'destdir sumszs'=. accumSvInfo y
   cmd=. (<'move "'),&.>(<y),&.>subds,&.><'\*.jpg" ',destdir
   cmd=. cmd,(<'move "'),&.>(<y),&.>subds,&.><'\*.avi" ',destdir
   rr=. shell&>cmd
   rr=. >,&.>/_3{.&.><;._2&.>rr
   destdir;(ratioKept destdir-.'"');sumszs
NB.EG 'destdir ratios totsz'=. getPixDir 'H:\DCIM\'
)

NB.* accumSvInfo: accumulate picture file information & save before altering.
accumSvInfo=: 3 : 0
   inf=. >,&.>/dir&.>(<y),&.>({."1 dir y,'*.'),&.><'\*.jpg'
   inf=. inf,>,&.>/dir&.>(<y),&.>({."1 dir y,'*.'),&.><'\*.avi'
   inf=. inf/:1{"1 inf                       NB. Order by ascending date
NB. +----------------------+----------------------+
NB. |G:\DCIM\102_FUJI\*.jpg|G:\DCIM\101_FUJI\*.jpg|
NB. +----------------------+----------------------+
   dtstr=. combineDates >~.3{.&.>1{"1 inf    NB. Date string, e.g. 20080628-29
   initDt=. 2{.>(<0 1){ inf                  NB. Characterize by initial date
   dsd=. (":0{initDt),'Q',":>.3%~1{initDt    NB. yyyyQq, e.g. 2008Q3<-3rd qtr 2008
   shell 'mkdir ',destdir=. '"','"',~PHOTOP,dsd,'\',dtstr
   saveInfoFl destdir;<(":&.>_2}."1 inf)     NB. Info file->new dir
   destdir;+/;2{"1 inf                       NB. Sum file sizes
)

NB.* ary2csv v represent enclosed array in csv format.
ary2csv=: 3 : 0
   ;(<@(,&LF)@}:@;)"1 ('"'&,@(,&'",')@(#~ >:@(=&'"'))) &.> ,&.> 8!:2 &.> y
)

NB.* saveInfoFl: save file of information on file sizes, save dates and times.
saveInfoFl=: 3 : 0
   'ddir inf'=. y
   ifl=. (ddir-.'"'),'\info.csv'   NB. Append w/o header if file exists.
   (ary2csv (fexist ifl)}.inf,~'Name';'Date';'Size') fappend ifl
   ifl
)

NB.* ratioKept: estimate ratio of photos taken to those retained.
ratioKept=: 3 : 0
   if. 1=#$y do.              NB. Single, enclosed item is dir name.
       fls=. {."1 jfi dir (y-.'"'),'\*.jpg'
   else. fls=. {."1 y end.
   if. 0=#nn=. countPicNums fls do. 0 0 0 else.
   (|0 1-(#nn)%>:(>./-<./)nn),#fls end.      NB. Ratio kept, discarded; # files
)

countPicNums=: 3 : 0
   nn=. (4{. 4}.])&.>y        NB. Get picture nums from file names.
   nn=. ".&>nn#~isValNum&>nn  NB. Only numbers
NB. 6667 and 3333 are arbitrary to catch break from 9999 -> 0 rollover.
   if. ((6667+./ .>~])*. 3333+./ .<~]) nn do. nn=. nn+10000*nn<:6667 end.
)

NB.* combineDates: combine list of dates into string from earliest-latest,
NB. combining common high-order date parts; see examples below.
combineDates=: 3 : 0
   dtlst=. ({.,:{:)4 _2 _2{.&.>"1]4 lead0s&.>/:~(_2{.1,$y)$,y
   deq=. *./\=/dtlst   NB. Where leading parts of dates are equal
   dtstr=. ;(deq#0{dtlst),}:,(<'-'),.~(-.deq)#"1 dtlst
   dtstr}.~-'-'={:dtstr
NB.EG    combineDates ,:2008 6 30
NB.EG 20080630
NB.EG    combineDates 2008 6 29,:2008 6 30
NB.EG 20080629-30
NB.EG    combineDates 2008 6 29,2008 6 30,2008 7 1,:2008 7 3
NB.EG 20080629-0703
NB.EG    combineDates 2008 12 30,2008 12 31,:2009 1 1
NB.EG 20081230-20090101
)

Parallel Photo Flipping Task Template

This is the template customized for each parallel task by the code above. This version implements mutex to prevent the multiple processes from over-writing each others’ information in their common timing file.

NB.* sampFlipPhotos.ijs: sample dedicated photo flipper.

load 'task images filefns ~Code/mutex.ijs'
coinsert 'fldir' [ coinsert 'mutex'
jpeg_quality_ima3_=: 99

NB.* regflipphoto: flip a .jpg pic 1/4 counterclockwise as is most common.
regflipphoto=: 3 : 0
   (1 0 2|:|."2 a.{~read_image y) write_image y
NB.EG
)

onlyRuntime=: 3 : 0
NB.* onlyRuntime: only invoke if not loaded via interactive session.
   if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
       1!:44 >IPD [ arg=. 1|.'""',}.;' ',&.>ARGV_z_ [ flout=. 'time.out'
       getHold ''
       (LF,~'Start flips (for ',arg,') @ ',":qts'') fappend flout
       releaseHold ''
       tm=. ":6!:2 'regflipphoto&.>IPD,&.>PHFLS'
       getHold ''
       (LF,~'Finished flips (for ',arg,') in ',tm,' seconds @ ',":qts'') fappend flout
       releaseHold ''
       2!:55 ''
   end.
)

Parallel Directory Parsing

The following code demonstrates another task run in parallel. This code parses the contents of a specified set of files, usually starting from the root of a hard-drive, to create several global vectors of information about all the files and directories under the starting point. These vectors can be used for a number of things but are most commonly used to implement a flexible backup routine which allows us to copy a certain amount of the most recently-changed files.

There are three scripts here. The main one, parseDir.ijs contains the original, serial directory-parsing code. The parallel extensions to this comprise two scripts: the main routine parallelParseDir.ijs which implements the parsing in parallel, and the template script pllPDSub.ijs which is customized for each parallel task by the main routine.

NB.* ParseDir.ijs: tools to parse directory listing->backup most recently changed.
NB. Will look for command-line arguments SZLIM (max bytes to backup) and TARGDIR
NB. (target directory to which to write backups).

require '~Code/locationInfo.ijs'   NB. On which machine are we running?
require '~Code/parallelParseDir.ijs'

DEBUGON=: 0         NB. Show messages as program progresses (1) or don't (0).
SESSWDW=: 0         NB. Open session window (1) or not (0)
DBGFL=: 'C:\Temp\parseDir.tmp'          NB. These 3 things allow logging at load
sepLF=: 13 : ';((": :: ])y),10{a.'      NB.  time: DBGFL (log file), sepLF, and
fappendDHM=: 4 : '(,x) (#@[ [ 1!:3) :: _1: (([: < 8 u: >) ::]) y'     NB. this.
3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:13';ARGV_j_,<6!:0 '') fappendDHM DBGFL end.
)

NB. "C:\Program Files\J602\j.exe" c:\amisc\JSys\user\code\parseDir.ijs -jijx SZLIM 10e6 RUNANYDAY Y RUNFROMSAVED Y TARGDIR D:\

NB.* nameSpan: given 2 names like pfxYYYYMMDD, combine->pfxYYYYMMD1-D2.
NB.* consolidateBkpsTowardPast: copy all contents of successive dirs into
NB.* coreRunDaily: core code for "runDaily": daily backups.
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
NB.* rmEndSep: remove terminal path separator from string.
NB.* extractExclusions: extract names of target, exclude dirs, and files from global ExcludeUsual.
NB.* excludeFiles: exclude designated files from list to back up.
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
NB.* cvtDt2Num: convert Y M D h m s date to single num: YYYYMMDD.day fraction.
NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
NB.* initFlsDir: parse memory-mapped file of directory listing->file, dir info.
NB.* getInfo: get directory into into file, memory-map and parse it.
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
NB.* extract1SubdirList: get first full sub-directory listing out of many.
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
NB.* mcopyto: text of DOS .BAT file to do multiple copies.
NB.* runDaily: backup to run every day: save some most recently changed files.
NB.* setGlobalParms: assign globals according to defaults or command-line overrides.
NB.* NYto01: convert 'N' or 'Y' to 0 or 1, respectively.
NB.* onlyRuntime: only invoke if not loaded via interactive session.
NB.* runFromSavedVars: Run backup assuming dir&file vars already saved.

NB.* nameSpan: given 2 names like pfxYYYYMMDD, combine->pfxYYYYMMD1-D2.
nameSpan=: 3 : 0
NB. y=. ({:,{.) dd  NB. Earliest, latest
   endlast=. 1 i.~ ~:/>y
   (>0{y),'-',endlast}.>1{y
)

consolidateBkpsTowardPast=: 3 : 0
NB.* consolidateBkpsTowardPast: copy all contents of successive dirs into
NB. earliest starting with oldest->consolidated backups with newest version
NB. overwriting older ones.
   y=. 2{.boxopen y           NB. Files' prefix, dir in which to work
   pfx=. openbox {.y          NB. Prefix is "WL" for laptop, "WD" desktop
   if. 0=#>1{y do.
       wrkdir=. 'C:\Temp\'
   else. wrkdir=. endSlash >1{y
   end.
   dd=. jd dir wrkdir,'*.'    NB. Just directories
   dd=. (<pfx,'[0-9]{8}$') rxfirst&.>dd   NB. Only names like {pfx}YYYYMMDD
   dd=. \:~dd-.a:             NB. Order from most to least recent.
   if. nameExists 'TOPDIR' do. svTD=. TOPDIR end.
   TOPDIR=: wrkdir
   2 moveDirOverAnother/\dd
   newnm=. nameSpan ({:,{.) dd
   cmd=. (TOPDIR{.~>:TOPDIR i. ':'),' && cd ',(TOPDIR}.~>:TOPDIR i. ':'),' && '
   cmd=. cmd,'ren "',(>{:dd),'" "',newnm,'"'
   shell cmd
   if. nameExists 'svTD' do. TOPDIR=: svTD   NB. Re-instate or
   else. 4!:55 <'TOPDIR' end.                NB.  remove global.
NB.EG consolidateBkpsTowardPast 'WD';'C:\Temp\WrkDesk\'
)

coreRunDaily=: 0 : 0
NB.* coreRunDaily: core code for "runDaily": daily backups.
   setGlobalParms ''     NB. Cmd-line size limit, target dir, if present.
   wkdy=. dow 3{.qts ''  NB. Weekday number: 0=Sunday, 6=Saturday
   wkdy e. 1 2 3 4 5     NB. Only Mon-Fri unless any
   RUNANYDAY+.wkdy e. 1 2 3 4 5
   sink=. runDaily '' [ 4!:55 <'ASSERR'
   nameExists 'ASSERR'
)

NB. Display commonly-used set of commands in usual sequence.
smoutput 0 : 0
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=. PllDirInfoEG ''C:\'''
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: getDirFlInfo SRCDIR'
(<'\Temp\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
'batfl cmds'=. buildBatFl_parseDir_ 700e6;'C:\Temp\Recent\'
shell batfl
)

coclass 'parseDir'
require 'jmf files dir dt logger filefns task'
coinsert 'base fldir'
USUVARS=: 'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
EV=: getEnviVars ''
WINDIR=: endSlash ,>EV{~<1,~(toupper&.>0{"1 EV)i.<'WINDIR'

buildBatFl=: 3 : 0
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
   'szlim targ'=. y
   xclud=. excludeFiles targ
   dtdord=. xclud-.~\:FLDTS        NB. File list indexes in date desc. order
   toobig=. szlim<:dtdord{FLSZS    NB. Exclude any single file > size limit
   toobig=. toobig#dtdord          NB.  as this will truncate list prematurely.
   dtdord=. dtdord-.toobig
   ss=. +/\dtdord{FLSZS
   cutoff=. 1 i.~ss>szlim
NB. Do better job noting which files excluded because too big; the following
NB. fails to account for cutoff.
   if. DEBUGON do.
       pl=. 's'#~1~:#toobig
       logMsg_logger_ 'Excluding file',pl,' singly > size limit: ',":#toobig
   end.
   ix=. cutoff{.dtdord
   cmds=. (0{DIRNMS),makeCopyCmds targ;<ix   NB. 0{DIRNMS gives disk to which
   tmpd=. getTempDir ''                      NB.  to copy from.
   cmds v2f batfl=. tmpd,'CDMDCopy.bat'
   batfl;<cmds
NB.EG 'batfl cmds'=. buildBatFl 700e6;'C:\Temp\Recent\'
)

locChildren=: 3 : '(<^:(L. = 0:)y),~(<#~0~:#) I. DIRDEP e. >0{y'
indicateSubdirs=: 3 : '~.;;locChildren^:_&.>I. y'

indicateSubdirs0=: 3 : 0
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
   xdix=. I. y
   childxd=. xdix-.~I. DIRDEP e. xdix
   while. 0<#childxd do.
       xdix=. ~.xdix,childxd
       childxd=. xdix-.~I. DIRDEP e. xdix
   end.
   xdix                       NB. Index into DIRNMS of all subdirectories.
NB.EG xdix=. indicateSubdirs DIRNMS e. 'c:\amisc';'c:\Program Files'
)

NB.* rmEndSep: remove terminal path separator from string.
rmEndSep=: 3 : '(]}.~[:-PATHSEP_j_={:)"1 dtb y'
extractExclusions=: 3 : 0
NB.* extractExclusions: extract names of target, exclude dirs, and files from global ExcludeUsual.
   targ=. rmEndSep y               NB. Exclude target to avoid unwanted recursion.
   sections=. '[ExcludeDirs]';'[ExcludeFiles]'
   xu=. <;._1 LF,ExcludeUsual-.CR
   xu=. xu#~&.>-.&.>+./\&.>(<'NB.')E.&.>xu  NB. Exclude comments
   xu=. xu#~0~:;#&.>xu
   whsect=. >+./&.>sections E.&.>/ xu
   secord=. /:sections i. ' '-.~&.>xu#~+./whsect
   targ;secord{(+./whsect)<;._1 xu
NB.EG 'targ xd xf'=. extractExclusions y
)

excludeFiles=: 3 : 0
NB.* excludeFiles: exclude designated files from list to back up.
   'targ xd xf'=. extractExclusions y
   whxd=. DIRNMS}.~&.>(2*;':'e.&.>DIRNMS)*;DIRNMS i.&.>':'  NB. Drop disk prefix.
   whxd=. (toupper&.>whxd)e. PATHSEP_j_,&.>toupper&.>xd,<targ    NB. Exclude target
   whxd=. (1) (indicateSubdirs whxd)}whxd                        NB.  to avoid recursion.
   whxf=. (toupper&.>FLNMS)e. toupper&.>xf                       NB. Exclude files.
   xclud=. I. whxf+.FLPARENT e. I. whxd
NB. xclud is list of indexes into FLNMS=files to exclude.
)

dirDependencies=: 3 : 0
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB. showing directory and subdirectories as parent-child relations.
   odir=. y
   DIRDEP=: (#odir)$_1   NB. Dummy entry for 1st node: _1->no parent.
   for_dc. i.#odir do.
       cd=. PATHSEP_j_,~&.>dc{odir
       len=. #>cd        NB. Which prefixes match only current?
       subs=. (;cd-:&.>len{.&.>odir)*.-.;PATHSEP_j_ e.&.>len}.&.>odir
       DIRDEP=: subs}DIRDEP,:dc
   end.
NB.EG dirDependencies 'c:';'c:\t1';'c:\t1\sb1';'c:\t2';'c:\t2\sb2';'c:\t1\sb3'
NB. _1 0 1 0 3 1         NB. Parent index for each input; _1 for no parent.
)

NB.* dirInfo: put directory info in more usable format: names, dates, sizes, dir flag.
dirInfo=: ([:((0{"1]) ; ([:>1{"1]) ; ([:;2{"1]) ; [:<'d'e.&>4{"1]) dir)

getDirFlInfo=: 3 : 0
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
   dskInf=. >(];[: dirInfo '\*',~]) generalWalkTree rmEndSep y
   DIRNMS=: 0{"1 dskInf                           NB. Full names of all paths
   DIRDEP=: (] i. (]{.~PATHSEP_j_ i:~])&.>) DIRNMS
   DIRDEP=: (_1) (I. DIRDEP=i.#DIRDEP)}DIRDEP     NB. Dirs' dependency tree: parent indexes
   DIRDEP=: (_1) (I. DIRDEP>:#DIRDEP)}DIRDEP      NB. "_1" is root (no parent).
   isfl=. -.;4{"1 dskInf                          NB. Exclude dir info->only files.
   FLNMS=: isfl#;1{"1 dskInf
   ned=. 0~:#&>2{"1 dskInf                        NB. No empty directories
   FLDTS=: isfl#;cvtTS21Num &.>ned#2{"1 dskInf    NB. Date as single num: YYYYMMDD.day fraction
   FLSZS=: isfl#;3{"1 dskInf                      NB. File size in bytes.
   FLPARENT=: isfl#;(#&.>3{"1 dskInf)#&.>i.#DIRNMS
   FLNMS;FLDTS;FLSZS;FLPARENT;DIRNMS;<DIRDEP
)

NB.* cvtDt2Num: convert Y M D h m s date to single num: YYYYMMDD.day fraction.
cvtDt2Num=: 3 : 0"1
   NB. 90065 = 5 + 24 60 60#.24 60 60 NB. Max secs/day+5 fudge for leap secs.
   (100#.3{.y)+90065%~24 60 60#._3{.y
NB.EG ' 20090803.32600899' -: 18j8":cvtDt2Num 2009 8 3 8 9 22
)
NB. Only distinguishes to about 1/10,000 second.

NB. Need to include a [regexp] exclusion section to apply to files, e.g. "saves-{d}*", "*~", etc.
NB. Exclude the usual files and directories from being copied given list of Files &
NB. Directories (result of munge_dir) & source Disk & target Disk[:\dir] names.
NB. Some DBs big enough to be done separately.

NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
ExcludeUsual=: 0 : 0
[ExcludeDirs]
$avg
.emacs.d
CFGSAFE
C_DILLA
Documents and Settings
MSOCache
Program Files
Recycled
Recycler
WINDOWS
amisc\DCIM
i386

[ExcludeFiles]   NB. Files to exclude from any directory
~
*.bz2
*.zip
AUTOEXEC.BAT
CONFIG.SYS
IO.SYS
MSDOS.SYS
NTDETECT.COM
NTLDR
SECURITY
SOFTWARE
SYSTEM
SYSTEM.ALT
Thumbs.db
boot.ini
eventlog.log
)
NB.*To do: actually use the wildcards in the file list above!!!

NB. Replace WINNT with actual Windows system dir from environment var.
ExcludeUsual=: ('WINNT';<'\'-.~WINDIR}.~WINDIR i. '\') stringreplace ExcludeUsual

NB. -------- Dir listing fns: parse text file directory listing:
NB. -------- this is a separate way to accomplish what has been done above.
initFlsDir=: 3 : 0
NB.* initFlsDir: parse memory-mapped file of directory listing->file, dir info.
   JCHAR map_jmf_ 'DIRLSTFL';y
   DBSTR=: LF,' Directory of '
   WHDB=. (CR,DBSTR) E. DIRLSTFL   NB. Where directory breaks are
   WHDB=: (I. WHDB),<:#DIRLSTFL
   DIRNMS=: FLNMS=: ''
   FLSZS=: DIRPARENT=: FLPARENT=: i.0
   FLDTS=: 0$0.0
)

3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:383';<6!:0 '') fappendDHM DBGFL end.
)

getInfo=: 3 : 0
NB.* getInfo: get directory into into file, memory-map and parse it.
NB.   winexec 'cmd /C dir /A /S C:\ > C:\allfls2.dir';1
   dirlstfl=. y
   if. 0=#dirlstfl do.
       dirlstfl=. 'C:\allfls2.dir'
       shell 'dir /A /S C:\ > ',dirlstfl
   end.
   initFlsDir dirlstfl
   for_ix. i.<:#WHDB do.
       ch=. extract1SubdirList WHDB{~ix+0 1
       process1Subdir ch
   end.
   unmapall_jmf_ ''
)

process1Subdir=: 3 : 0
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
   ch=. y
   thisdir=. (($DBSTR)}.ch) {.~ 1 i.~ LF E. ($DBSTR)}.ch
   'isnew thisdn'=. addPath thisdir
   ch=. (<;._1 ch)-.a:             NB. break into lines; no empty lines
   ch=. ch#~' '~:;{.&.>ch          NB. Get rid of lines beginning with space.
   ch=. <;._1&.>' ',&.>dsp&.>ch    NB. break apart lines by spaces
NB. re-join any names with embedded spaces
   ch=. |:>(3{.&.>ch),&.><&.>(}.@;)&.>(' '&,)&.>&.>3}.&.>ch
   chtit=. 'DATE';'TIME';'SIZE';'NAME'  NB. row titles for "ch"
   whmootdirs=. (ch{~chtit i. <'NAME')e. ,&.>'.';'..'
NB. "SIZE" column has "<DIR>" indicator for directory, size for file.
   whdir=. ((ch{~chtit i. <'SIZE')e. <'<DIR>')*.-.whmootdirs
   addPath&.>(<thisdir,'\'),&.>whdir#ch{~chtit i. <'NAME'
   whfls=. -.whdir+.whmootdirs
   ch=. whfls#"1 ch
   FLNMS=: FLNMS,ch{~chtit i. <'NAME'
   FLPARENT=: FLPARENT,(+/whfls)$thisdn
   FLSZS=: FLSZS,;n2j&.>(ch{~chtit i. <'SIZE')-.&.>','
   FLDTS=: FLDTS,;DateTimeCvt &.>,&.>/' ',&.>ch{~chtit i. 'DATE';'TIME'
   thisdn
)

extract1SubdirList=: 3 : 0
NB.* extract1SubdirList: get first full sub-directory listing out of many.
   'st end'=. y
   ch=. CR-.~(st+i.>:end-st){DIRLSTFL
)

addPath=: 3 : 0
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
   p2a=. <;._1 '\',y    NB. Path to add, e.g. 'C:\top\mid\bottom'
   p2a=. p2a-.a:
   isnew=. 0 [ wh=. _1
   for_nm. p2a do.                 NB. "wh" is parent index of current node...
       if. 0=#wh2=. I. DIRNMS e. nm do. NB. new subdir
           DIRPARENT=: DIRPARENT,wh
           wh=. {.<:#DIRNMS=: DIRNMS,nm NB. will be parent of next node, if any
           isnew=. 1
       else.
           if. 0=#wh2=. wh2#~wh=wh2{DIRPARENT do. NB. Name exists but with
               DIRPARENT=: DIRPARENT,wh           NB.  different parent.
               wh=. {.<:#DIRNMS=: DIRNMS,nm
               isnew=. 1
           else. wh=. {.wh2 end.                  NB. Name exists with
       end.                                       NB.  same parent
   end.
   isnew,wh                              NB. 0 if path was already here
)

Tst0addPath_tests_=: 3 : 0
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
   coinsert 'parseDir base'
   d0=. DIRNMS=: 'C:';'Aegis';('\'-.~WINDIR}.~WINDIR i. '\');'Web';'printers';'foo';'Web'
   dp0=. DIRPARENT=: _1 0 0 3 4 0 0

   assert. 0 2-:addPath WINDIR     NB. Shouldn't add it again.
   assert. d0-:DIRNMS              NB. Should not have changed
   assert. dp0-:DIRPARENT          NB. Should not have changed

   td=. 'D:\foo\bar'               NB. New path starting from new root
   assert. 1 9-:addPath td         NB. but with same-named sub as existing
   assert. (d1=. DIRNMS)-:d0,<;._1 '\',td
   assert. (dp1=. DIRPARENT)-:_1 0 0 0 3 4 0 0 _1 8 9
   assert. 0 9-:addPath td    NB. Shouldn't add it again.
   assert. d1-:DIRNMS         NB. Should not have changed
   assert. dp1-:DIRPARENT     NB. Should not have changed

   assert. 1 10-:addPath WINDIR,'foo'
   1
)

NB.* mcopyto: text of DOS .BAT file to do multiple copies (in case it's missing).
mcopyto=: 0 : 0
Rem MCopyTo.bat: Multiple COPY TO %1: copy %2, %3, %4, etc.
:START
If %1/==/ goto SHOWHOW
Set tmpnm=%1
:DO1
If %2/==/ goto BYEBYE
Copy %2 %tmpnm% > nul
Shift
Goto DO1
:SHOWHOW
Echo on
Rem  MCOPY target source1 source2...sourceN
Echo off
Goto BYEBYE
:BYEBYE
Set tmpnm=
)

NB.* setTargetDir: Different target dir according to which known machine backed up.
setTargetDir=: 3 : 0
   if. -.nameExists 'TARGDIR' do. TARGDIR=. '' end.
   if. -.nameExists 'TMPDIR' do. TMPDIR=. getTempDir '' end.
   if. 0=#TARGDIR do.             NB. Assign dir prefix based on machine, if known.
       pfx=. KMPFX
       'machnm usernm'=. whoami ''
       pfx=. >pfx{~KNMACH i. <machnm         NB. Prefix based on computer name
       today=. ;4 2 2 lead0s&.>3{.qts ''     NB. today's date as 'yyyymmdd'
       dirnm=: TMPDIR,pfx,today,'\'
   else. dirnm=: endSlash TARGDIR end.
   TMPDIR;dirnm
)
NB. ARGV_z_=: ARGV_z_,<'-jijx'

NB.* runDaily: backup to run every day: save some most recently changed files.
runDaily=: 3 : 0
   TMPDIR=: 'C:\Temp\' [ setGlobalParms ''
   today=. ;4 2 2 lead0s&.>3{.qts ''    NB. today's date as 'yyyymmdd'
   'machnm usernm'=. whoami ''
   if. DEBUGON do.
       logMsg_logger_ 'Running daily backup for machine, user: '
       logMsg_logger_ ' ',machnm,', ',usernm,' at ','.',~showdate ''
   end.
   'TMPDIR dirnm'=. setTargetDir ''
   ASSERR=: '*** Directory "',dirnm,'" does not exist.'
   dodircheck=. 1
   if. 3=#dirnm do.                     NB. if dir is root: 'C:\'
       if. (PATHSEP_j_={:dirnm)*.':'=1{dirnm do. dodircheck=. 0 end.
   end.
   if. dodircheck do.
       if. -.dirExists dirnm do. assert. createdir dirnm end.
   end.
   if. DEBUGON do.
       logMsg_logger_ 'Created directory ',';',~dirnm
       logMsg_logger_ 'starting "getDirFlInfo" at ','...',~showdate ''
   end.

   if. RUNFROMSAVED do. ASSERR=: '*** Error in runDaily at "if. RUNFROMSAVED".'
       TMPDIR=. 'C:\Temp\' NB. [ 4!:55 <'TARGDIR'
       'TMPDIR dirnm'=. setTargetDir ''
       setGlobalParms ''
       TMPDIR runFromSavedVars SZLIM;dirnm
   else. ASSERR=: '*** Error in getDirFlInfo.'
       tmpbkp=. 6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: getDirFlInfo SRCDIR'
NB.       'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=. PllDirInfoEG 'C:\'
       nm=. 'tmpbkp',today
       ".nm,'=: tmpbkp'
       if. DEBUGON do.
           showtime tmbkp
           logMsg_logger_ 'Saving file and dir info in ',TMPDIR,'...'
       end.
       ASSERR=: '*** Error 0 saving file and dir info.'
       nms=. nm;USUVARS
       ".&.>nms,&.>(<'_base_=: '),&.>nms
       ASSERR=: '*** Error 1 saving file and dir info.'
       xx=. (<TMPDIR) fileVar_WS_&.>nms
       coclass 'parseDir'                   NB. Get back to this namespace after
       ASSERR=: '*** Error 2 saving file and dir info.'
       ".&.>nms,&.>(<'=: '),&.>nms,&.><'_base_'  NB. Ensure local names are same
       coinsert 'base'                      NB. fileVar_WS_ moved us to "base".
       ASSERR=: '*** Error 3 saving file and dir info.'
       if. DEBUGON do.
           logMsg_logger_ 'Save result:'
           logMsg_logger_ >xx
       end.

       if. -.nameExists 'SZLIM' do. SZLIM=. 1e7 end.
       ASSERR=: '*** Error building batch file.'
       'batfl cmds'=: buildBatFl SZLIM;dirnm,~'C:'#~-.':'e.2{.dirnm
       if. DEBUGON do.
           logMsg_logger_ 'About to run batch file at ','...',~showdate ''
       end.
       ASSERR=: '*** Error running batch file ',batfl,'.'
       shell batfl
   end.
   closeLog_logger_ '' [ wait 5
   4!:55 <'ASSERR'       NB. Eliminate global error message if we got this far.
)

NB.* setGlobalParms: set globals according to defaults or command-line overrides.
setGlobalParms=: 3 : 0
   SZLIM=: ''$>1e7 lookupValAfterName ARGV_z_;<'SZLIM'
   if. -.isNum SZLIM do. SZLIM=: ".SZLIM end.
   RUNFROMSAVED=: NYto01 {.>0 lookupValAfterName ARGV_z_;<'RUNFROMSAVED'
   TARGDIR=: ,>lookupValAfterName ARGV_z_;<'TARGDIR'
   SRCDIR=: ,>'C:\' lookupValAfterName ARGV_z_;<'SRCDIR'
   RUNANYDAY=: NYto01,>'N' lookupValAfterName ARGV_z_;<'RUNANYDAY'
   if. -.nameExists 'DEBUGON' do.
       DEBUGON=: NYto01,>0 lookupValAfterName ARGV_z_;<'DEBUGON'
   end.
   ans=. ans,:".&.>ans=. 'SZLIM';'RUNFROMSAVED';'TARGDIR';'RUNANYDAY';'DEBUGON'
)

NB.* NYto01: convert 'N' or 'Y' to 0 or 1, respectively.
NYto01=: 3 : 'if. -.isNum y do. ''Y''=toupper {.y else. y end.'
)

NB.* onlyRuntime: only invoke if not loaded via interactive session.
onlyRuntime=: 3 : 0
   if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
NB. Only invoke for runtime, not interactive (edit) session.
       if. -.nameExists 'DEBUGON' do. DEBUGON=: 0 end.
       if. -.nameExists 'SESSWDW' do. SESSWDW=: 0 end.
       setGlobalParms ''      NB. Cmd-line size limit, target dir, if present.
       wkdy=. dow 3{.qts ''   NB. Weekday number: 0=Sunday, 6=Saturday
       if. RUNANYDAY+.wkdy e. 1 2 3 4 5 do.      NB. Only Mon-Fri unless
           try. sink=. runDaily '' [ 4!:55 <'ASSERR'   NB.  any day allowed.
           catch. if. nameExists 'ASSERR' do. logMsg_logger_ ASSERR
                  else. msg=. '*** Unspecified error in "runDaily" at '
                        logMsg_logger_ msg,'.',~showdate ''
                  end.
           end.
       end.
       if. -.SESSWDW do.
           closeLog_logger_ '' [ wait 5   NB. 5 secs to read msg.
       end.
   end.
)

NB.* runFromSavedVars: Run backup assuming dir&file vars already saved.
runFromSavedVars=: 3 : 0
   '\amisc\' runFromSavedVars y
:
   bkpargs=. y          NB. 'szlim targ'=. y
   vn=. USUVARS
   (<x,'\'#~'\'~:{:x) unfileVar_WS_&.>vn
   ".&.>(<'parseDir_'),&.>vn,&.>(<'_=. '),&.>vn
   'batfl cmds'=. buildBatFl bkpargs
   shell batfl
   wait 2                                    NB. Give .BAT chance to start.
NB.EG runFromSavedVars 5e6;'C:\Temp\WL20050315\'
)

3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:638';<6!:0 '') fappendDHM DBGFL end.
)

coclass 'base'
coinsert 'parseDir'
onlyRuntime ''

locationInfo.ijs

This script sets an arbitrary string to distinguish between the different machines on which I usually work. It has been anonymized here to use only made-up example names.

NB.* locationInfo.ijs: information to distinguish run-time location

NB.* KNMACH: Known machines: network IDs to distinguish different machines I use.
NB.* KMPFX: Known-machine prefixes to de-couple network ID from functionality.
NB.* KNIDS: Known-IDs associated with each machine - good idea?
NB.* whoami: find what machine and user-id I'm on (assume Novell net command).

KNMACH=: (<'\\'),&.>'OldDesktop';'LapHome1';'LapHome2';'WorkDesktop'
NB.* KMPFX: Known-Machine prefixs to use for backup dir.
KMPFX=: 'OD';'LH1';'LH2';'WDsk'
KNIDS=: 'MyOldHomeLogon';'MyHomeLogon';'MyHomeLogon';'MyWorkLogon'
NB. Home: old desktop, laptops;  Work Main Machine (desk),

NB.* whoami: find what machine and user-id I'm on (assume Novell net command).
whoami=: 3 : 0
   sess=. spawn 'net config workstation'
   sess=. dsp&.><;._1 LF,sess-.CR            NB. Vec of lines
   sess=. ><;._1&.> ' ',&.>,sess             NB. Mat of words
   strs=. 'COMPUTER';<'USER'                 NB. Strings to key on
   wh=. (<toupper&.>0{"1 sess)e.&.> <&.>strs NB. Which lines start with strings
   wh=. ;b2i&.>(<(toupper&.>1{"1 sess)e. <'NAME')*.&.>wh
   wh{2{"1 sess
NB.EG 'machine userid'=. whoami ''
)

Parallel Directory-Parsing Task Template

This is the template modified by the main routine and run as a distinct task for each top-level directory.

NB.* parallelParseDir.ijs: gather directory info in parallel.

usuUse=: 0 : 0
   1!:44 dd=. 'C:\amisc\J\Parallel\ExampleProblems\DirInfo\'
   load 'parallelParseDir.ijs'
   6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: PllDirInfoEG ''C:\'''
)

PllDirInfoEG=: 3 : 0
NB.* PllDirInfoEG: gather dirs' info accumulated in parallel.
   'tmpd srcd flnms fldts flszs'=. pllParseDir y=. endSlash y
   flszs=. ;flszs [ fldts=. cvtTS21Num&>fldts
   flparent=. 0$~#flszs
   dirdep=. _1,0$~#srcd [ dirnms=. y;srcd
   vnms=. <;._1 ' FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'
   for_ii. i. #tmpd do.
       vals=. (3!:2)&.>fread&.>(<endSlash >ii{tmpd),&.>vnms,&.><'.DAT'
       'flnms fldts flszs'=. (flnms;fldts;<flszs),&.>3{.vals
       flparent=. flparent,(#dirnms)+>3{vals
       dirnms=. dirnms,>4{vals
       ddep=. (#dirnms)|dirnms i. (]{.~PATHSEP_j_ i:~])&.>dirnms
       dirdep=. (ddep=i.#ddep)}ddep,:_1
   end.
   cleanupTempDirs 'PllDTmp'
   flnms;fldts;flszs;flparent;dirnms;<dirdep
NB.EG 'flnms fldts flszs flparent dirnms dirdep'=. PllDirInfoEG 'C:\'
)

NB.* endSlash: ensure path has ending slash.
endSlash=: 13 : 'y,PATHSEP_j_#~PATHSEP_j_~:{:y'
SUBTASK=: <jpath '~Code/pllPDSub.ijs'
dq=: '"','"' ,~]                        NB.* dq: put double-quotes around y.

pllParseDir=: 3 : 0
NB.* pllParseDir: launch sub-tasks to parse each sub-directory under y.
   if. 0=#y do. y=. 'C:\' end.
   srcDirs=. jd dir '*',~y=. endSlash y
   'flnms fldts flszs'=. <"1|:0 1 2{"1 jfi dir '*',~y
   cleanupTempDirs 'PllDTmp'
   tmpDirs=. (<'C:\Temp\PLLDTmp'),&.>":&.>i.#srcDirs   NB. Don't end w/slash->escapes "
   args=. SUBTASK,&.>(<' SRCDIR '),&.>dq&.>(<y),&.>srcDirs  NB. Build command
   args=. args,&.>(<' RESULTDIR '),&.>dq&.>tmpDirs          NB.  for each dir.
   if. -.fexist BINPATH,'\PllPD.exe' do. (BINPATH,'\J.exe') fcopy BINPATH,'\PllPD.exe' end.
   fork&>(<dq BINPATH,'\PllPD.exe'),&.>(<' -jijx '),&.>args
   checkIfDonePPD 'PllPD'
   tmpDirs;srcDirs;flnms;fldts;<flszs
NB.EG 'tmpd srcd flnms fldts flszs'=. pllParseDir 'C:\'
)

checkIfDonePPD=: 3 : 'while. 5<LF+/ . =shell ''pslist|egrep '',y do. wait 1 end.'

cleanupTempDirs=: 3 : 0
   tmpdirs=. (<'C:\Temp\'),&.>jd dir 'C:\Temp\',y,'*'
   shell&>(<'echo Y|del '),&.>tmpdirs,&.><'\*'
   1[shell&>(<'rmdir '),&.>tmpdirs
)

0 : 0
NB.* Only invoke this if not loaded via interactive session.
   if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
       'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=. PllDirInfoEG 'C:\'
       (<'\Temp\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
       'batfl cmds'=. buildBatFl_parseDir_ 12e6;'C:\Temp\Recent\'
       shell batfl
   end.
)

Mutual Exclusion

Here is the script implementing mutex by means of a semaphore file using the file-locking primitives in J. There are two scripts: the basic routines for mutex and a set of test cases to validate the correctness of these routines.

NB.* mutex.ijs: implement mutual exclusion via file locking.

coclass 'mutex'
require 'task'      NB. Only for "spawn" in "whoami" - not necessary.

fopen=: [:1!:21<    NB. Open file named by y.
fclose=: [:1!:22<   NB. Close file name or number y.
flock=: 1!:31       NB. File number, index, length of region to lock
funlock=: 1!:32     NB. File number, index, length of region to unlock
whlocks=: 1!:30     NB. Which locks - 3-cols: file #, index, and length
whfiles=: 1!:20     NB. Which files - 2-cols: file #; name
getpid=: 2!:6       NB. Current process ID
qts=: 6!:0          NB. Timestamp -integers: Y M D h m s.s (millisec?)

LOCKFL=: 'Lock.fl'  NB. Central hold file
LFINFO=: (i.0);''   NB. Hold file details while holding

NB.* getMyInfo: nice to have info identifying who has the lock and when.
getMyInfo=: 3 : '100{.(;'' '',~&.>whoami''''),":(getpid,qts)'''''

NB.* getHold: put hold on lock file.
getHold=: 3 : 0
   fid=. fopen LOCKFL [ myinfo=. getMyInfo ''
   if. -. fid e. ;0{"1 whlocks '' do.             NB. If we don't have lock,
       while. -.flock fid,0,100 do. wait 1 end.   NB.  keep waiting until we
       myinfo fwrix_fldir_ fid;0                  NB.  get it; put ID info in
       LFINFO=: fid (]{~[i.~[:;0{"1]) whfiles ''  NB.  file & update global.
   end.
   fid;myinfo
NB.EG 'fid myinfo'=. getHold ''  NB. File ID, my ID info.
)

NB. funlock (>0{LFINFO) (]{~[i.~[:;0{"1]) whlocks ''
releaseHold=: 3 : 0
   if. 0~:#>0{LFINFO do. 0*./ . =#&>LFINFO=: (i.0);'' [ fclose >0{LFINFO end.
)

NB.* getMUP: get Machine name, User ID, Process ID from my info string.
getMUP=: 13 : '3{.<;._1 '' '',y'

NB.* dsp: Despace - remove leading, trailing and redundant spaces.
dsp=: [:(#~(+.(1:|.(></\)))@(' '&~:))"1 (#~([:(+./\*. +./\.)' '&~:))"1

NB.* whoami: get machine and user ID - Windows only.
whoami=: 3 : 0
   assert. 6-:9!:12 ''                       NB. Fail if not Windows
   sess=. spawn 'net config workstation'
   sess=. dsp&.><;._1 LF,sess-.CR            NB. Vec of lines
   sess=. ><;._1&.> ' ',&.>,sess             NB. Mat of words
   strs=. 'COMPUTER';<'USER'                 NB. Strings to key on
   wh=. (<toupper&.>0{"1 sess)e.&.> <&.>strs NB. Which lines start w/ strings?
   wh=. ;I.&.>(<(toupper&.>1{"1 sess)e. <'NAME')*.&.>wh
   wh{2{"1 sess
NB.EG 'machine userid'=. whoami ''
)

Mutual Exclusion Test Cases

Here are routines to test various aspects of the mutex implementation.

NB.* mutex_TCs.ijs: test cases for mutex.ijs.

coclass 'mutexTest'
require '~Code/mutex.ijs'
coinsert 'mutex'

checkIfSelfLocked=: 3 : '(getMUP getMyInfo '''')-:getMUP y'
endSlash=: ],'\'-.{:
wait=: 6!:3

TC0=: 3 : 0
   'fid myi'=. getHold ''
   assert. -. 0*./ . =#&>LFINFO
   releaseHold ''
   assert.    0*./ . =#&>LFINFO
)

TC1=: 3 : 0
   'fid myi'=. getHold ''
   assert. checkIfSelfLocked fread fid
   releaseHold ''
)

TC2=: 3 : 0
   'fid myi'=. getHold ''
   assert. checkIfSelfLocked myi
   releaseHold ''
)

TC3=: 3 : 0
   'fid0 myi0'=. getHold ''
   'fid1 myi1'=. getHold ''
   assert. fid0-:fid1
   assert. checkIfSelfLocked myi0
   assert. checkIfSelfLocked myi1
   releaseHold ''
)

TC4=: 3 : 0
   'fid myi'=. getHold ''
   assert. fid=>0{LFINFO
   assert. fid e. ;whlocks ''
   assert. 3=#fid (]{~[i.~[:;0{"1]) whlocks ''
   assert. LFINFO-:fid (]{~[i.~[:;0{"1]) whfiles ''
   releaseHold ''
)

NB.* holdAndWait: hold file for a random time, then release.
holdAndWait=: 3 : 0
   tm=. tm,qts'' [ 'fid myi'=. getHold '' [ tm=. ,:qts ''
   rc=. *./checkIfSelfLocked &> myi;fread fid
   rc=. rc,wait >:?y
   releaseHold ''
   rc;tm       NB. If self-held, holding time; when asked and got hold.
)

NB.* holdsAndReleases: do number of holds and releases, waiting between each.
holdsAndReleases=: 3 : 0
   'ni wl mhl'=. y  NB. Number of iterations, wait length, max hold length.
   rc=. 0 2$0 2;2 7$2
   while. _1<ni=. <:ni do. wait wl [ rc=. rc,holdAndWait mhl end.
   rc
)

NB.* startHRs: start holders and releasers as separate processes.
startHRs=: 3 : 0
   if. 0=#y do.
       args=. |.(10+|.i.10),. 2 2,2 3,3 2,3 3,2 4,4 2,4 4,2 5,5 2,:5 5
       dd=. 'C:\amisc\J\Parallel\ExampleProblems\Holds\'
   else. 'dd args'=. y end.
   if. 0=#(1!:0)@<(]}.~[:-'\'={:) dd do.     NB. Create dir if doesn't exist.
       try. 1!:5 <dd catch. assert. 0 end. end.
   start1HR"1 dd;"1 args
)

start1HR=: 3 : 0
   'rundir arg'=. y
   1!:44 rundir=. endSlash rundir [ svdir=. 1!:43 ''   NB. Run in target dir.
   basis=. fread jpath '~Code/mutex_TCs.ijs'      NB. Get base code.
   rndnm=. 8 (]{~[:?[$[:#]) 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
NB. Obfuscate rand name place-holder to avoid replacement here.
   rplargs=. '{',&.>('RN';'arg';'rundir'),&.>'}'
   addon=. (,rplargs,.rndnm;(":arg);rundir) stringreplace addon
   if. fexist hrfl=. 'HR_',rndnm,'.ijs' do. ferase hrfl end.
   (CR-.~basis,addon) fwrite hrfl                 NB. Create unique HR file.
   exe=. (<'"',(jpath '~bin'),'\j.exe" -jijx "'),&.><endSlash rundir
   fork&.>exe,&.>(<hrfl),&.>'"'                   NB. Run HR file.
   1!:44 svdir
)

addon=: 0 : 0
varnm=. 'rctm{RN}'
(varnm)=: holdsAndReleases {arg}
(varnm)=: ('{RN}';(getpid ''),qts''),".varnm
(3!:1 (".varnm)) 1!:2 <'{rundir}',varnm,'.dat'
2!:55 ''
)

Checking Results of Mutual Exclusion Tests

The results of the startHRs tests need subsequently to be analyzed to ensure that no holds or releases overlap between separate processes, as shown here:

NB.* checkMutexOverlap: examine mutex hold and release times to find overlap.

NB.* Regularize vec of mats to 3-cols: time, ID, "H" (hold) or "R" (release),
NB. ordered by timestamp.
regMatz=: [:(] ,. ('H';'R')$~#) ([:<"1 [:,/[:>[:,/1{"1}.),.(<0 0){]
checkHRSeqc=: (2 {"1 ]) -: ('H';'R') $~ #
checkIDSeqc=: [:*./_2=/\1{"1]
egUse=: 0 : 0
   dd=. 'C:\amisc\J\Parallel\ExampleProblems\HoldsWL2M100713\'
   fls=. 0{"1 dir dd,'*.dat'
   hldtms=. 3!:2&.>fread&.>(<dd),&.>fls
   hh=. >,&.>/regMatz&.>hldtms
   rc=. 0
   assert. checkHRSeqc hh
   assert. checkIDSeqc hh
   rc=. 1
)