User:Marshall Lochbaum/Parallelize

This code gives a relatively clean way to parallelize "embarassingly parallel" problems--that is, those which are represented by verbs u with u-:u"_1. It can be run simply as u parallelize n y, and will fork off n tasks to run equally spaced sections of y. If one of these returns an error, the error is spat back using debug foreigns, and another thread is spawned to kill off the remaining processes as they terminate.

If it is necessary to load scripts before execution of u, they can be set with setloadfiles, which has the same arguments as load.

Tested on Linux and previous versions worked on Mac and Windows. Windows execution will give console windows that pop up for the duration of the computation. Currently does not support dyadic evaluation.

load 'strings files task'

fork=: ([: 2!:1 '(',,&')&')`(fork_jtask_)@.IFWIN
JEXE=: jpath IFWIN{:: '~bin/jconsole';'~bin\jconsole.exe'
WAITTIME=: 0.05

NB. read, write a boxed J noun
oread  =: [: <@(3!:2)@:(1!:1)"0 boxopen
owrite =: (3!:1@>@[ 1!:2 ])"0 boxopen

getfname =: ((jpath '~user/')&,) : (((jpath '~user/') , ({.~i.&'.')@[) , ":@] , (}.~i.&'.')@[) &.>

NB. (u parallelize n) y
NB. assumes (u -: u"_1) y
NB. divides y into n parts and executes u on them in parallel.
parallelize=:2 :0
  parts=. (</.~ [:<.n* (%~i.)@#) y
  'inputs flags outputs kernels'=. <"_1|: files=. |: ('input';'flag';'output';'kernel.ijs') getfname/ i.n
  parts owrite inputs
  '0' 1!:2"0 flags

  (u buildtemplate) runkernels files

  result=.$0
  while. +./ '1'~: read=. {.@fread"0 flags do.
    if. '2'e.read do.
      err =. ; oread (read i.'2'){outputs
      ferase (read i.'2'){files
      cleanup (read='0') # files
      13!:8&>/ err
    end.
    6!:3 WAITTIME
  end.
  result=. ; oread outputs
  ferase files

  result
)

NB. run first y kernels
runkernels=:4 :0"_ 1
  'input flag output kernel'=. y
  sentence=. x rplc  '{input}';input;'{flag}';flag;'{output}';output
  sentence fwrite kernel
  fork JEXE,' "',kernel,'"'
)

NB. built a template string to use for runkernels.
NB. assumes u is monadic.
buildtemplate=:1 :0
  if. _1= 4!:0 <'loadfiles' do. loadfiles=:'' end.
  if. 3 = 4!:0 <'u' do. u=.5!:5<'u' end.
  template rplc '{loadfiles}';loadfiles;'{verb}';u
)
template =: ('\)';')') rplc~ 0 :0
{loadfiles}
f=: {verb}
3 :0 ''
try.   '{flag}' (1!:2<)~ '1'[ '{output}' (1!:2<)~ f&.(3!:2) 1!:1 <'{input}'
catch. '{flag}' (1!:2<)~ '2'[ '{output}' (1!:2<)~ 3!:1 (13!:12'') ; (13!:11'')
end.
\)
2!:55 ]0
)

cleanup =: 3 :0
  c =. >getfname <'cleanup.ijs'
  c fwrite~ cleanupscript rplc '{files}';(5!:5<'y');'{WAITTIME}';(5!:5<'WAITTIME');'{c}';c
  fork JEXE,' "',c,'"'
)
cleanupscript =: ('\)';')') rplc~ 0 :0
(3 :0) {files}
try.
  while. #y do.
    for_i. y do.
      'input flag output kernel'=.i
      if. '0'~:{.1!:1 <flag do.
        1!:55 :: _1:"0 i
        y=.y-.i
      end.
    end.
    (6!:3) {WAITTIME}
  end.
catch.
  1!:55 :: _1:"0 y
end.
\)
1!:55 :: _1: <'{c}'
2!:55 ]0
)

NB. y is list of files to load (arguments to the verb load).
setloadfiles=:3 :0
  loadfiles=: 'load ''',y,''''
  i.0 0
)

---

Contributed by Marshall Lochbaum.