User:Devon McCormick
Devon McCormick
Founder - New York City J Users Group
I curate the SIGAPL (Special Interest Group on APL) page under the auspices of ACM's SIGPLAN (Special Interest Group on Programming LANguages.)
Email: <<MailTo(devon AT SPAMFREE acm D0T org)>>
J Wiki Pages
Following are largely ones I've written but also includes those related to areas in which I'm interested.
o Trees: an exposition on the great utility of using the parent-index method of representing trees in J.
o Array-thinking: an explanation of the fundamental difference between array-thinking and scalar-thinking and how the former helps us think about algorithms.
o An attempt at a minimal subset of J to ease the shock of learning all that there is.
o An example of using J for beginning financial data analysis.
o OLEExcel - manipulate Excel worksheets using OLE interface: Scripts/OLEExcel
o WS - save J variables to file and retrieve them: Scripts/File J Variables
o An early (unsuccessful) attempt to use the J database "JDB" on the "Netflix Challenge" data. It appears to have been unsuccessful because this dataset was too large for JDB to handle.
o Start of an attempt to detail J equivalents of basic operations in Microsoft Excel.
o An essay contrasting visual understanding with symbolic understanding.
o BigFiles - work with files larger than 2 GB: Scripts/Working with Big Files. This is now largely obsolete with the advent of 64-bit versions of J that have extended the scope of the file primitives to handle larger files.
Parallelizing J
o Some general limitations to consider when parallelizing code.
o General discussion of multi-core parallelization in J; general ideas on running threads from J.
o Simple example of writing parallel code in J.
o Detailed example of how to simply parallelize code in J. This assumes the (common) sort of problem where coordination of the various parallel processes can be handled at the point where a "master" process spins them off and that the problem domain lends itself to "coarse" parallelism. I currently run two multi-core J routines almost every day using this technique and it's worked well for years now with substantial speed-ups and little difficulty.
o Complete examples of parallel J code presented at the APL2010 conference in Berlin, Germany, in September, 2010. These are the two examples of parallel code tasks, mentioned above, as well as an implementation of mutex (mutual exclusion), all written in J (version 6.02).
o Ideas on constructing arbitrarily-large datasets for comparing the effectiveness of different parallel implementations.
o "Red or Black" game simulation -> parallelization example NYCJUG/2011-04-12/RedOrBlackGameSimulation
Presentations About J
Feel free to take anything from these for your own presentation.
o Slides of my talk on "Array Thinking" at the 2014 J Conference in Toronto.
o An introduction to J presented to Hacker School in New York in September, 2014.
o "J: the Hard Parts" - a fairly lengthy talk explaining some of the parts of J that most people find the most difficult to comprehend, given to the N-Languages Meetup group on January 21, 2014.
o Presentation of J to a "Functional Programmers" Meetup group on September 20, 2011.
o Short (five minute) introduction of J to general programming group at a language slapdown which was supposed to be held on November 17, 2010 but was canceled.
o Development of code for "Diffusion-Limited Aggregation" from an initial version coded in a more conventional format to a more "J-like" version
o "Short is Sweet" - An exposition on the importance of succinct expression (from a talk for the Kdb User Group on 12/9/2009).
Other Links: Wiki Tips, Foreigns, Starting J, Command-line Options
Here are some tips for working with pages on this wiki.
This page lists all J foreign conjunctions. These are used to access resources external to the J language itself, things like I/O, times and timing, object characteristics, and such.
This is the list of steps J follows upon start up.
Here are command-line parameters for starting J with various options.
Non-breaking Space (NBSP)
Sometimes when loading a text file as a J script, we find that it contains characters which give a "spelling error". Here is an example of how to fix the file when it contains a non-breaking space. This character, also called "NBSP", is meant to prevent an automatic line-break on the character, which looks like a space (32{.a.), to prevent breaking up parts of a token represented by multiple space-delimited words.
It can get into text that was cut-and-pasted from a web page and can be annoying because it's hard to see what the problem is. Here is an example of discovering and fixing this problem in a file.
 |
Here we see the initial error upon loading the file: it fails with a "spelling error".
If we copy an instance of the bad character, we can put it in quotes and look it up in a.. Looking up this character in a., we see that the result is two numbers, showing us that it is a multi-byte character. We can then replace this pair of bytes with a single space using the standard rplc function and write it back into a new file. The subsequent successful loading of this new file without error shows us that our fix worked. |
Solving Problem Reading Images in JConsole
I've had a recurring issue with many J upgrades: they break the code I most often use. What breaks is the call to underlying verb wdreadimg, like this:
herbs=. read_image 'Herbs1.jpg' |domain error: cd | m=. wdreadimg(utf8,y);wh=.2$2-2
The immediate cause of the problem is in the definition of wdreadimg:
wdreadimg '"jqt.dll" wdreadimg >+ x *c *i'&cd
In the 8.04 version of J where this code still worked, that verb was defined this way:
wdreadimg '"E:/Users/DevonMcC/j64-804/bin/jqt.dll" wdreadimg >+ x *c *i'&cd
However, the exact sequence by which this gets defined differently is obscure, and doing the equivalent assignment in J 8.07 revealed the absence of the jqt.dll file in the 8.07 bin sub-folder. Copying this there did not help.
The break-through came from looking at an older email chain where I'd written to the J Forum about my problems. In this chain, Ric Sherlock discovered that the underlying image reader from the jqt package, readimg_jqtide_, worked for .png files but not for .jpg files. He eventually came up with this fix:
from: bill lam <bbill.lam@gmail.com> via forums.jsoftware.com
to: Programming forum <programming@jsoftware.com>
date: Jun 15, 2017, 12:49 AM
subject: Re: [Jprogramming] Location of "readimg"
I had tried again and might had found a solution. Apparently just
loading qtlib under jconsole failed to locate the qt plugin folder. It
needs to set env QT_PLUGIN_PATH to the path of J bin directory where
contains the Qt dll files. This can be automated by adding a line
into jconsole.cmd
@rem load J console
@rem run from the J install directory
@cd /d %~dp0
@set QT_PLUGIN_PATH=%~dp0bin
@bin\jconsole.exe
then click on jconsole.cmd to launch
2!:5'QT_PLUGIN_PATH'
D:\j805\bin
load'ide/qt/qtlib'
$readimg_jqtide_ jpath '~temp/toucan.png'
144 200
$readimg_jqtide_ jpath '~temp/toucan.jpg'
144 200
With this insight, I modified my J-launcher utility J64-807.bat to look like this:
Rem J64-807.bat: start 64-bit jconsole v8.07 of J (under emacs). %BASEDSK% Rem cd "%BASEDSK%\Users\devon_mccormick\j64-807\bin" cd "%BASEDSK%\Users\DevonMcC\j64-807\bin" Rem "-interactive" signals not a batch run ..\jconsole.cmd %BASEDSK%\amisc\JSys\J8\DHMConfig.ijs -interactive -js "1!:44 '\amisc\work\neuralNets\'"
(The bit at the end following "-js" is something I change as my default working area changes)
To make this work properly by making jconsole.cmd process the command-line arguments I typically provide, I had to modify that file in the 8.07 installation directory to be like this:
@rem load J console @rem run from the J install directory @cd /d %~dp0 @set QT_PLUGIN_PATH=%~dp0bin @bin\jconsole.exe %1 %2 %3 %4 %5 %6 %87 %8 %9
So, it looks like I now have a 8.07 installation that does not break my most-used code.
Useful Utilities
One set of useful utilities for visiting nodes in a directory tree we explain here.
Walk Directory Tree
This idea of visiting each sub-directory under a particular one and doing something in each is commonly known as the visitor pattern because we visit each directory and perform a specified action in each one.
We implement this in J as shown here; notice how an optional left argument specifies either breadth-first (the default) or depth-first order of visiting the directories. Notice how the difference between the two is entirely determined by the order in which we accumulate the sub-directories.
NB.* generalWalkTree: general directory tree walk: perform "u y" at each node:
NB. 0{x: (0) depth first or (1) breadth first;
NB. 1{x: (0) flattened or (1) nested result matching input tree structure.
generalWalkTree=: 1 : 0
(1 0) u generalWalkTree y NB. Default: breadth-first, flattened result.
:
ct=. 0 [ rr=. '' [ stack=. ,boxopen y [ x=. 2{.x
ctr=. (0{x){_1 0 NB. First we build the stack.
while. ct<#stack do. NB. Get subdir names:
subds=. ((('d'e.&>4{"1])#0{"1])@:(1!:0@<)) (>ctr{stack),'\*'
subds=. subds (],&.>'\',&.>[) ctr{stack NB. -> full path names
if. 0{x do. stack=. stack,subds NB. Breadth or
else. stack=. subds,stack end. NB. depth first
ctr=. ctr+(0{x){_1 1 [ ct=. >:ct NB. Go forward or backward
end.
dpth=. (]-<./)'\'+/ . =&>stack
if. 1{x do. ;&.>(<:~.dpth) depthEnc dpth </. u&.>stack NB. Preserve tree
else. u&.>stack end. NB. or flatten it.
NB.EG ([:;2{"1 [:dir '*',~],'\'-.{:) generalWalkTree 'C:\' NB. All file sizes
)
The two sub-functions used are defined thusly:
depthEnc=: 4 : '<^:(>:x)]y'"0 boxopen=: <^:(0&(>: L.))
The latter one, boxopen, is from J's standard library: it boxes only an unboxed item.
Examples of the Visitor Pattern
Here are a couple of examples of common uses of the generalWalkTree adverb.
Find String in File
Here we implement a simple search into file contents of each directory.
NB.* lookUnder: look for x in files under dir y and its subdirs. lookUnder=: 4 : '(x&findInFiles) generalWalkTree y'
The verbs we apply at each directory, findInFiles and its sub-functions jfi and searchForInText, look like this:
NB.* findInFiles: look in all files in directory for given string.
findInFiles=: 13 : 'x(]#~ ] searchForInText&>~[:<[)(([: < endSlash) ,&.> [: {."1 [: jfi [: dir ''\*'',~ ])y'
jfi=: 3 : '(-.''d''e.&>4{"1 y)#y' NB. Return just files in a given directory listing from J standard 'dir' command.
NB.* searchForInText: search for string in file.
searchForInText=: [ ([: +./ E.) [: fread ]
The last one of these uses the J standard fread verb.
The left argument to findInFiles is the string we wish to find. We use the J standard library dir command to list all the file names.
As an example of using findFiles, here we look for instances of the string whoami within all files under a single directory, here C:\amisc\Jsys\user\code:
'whoami' lookUnder 'C:\amisc\JSys\user\code'
The generalWalkTree adverb applies this to all sub-directories as well as the top-level one supplied as its argument.
Delete Particular Files
Here we delete all *.png files, using the underlying OS's del command, in all sub-directories under D:\amisc\pix\Sel\2007, using a couple of defined verbs delPNG and chgToDir:
delPNG=: 3 : 'shell ''del "*.png"''[chgToDir y' chgToDir=: 3 : 'try. 1!:44 y catch. '''' end.' delPNG generalWalkTree 'D:\amisc\pix\Sel\2007'
Feedback -- leave me /Comments
...