NYCJUG/2013-07-09

From J Wiki
Jump to: navigation, search

ordering files, conjunction/adverb/verb/noun hierarchy, card game simulation, software development estimation, arc consistency, code presentation, javascript, astronomy, Lobster games programming, Harlan GPGPU programming


Location:: BEST

Meeting Agenda for NYCJUG 20130709

1. Beginner's regatta: a simple exercise of renaming a large number of files:
see "Numbering Files to Put Names in Date Order.doc".

J's implicit hierarchy of "types of functions":

           conjunction                       adverb
        verb          noun or verb       noun      verb
   [noun]   noun

See "Dan Explains Adverbs".


2. Show-and-tell: building a simulation - how long does it take to get some
useful results using J?  See "Quickly Building a Simple Simulation of 727.doc"
and "simulate727IJS.doc".


3. Advanced topics: Software development estimation - intra-person
communications of overwhelming importance: see "Why Are Software Development
Estimations Regularly Off by a Factor of 2.doc".


4. Learning, teaching and promoting J, et al.: a criticism - how not to present
J code: see "Arc Consistency for Constraint.doc"; also, see "Favorable Review
of Programming Book.doc".

JavaScript as a "full-stack language" - see "JavascriptFullStackLanguage.doc".

See "Astronomical Calculations on a Phone.doc".

Some other languages of interest: see "LobsterGameProgrammingLang.doc" and
"HarlanGPUProgrammingLanguage.doc".

Proceedings

Beginner's Regatta

Numbering Files to Put Names in Date/Time Order

We have a directory of photos but our slideshow program will only show them in alphabetical order which is a problem if we’ve selected some photos and named them meaningfully. However, a little J code will quickly fix this.

First, let’s move to the directory where we have these photos.

   qts''
2013 7 8 19 2 49.037
   1!:44 '../Vacation201304'

   1!:43''
C:\amisc\pix\Photos\2013Q2\Vacation201304

Now extract the names and timestamps of the photos from the directory listing.

   'jfls dtms'=. <"1|:0 1{"1 dir '*.jpg'
   $jfls
892

Check that these look correct.

   >3{.&.>jfls;<dtms
+------------------+------------------+-----------------+
|LeavingNYC.jpg    |LeavingNYC2.jpg   |LeavingNYC3.jpg  |
+------------------+------------------+-----------------+
|2013 6 16 23 48 20|2013 6 30 13 27 22|2013 5 26 13 6 58|
+------------------+------------------+-----------------+

Order the names by the timestamps and check that the resulting start and ending files look like the ones we expect. We'll look at the first and last three files:

   (3&{.,:_3&{.) jfls=. jfls/:dtms
+------------------+-------------------+-------------------+
|LeavingNYC.jpg    |LeavingNYC2.jpg    |LeavingNYC3.jpg    |
+------------------+-------------------+-------------------+
|returningToNYC.jpg|returningToNYC2.jpg|returningToNYC3.jpg|
+------------------+-------------------+-------------------+

Create the list of new names which are the same as the existing ones but prefixed by ascending numbers padded with leading zeros so they’ll alphabetize correctly.

   newnms=. (4 lead0s 10*>:i.#jfls),&.>jfls

where

   lead0s
[: ]`>@.(1 = [: # ,) ('r<0>','.0',~ [: ":[) 8!:0 [:|[: ".`]@.(0=[: {.0#]) ]

Why "4"? Because of how many numbers we generate (which count up by tens in case we want to insert other photos later):

   10^.10*#fls
3.94988
   <.>:10^.10*#fls
4

So we need four digits. Check that the names look as we expect:

   (3&{.,:_3&{.) newnms
+----------------------+-----------------------+-----------------------+
|0010LeavingNYC.jpg    |0020LeavingNYC2.jpg    |0030LeavingNYC3.jpg    |
+----------------------+-----------------------+-----------------------+
|8900returningToNYC.jpg|8910returningToNYC2.jpg|8920returningToNYC3.jpg|
+----------------------+-----------------------+-----------------------+

Build the DOS copy commands and check that they look right.

   cmds=. (<'copy "'),&.>jfls,&.>(<'" "..\orderedSlides\'),&.>newnms,&.>'"'
   ({.,:{:) cmds
+---------------------------------------------------------------------+
|copy "LeavingNYC.jpg" "..\orderedSlides\0010LeavingNYC.jpg"          |
+---------------------------------------------------------------------+
|copy "returningToNYC3.jpg" "..\orderedSlides\8920returningToNYC3.jpg"|
+---------------------------------------------------------------------+

Make sure our destination directory exists:

   shell 'mkdir ..\orderedSlides'
   dir '..\*'
+----------------+------------------+-+---+------+
|orderedSlides   |2013 7 8 19 5 57  |0|rw-|----d-|
+----------------+------------------+-+---+------+
|Vacation201304  |2013 7 8 11 54 22 |0|rw-|----d-|
+----------------+------------------+-+---+------+

Make sure we’re still in the correct source directory:

   1!:43''
C:\amisc\pix\Photos\2013Q2\Vacation201304

Run the commands and time how long they take:

   6!:2 'shell&.>cmds'
76.8691

Perhaps more important than this reasonably quick run time is the fact that it took less than seven minutes to produce this code.

Dan Explains Adverbs

from:	 Dan Bron <j@bron.us> via srs.acm.org
to:	 chat@jsoftware.com
date:	 Thu, Jun 13, 2013 at 12:48 PM
subject: Re: [Jchat] "|value error: m | x m write_jpeg y" - what???

Alexander Epifanov wrote:

> but I did not understand what is the different, I mean how it works if it is adverb.

To understand this error, we must first discuss what adverbs are, how they behave, and how they differ from verbs. So let's start there.

Adverbs are a different class, or order, of words than verbs. In particular, they have higher grammatical precedence than verbs, and so gobble up any suitable arguments lying around before verbs can even see them. Conjunctions are in this same high-precedence class, but whereas adverbs only take one argument (on the left), conjunctions take two (one on the left, the other on the right). You can think of adverbs and conjunctions as higher-order analogs to monadic and dyadic verbs respectively.*

Adverbs are called adverbs because they normally modify verbs: that is, in typical use, they accept a verb argument and produce verb result, which is related in some (consistent) way to the argument. The most famous example is /  :

 +/ 2 3 4  NB.  Sum of data (Σ s[i])
 */ 2 3 4  NB.  Product of data  (Π s[i])
 ^/ 2 3 4  NB.  Tetration ("power tower") of data

Here, / takes a dyad (two-argument verb) as an argument, and produces a monad (one-argument verb)*. The output is related to the input in the following sense: when the output verb is provided an noun, it inserts the input verb between each pair of items in the noun, such that: +/ 2 3 4 is 2+3+4 */ 2 3 4 is 2*3*4 ^/ 2 3 4 is 2^3^4 NB. J executes right-to-left, so this is 2^(3^4) and

   +/ 2 3 4 , 5 6 7 ,: 8 9 10

is:

      2  3  4
         +
      5  6  7
         +
      8  9 10

which, because + is rank 0 (scalar), is:

      2  3  4
      +  +  +
      5  6  7
      +  +  +
      8  9 10

etc.

But bear in mind that taking verb arguments and deriving (consistently) related verbal results is only the typical case for an adverb. Adverbs can also take a noun for an argument (an "adjective"); the most common example is  } , which normally takes a noun argument specifying which indices the derived verb should modify (when it, itself, is applied to nouns):

  putFirst        =:    0}
  putLast         =:   _1}
  putFirstAndLast =: 0 _1}

  '*' putFirst '12345'
*2345
  '*' putLast 'ABCDE'
ABCD*
  '*' putFirstAndLast 'ABCDE'
*BCD*

So adverbs can take verbs or nouns as inputs, and normally produce verbs as outputs. But adverbs are not restricted to verbal output; they can produce anything, including verbs, nouns, and even other adverbs and conjunctions. Primitive adverbs which produce non-verb results are unusual (primitive conjunctions are a little more diverse in this regard), but they exist. For example, when the adverb ~ is applied to a string, it treats the string in as a name and evokes it, such that 'someName'~ is equivalent to someName. Therefore ~ can produce anything at all:

   someNoun =: 42
   someVerb =: +
   someAdverb =: /
   someConjunction =: @

   'someNoun'~
42
   'someVerb'~
+
   'someAdverb'~
/
   'someConjunction'~
@

Of course user-defined adverbs will produce anything they're defined to produce, so you can't know what they'll do without reading the definition or documentation. That said, user-defined adverbs tend to follow the same patterns as primitive adverbs: they're almost always abstractions over verbs which produce verb results; sometimes they take noun arguments and/or produce noun results, and only very rarely do they produce other adverbs or conjunctions.

Ok, with that as a background, we're ready to discuss write_image and the error you observed.

The word write_image falls into this "user defined adverb" category. The reason it was defined as an adverb instead of a verb is so that it can accept up to 3 arguments (filename, data to write, and a set of options like image quality or scaling), whereas if it were defined as a verb, it could accept no more than two arguments. Meaning if write_image had been defined as a verb, it would have to find some way to pack two arguments into a single noun, and unpack them inside the definition, which can sometimes lead to convoluted code. Keeping it as an adverb with three distinct arguments is very clear and clean.

But it does stymie attempts to use it like a verb, as you discovered. In particular, when you embedded it in

   (('small/'&, (write_image)~ ((3 3)&resize_image)@:read_image)@:>) i

, its higher grammatical priority caused the adverb to seek out an argument immediately, and since the verb 'small/'&, was on its left and suitable (because verbs are perfectly acceptable arguments for adverbs), the result was that write_image bound with 'small/'&, .

Now, the specific coding style** of write_image prevented it from being executed immediately (if it'd been executed, you'd know it, because you would have gotten an error: write_image is expecting data [a noun] as an argument, not a verb like 'small/'&,), but it also allowed the J interpreter to infer that when it is executed, it will produce a verb.

So write_image woke up, looked around for an argument, found 'small/'&, , bound with it, and though it didn't actually execute, the J interpreter knew its product would be a verb. Knowing this, J proceeded parsing the sentence, found another verb ((3 3)&resize_image)@:read_image)@:>, and hit a close paren. Since it had found two verbs in isolation (nestled inside a cozy pair of parens), it interpreted the train as a hook. This is really no different from the sentence (%~ i.) 10 where ~ immediately binds to %, the product of that binding and i. form a hook.

After forming the hook, the interpreter hit the noun i and applied the hook as ('small/'&,write_image~ 3 3&resize_image@:read_image)@:> i . The interpreter executed 3 3 resize_image read_image > i. and got a result. Up to this point, everything was fine. But now it came time to use the results it had calculated, and actually execute write_image . That's where the problem occurred: and it was exactly the error I mentioned earlier, that the interpreter avoided by deferring the execution of write_image (you can delay the inevitable, but you can't avoid it).

That adverb was written expecting that its argument be a noun, and refers to m, which is the name for the noun argument to an adverb (or conjunction). But given how you expressed your sentence, in this case argument to write_image was a verb: 'small/'&, . Therefore m (the name for a noun argument to an adverb) was undefined, yet write_image tried to use it anyway.

J calls the use of undefined names a "value error". This is the same error as when you type

    someNameIHaventDefinedYet
|value error: someNameIHaventDefinedYet

in the session manager. But a closer analogy is the value error you'd get if you tried to use x (which names a left argument) in a monadic verb which only has a right argument:

   monad def 'x + y' 4
|value error: x
|       x+y

You get a value error because x is undefined, and x is undefined because monadic (valences of) verbs don't have the concept of a left argument: x is literally meaningless.

Similarly, when write_image referred to the noun argument m, the J interpreter balked: "What noun argument? Your argument is a verb, 'small/'&, . I don't know what you're talking about." . The name for the (non-existent) noun argument to write_image, m, was literally meaningless. All because adverbs have higher precedence than verbs and can accept verbs as well as nouns as arguments.

Well, actually, because Cliff decided to define write_image as an adverb so he could have three separate arguments, without boxing. I know that's a lot to digest. I'm not known for my laconic style (cf Roger Hui), but I hope this helps.

-Dan

--

  • Technically, all verbs in J are ambivalent; that is, they can be called with either one argument (on the right) or two arguments (one on the right, and one on the left). The words "monad"/"monadic" and "dyad"/"dyadic" are just shorthand for the "one-argument valence of the verb" and "the two argument valence of the verb" respectively.

Note that some valences of some verbs have empty domains, such as the dyad ~. or the monad E. or the monad 4 : 'x + y' etc. That doesn't mean the valence doesn't exist; it does exist, but it rejects all arguments (a generalization of the concept that e.g. + rejects any argument that's not a number).

Now adverbs and conjunctions (collectively called operators) are analogous to the monadic and dyadic valence of a verb respectively, but it is exactly because of their higher grammatical precedence that there is no operator analog to an ambivalent verb. That is, there is no operator that can take either one argument or two arguments. Operators' higher binding power requires that we treat these cases separately - and, incidentally, is the reason adverbs (monadic operators) take their argument from the left, as opposed to monadic verbs which take their argument from the right.

---

from:	 bob therriault <bobtherriault@mac.com>
date:	 Thu, Jun 13, 2013 at 1:08 PM

Great explanation Dan,

A couple of years ago I put this video together about the adverb '~'  : http://bobtherriault.wordpress.com/2010/11/17/those-tricky-adverbs/

Let me know if you have any interest in developing your explanations into a more multimedia mode. I don't have huge amounts of time (and it does take some time), but I do enjoy doing this stuff.

Cheers, bob

---

from:	 Dan Bron <j@bron.us>
date:	 Thu, Jun 13, 2013 at 1:45 PM

Yes, I remember your video series warmly. Personally, I'm stuck in the text age. I don't have the tools or ambition to go multimedia. And a related pet peeve is when I click through a news story or other link and am presented with a video, which can only be scanned linearly, as opposed to text, which is random access.)

---

from:	 bob therriault <bobtherriault@mac.com>
date:	 Thu, Jun 13, 2013 at 2:36 PM

Hi Dan,

You explain things beautifully in your medium of choice (but I would hardly think of you as stuck in text) and if you do want to add some other media at some point, let me know.

I think that your related pet peeve is a result of media not matching its audience. You want a sound bite, but you are given a lecture. If you combined a series of short videos to be displayed beside text, you would have structured the sound bites to a form that would be closer to your needs. In the text world, you might consider this related to the effect of white space on the structure of your writing. It is a matter of packaging more than a matter of content, but it can affect comprehension and retention.

I believe Ian Clark was pushing for this in the short animations he was using in new vocab a few years ago.

http://www.jsoftware.com/jwiki/NuVoc with http://www.youtube.com/v/aTRONIqXFVI as an example of a short form video. In any case, your ability to explain is a valuable commodity in the world of J (or any other programming language). I look forward to your future contributions.

Cheers, bob

Show-and-tell

Quickly Building a Simple Simulation

We want to build a File:Simulate727.ijs to get an idea of the distribution of starting hands.

The rules of the game are as follows:

. 1) two cards are dealt face-down initially. . 2) face cards are half a point, . 3) aces are one or eleven points (at the player's discretion), . 4) tens are zero or ten points, . 5) all other cards are "face value" points, e.g. a five is five points. . 6) Two aces and a five constitute a special hand that wins the whole pot.

The value of a hand is the sum of its points. The object is to be as close to either 7 or 27 (or both). The players closet to these values split the pot.

Here’s how we might start to build a simulation of this game:

   $deck=. (0.5#~*/3 4),4#>:i.10   NB. Initialize aces and tens at face value
52
   +/deck
226
   mean deck
4.34615

   _2<\deck{~10?#deck
+-----+------+---+---+-----+
|9 0.5|0.5 10|4 1|8 1|5 0.5|
+-----+------+---+---+-----+

   hands=. _2<\deck{~(2*5)?#deck
   +/&>hands
11 1.5 12 12 9
   hands
+---+-----+---+---+---+
|3 8|1 0.5|4 8|3 9|3 6|
+---+-----+---+---+---+

   init727Hands=: 13 : '_2<\x{~(2*y)?#x'
   init727Hands
_2 <\ [ {~ (2 * ]) ? [: # [

   deck init727Hands 5
+-----+----+---+-----+---+
|6 0.5|10 8|9 1|0.5 4|2 5|
+-----+----+---+-----+---+
   sums=. +/&>&> h0=. (<deck) init727Hands &.> 10000$5
10000 5
   3{.sums
 16   6  12 2.5 18
8.5  12   5 6.5 12
  9 5.5 4.5 1.5  4

   BKTS=: i.21 [ PCT=: 0   NB. Globals for "plotHistoMulti": fixed buckets, not % numbers.
   ss=. '10K 727 Hands' plotHistoMulti ,sums

Here’s an initial look at the distribution of hands:

initial distribution of "727" hands (Apologies for the "bar-creep" - the bars are supposed to line up with the integer intervals but do so only in the center of the chart.)

We can examine these 10,000 cases manually to get a rough idea of some of the distributions. For instance, how many deals have one hand that hits seven exactly on these first two cards?

   1+/ . =7+/ . = |:sums
1629

But how many deals have two hands that hit seven exactly on these first two cards?

   2+/ . =7+/ . = |:sums
134
   3+/ . =7+/ . = |:sums
1

So, how many deals have one hand that is only half a point away on the first two cards?

   1+/ . =6.5+/ . = |:sums
1630
   2+/ . =6.5+/ . = |:sums
86
   1+/ . =7.5+/ . = |:sums
1642
   2+/ . =7.5+/ . = |:sums
101
   3+/ . =7.5+/ . = |:sums
1

From the histogram, it looks like the most common first two cards total is around ten or eleven – which is it and how common is this?

   1+/ . =11+/ . = |:sums
2309
   2+/ . =11+/ . = |:sums
285
   3+/ . =11+/ . = |:sums
20
   4+/ . =11+/ . = |:sums
0
   1+/ . =12+/ . = |:sums
2152
   1+/ . =10+/ . = |:sums
2230

Now let’s formalize the initialization step of dealing: we’ll want to split the deck into a set of hands and the remaining deck.

init727=: 4 : 0
   ixs=. (2 * y) ? # x
   (<x#~(0) ixs}1$~#x),<_2 <\ x {~ ixs
)

   'd2 h0'=. deck init727 5
   $d2
42
   $h0
5
   h0
+-----+-------+----+----+---+
|0.5 2|0.5 0.5|10 3|7 10|4 5|
+-----+-------+----+----+---+

Now we start to hit the more difficult part of the simulation: how do we proceed? Which player will take another card in the hope of improving? How does a player decide what to do? Let’s start with a simple rule: if we’re within one point of 7, we stand; otherwise, we take another card.

hit0727=: 4 : 0
   wh=. 1<|7-+/&>y   NB. Which will hit (initially)?
   ixs=. (+/wh)?#x
   y=. (--.wh)}.&.>y,&.>wh#^:_1 ixs{x
   x=. x#~(0) ixs}1$~#x
   (<x),<y
)

Going forward, we’ll want to test different rules – this implies that we should come up with a good way to represent these rules so we can generate our search space programmatically. But first, we have another problem to solve: the ambiguity of scoring the hands based on the dual points possible with some of the cards: an ace can be one or eleven and a ten can be zero or ten, based on the player’s preference. How do we score hands with this consideration?

After some experimentation, we come up with this:

   scoreHand=: 3 : '~.+/&>,{(y=1)}((y=10)}(<"0 y),:<0 10),:<1 11'
It passes our initial test cases:
   scoreHand 1 1    NB. Two aces
2 12 22
   scoreHand 1 10   NB. An ace and a ten
1 11 21
   scoreHand 10 10  NB. Two tens
0 10 20
   scoreHand 3 2    NB. Some non-ambiguous cards
5
   scoreHand 1 1 1  NB. Could get to three aces after one draw
3 13 23 33
   scoreHand 1 1 10 NB. Or two aces and a ten
2 12 22 32

It took about an hour to get this far in the simulation.

Advanced topics

We looked atan essay about explaining why software development estimates are regularly off by a factor of 2 or 3, though some of us thought this factor was generously low.

The essay uses the analogy for software estimation of planning a hike down the the California coast from San Francisco to Los Angeles. The author illustrates all the sorts of things that are likely to go wrong. The comparison to software development projects is implicit but does a good job of showing how a 10-day estimate can quite reasonably balloon to 60 or 70 days once a project has started.

Some of the more pertinent comments pointed out things like "Developers are also the only group where they are asked to do something which has never been done before, and tell someone else how long it will take before they even know what actually needs to be done." (Devdas Bhagat)

Several others pointed out how much communications overhead between developers and others enormously adds to the time. One commenter (Walt Howard) with 30 years of experience suggested multiplying any rough estimate by the number of developers involved. This is pertinent to the expectation of much smaller teams with which one might run a project using more powerful tools like J.

Learning, teaching and promoting J

However, as we know, writing in J is no silver bullet. We looked at an example of some J code written in a style one ought not emulate. The code attempts something that would be interesting ("Arc Consistency") if it worked but apparently it does not.

This does not appear to be an isolated case in the world of software. We next looked over a (favorable) review of a book (on PHP) that was notable for the multitude of errors it noted. One interesting comment in the review was that although there was extensive example code, "[s]ome of it is concise enough so as not to distract from the narrative flow, but far too many examples involve much more code than necessary. This at first glance might seem to be an advantage, but it actually makes it more difficult for the reader to see the parts of the code relevant to the topic at hand."

We fans of concise code are not surprised by such a conclusion.

Astronomical Calculations on a Phone

We noted with amusement and awe the following communication from the astronomer J. Patrick Harrington.

from:	 J. Patrick Harrington <jph@astro.umd.edu>
to:	 Programming forum <programming@jsoftware.com>
date:	 Fri, Jun 14, 2013 at 12:36 PM
subject: [Jprogramming] Limits of J iOS - not!

I continue to be surprised by the ability of J on the iPhone to run rather substantial programs. I have a J version of a program to compute photoionization models of planetary nebulae. It has a main section of 500 lines or so and calls about 30 other verbs and data sets. One of these (2D grids of the collisional cooling by each of 16 ions) has dimensions of 55 250 16 4. The program integrates a system of 28 differential equations.

Last night, expecting the worst, I loaded it into my iPhone (on my jailbroken iPhone 4S, I used the freeware "Fugu" to make transfers from my MacBook Pro). And the thing runs! I do have to make use the unofficial "Insomnia.app" to keep the phone from going to sleep during the computations, which can take 10's of minutes. I'm just writing this to encourage others to explore the limits of J on your phone - you may be surprised.

Patrick

Materials

-- Devon McCormick <<DateTime(2015-02-08T18:49:51-0200)>>