Wiki/Report of Meeting 2024-04-25

Report of Meeting 2024-04-25

Present: Art Anger, Ed Gottsman, Raul Miller, Devon McCormick and Bob Therriault

Full transcripts of this meeting are now available below on this wiki page.

1) Ed gave a report on dendrograms as a way to display information and has decided to move on to using tree maps as a way of displaying the tags. He provided a working prototype although he warned that it was in early stages of development. Ed said that he's not a fan of tree maps and wonders if this is going to be useful, but will pursue a bit further to see if there is promise. Raul suggested smaller boxes to allow space for the category names. Bob suggested the APL approach of including the shape in the border of the frame, except in this case it might be the name of the category. Ed says that a lot of these visualizations look cool, but he feels that the information value is not as good as you would hope when the cool factor wears off. Devon feels that an outline format is more compelling and Ed agrees. Bob talked about the Pixar effect of having to protect ideas in the early stages before they were given full critique. Ed feels that the tree map may be best to analyze whether the tree is balanced or if the structure is appropriate for displaying information to the user.

2) Bob showed some preliminary work that he had done on the template to display the category trees at the bottom of each page. Earlier attempts to incorporate SVG had failed because MediWiki turns SVG's into PNG's, but this takes away the ability to scale and the interactivity. The option of SVG would still be available for the J Viewer because JQt can display SVG. Ed pointed out that his tree maps were done in isidraw with JQt, J and some low level graphics calls. Bob explained that his naming of the links can shorten the category page identifier. Th other functionality that Bob is exploring is the has: pseudo class in CSS which allows action at distance that may allow information to be displayed in a separate area when a category is hovered over. Ed felt that this is a summary detail interaction and feels that has some promise. Bob is now working on the display to show the hierarchical structure. Ed thinks that the CSS of grid display can show the hierarchy automatically with grids within grids. Bob thinks that the work that is done in the template is worthwhile because it can be used in multiple areas and would not need to be changed too often. Ed reiterated his offer to help with JavaScript because that would certainly work within the wiki. Raul felt that a lot of the resistance in MediaWiki could be the result of legacy code that has grown organically over the years.

3) Jan Jacobs had sent an email with some preliminary categories suggested by his data mining. Ed says that the categories Bob has created have more semantic meaning at this points, but Bob thinks that this is early days. Jan's email:

Hi Bob, Ed,
since last time's meeting I have pursued 2 directions for compressing the large (sparse) document space that is spun up by all the terms in the Jwiki. Compression is necessary to keep  SOM training times down to acceptable limits.
The 2 directions are:
using the word embeddings of GloVe (100D)
using the random mapping approach used in the WebSOM experiment
Both directions demand for an adaptation of the hierarchical clustering approach because I didn't support compression 'well' enough.

I started with GloVe and this time with more terms (15K in total) in order to come up with a lot of them to look up in the GloVe embedded vectors array. Although I managed this time to  keep all wiki pages in, however the number of looked up terms is very low for some pages. This again shows that some pages lack enough *standard* terms. Nevertheless I have a base map  representing the 'bottom' of the tree with 118 clusters. I included the descriptive words of all clusters, with the contained page numbers (and implicit count) as appendix to this  mail. The number of descriptive words (characterising and discriminating ones) is to be specified.
At this point a warning should be given: word embeddings (~text AI) do code for some semantics in a text and one could derive *in theory* better descriptive words on all intermediate  hierarchical levels (direction1) than a pure statistical approach (direction2). When going for 'statistics' you will stick with words as they occur in the data set. So no clever  exchange of e.g. terms England, Wales, Scotland, ... by United Kingdom as a text AI simply could do. But one, could of course do this by yourself when given enough terms that describe  hierarchical clusters!

I will now continue with direction2. This also allows for including the Jenglish terms. When I have the base map I will provide you with a similar csv as below and we could discuss the  construction of the tree in an agglomerative way.
Jan.

Ed feels that the technology that is available to Jan is the limiting factor of what he can find. A server cluster on Amazon might solve that problem, but that might be an option later in the process. There are certainly many possibilities with this.

4) Ed asked Raul if the forking of processes issue had been solved with JQt. Raul had not seen any resolution to this issue. Ed said he was happy to explore tree maps for now and when those issues were resolved, Ed will return to the stand alone J Wiki Viewer in the future.

5) Raul showed an interface that he had developed based on trace to be able to follow the execution of J verbs as an educational tool. Raul is working on a bug that seems resistant to the debug stack facilities in J. Ed observed that this will feed an independent interface that will allow the user to understand the processes. Ed had thought of using a scroll wheel driven odometer model of going through a sequence of operations. Bob said that Marshall Lochbaum had once told him that a good illustration is much better than an animation for viewer understanding. Ed agreed that animations are often used by those who have not had the time to do a proper illustration.

For access to previous meeting reports https://code.jsoftware.com/wiki/Wiki_Development If you would like to participate in the development of the J wiki please contact us on the J forum and we will get you an invitation to the next J wiki meeting held on Thursdays at 23:00 (UTC) Next meeting is May 2, 2024.

Transcript

Alrighty then, when we last met, Ed, you were going to investigate something for the Jay

viewer.

I was going to do that.

I did do that.

Actually, let me, let me share my screen.

So I started out fooling around with dendrograms, which are basically, I don't have a picture

handy, but they're basically bubble trees.

And the problem with dendrograms for the Jay Wiki hierarchy is that when you lay the tree

out, it's a very inefficient use of space and there are just too many nodes and the

nodes wind up being pretty tiny.

You can't convey much with them.

So I reluctantly decided to investigate tree maps, which is what I'm in the middle of now.

And so what we're looking at here is a tree map of the categories.

And what happens is when you hover on a rectangle, there's a nested red highlight of rectangles.

So whatever rectangle you're in is highlighted red and its parent rectangle is highlighted

red and its parent is highlighted red and so on.

And then up here, there's a title bar that shows you the full path, the fully qualified

path from home to reference to files, from home to reference to interfaces, to R, to

interfaces, to Python, and so on.

And then what happens, the intensity of the green indicates the number of documents associated

with this node.

So there are a lot of scripts, but relatively few pages associated with the interfaces node.

And then the only other mechanism I've got is that if you click on a node, so you click

on home reference frameworks general, for example, it fills up this menu with all the

pages in that category.

So if I clicked on home essays miscellaneous, I get a whole boatload of pages.

And that is as far as I've gotten.

I have to say, I've never liked tree maps.

I sort of, I admire them.

I recognize their value.

And I guess if you were looking at a structure where the structure itself was of interest,

in other words, there were aspects of the structure that told you something interesting,

I think it would be great.

But as a navigation mechanism for the jwiki category hierarchy, I'm not quite sure I'm

seeing it.

These rectangles are not nearly big enough to put labels for the most part.

So that's out.

I'm going to keep fooling with it.

But I I'm a little skeptical at this point, unless somebody has a brilliant idea for me

to follow up on here, which I would very much appreciate.

Well, I agree that that's kind of discouraging in terms of information presentation.

You could probably do a prefix of the last part of the of the name, like J602 for that

one and maybe JData for that or for that one, you know, SQL.

That's true.

I could just do the leaf node name.

Just do the prefix, yeah, whatever fits.

And if you hover outside, so I'm in, I'm now at home reference framework SQLite.

But if you go out in between the leaf nodes, you just get, in this case, home reference

frameworks.

So that and here we've got a case where databases only contain SQL, apparently.

That's the only category.

So there's this weird effect where-

I was going to ask you about the double line.

Yeah, yeah.

Yeah.

You've got a, you've got what looks like a leaf node with that's got actually a real

leaf node inside it.

So because there's only one category under databases, which is SQL, at least that's what

the parser found.

I don't know if that's actually accurate.

I can go back and check.

But because of that-

You'd have to shove more-

You'd have to get a nested node.

Sorry?

For non-leaf nodes, you'd have to almost have to shove things down a little bit to make

room for a label at which point-

For the title, yeah.

And that's a, that's a common thing.

I've looked, I've looked at a number of algorithms for doing layout.

This is the least offensive so far.

That's a promising insight.

But yeah, I think you're right.

I need to, I need to shrink, I need to shrink the top of child nodes so as to make room

for the label for the parent node.

And that's probably what I'll fool with next.

And it's going to be, you know, 10 point or 12 point type.

It's not going to be, and clipped.

It's not going to be real legible.

I've got some nodes over here I haven't figured out yet.

Teeny tiny things that I think are, oh, books.

Yeah, I can't hit them.

They're so tiny, but I can't even get them to act as targets for the mouse.

So there's, you know, plenty to explore, plenty to figure out, but I don't, I don't know.

I think in the end, you're probably better off with an outline style of interface for

a, certainly for a hierarchy as small as this one is.

One thing I've seen in the boxes is in the upper left corner, I think APL does this in

some cases, they've actually like widened the line or at least widened the gap between

lines and they'll actually put the name of the, like in the case of, of home, which is

the box that encircles everything.

Right.

Yeah.

That, that was labeled would go up in the upper right corner or upper left corner.

Another thing you could do, I, since it was my suggestion, I'm going to unsuggest it is

if you just, just show the, the, the leaf node prefix and leave the, the hierarchy above

it to inference.

See how that I'd go there before I made too many other changes.

Oh, I see what you mean.

Yeah.

It's a little dicey because if you're in between leaf nodes, in other words, you're essentially

touching a parent or a grandparent node, then what do you show?

Well then you show what you're currently showing.

I mean, I, yeah, you're, you present too much information becomes noise, present not enough

and it becomes gloss.

No, darn it.

It should be, there should be a rule I can follow that works in all situations.

Just like the Bible is inerrant.

Damn it.

All right.

We can return to that another time.

Yeah.

Okay.

Fair enough.

That's, I'm sorry, go ahead.

I was going to say that that's actually pretty cool looking work though.

I mean, it may not have gotten you where you wanted to go, but boy, you've done a lot with

it.

That's the same problem that 3D visualizations have.

They have about a 15, 20, 30 second cool factor.

And then when you try to figure out what you can do with them, it sort of starts to break

down.

Yeah.

So I agree with you that it's cool and it's sort of fun to putz around with and there

are definitely more things that could do.

But I'm, I'm a tad skeptical at this point, I admit.

But we'll see.

This doesn't really grab me.

I mean, it looks very opaque.

Yeah.

Agreed.

I think if there were more labels, it would look really noisy.

There'd be more information and you'd still find it to some degree repellent.

So I don't quite know what to do.

I mean, you could imagine playing games with drill down.

So or with expanding the immediate parent or grandparent rectangle that you're in and

shoving everything else to the sides, you have room to show a lot of, a lot of text

for a few nodes or whatever you're focused on now.

That's not impossible.

You can certainly do that kind of thing.

I just, it hasn't, I haven't, I haven't fooled with it enough yet, I guess.

I'm certainly not giving up.

I'm sure I'll have something different to show next week.

My initial report is interesting, but probably not compelling.

Yeah.

I think the outline format just looks much more informative and useful.

I would agree.

And do you think it might be, go ahead.

Yeah.

And do you think it's a thing about tree maps?

Because I think people have to get used to looking at tree maps and almost looking at

them as a contour map, right?

Well, but then what does that tell you about the hierarchy?

Yeah.

What, what information do those contours convey that is of any interest to you whatsoever?

That's where I lose the thread.

That's what I meant by, if the structure is intrinsically interesting, then this is intrinsically

interesting.

This is potentially interesting.

But the structure, cross-structure, I don't really care.

I mean, if it's a balanced hierarchy, then it's just going to be a completely vanilla

monochrome map.

And what do you learn from that?

You don't learn anything.

So it's not like a map of a physical territory you intend to traverse.

That's different.

That's solving a different problem.

So not to sound, I shouldn't sound negative because I'm going to keep pushing on this.

I think it's nifty and fun.

But I, at this point, I think I'm, I'm not, I'm not terribly enthused.

There's an interesting thing I remember reading in Ed Catmull's book about creativity and

writing about Pixar.

I don't know whether you've had a chance to read it, have you?

Oh, it's a very good book.

Okay.

One of the things he talks about is the fact that when people would come to their storyboards,

they'd put their storyboards up and they, they would, there'd be a champion for the

storyboard.

So essentially the director would come up, put the storyboard on the wall, and then the

whole group would attack the story.

It's not working here.

Not working here.

Because it was really important was there needed to be at least a couple of people who

would go to the wall to save the baby, basically, because it was too easy to wipe it out.

Because it's easy to knock holes in all this.

But that's why they tell such great stories is because by the time they get to their final

iterations, they've had so many people look at something that the story just comes out

with great power.

And there's all these resonances that happen.

But that wasn't how it started out.

And he said, what they found is they were losing too many stories right off the bat,

because they could easily punch holes in them and everybody would go, oh, well, that's not

going to work.

And then it gets set aside.

And then three years later, well, it was Toy Story was one of the ones that made it through,

but like Cars comes out.

Well, Cars was a different idea.

And it got knocked down right away.

But then they, when they saw it work for other things, they said, well, we should.

And then when they had a champion, they could move it along at least the first couple of

stages and then it could stand up.

But he said, before it stands up, it's like punching a baby.

There's not going to be a defense.

There's not going to be any sense of what it could do.

It's potential.

And so you've got to be, you do have to be critical, but you have to be careful of how

you're critical and not throw up too much to begin with.

I think that that is wise and excellent advice.

And I certainly, I certainly agree with it.

I've seen a lot of tree maps over the years.

I have yet to run across one that made me say, oh yeah, this, this is great.

This I love.

So I am coming at this with a certain amount of history.

But as a programming challenge, it's very interesting.

So I'm going to keep pushing on it.

Maybe something will happen.

Yeah.

It is very interesting.

And the reaction is very interesting too.

The fact is you say that there's, it feels like there's an awful lot of information there.

And then when you look at it, there's really not that much that's useful.

It doesn't structure itself for you.

You could imagine, I don't remember quite what my color coding is and more, I guess

it looks like the bigger the rectangle, the more documents are associated with that category.

And also the more intense the green, the more documents are associated with that category.

So I'm redundantly showing document counts for each category node.

Yeah.

But you've got somewhere, you have a big rectangle that's white and a smaller rectangle that's

greener.

Yeah.

Some of these dropped out with, I don't know, no documents.

No, that doesn't make any sense.

Okay.

I'm going to back off interpreting this.

Okay.

Yeah.

So that's where I am.

Yeah, no, there's something there for sure.

But we'll see where it goes.

I think it might be of most interest in some ways, Bob, to somebody who's in your role

that is sort of responsible for the overall health of the structure in some sense.

So you're after a balanced hierarchy.

You're after a hierarchy where there aren't too many pages in any one place and too few

in any other place.

You'd like to have a really good discriminating hierarchy so a person doesn't need to follow

too many hops to get to what they're interested in.

Yeah.

I think on a general level, exactly.

But then there's specific areas where I wouldn't mind drill down at all if there's interest

in those areas.

Yeah.

Sure.

Yeah.

I guess that kind of, well, it steers me around to the stuff I've been looking at.

I'll just see if I can pull some of that up.

And I'll share a screen.

It's where we want to get.

Okay.

So two things.

This is actually my version, a very less sophisticated thing than what you've done.

But essentially this is my hierarchy here.

And all I've done is I've created a grid to display it.

But I haven't got to the point yet where I'm organizing the grid really.

So this group in here is all the newcomers areas.

So that'll get organized into probably this is the overall category.

And then within these, these will be the subcategories.

And I have to decide with the grid display how to do that.

Where I started with this earlier in the week was to say, you know, like you guys were talking

about SVG and I was thinking, yeah, this could be.

I really dug in and I was doing stuff with SVG.

This could be really cool.

And then I went to put it in the template for MediaWiki.

And did you know MediaWiki really doesn't like SVG at all?

What it does is it'll take an SVG and it'll create a PNG.

And that's how it displays it.

So suddenly it's not reactive and it's not, you know, you can't resize it.

Oh, well you can, but it gets, you lose the detail.

It pixelates it out.

PNG is not bad, but it's not SVG.

So you lose all of that.

And I was looking at it going, there must be a way.

And the only really way you can do it is if you have a media file that's SVG, it almost,

you sequester it off an area and you display it there and it can exist as an SVG.

But it's like, it cannot live on a MediaWiki page.

It can't live with other things.

What lives with other things is the PNG.

And that's the best you can do.

So that kind of put the kibosh on SVG, at least for that.

Although I don't know whether you'd be interested in using SVG for some of your viewer stuff,

because of course, JQT can run SVG and it runs it quite well.

So the option is still there for the viewer.

Okay.

Well, that's good to know.

Yeah.

Thank you.

I think you're onto something here.

That's quite cool.

I think you're going to have to do a lot of manual layout probably.

Yep.

But it might be worth the effort.

Well and that when I was playing around with the SVG, that's exactly sort of what I came

to was I can do this once and drop it into a template.

And I mean, honestly, I can do all sorts of things in the background and the way the structure

shows and all the different things I can do with color and shape.

And I can actually really build something.

But the problem is as soon as you dump it to the media wiki, it's no longer reactive

at all.

So all you get is a pretty picture that if you zoom in on it, looks worse.

But that's not the case with JQT.

You can do things with viewports and move stuff around.

Yeah.

No, I understand that.

I am more comfortable with the flexibility of an honest to God Turing complete programming

language for the graphics than I am with, I mean, SVG is one of those cases where the

discipline is liberating.

In other words, there are restrictions on what you can do, the degree of malleability,

the degree of dynamism that you can have.

And those restrictions make it a lot easier to do certain things.

But if you want to do something that the SVG designers did not think you might want to

do, you fall off a cliff.

And I try to avoid the cliff.

And no, I don't have an example offhand, but I know that it is a restrictive environment.

I think I can do more without it than I can do with it, at least for the kinds of things

I want to do.

But it is good to know that JQT handles them.

Yes, it does.

And it does allow some things with interactivity that you work really hard if you're in the

CSS or HTML or JavaScript areas.

Well, those are more discipline is liberating environments.

They have their own restrictions.

But if you're comfortable with a Turing complete language and a low level graphics library,

the world is your oyster is what I have found.

And you can position SVG within other, like aside from MediaWiki, you can use a grid display

and you could put SVG as the contents for each grid if you wished.

So it does play fairly nicely with a lot of the standard CSS and HTML stuff.

Right.

Just to be clear, everything that I showed a minute ago, not HTML, not CSS, not nothing.

That was just a couple of low level graphics calls and J.

So there's no that wasn't a web page that you were looking at.

That was a straight ahead is a graph or is it draw, I guess.

Well, that's good to know, too, because I think that shows some flexibility there that

I wasn't sure was existed.

But let's the problem is that the flexibility is overwhelming.

Right.

So you can you can do anything.

And as a result, anything you try to do is going to be more effort than it would be if

you did it in SVG, assuming SVG was capable of doing it or CSS or HTML.

Yeah.

Where I am right now at this stage with this with this layout of the different categories.

Is I what I've done is I've taken the categories and because you can create a link with a different

name.

What I'm actually looking to is category newcomers and there you can see it on the tool tip.

All I need to display is newcomers.

So I've been able to do an abbreviation there.

The other thing I found is with CSS, you can use the has I think it's a pseudo class.

And you can actually by hovering or clicking on something, you can have something else

unrelated to display as long as it has the right property.

So, in other words, I can change.

I haven't done it with this group yet.

But say, for instance, as I was on newcomers and I had a blank area out here to display

a bigger page and I could when I hovered over here, actually change the opacity of whatever

information I wanted here.

So suddenly it shows up.

And then using that in combination with the not pseudo class, if I went over here and

I went to something else, I could close this one.

And then I could open a new one.

And what that would allow me to do is I could this becomes sort of semi permanent.

So I can actually go over here and click on things.

If I just do it by hovering, it only shows up while I'm hovering.

So the information is there.

But by the time I got over here, it would have disappeared.

So that's some of the stuff I'm playing around with right now is to try and figure out I

think that's probably the next stage is to try and figure out whether I can you were

talking about when I was I was talking about sort of exploding or doing a fish eye and

part of it.

What I came to is if I have extra space over here, that can be my fish eye.

And this doesn't have to change as I'm moving around it.

Yeah, no, you're you're talking about a summary detail interaction experience.

Yeah, that's that's exactly right.

Yeah, just.

Okay, I'm sorry.

Go ahead.

Nope, that's that you're at.

You're exactly right.

That's what I'm talking about.

And that just provides an ability to provide more information than whatever this heading

would be.

Right.

And this would set as a template at the bottom of each page.

Right.

So you would have the option to look at your different categories.

And now the place I'm really at right now is how to organize these so that the categories

the hierarchical structure shows up.

Because right now it really doesn't.

No, so I have a thought on that.

You have a thought on that?

Yeah.

So absolutely.

Yeah.

There's nothing to stop you with the CSS grid facility from nesting.

Yeah.

So you could in CSS define the hierarchy, define the category hierarchy as grids within

grids within grids.

And then just let it go and see what it comes up with in terms of layout.

You might be surprised and maybe unpleasantly surprised, but you could also edit it.

You could change its properties.

Yeah.

And I'll just quickly go to the edit of this page.

So I've sort of started down that path with defining different classes for the different

areas.

Now, I may not stay with those classes.

I just went with levels at this point.

And so that's how I'm producing my borders and everything is I'm just saying all these

levels.

But you're right within each section of the grid, I can have its own little grid and explain

it, you know, explode it in that area.

Then it just becomes a question of how I use the space.

But that probably is the next step and use classes to be able to group things.

Yeah.

I mean, it won't be right the first time you do it.

And it's still a manual effort, but at least you're not doing manual layout.

You're just doing manual hierarchy specification.

Exactly.

And the other thing I'm doing it in a template.

So I only have to do it once.

And I don't I mean, it's a fair amount of work to organize it to start with.

But once it's done, it's done.

And actually, the other thing it means is that it might conceivably behave OK on a smaller

display.

Yeah, it should, because with grid, I can I can make some adjustments for the size of

the display.

Well, you just you just yes, you can.

But even without specific adjustments, if you just tell the grid, hey, you've only got

this much horizontal room to work with, it will reflow appropriately.

Well, yeah, you have to give it hints about how you want it to reflow.

But you're exactly right.

If you went to a narrower screen, you might find that it has more of a vertical format

because that's what you do when it's too narrow.

But that's all depending on the hints, the min max hints that you put in.

Right.

The goal would be to to minimize the number of hints that you signed yourself up for.

Just because the less work you have to do in that regard, the better as you have to

maintain the hierarchy in later years.

Yeah, I well, yeah, again, you don't want to complicate it.

Absolutely.

But I think there's a certain amount you have to do to make it work for different sizes.

I can't get around that.

I haven't used grid in a long time.

Okay.

Yeah, no, I think you're right.

I think the next step is to bite the bullet and just do the whole grid and let it rip

and see what it comes up with in various screen sizes.

Yeah.

And then the second part will be to see if I can use that display, you know, off to the

side and pull information out that way and see if I can get that working.

But all of that is it would be done through CSS and it'll all work within a template form

as opposed to SVG, which unfortunately doesn't.

Yeah, right.

So that's where I am right now.

So a lot of the same questions you're facing with regard to hierarchies and stuff I'm looking

at, too.

Yes, absolutely.

Yeah.

Yeah, it's it's yeah, you're gonna need to you're gonna need to think about basically

exactly the same stuff I do.

Your implementation mechanism is different, but yeah, the same set of questions.

You're right.

Yeah.

And on intermediate nodes and how to handle nodes.

Yeah.

Yes.

And I'm things I'm kicking around in my head is whether it's it's a shading mechanism so

that you go starting from light to dark that you end up using the gradation to show as

you go deeper, sort of like along the lines of your shades of green.

Yeah.

You know, that's might indicate as you're moving down a tree or moving up a tree.

But again, I won't know until I can dig into it and see how that works.

But I can do that with a block background, I mean, or changing the thickness of the the

lines around the different areas.

That's another way to do it is just do it by strictly by the thickness of the lines

that as you go deeper, the line gets thinner.

Yeah, I actually had that at one point, big, thick lines for upper level nodes and much

thinner lines for lower level nodes.

And I that didn't work out so well for me, but I would definitely encourage you to experiment.

So far, the nesting that I did with just one pixel borders was the clearest in terms of

conveying the structure.

But that could be just an accident of the way I did it.

Anyway, that's the stuff I found out, which was disappointing with SVG, but maybe shows

the promise using the grid and.

It's a brave new world for you, Bob.

Very powerful.

Plus, if you want to, you can easily use JavaScript.

I don't know if that's the case with SVG.

Yeah, you can.

Absolutely.

Oh, yeah.

Yeah.

And and you kindly offered to help out with JavaScript.

And I will if I need to go down that road, I will take you up.

You could actually do a tree map in JavaScript if the tree map thing works out.

If I can come up with a design, a visual design and an interaction design that works.

There's no reason in principle why we couldn't implement it in the wiki.

Yeah.

Well, and that's because I know the wiki, it's, you know, JavaScript isn't a problem

for the wiki.

They don't seem to have a problem with that.

Just they don't like I think some of the ways that SVG brings stuff in.

Yeah, the idea of treating SVG as, oh, we'll take a picture of it and put the picture up

just seems completely to defeat the purpose.

It's it's remarkable.

I think it's a old it's it's browser support back when it was early on and nobody's gone

gone back to, you know, nobody likes to deal with old code.

So it's kind of just stuck there until somebody gets motivated to either rip it out and replace

it or do some archaeology work on the code or whatever else.

Yeah.

That's pretty much for my part of it, looking at those things.

The one extra thing that's come up that I was aware of, and I think you've probably

saw the email as well, Ed, from from Jan.

I did.

I couldn't I couldn't entirely interpret it.

But I did look at the spreadsheet, which had categories and subcategories.

You had a two level hierarchy.

And I guess what I would say is that it makes a lot of sense.

I can understand how you would have grown that hierarchy by sort of bubbling up by examining

all the documents flat initially.

And I would say that your hierarchy is more expressive of the content.

That is to say, if I were looking for something, I would probably prefer to use that hierarchy

versus the one that was developed automatically.

But I he doesn't have to apologize for it.

I understand how he got it.

It makes sense.

But I think that as a navigation mechanism, if that's the problem you're trying to solve,

I think the hierarchy you've come up with at this point is better.

Yeah, I agree.

But I also think of the same back to the Pixar thing.

It's a baby.

And it's quite possible that one will go far beyond.

Yes, exactly.

Yeah, exactly.

No, this is just his, I don't know, first, second cut at it.

So right.

He's just like me with the tree map.

He's still got some hope.

So we should both continue.

We should all continue.

Nobody should smother any babies.

No.

None of that here.

Yeah, definitely not.

Anyway, so that's the only other thing.

And well, I guess I could read out the email.

He said, since last time's meeting, I've pursued two directions for compressing a large, sparse

document space that's spun up by all the terms in the jwiki.

Compression is necessary to keep some training times down to acceptable limits.

The two directions are using the word embeddings of glove, which he's got parentheses 100D,

which I'm guessing is either the depth or some other dimensionality.

100 dimensions.

100 dimensions, OK.

Or two, using the random mapping approach used in the WebSOM experiment.

Both directions demand for an adaptation of the hierarchical clustering approach because

I didn't support compression quotes well enough.

I started with the glove and this time more terms, 15,000 in total, in order to come up

with a lot more of them to look up the glove embedded vectors array.

Although I managed this time to keep all wiki pages in, however, the number of looked up

terms is very low for some pages.

This is again shows some pages lack enough standard and he's got emphasis on standard

terms and I think that that we've all noticed that in the wiki.

It's erratic in its depth.

Nevertheless, I have a base map representing the bottom of the tree with 118 clusters.

I included descriptive words of all clusters with the contained page numbers and implicit

count as appendix to this email and that was what he included was the CSV of it.

The number of descriptive words at this point, sorry, characterizing discriminating ones

is to be specified.

At this point, a warning should be given.

Word embeddings do code for some semantics in a text and one could derive in theory better

descriptive words on all intermediate hierarchical levels.

He's got direction one than a pure statistical approach direction two and I guess that refers

back to is using the word embeddings for glove as one and using the random mapping approach

for two.

When going for statistics, you'll stick with words as they occur in the data set.

So no clever exchange of examples of terms like England, Wales, Scotland, by United Kingdom

as a text.

But one could of course do this by yourself when given enough terms to describe hierarchical

clusters.

He's going to continue with deck direction two, which was the web some random mapping

approach.

And this also allows for including the J English terms.

When I have the base map, I'll provide you with a similar CSV as below.

We could discuss the construction of the tree in an algorithmic or a glomerous way.

Yes, a glomerous.

So I think he's made some progress too in finding out what does and doesn't work, which

is great because we all may end up in the same space asking the same questions, which

is pretty cool.

Yeah.

You know, it's a shame.

The technology that he's using is basically badly constrained by the resources he can

throw at it.

He's got a desktop box, something like that, maybe without even a GPU.

If he were willing, if we were willing to put it up on Amazon web services or Google

cloud platform or whatever, and rent a GPU cluster, I think the scope of what he could

do would be much greater.

That is a good way to max out your visa card.

So I'm not recommending it.

But it's sort of a shame that we're still living in a world where he has to live with

that constraint.

And I've certainly seen areas where that constraint is very real and no amount of cleverness will

get you past that barrier because it's just a scaling thing.

Yeah.

But we'll see where he gets.

And if it's promising, then maybe we look to drum up some resources and see what we

can do with that.

Yes.

Yeah.

Yes.

And I think that's about all I have for the stuff that I've done.

And I'll leave the floor open for anything anybody else would like to...

I have a question for Ro.

The bug report you put in regarding forking off a Windows process program, has that been

resolved?

Not to my knowledge.

Okay.

And that's not the only bug I'm stalled on.

I'm sorry.

It's that old code issue, you know.

There's so many moving parts that you're afraid of breaking something or you're not afraid

of breaking something and you break something.

Then things are broken and you've got to keep breaking things until they're fixed.

It's a definite hot potato issue.

I did include it in the report, last week's report.

And I'm pretty sure Bill does take a look at them.

So it might have twigged him a bit, but I don't know.

I haven't seen any other response from him.

I would rather work on tree maps than work on launching an independent or standalone

JViewer.

So I'm just as happy if the bug doesn't get fixed soon, to be perfectly honest.

Yeah.

Well, that's the advantage of working as a volunteer.

You get to pick the areas of interest.

Somebody should have told me about this years ago.

So much better than working for a living, I'll tell you.

I can't begin to explain.

I had a lot of discussions when I was managing people and managing volunteers.

I had a lot of discussions with volunteers saying, "You don't understand the power you

have."

Yeah, right.

You get to choose your projects.

Good and not evil.

Yeah, exactly.

I have to do what my bosses tell me to do and I have to do what my job tells me to do.

You get to do what you'd like.

Anyway, anything else?

My apologies, Devin, for not being at the JUG meetings because I find them really cool.

But we've gone into that sequence where the Tuesdays we do the podcast or the Tuesdays

that you do readings.

And until we split from that, we'll probably go three or four months that I can show up

and then I'm off again because there's just too much to do on the day of.

Yeah, you put a lot of work into those.

Well, yeah.

There's a crunch to get them done.

Well, you mentioned last week that you were working on some things that you weren't ready

to talk about yet, but are you blocked on all of them?

I don't know if I'm blocked.

I can show you.

Let's share my screen here.

It's like I said, I got bugs that I'm dealing with or not dealing with or whatever.

I even got this here.

So I was working on a trace like.

So here's that in action.

And what it's doing is when it I'm not showing the results of evaluations, but when there's

an evaluation going on, I'm I'm showing the details.

And so in principle, in this case, doesn't do it.

But if you have a compound verb.

Like here we're doing plus reduce and.

It goes in and shows the pluses.

I could I could also show the results of these.

I haven't got to that point.

The thing that's bugging me is my code has so many extra moving parts that I should probably

just get rid of.

I'm doing things like towards the end, I'm displaying things three times for no good

reason.

It's always something to do with my code structure.

I haven't haven't gotten into figuring out why that is.

But anyways, that's that's that's what I was working on.

And I haven't this is really more of a motivation issue than as much as anything else.

But there's a lot of junk in it.

Remember, you can either fix your code or you can repair it after it makes mistakes.

So the easy fix to showing the same thing three times is to check for dupes on the output

and just discard anything that looks the same.

I know that's got problems, too.

But yeah, that that does bad things when I have dupes in the input.

Yeah, yeah, I know.

I know.

And it doesn't.

And it also leads me into a further situation since I don't know what caused it.

I don't know what further symptoms that underlying problem would cause.

And would mask and then filter it with mask.

Oh, this is cool.

Yeah.

But I was it is kind of an attempt to combine debug and trace.

And the problem is I got too ambitious and I'm running into into remember that bug that

I posted forums about how I couldn't do a stack trace.

Yeah, that that was related to this trying to trying to analyze what's going on with

this better code.

I have.

Although it's, you know, conceptually a replacement for the trace, I wrote it in such a way that

I could I could rip out my part of the rig and replace it with an event driven structure.

And right now I'm trying some might it's possible that my bug has to do with what I've done

there, which means I should probably just rewrite that entire thing.

And of course, the other thing is I'm using that that rap eval stuff, which has a whole

bunch of things that's tracking, which I just don't claim don't use here.

So that just gets in the way of reading the code.

Yeah.

So all together, I think I've got twice as much code as I need.

I'm using about half or a third of the code that's actually in place.

And as you can see, it's doing something that I don't think it should be doing.

And I have don't have the since I waited for a while to get to this point, I don't have

the insight of what what my code might be doing that would be triplicating.

It's not always triplicating either.

In some cases, it's four times.

It depends on how many how many items you have in your argument, because you've got

nine items.

Yeah, I can if I do two items, it's very similar.

It still does three.

Wow.

Yep.

So if I'm going to do here, bring this down, change it to two.

Yeah.

But you can see here that I've got four of these.

Yeah.

And it's it's right off the bat, it's doing a count on the two, three, three times.

So whatever it's doing, it's doing three times.

Now is that that's based off the trace?

Yeah, that's this this.

What you're seeing here is this evaluation right here, how to do is divided by pound

of two, three.

So it's doing his pound to three, three times, but it's really just that it's doing the plus

reduce to three is doing for those.

And then it's doing the pluses.

Actually, here's a plus and here's a that's five, only one of those for that reason.

And then it's doing the divide three times.

So something screwy there in my logic.

I think it's got to be my logic and not and not Jay here.

But Jay is a bug because it won't let me do a stack trace.

I was going to say, what's a stack trace?

So I can just see what the call tree is for each of these.

But I can't.

Jay will not do that for me because of some bug that Henry thinks is a JQT bug.

So unfortunately, that same bug shows up under J console.

It does sound like you could fruitfully get rid of code.

Yeah, get rid of the stack trace.

But I just need to bite the bullet and rewrite it.

Yeah, you've learned a lot.

It's time to rewrite it.

It's time to spread.

Brooks said you should plan to throw one away.

You will anyway.

So what the hell?

Yeah.

I should probably do a specialized just to do trace.

Because this part of the point of all this rigging here was to do something else.

And I'm not getting anywhere near that point because I can't figure out how to manage the

screen real estate for the general case.

So this is easier.

You're going to have-- it sounds like you're building this in such a way that there will

be an engine independent of the output user interface that could be leveraged and exploited

by some different user experience.

Yeah.

But the thing there is if I want it to be a graphical display, for something like this,

I can do a loop.

It's just sequential code in a loop.

And it goes out to the screen.

That's fine.

But if I want to have it going-- this all flies by real fast.

And unless I want to throw quad DL into there to have it still be an animation but happening

slower, it's going to be a key.

What's quad DL?

Sorry.

I'm not an APL-er.

Delay.

It's--

No, that's not a solution.

Yeah.

Well, I-- I don't have-- the slide is somewhere.

I won't try to dig it up.

I won't waste your time.

But I did have this notion of a sort of odometer model for executing J code where the scroll

wheel-- where the cursor could be positioned over a verb, and the scroll wheel would move

things forward with respect to the inputs and outputs of that verb.

And that's kind of what I was building towards here.

OK.

Except, of course, it got too complicated to debug for the amount of motivation I'm

putting into it.

And plus, the debugging requires that I debug not only my code but somebody else's code.

Yeah.

And some of it's in C.

Yeah.

When Marshall Lockbaum was talking about-- we had a discussion at one point about his

BQN documentation.

And I think we were talking about animations.

And he pointed out-- and I think he's-- the more I think about it, the more I reflect

on it, the more I think he's exactly right.

A good illustration is way better than a good animation.

Yeah.

Absolutely.

And what you may be looking at is a way to create an illustration that you can interact

with, but it will give you the information.

Well, yeah.

It's still an animation, but it's an animation that's very much under your control.

Yeah.

And so there are still differences over time.

That's basically what an animation is.

But you decide what they are going to be.

And what aspect of the illustration you're looking for, the information from.

And what part you're going to be changing, tweaking, in order to create those changes.

Yeah.

So, no, Marshall's absolutely right.

Animations are often a crutch for somebody who couldn't spend the time to do a static

illustration that conveys the same information better in less time.

Well, and I think the illustrations tend to be much more user-friendly because you can

sit and look at them.

You're not having to play it back to see what this evolution is happening.

Yeah.

You don't have to remember anything.

No.

You're not trying to build up a picture in your mind.

It's, as you say, all right in front of you.

Yeah.

A picture's worth a thousand words and it costs like $10,000.

Yeah.

Well, right.

Yeah.

That information doesn't come from nowhere.

Bro, I am extremely happy that you're still working on this because I still think there's

a lot of value, potential value to it.

Both for debugging and for pedagogy.

Yep.

I think it's very cool.

You were going to say something, Devin?

No, I was just going to reaffirm what Ed was saying.

Does anybody else have anything they want to throw out there?

That was very cool, Earl.

Yeah.

Thank you.

Stop sharing.

All right.

Well, I am-

Unrelated topics.

Unfortunately, it always devolves into politics and I respectfully refuse-

I can do physics.

Oh, physics is also good.

Yeah.

Cosmology or something.

Cosmology is good.

Well, the smothering the babies remark just reminded me of an article I read in the New

York Times.

I was like, "Oh, I'm going to be a physicist."

I was like, "Oh, I'm going to be a physicist."

I was like, "Oh, I'm going to be a physicist."

I was like, "Oh, I'm going to be a physicist."

that he was expecting some punk kid in the driver's seat with a meth lab in his trunk,

and he thought he was on to something. So this sweet, older lady, you know, rolls down her tinted

window and smiles at him and he had to come up with something, so he said, "Well, there's mud on

your rear license plate, I couldn't see it, so I have to give you a ticket." So she came home and I

said, "We're getting rid of it." Donated it to National Public Radio. It's just not worth the

traffic stops. And it was a great car for, you know, 20 years. Yeah. Well, and then you think

about the profiling that can go on in other areas and just realize that... Oh yeah, no, right. You

don't want to mess with the cops. Yes, based on skin tone, largely, at least in this country. Yeah,

no, it's true. This country as well. Yeah. Well, okay, somebody, we got two minutes in which

somebody can lighten the mood a little bit and I would appreciate it for one. I have a... mine

was going to be about heavy universes. No, Raoul, no. Not lighten enough, huh? Well, I'll tell you,

I'm still getting rid of stuff and I have a stack of APL books here I'm going to get rid of, so I'm

going to put out some more notes. But I was actually at an APL meeting a week before last

and it was kind of a funny coincidence because they, the APL, the dialogue people came to the

US in part to see the eclipse and they figured they should do a meeting while they're here.

And Chris Lincoln was at the meeting because he was in from Bonaire to see the eclipse. And so,

he usually doesn't get to the United States much anymore. And so, we had a nice meeting with a

bunch of people I hadn't seen in a long time. And someone I talked about getting rid of the books

gave me his card, so I have to remember to let him know. Oh, I have a thought on that, Devin.

I faced a similar problem when my father passed away just over a decade ago. He, by education,

was a classical philologist. That is not how he supported his family, but that's what he did. Had

he been independently wealthy, he would have gotten his druthers, he would have wound up

teaching classics at Yale. That is not what happened. But I inherited a huge number of books

from the 40s and 50s in Greek and Latin. And I thought, well, I actually checked and it turns

out that the University of Texas, which is the university closest to my house, to my shock,

has a classics department. This is a university founded with oil money. These are wildcatters.

These are not guys who are into classical philology typically. And no offense to them,

it's just sort of not their background. I thought, well, I will call up the department and I will ask

them if they might be interested in a bunch of 60-year-old, falling apart, hardcover,

Greek and Latin classics. And then I thought, no, that's the wrong thing to do. And I put all

of the books into a gigantic box and I addressed it to the classics department at the University

of Texas and I shipped it to them, third or fourth class book rate. I don't know what they did with

it, but I discharged my obligation. I did the right thing by my father's memory and I didn't

need to argue with anybody. So I would say that maybe if you've got this guy's card and it's got

an address on it, your path is clear. - I could do that, yeah. I also have a,

one of the things I was getting rid of is an old Kindle. And I gave it to someone,

then I realized it was passcode protected and I didn't have the passcode anymore.

So that was a problem. And then the same week, my current Kindle stopped working.

It would no longer connect to wifi. So all this stuff that was on it, I could continue reading,

then I did a factory reset and I lost it all. So it still functions as an electronic reader.

You can put a cable into it and download stuff off of Project Gutenberg or whatever.

As long as it's an EPUB format, it's just a lot less convenient to use without having the wifi,

but it's still a perfectly good device, given that limitation. So I don't know,

I would like to throw it away. - I had the inverse experience

Kindle wise in the last week. I forgot that I owned one because I haven't looked at it in years

and I bought another one. And while I was waiting for it to arrive, found the original. And I

thought, geez, well, what do you know? I guess I'll just return the new one when it comes.

Boy, they're nice now. The 11th generation Kindle is larger, has a smaller bezel, which is sort of

nice. But the really good part is it has this wonderful, very soft sunset orange glow.

If fireflies were orange, this is the color they would be. And you can adjust the degree of

intensity or luminance. And the effect of this is you can read your Kindle in bed without having

to turn on a light. And it is just the softest, nicest reading experience there is. So I have

two Kindles and I'm perfectly happy keeping both of them. - Well, I really liked my Kindle, but

if you try and download technical stuff in PDF form or something, you can't really,

you know, unless the font placement and size is exactly right. If you try and expand it,

you can't read the whole page. And, you know, so PDFs don't really work that well with it

because you can only expand in discrete increments. - Well, I got it mainly so I could read cheap PG

Wodehouse novels and short stories. And it turns out all of Wodehouse's work he wrote in the early

20th century, all of his work is out of copyright. So Amazon will sell you like 5,000 pages worth for

a dollar and a half. - And I know Gutenberg has a bunch of those too. - Oh yeah, no, it doesn't

surprise me. I'm sure that's true. But this is supposedly the complete Wodehouse as one giant

wodge of data. So I'm slowly working my way through it. God, he was wonderful.

- I've heard of him. I have not read, I should, because of the stuff I've heard of.

- He's the Jeeves guy, right? - He created Jeeves and Wooster. Yeah,

yeah. But he did a lot more than that. All right. I feel the mood has been sufficiently lightened.