Jd/Release

From J Wiki
Jump to: navigation, search
Jd | Overview | General | Ops | Admin | Guide | Technical | Release | License | Support

The Jd wiki is dynamic. It has the latest corrections, expansions, and changes. It reflects not just the latest release, but development work that hasn't been finalized or released. It reflects the bleeding edge and may not be relevant for the version you are running.

A snapshot of Jd wiki is converted to html and included in a release.

For the latest on Jd, use the wiki. For details relevant to your release, use the release html.

2017 December 11 - version 4.7

linux cd folder contains libjpcre.so as it is not included in J deb install

cd folder contains only jd platform binary

where range clause runs faster

FLUSHAUTO has been decommitted. Previously FLUSHAUTO_jd_ (default value 1) flushed buffers when a close, csvrd, createcol, ref, or set was done. This slowed things down (e.g., jdtests took twice as long) and wasn't required except for critical, production systems.

Use jdcloseflush to mark a db to require a flush when a close is done.

Use flush to flush buffers explicitly when it make sense for the application.

2017 November 29 - version 4.6

documentation reorg

csv colsep/rowsep AUTO support added back in

damaged db

repair tool can repair a damaged db

delete and update ops can use keys

sort op

upsert op

key op

keys - one or more cols form a key

col-data-pairs reworked

2017 November 4 - version 4.5

new tutorial - quandl_eod_stock_data

jdtests‘‘ NB. run tests

rationalize, reorganize, and clean up folders

2017 November 1 - version 4.4

old keys no longer work - new purchased keys are required

quandl_ibm - new tutorial - load table with historical ibm pricing from quandl csv

jdrt/jdex cleaned up

validate streamlined to run faster

docs updated to reflect join changes

jd'reads from a,a.b where b.id=2' - bug fix - incorrectly included missing rows

jd'ref /left ...' - left join - left1 and inner derived from left

left1 - preferred join if apppropriate - fast

jdadmin'none' - proper error report

2017 October 26 - version 4.3

join tutorial has example of multiple refs between tables (bug fixed)

some problems with jdex examples fixed

Jd paypal page added to web site and docs updated

new style keys supported

old style keys still work

OSX problems with unicode table and column names

  • previously failed because of OSX file system decomposed unicode
  • jdtests now flag this OSX error
  • will be fixed in a future release

806 new labs

  • tutorials use new 806 labs if available
  • sp.ijs loaded only if spx required to run labs
  • sptable defined (in case sp.ijs not loaded)

2017 September 28 - version 4.2

Snapshot of Jd wiki with new proposed licensing and prices.

Addition of tutorial server-zmq - high-performance, scalable zmq based Jd server.

2017 May 17 - version 4.1

Jd 4.1 is a major release and is incompatible with previous versions. A db from a previous version has to be converted to be used by this version.

See Convert for info on converting a Jd3 database to Jd4.

Overall the changes were to simplify and streamline code for the use cases that have proved common and to remove unwarranted complications. The big changes were removing jdactive and support for other than a simple join.

pacman 2.3.7

  • rewrite createcol
  • rewrite insert/update data validation to have simliar conformabilty rules
  • fix createtable parse problem with , separated coldefs
  • toJ for builddemo jdcsvrefs.txt

pacman 2.3.6

  • new Jd4 jdkey required (Jd4 key in ~confg and Jd3 key in ~addons/data/jd/config)
  • new shared library - version 8
  • jdactive col decommitted
    • delete immediately removes rows from the table
    • not dealing with rows marked as deleted is a major code simplification
    • everything (except delete) runs faster and cleaner
  • left1 is the only join - all others are decommitted
  • ref now supports multiple cols
  • reference col decommitted
    • was required for non left1 joins
    • creating reference was slow on large tables
    • code to dynamically track changes was complicated
  • hash col decommitted (primarily for reference)
  • update in place (rather than delete/insert)
    • much simpler code and much faster
    • update varbyte done in place (if space) or new data appended to val end
  • modify decomitted (update does it all)
  • insert
    • errors normally caught before insert starts - table unchanged on error
    • an error after the insert starts now reverts - table unchanged on error
  • when col is opened, its length is validated against Tlen - db damaged if wrong
  • tableappend decommitted (tableinsert does it cleaner and faster)
  • info changed to reflect other changes (deleted/reference/dynamic)
  • csvdump /replace
  • csvrestore /replace
  • csvdump creates jdcsvrefs.txt and csvrestore uses it to create refs
  • dropdynamic and createdynamic decommitted

2017 April 27 - version 3.3

  • Jd wiki - latest Jd documentation at: http://code.jsoftware.com/wiki/Jd/Index
  • Jd wiki snapshot (release specific) at: ~addons/data/ide/doc/Index.htm
  • update bug - bad shape deleted record
  • modify bug - bad shape truncated byte data
  • info validate/validate - col c marks bad tlen - col s marks bad fsize
  • new jd shared library version 7
  • db log.txt limited in size to 16mb
  • table... changes - srcdan removed as arg from all except tablemove
  • csv column names - bugs in utf8 and special chars fixed
  • csv cdefs cols names can now be any valid Jd col name
  • csv BOM UTF8 (239 187 191{.a.) skipped
  • csv cdefs # starts a comment line
  • csv csvcdefs creates cdefs with default " and NO for escape
  • csv cdefs AUTO no longer supported (handled in csvcdefs)
  • csvprobe op to examine csv file
  • csvscan - scans csv file and updates cdefs to have proper byte widths
  • csvcdefs /u no longer takes value - value is derived from data
  • csv tutorial reworked
  • csv_load tutorial added

2016 May 12 - version 3.2

  • partitioned table - significant new facility - see tutorial ptable
  • invalid epoch data passes precision assertions
  • default epoch and sfe display does not have Z suffix
  • table/column/dan name validation (General|Names)
  • read op parsing reworked and some bugs fixed
  • documentation of ops reorganized and improved
  • readtc requires jdtc clause (:::s::: no longer supported)
  • validate no longer done before/after insert/update/modify/delete
  • Tlen (active+deleted rows) maintained in table state
  • S_deleted (deleted rows) maintained in table state
  • info summary uses Tlen/S_deleted - fast as no read of jdactive
  • info improved and enhanced to handle ptable
  • csv loader bug with fields longer than 1000 fixed
  • where clause not bug fixed
  • datatune support removed
  • regex cd problem with linux debian installs fixed

2016 January 12 - version 3.1

  • showbox bug fixed for jconsole (non linedraw) box chars
  • showbox replaced by sptable (from simple project)
  • insert/update datatune bugs fixed
  • modify now does datatune
  • tutorial datatune updated
  • datatune doc
  • key enforced by shared library
  • Jd binary and code version match enforced
  • csv now supprts epoch cols (Ops csv)
  • csvrd default reads all rows (rather than just 10)
  • tutorial csv updated
  • install over previous install (orphans) now OK (scriptlists.ijs)
  • JAL version no longer tied to jd'list version'

2015 December 12 - version 2.2.11

  • dropcol does not update column_create_order.txt - bug fix
  • csvappend fails in some cases - bug fix
  • min-max/epochdt with no records works (not domain error)
  • logging and db validation improved (Technical|Damaged)
  • tutorial log
  • validate - detects db problems (Ops misc|validate)
  • info validate - info on all db mappings (Ops info)
  • info validatebad - validate mappings with problems (Ops misc|validate)
  • jdadmin does validate
  • validate done before/after insert/update/modify/delete
  • insert revert bug - detailed log and marks db damaged

2015 October 1 - version 2.2.10

  • linux lock bug fixed - concurrent Jd task not locked out
  • info dynamic - (Ops info)
  • createhash/createunique 2 to 10 times faster on large tables
  • createhash /nc
  • createunique simplified and changed
  • insert unique simplified and changed
  • flush - flush mapped file dirty buffers (Ops misc|flush)
  • ref changed and improved
  • performance tutorial - table size vs ram

2015 June 18 - version 2.2.9

  • where cola,colb = 1,2 doc and implementation bug fixed
  • jddropstop prevents drops (General|Drop/Delete)
  • jddeletefolder fails if criteria not met (General|Drop/Delete)
  • jddeletefolder more robust vs background tasks (Technical|Windows search service)
  • jddeletefolderok allows otherwise blocked jddeletefolder
  • welcome replaced by short message after jdtests run
  • modify with indexes as well as where clause - jdex_jd_'modify'
  • bugs fixed in where col in ()
  • rough version of pm (performance measurement) - pmhelp_jd_
  • droptable /reset
  • info jd - info on jd... cols (Ops info)
  • better createhash/createunique performance
  • fix bug in createhash/createunique for multiple cols
  • libjd.so built with stable linux (RH 4/Centos 5) (Technical|libjd.so)

2015 May 11 - version 2.2.7

  • where cola range (23,64) - tutorial reads
  • where cola,colb = 1,2 - tutorial where
  • jdadminx'a' (single letter) bug fixed
  • new doc page (General)
  • consistent rules for dan/table/col names (General|Names)
  • modify bugs fixed
  • createtable /a option must be before table name
  • create table from array - tutorial table_from_array
  • better error messages

2015 April 14 - version 2.2.6

  • CDEFSFILE (tutorial csv)
  • gethashlen bug fixed
  • where in () bug fixed
  • jdtests reports elapsed time
  • performance measurement - pmhelp_jd_
  • read.ijs sortfrom bug fixed
  • createcol shape validated
  • droptable allowed with hash (fails with reference)
  • some aggregates sped up
  • better error message for insert/update/modify
  • jdlast last line shuffled
  • renametable implemented
  • renamecol implemented
  • modify implemented - update in place
  • libjd_centos.so (Technical|libjd.so)

2015 March 10 - version 2.2.1

  • dropdynamic for specified columns
  • info unique

2015 March 9 - version 2.2

  • ref join - simpler than reference but with restrictions (tutorial join,Technical|Ref)
  • improved handle management and related errors (Technical|File handles)
  • epochdt new features - tutorial epochdt
  • epochdt allows blank (as well at T) as delimiter of time fields
  • epochdt bug with UTC offsets fixed
  • join with empty table bug fixed
  • join with createunique crash now signals error
  • csvrd bug with float .x vs 0.x fixed

2014 Jan 5 - version 2.1

  • key required to run Jd
  • epoch datetime col types (tutorial epochdt, General|Types)
  • smarter file allocation and resize
  • all column files mapped when db is opened
  • list lists version and open databases (Ops info|list)
  • info works only on current database

2014 April 22 - version 2.0

  • Jd released as addon