Complete backup or restore is just a copy of the db file folder. Host shell scripts can provide full backup/restore. With large databases and suitable hardware it might be worthwhile to use multiple tasks and use compression.
CSV dump/restore also provides complete backup.
See tutorial log.
There are many ways a database can be damaged: hardware crashes, OS crashes, JD bugs, application bugs...
A simple example is a table where different cols have different counts. A db with such tables can work in some cases, get unexpected errors, result in further damage (that obscure the orignal problem), or in the worst case give wrong answers.
When a col is used that has a bad count (not the same table TLen) the db is marked as damaged.
jd'validate' checks the integrity of the db (all tables, columns, files) and if there is a problem, marks the db as damaged and signals an error.
jdadmin does a validate to ensure the db is ready for use.
Validation details can be seen with:
jd'info validate' NB. complete info jd'info validatebad' NB. info about problems
The db is marked as damaged and a jdlog record is written if a validation fails.
Jd is distributed with JAL (Package Manager) and the Jd library is at ~addons/data/jd and is accessed with the following equivalent lines:
A developer works with a local repo. Use the development library with something like:
Loading jd.ijs sets JDP_z_ as the path to the Jd library and this is used for all library references.
An automated process copies the developer repo to the addon svn repo to build a new Jd release.
Jd requires lots of file handles. Using thousands of columns requires thousands of handles.
Jd fails badly if it runs out of handles. Unable to access a file, an error is signaled, perhaps in the middle of an operation that will leave the database damaged.
Windows user does not have a limit on file handles.
Linux/Mac user often has low soft and hard limits on handles and this must be increased for serious use of Jd. There is no reason to not raise the limit to 100000.
See the soft and hard limits with:
...$ ulimit -n
If hard limit is high enough, it might be easiest, before starting J, to do:
...$ ulimit -n 100000
To increase file handle limit for Linux Jd user fred: ...$ ulimit -n # show current file handle limit
run superuser text editor and open /etc/security/limits.conf add following 2 lines at the end fred soft nofile 200000 fred hard nofile 200000 save the file, restart system, and verify new ulimit
To increase file handle limit for Mac the steps are similar, but of course different, and details are left to the reader. Yosemite has a low soft limit and a high hard limit.
Folder symbolic links (Windows folder junctions) are used to place db cols on different drives.
- i/o load balancing across drives
- some cols can be placed on ssd drives
- total db size irrelevant - only limit is biggest col must fit on a drive
See Admin jdlinkmove, jdlinkset, and jdlinktargets for details.
See tutorial link.
Jd linux libjd.so shared library will run on most modern linux systems.
If Jd gets an error loading the linux shared library, please report the following to the J database forum:
...$ cat /proc/version ...$ cat /etc/issue ...$ ldd .../libjd.so
Parts of a database (tables, cols, data) correspond directly with the file structure. That is, a table is a folder in the database, each col is a folder in its table folder, and data is a file in its col folder.
When a database is opened, J locales are created that correspond to the database structure. Each table has a locale with metadata and each col has a locale with metadata and mapped file(s) with the data.
Sometimes it can be useful to dig into the internals.
jdadminx'test' jd'gen test f 3' jd'reads from f' t=. jdgl_jd_'f' NB. get locale for table f NAME__t NB. table name NAMES__t NB. col names in table c=. jdgl_jd_'f x' NB. get locale for col x in table f typ__c NB. column type PATH__c NB. path to col dat file dat__c NB. mapped file data
See pmhelp_jd_ for info.
Folder pm has scripts for performance measurement.
.../db/tab/refcol/jdstate has dirty flag and starts as 1
set 0 whenever ref col is recalculated
set 1 when delete/update/insert done to either table
jdstate written whenever dirty flag changes db can be closed with dirty 1
Windows search service
Windows Search Service (content indexing, ...) can cause lots of disk activity and can interfere with Jd file operations and if possible should be disabled when using Jd.
Disable Windows Search Service as follows:
- command prompt ...>services.msc
- scroll down and right click Windows Search
- click Properties
- click Stop button to stop service if it is running
- change Startup type: to Disabled
- click Apply