David Lambert/dwldb and cryptocurrency
Jump to navigation
Jump to search
Warning: the file paths are oriented toward my personal configuration, including linux.
These two files are implemented as classes. The database. appendcol automatically resizes as needed. There is not an "append rows" functionality implemented. When open rw, the database first makes a tar archive of the columns. Otherwise, error recovery is not implemented.
host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR
Note'00README'
To the Hands
Caroline Shaw
(3333$a)appendcol__D'a'[D=:'/tmp'conew'dwldb'[load'~/j.ijs'
setup: make a directory, install files fieldName.col using createjmf_jmf_
load this file into a session
all files in the directory with extension name.col are mapped as columns.
VALID is a reserved column name
require'~lambertdw/src/j/dwldb.ijs'
db=: DIRECTORY conew dwldb
Setup for 1st use.
put *.col files into a directory. Each of these will be a database field.
for example, date.col will map to the variable date.
These files are created with
require'jmf'
createjmf_jmf_'date.col';666
subsequent uses
create takes arguments dirpath ; readonly flag
The default flag is 0, ie. rw.
It maps all the columns, storing the variables into the base locale.
)
coclass'dwldb'
require'jmf'
Filter=:(#~`)(`:6)
Dyad=: [: :
Monad=: :[:
ls=: 1!:0@boxopen Monad
read=:1!:1@boxopen Monad
write=:(1!:2 boxopen)~ Dyad
append=:(1!:3 boxopen)~ Dyad
mkdir=:1!:5@boxopen Monad
open=:1!:21@boxopen Monad
close=:1!:22@boxopen Monad
pwd=:1!:43 Monad
cd=:1!:44@boxopen Monad
rm=:1!:55@boxopen Monad
host=:2!:0 Monad
spawn=:2!:1 Monad
getenv=:2!:5 Monad
getpid=:2!:6 Monad
exit=:2!:55 Monad
nameClass=:4!:0@boxopen
erase=: [: 4!:55 ;: ::]
format=: ;@:((a: , ":&.>)@:[ ,. '{}'&([ }.L:_1@:(E. <;._1 ]) ,)@:])
fail=: format assert 0:
rank=:#@$
showmap=: ,:^:(2>rank)@:showmap_jmf_
create_dwldb_=: 3 : 0
NB. default is open for rw
'dir ro'=. 2 {. (boxopen y) , < 0
'readonly flag must be 0 or 1'assert ro e. 0 1
DIR=: PATHJSEP_j_ ,~^:(~:{:) jpath dir
a=. ls }: DIR
'expected 1 match' assert 1 = # a
'not a directory' assert 'd' = {. (0 _1) {:: a
opendb ro
)
valid_columns=: 3 : 0
NB. y is the desirable nameclass, 0 for existing nouns, _1 for unused names
a=. ([: -. (1 e. '_'&e.)&>)Filter {."1 ls DIR , '*.col'
(_2 < ([: nameClass ('_' , '_' ,~ > COCREATOR) ,~ _4&}.)&>)Filter a
)
closedb=: ([: unmap_jmf_ _4&}.)&>@:valid_columns
destroy=: 3 :0
closedb''
coerase y
)
opendb=: 3 :0 :.closedb
NB. y is readonly flag
if. y = 0 do. NB. make backup
host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR
end.
files=. valid_columns _1
([: map_jmf_ ('' ; y) ,~ _4&}. ; DIR&,)&> ([([:echo 'opening '&;)) files
)
appendcol=: 4 :0 ::resize
name=. fullname_jmf_ y
if. # name~ do. (name)=: name~ , x else. (name)=: x end.
EMPTY
)
resize=: 4 :0
name=. fullname_jmf_ y
map=. showmap''
({. map) =. map&(((i.~ {."1)~ <) { [) ::(fail&'{} not mapped') name
tempname=. DIR,'growth_here'
createjmf_jmf_ tempname ; {. fsize + msize
map_jmf_'growth_here';tempname
growth_here_name=: fullname_jmf_ 'growth_here'
(growth_here_name)=: name~
unmap_jmf_&> y ; growth_here_name
host 'mv {} {}' format~ (; ('bak' ,~ _3&}.)) fn
host 'mv {} {}' format~ tempname ; fn
map_jmf_ y ; fn
x appendcol y
)
Example using dwldb, uses wget to load the json data from the www.coindesk.com/price/, making some gross assumptions that they won't change their web page format.
coclass'cryptocurrency'
Note'use'
$ ~/downloads/installs/j903/jconsole.sh
if the ro flag is 0, update the database, then open read only.
if the ro flag is 1, just open the db read only.
roflag conew
c=:0 conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs'[13!:0[1
$@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME'
avg=:(+/%#)
avg HIGH
+/CIRCULATINGSUPPLY
NAME{~(i.>./)PERCENT
10{.(\:PERCENT){NAME
10{.(\:PERCENT){PERCENT
)
NB. c=:''conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs'
require'files'
require'~addons/convert/json/json.ijs'
load'/home/lambertdw/src/j/dwldb.ijs'
DBDIR=: '/home/lambertdw/src/j/cryptocurrency/'
Dyad=: [: :
Monad=: :[:
ls=: 1!:0@boxopen Monad
read=:1!:1@boxopen Monad
write=:(1!:2 boxopen)~ Dyad
append=:(1!:3 boxopen)~ Dyad
mkdir=:1!:5@boxopen Monad
open=:1!:21@boxopen Monad
close=:1!:22@boxopen Monad
pwd=:1!:43@boxopen Monad
cd=:1!:44@boxopen Monad
rm=:1!:55@boxopen Monad
host=:2!:0 Monad
spawn=:2!:1 Monad
getenv=:2!:5 Monad
getpid=:2!:6 Monad
exit=:2!:55 Monad
nameClass=:4!:0@boxopen
erase=: [: 4!:55 ;: ::]
create=: 3 :0
NB. y is read only flag
if. 0 ~: y do.
db=: (DBDIR ; 1) conew'dwldb'
else.
if. fexist DBDIR,'ISO.col' do.
db=: DBDIR conew'dwldb'
else.
([: createjmf_jmf_ 1024 ;~ '.col' ,~ DBDIR&,)&> ;: 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS'
db=: DBDIR conew'dwldb'
ISO =: 0 5$''
NAME =: 0 21$''
OPEN =: 0 1$0.0
HIGH =: 0 1$0.0
LOW =: 0 1$0.0
CLOSE =: 0 1$0.0
PERCENT =: 0 1$0.0
VALUE =: 0 1$0.0
MARKETCAP =: 0 1$0.0
CIRCULATINGSUPPLY =: 0 1$0.0
TIMEPERIOD =: 0 3$''
TO =: 0 1$00
FROM =: 0 1$00
TOTALEXCHANGEVOLUME=: 0 1$00
RETURNS24 =: 0 1$0.0
TS =: 0 1$00
end.
acquire db
closedb__db''
erase 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS'
opendb__db 1 NB. readonly
end.
)
NB. location of the temp file for wget
DIR=: '/tmp'
FILENAME=: 'bitcoin'
Note '00README'
wget https://www.coindesk.com/price information source
https://www.CryptoDataDownload.com historical data in .csv, many coins
plan
use a home-brew non-sql database
create a directory for the database
auto-extending
data fields and types
dates stored as int, see timestamp verb
data comes from json section of the web page
)
rm=: 1!:55@:boxopen
pwd=: 1!:43
cd=: 1!:44
host_command =: 2!:0
recent_error=:13!:12
ufail=: 'verb u failed'"_
MapFile=: adverb define
NB. u is a verb to act on the file named as y
NB. MapFile wraps the mapping
JCHAR map_jmf_'A' ; y
result=. u (fullname_jmf_'A')~
unmap_jmf_'A'
result
)
UnderDir=: conjunction define
savedir=. pwd''
cd n
result=. u ::ufail y
cd savedir
result
)
UnderTmpFile=: adverb define
rm ::empty y
result=. u ::ufail y
rm ::empty y
result
)
get_data_from_internet=: 3 :0
host_command'wget https://www.coindesk.com/price/',y
([: , (13&}.;.1~ '"currencies":[{'&E.) MapFile) y
)
yyyymmddhhmmss=: 0 ". 1 1 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 &#
yyyyxmmxddxhhxmmxss=: _ 100 100 100 100 100&#.inv
timestamp=: yyyymmddhhmmss :.yyyyxmmxddxhhxmmxss
assert 2021 4 7 23 4 47 -: ]&.timestamp'2021-04-07T23:04:47.851+00:00'
acquire=: 3 :0
a=: get_data_from_internet UnderTmpFile UnderDir DIR FILENAME
b=: dec_json_json_ ,@((];._2~ '},"dataLayer":'&E.)) a
NB. make sure all fields are available
$@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME'
NB.literal 5 iso (<1;0)&{::S:_1
'ISO'appendcol__db~(<1;0)&{::S:_1 b
NB.literal 21 name (<1;1)&{::S:_1
'NAME'appendcol__db~(<1;1)&{::S:_1 b
NB.floating open ([: |: [: > 1 {"2 (<1;6)&{::S:_1)
NB.floating high
NB.floating low
NB.floating close
(([: |: [: > 1 {"2 (<1;6)&{::S:_1) b)(appendcol__db >)"1 0;:'OPEN HIGH LOW CLOSE'
NB.floating percent ([: |: [: > 1 {"2 (<1;7)&{::S:_1)
NB.floating value
(([: |: [: > 1 {"2 (<1;7)&{::S:_1) b)(appendcol__db >)"1 0;:'PERCENT VALUE'
NB.floating marketCap (<1;8)&{::S:_1
'MARKETCAP'appendcol__db~(<1;8)&{::S:_1 b
NB.floating circulatingSupply
'CIRCULATINGSUPPLY'appendcol__db~(<1;9)&{::S:_1 b
NB.literal 3 timePeriod ([: |: [: ,/ _ _1 _ {. (<1;_6)&{::S:_1)
NB.floating 29 from timestamp
NB.floating 29 to timestamp
t=. ([: |: [: ,/ _ _1 _ {. (<1;_6)&{::S:_1) b
'TIMEPERIOD'appendcol__db~ > 0 { t
'FROM'appendcol__db~ timestamp S:0 ] 1 { t
'TO'appendcol__db~ timestamp S:0 ] 2 { t
NB.integer totalExchangeVolume ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1)
NB.floating returns24
t=. ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1) b
'TOTALEXCHANGEVOLUME'appendcol__db~ <. {. t
'RETURNS24'appendcol__db~ {: t
NB.integer ts (<1;_2)&{::S:_1
'TS'appendcol__db~ (<1;_2)&{::S:_1 b
)