Inexact numbers

From J Wiki
Jump to navigation Jump to search


<=   =>

The way numbers are stored in a computer limits the maximum number of digits in the number. This maximum depends on the hardware, but typically a pps argument of 20 guarantees that all the digits available for a value will be displayed.

   pps 6
   %3
0.333333
   pps 12
   %3
0.333333333333
   pps 20
   %3
0.33333333333333331483


At 20 digits of precision the result of %3 displays all the detail it has on the number in 17 digits. The result of %3 is not the exact mathematical result, but is the closest number to that exact result that can be stored in the computer. This difference between what you would expect from exact math and the limitations on how numbers are stored in computers can be confusing.

The fact the %3 isn't stored exactly in the computer may not surprise you too much if you realize that an exact representation in decimal digits would take an infinite numbers of 3's.
There is an additional source of confusion due to the fact that computers store numbers internally in a binary format where each digit is a 0 or 1, rather than the decimal format you are familiar with where the digits range in value from 0 to 9. A consequence of this is that even very simple decimal numbers, exactly expressed with a few digits, when converted to the computer's binary format, are stored as an inexact value.

   pps 20
   0.5 0.25 0.1
0.5 0.25 0.10000000000000000555


The 0.5 and 0.25 are stored exactly, but the 0.1 is stored inexactly, and when displayed with maximum precision shows as 0.10000000000000000555 .

These are facts of life with the way computers store floating point numbers and apply to all computer languages, not just J. Usually you can ignore these details, but they can sometimes cause problems or confusion if you don't have an idea about what is going on.

<=   =>

Primer Index               Hover to reveal titles   -   Click to access   -   Current page is highlighted
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
45 46 47 48
50 51 52 53 54 55 56 57
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
97 98 99 100 101 102 103 104 105 106