[OPEN-ILS-DEV] SPAM: PATCH: storing numbers as strings in jsonObjects

[Scott McKellar]

These patches are the culmination of several postings on this subject.
The overall effect is to store numbers in jsonObjects as strings,
rather than as doubles, in order to avoid needless loss of precision
in translating back and forth between text and floating point 
representations.

I shall not repeat the details outlined in previous posts, but rather
focus on what's new:

1. A new extern function jsonNewNumberStringObject constructs a
JSON_NUMBER from a character string.  If the string is not numeric
according to JSON rules, the function returns NULL.

2. A new extern function jsonScrubNumber accepts a character string
and reformats it, if possible, into a numeric string that is valid
according to JSON rules.  For example, it transforms "  +00.42" 
into "0.42".  The transformed string is returned as a char* that 
the caller is responsible for freeing.

jsonScrubNumber performs this transformation by manipulating text, 
not by passing the value through a double.  Therefore it can handle
numbers that would be too long, too large, or too small for strtod()
and its kindred to handle.

It accepts leading white space and scientific notation, but not 
trailing white space, hex, or octal.

If the input string is not numeric, jsonScrubNumber returns NULL.

3. The doubleToString function now translates the incoming double 
to a character string with up to 30 decimal digits of precision.
That should be enough to minimize the impact on existing code,
depending of course on how faithfully snprintf() does the formatting.

4. In osrf_json.h: I changed the signature of the next-to-last
function pointer in a jsonParserHandler, so that it accepts a
character pointer instead of a double.  Likewise for the corresponding
declaration of _jsonHandleNumber in osrf_json_utils.h.

5. In osrf_json_parser.c: I construct a JSON_NUMBER from the input
character string without passing it through a double.  If the input
character string is not valid according to JSON rules, I try to use
the new jsonScrubNumber() to normalize the formatting so that JSON
can accept it.

----------

Arguably the parser should not trouble itself to accept numeric
strings that don't follow the JSON rules.  For now, however, I thought
it prudent to allow some leeway for existing applications.  Maybe 
someday we can be less forgiving, if we choose to.

The next step is to look for places that pass doubles to set the
value of a JSON_NUMBER.  Probably in many cases the double originally
came from a character string, and we can eliminate the round trip.

----------

I have unit-tested these patches and everything seems to work.
However I am not in a position to system-test with realistic data,
nor to assess the impact on existing applications.

Also it's hard to be sure that the new jsonScrubNumber function will
always detect non-numeric garbage, because it's hard to anticipate
all the possible kinds of garbage.  Some independent torture testing
might be worthwhile.

Scott McKellar
http://home.swbell.net/mck9/ct/

Developer's Certificate of Origin 1.1 By making a contribution to
this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license indicated
in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source license
and I have the right under that license to submit that work with
modifications, whether created in whole or in part by me, under the
same open source license (unless I am permitted to submit under a
different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person
who certified (a), (b) or (c) and I have not modified it; and

(d) In the case of each of (a), (b), or (c), I understand and agree
that this project and the contribution are public and that a record
of the contribution (including all personal information I submit
with it, including my sign-off) is maintained indefinitely and may
be redistributed consistent with this project or the open source
license indicated in the file.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osrf_json_utils_h_3.patch
Type: text/x-patch
Size: 730 bytes
Desc: 3600244231-osrf_json_utils_h_3.patch
Url : http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20071222/9ce5fa68/osrf_json_utils_h_3-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osrf_json_h_10.patch
Type: text/x-patch
Size: 2902 bytes
Desc: 3168948053-osrf_json_h_10.patch
Url : http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20071222/9ce5fa68/osrf_json_h_10-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osrf_json_object_c_7.patch
Type: text/x-patch
Size: 11053 bytes
Desc: 3745626656-osrf_json_object_c_7.patch
Url : http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20071222/9ce5fa68/osrf_json_object_c_7-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osrf_json_parser_c_4.patch
Type: text/x-patch
Size: 2354 bytes
Desc: 3733242396-osrf_json_parser_c_4.patch
Url : http://list.georgialibraries.org/pipermail/open-ils-dev/attachments/20071222/9ce5fa68/osrf_json_parser_c_4-0001.bin