So there are some
big challenges right out of the gate when testing statistical software, and so
far I've looked at rounding errors. The
bigger question, given that computers have this innate limitation, is how accurate
can and should we be?
On one hand,
statistics gives probabilities, not truth.
So having a routine give a 86.4532%
chance compared to second routine giving an 86.4533% seems like we are splitting hairs. But we have some trust in
our computers and we need to get the answer as "right" as possible.
My favorite stat on
this is Nasa using 15
digits of accuracy for pi. That is
as much as they need to track everything they have ever launched, so that is
one benchmark to consider. For the
software on which I work, I doubt many folks are tracking interstellar traffic,
though. It's still a good data point.
Financial markets
are something we handle. This gets
tricky really quickly, though. In the
USA, we track (pennies) to two decimal places and have rules about rounding the
third decimal place. Other countries use
more decimal places, so this doesn't help much.
(Side note: Ohio has a rounding methodology for sales tax that strongly
skews to higher taxes: http://www.tax.ohio.gov/sales_and_use/information_releases/st200505.aspx)
There is also a
performance (of the computer) vs. precision factor that we need to
consider. While we could get much more
precision, that comes at the cost of run time.
One way of having more precision would be to allocate more memory for
each number we are using. Also, there
are some libraries that the National Institute of Standards and Technology
makes available that really help, and companies like Intel also provide these
tools. They generally run more slowly
than code that doesn't push for that level of precision. More on this later.
Looking ahead, once
we settle on a standard, then the test team has to ensure we meet that goal
100% of the time. I'll finally get
around to covering that aspect of the test team's goals as we go along.
Questions, comments,
concerns and criticisms always welcome,
John
No comments:
Post a Comment