Accuracy of converting semicircles to degrees

Discussion in 'General GPS Discussion' started by Bruce Stemplewski, Nov 27, 2004.

  1. Garmin's formula for converting degrees to semicircles to degrees is:

    d = s * (180.0D / 2^31)

    When s = 119304640

    d = 9.9999994039536

    But 10 was entered on the GPS. Does anyone know of a more accurate
    formula for converting semicircles to degrees?
     
    Bruce Stemplewski, Nov 27, 2004
    #1
  2. Bruce Stemplewski

    Fred Hiltz Guest

    The difference lies not in the formula, but in the representation of
    an angle by 31 bits (plus sign) in the computer. Here the difference
    is less than one millionth of one degree.
     
    Fred Hiltz, Nov 27, 2004
    #2
  3. Bruce Stemplewski

    Dale Andrews Guest

    d = s * (180.0D / 2^31)+0.000002
    When s = 119304640 and D=1 then d = 10.000001403953552525650000000000

    Does that look better?

    This reminds me of an operator that was continually writing up a
    system that was supposed to maintain 1500 pounds and the digital meter
    kept reading 1499. Tweak the meter to read 1501 and all the complaints
    stopped. How important is the difference other than it doesn't look
    right when there are a string of 9's rather than a string of 0's?

    You should never expect more accuracy from an answer to a formula than
    the accuracy of the input. Therefore your d should be no more than 9
    significant digits or 9.99999940. If we consider the 180.0 to really
    be 180 and an infinite number of zeros then the real max accuracy of
    the formula would be limited by 2^31 or again 9 digits. 9 digit
    accuracy is VERY good. It is the difference between 30 years and 30
    years plus 1 second. Do you really need more than that or is it just a
    string of 9's isn't as personally satisfying as a string of 0's??
     
    Dale Andrews, Nov 28, 2004
    #3
  4. Bruce Stemplewski

    Ted Edwards Guest

    Or almost TWOmm in the distance from LAX to JFK. How can we possibly
    condone such GROSS inacuracy in our hand-held GPSRs? ;-)

    Ted
     
    Ted Edwards, Nov 28, 2004
    #4

  5. I corrected it by taking a millimeter off each end of my handheld with a
    dremel tool.

    --Gene
     
    Gene S. Berkowitz, Nov 29, 2004
    #5
Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.