Still studying navigation and am trying to consolidate my knowledge of the navigational triangle. Current interest in particular is how to establish the distance of the unknown side of the PZX triangle. From the assumed position of the ship (Z) to the geographical location of a star (X).

Lets throw in a couple of positions to work with:

54 degrees 13'.45N
003 degrees 14'.62E

47 degrees 17'.95N
019 degrees 37'.82W

I am familiar with the following formula as a result of help received on Jiskha, thanks Damon.

cosp = cosx x cosz + sinx x sinz x cosP

and a minute or so on the calculator will yield an answer.

I have now become curious about another formula I have come across to calculate the same information and need a lot of help AGAIN! applying this new formula.

New formula as follows:

Distance=60.ACOS(sin(latX)x sin(latZ)+cos(latX)x cos(latZ)x cos(longX - longZ))

Too many brackets for my liking. Obviously if both formulas are for the same purpose they should yield the same answer working with positions given above.

Please can someone have a go at the calculations using both formulas to see if it is possible to achieve the same answers.

I would also be interested to learn of any other formulas suitable for calculating distances between 2 positions on earth.

Thanks a million


  1. 👍
  2. 👎
  3. 👁
  1. The first one is the formula I used yesterday and agreed with your answer. I also checked it against the online navigation calculators.

    However I remember you did something like take 90 - the latitude before you ran your formula.

    If you did that, then since cos (90-x) = sin (x) and sin(90-x) = cos (x)
    the two formulas would be the same.

    The 60 is to get nautical miles instead of degrees.

    For the data you gave above the online software gives 956.7 nm

    I will calculate roughly using your data and the second formula since I did not use that yesterday. I am dividing minutes by 60 to get fractional degrees for my calculator which does not do base 60 arithmetic.:
    sin Lat X = sin 47.299 = .7349
    sin Lat Z = sin 54.224 = .8113

    cos Lat X = .6782
    cos Lat Z = .5846

    cos (long X -long Z) = cos (19.630+3.244) = cos(22.874) = .9214
    note: The Plus sign here is because the Z longitude is East - - =+
    cos^-1[ .7349 * .8113 + .6782 * .5846 * .9214 ]

    cos^-1 [ .5962 + .3653 ]

    cos^-1 [ .9615 ]

    15.95 degrees

    times 60 = 957 nm

    So the second equation agrees with the online software navigator

    1. 👍
    2. 👎
  2. Go to wikipedia on the subject of great circle navigation. They have several of the equations, the one you just gave being on the top. They give examples of where one might be accurate and another inaccurate.

    1. 👍
    2. 👎
  3. Here is an excerpt from the online aviation navigation primer. If it prints here, it also shows your second formula as first choice and the haversine formula second:

    Some great circle formulae:
    Distance between points

    The great circle distance d between two points with coordinates {lat1,lon1} and {lat2,lon2} is given by:


    A mathematically equivalent formula, which is less subject to rounding error for short distances is:

    d=2*asin(sqrt((sin((lat1-lat2)/2))^2 +

    1. 👍
    2. 👎
  4. No one is listing the one I had from my old books and I can not get an accurate result from it on your second problem. I suggest using the ones I am seeing in the present day texts.
    The Wikipedia article on marine navigation and the aviation source agree on the
    cos^-1 [sin sin + cos cos cos] form, so I would stick to that.
    They also agree on the haversine form, but I do not think you really need that unless just for fun. You can always carry more places with your calculator if you want more accuracy for special cases.
    By the way, the online calculators allow you to use different earth models. We are assuming a perfect sphere as have navigators for centuries. However greater accuracy is possible using more accurate models for the geometry of earth, but no way you are going to figure them out with a pocket calculator.

    1. 👍
    2. 👎

Respond to this Question

First Name

Your Response

Similar Questions

  1. Computers

    1. Which of the following study tools would work well for studying Vocabulary? A. making an outline B. summarizing the textbook C. flash cards*** D. comparing notes to the test 2. Which of the following is helpful when studying

  2. Math-Algebra

    (Engineering) The current / varies as the electromotive force E and inversely as the resistance R. If in a system, a current of 20A flows through a resistance of 20 ohms with an electromotive force of 100V, find the currect that

  3. History

    What was the purpose of the sextant during the Age of Discovery? A. It was a navigational tool used to determine longitude and latitude by calculating the angle between the horizon and a celestial body such as the moon, sun, or a

  4. Statistics/probability

    Of 120 students, 60 are studying French, 50 are studying Latin and 20 are studying French and Latin. Find the probability that a student is: (a) studying French or Latin and (b) studying neither.

  1. History

    How did an astrolabe serve European explorers during the Age of Discovery? it was a navigation tool used to determine longitude an latitude by calculating the angle between the horizon and celestial body such as the moon, sun, or

  2. History

    How did an astrolabe serve European explorers during the Age of Discovery? A) It was a navigational tool used to determine longitude and latitude by calculating the angle between the horizon and a celestial body such as the moon,

  3. Statistics

    A car rental agency currently has 41 cars available, 28 of which have a GPS navigation system. Two cars are selected at random from these 41 cars. Find the probability that both of these cars have GPS navigation systems

  4. ART Check

    How would a visual effects designer working on an animated movie with human characters best use knowledge of biology? Knowledge of biology can help the designer to create accurate human features.**** Knowledge of biology can help

  1. English

    A team of explorers ___________ the ocean floor. A.were studying B.are studying C.have been studying D.has been studying I think it is D...?

  2. Finance

    which of the following is true with regards to rising interest rates? A. Use Long-term loans to take advantage of current low rates. B. The term of the loan option is not impacted by rising interest rates. c. Use Short-term to

  3. History

    How did a chronometer help European explorers during the Age of Discovery? It was a navigational tool used to calculate a unit of speed equal to one nautical mile per hour, helping to gauge a ship’s speed. It was navigational

  4. Advanced Accounting

    GAAP rules are clear about when a company needs to consolidate or not, but companies tend to find loopholes to circumvent this rule. GAAP clearly indicates that consolidated financial statements are “usually necessary for a fair

You can view more similar questions or ask a new question.