In Dune, was the Golden Path really necessary?

I am trying to understand the reason why Leto III merged with the sandtrout in Children of Dune and why he was so committed to the Golden Path.

I just finished the first three books and I fear that I have missed the point of why prescience is so bad. According to the wikipedia entry, Leto wanted to teach humanity a lesson to avoid stagnation, so much so that he punishes them for 3,500 years by being a brutal emperor.

What isn't clear to me is how this actually saves humanity? Given the horrors of Muad'Dib's holy war and Leto's rule you'd think that having a few prescient noblepeople running around wouldn't be so bad. And why does he need to merge with a sandtrout and then let the Sandworms go extinct in order to accomplish this?

I checked out the Dune and Children of Dune miniseries in the hopes that the "made for TV" aspect would simplify the explanation of Leto's commitment but it doesn't do a good job explaining that either, it just seems to be a rehash of Leto's speeches on why the path is necessary.

Answer

The golden path was necessary due to the events that are seen through prescience and not fully realized until after Chapterhouse: Dune.

Without the Golden Path, humanity stagnates: it grows ever more dependent on melange and Arrakis, on the Spacing Guild, the Bene Gesserit, and the other powers-that-be. Realize that, through the control of melange and the Kwisatz Haderach breeding program, the main powers of the Dune Universe literally control humanity's continued evolution.

Paul Muad'Dib and Leto II both foresee the final destruction of mankind by an unknown power due to its complacency. So, Leto attempts to instill such a hatred of the centralized power structure that's at the heart of humanity's stagnation so humans would do everything in their power to "scatter": to go to the farthest reaches of the known Universe to avoid being tied to one central authority. This causes the established powers to lose their hold on humanity's progress and future.

The Scattering has one major additional effect: the development of humanity's ability to avoid prescience, which allows a section of humanity to elude the unforeseen power in the universe that will ultimately destroy them. This side effect plays heavily into the unwritten final chapter of the series, later adapted into Hunters of Dune and Sandworms of Dune.

In it, the unknown power uses a form of prescience to detect every possible move humans can make and provide the appropriate counter. That is, if I know what you're going to do before you do it, you can never defeat me. Developing the ability to avoid prescience allows humanity to have a fighting chance against the unknown power.