tolkiens legendarium - How well-known is the theology of Middle-earth, in Middle-earth?

One thing that is notably absent in Middle-earth is any mention of temples, churches, priests or religious rites. There are passing mentions of Men who would "worship" Sauron, but generally the world of Middle-earth seems much less religious than medieval Europe, or even classical antiquity. One never hears about "the cult of Aule" or "the cult of Yavanna," although historically there were many "cults of Athena" or "cults of Poseidon." Aragorn doesn't (to my memory) make an offering to Tulkas before battle, nor does he pray to Eru Ilúvatar for victory, and all the holidays and solemnities seem to commemorate historical figures, not religious ones.

This is especially strange in a world where there are still living witnesses, like Galadriel, to the actions of the Valar and other "divine" beings, and where there are still Maiar wandering around talking to people and putting on flashy firework shows.

Now, a lot of people talk about how Tolkien didn't want the Valar to be considered "gods," but at the same time he did use phrases like "the gods of old" numerous times. Clearly, these beings would have qualified for worship in most cultures. And even without the Valar being considered a true pantheon, there is still Eru, a closer analogue to the Christian God of Tolkien's beliefs, and we know the denizens of Middle-earth knew of him (Him?) because of the line "Eru, who in Arda is called Ilúvatar." And yet, Ilúvatar seems not to be terribly important to anyone on a daily basis.

So my question is this: how familiar with the "theology" of Middle-earth were the citizens of Middle-earth? Would the average merchant in Rohan know about the Valar and the Maiar and Iluvatar and the Music of the Ainur? The average noble in Minas Tirith? The younger Elves, like Legolas? The Dwarf Lords?

Were the theological truths about the formation of the world, the Valar and the Maiar just so well known that they weren't worth commenting on, or had they been forgotten by most? And if they had been forgotten, why wasn't the knowledge (whether through deliberate teachings or casual conversation) re-invigorated by witnesses like Gandalf or Galadriel?

Answer

Tolkien deliberately omitted references to "religion" in his works; see Letter 142:

I have not put in, or have cut out, practically all references to anything like 'religion', to cults or practices, in the imaginary world.

Letter 153, which he wrote to a reader who one must feel took many things entirely too seriously, elaborates further, and it's worth quoting the entirety of a footnote to it, although I'll break it up to avoid a "wall of text".

First a general overview:

There are thus no temples or 'churches' or fanes in this 'world' among 'good' peoples. They had little or no 'religion' in the sense of worship. For help they may call on a Vala (as Elbereth), as a Catholic might on a Saint, though no doubt knowing in theory as well as he that the power of the Vala was limited and derivative. But this is a 'primitive age': and these folk may be said to view the Valar as children view their parents or immediate adult superiors, and though they know they are subjects of the King he does not live in their country nor have there any dwelling.

On Hobbits:

I do not think Hobbits practised any form of worship or prayer (unless through exceptional contact with Elves).

Numenor, both before and after Sauron's arrival, including other Men who didn't go there:

The Númenóreans (and others of that branch of Humanity, that fought against Morgoth, even if they elected to remain in Middle-earth and did not go to Númenor: such as the Rohirrim) were pure monotheists. But there was no temple in Númenor (until Sauron introduced the cult of Morgoth). The top of the Mountain, the Meneltarma or Pillar of Heaven, was dedicated to Eru, the One, and there at any time privately, and at certain times publicly, God was invoked, praised, and adored: an imitation of the Valar and the Mountain of Aman. But Numenor fell and was destroyed and the Mountain engulfed, and there was no substitute.

And the Dunedain in Middle-earth:

Among the exiles, remnants of the Faithful who had not adopted the false religion nor taken part in the rebellion, religion as divine worship (though perhaps not as philosophy and metaphysics) seems to have played a small part; though a glimpse of it is caught in Faramir's remark on 'grace at meat'.

Letter 156 covers High Elves:

The High Elves were exiles from the Blessed Realm of the Gods (after their own particular Elvish fall) and they had no 'religion' (or religious practices, rather) for those had been in the hands of the gods, praising and adoring Eru 'the One', Ilúvatar the Father of All on the Mt. of Aman.

There are many other mentions of religion in Middle-earth scattered throughout the Letters, but the general outcome and completely consistent observation is that Middle-earth has no religion as we would understand it today.