the lord of the rings - Why didn't Gollum kill Frodo and Sam?

I'm only basing this question on the LotR movies, since I haven't read the books.

Gollum starts following Frodo and Sam, and then one night jumps out and tries to kill them and take the Ring. They subdue him, but only because they expected him and were thus able to defend themselves. From then on he kind of decides to help them get to Mordor to destroy the Ring.

During this period he seems to have split personalities: Gollum vs. Smeagol, with Gollum wanting to kill them and Smeagol seeming to be at least neutral towards them, from the conversations the movie shows between the two.

At some point though, Gollum seems to win over Smeagol and convince him to lure them to Shelob to kill them.

But why didn't he just kill them while they were sleeping? In the movie, he has dozens of opportunities where he could definitely cut one hobbit's throat, bash their head in with a big rock, push them off a ledge, etc.

Answer

To begin with, Gollum does not know Frodo and Sam are trying to destroy the Ring, only that they want to enter Mordor. In fact he reacts very angrily when he learns they want to destroy his Precious. In the film, Frodo tells him just outside Shelob's lair; in the books, Gollum doesn't work it out until he attacks them on the slopes of Mount Doom itself:

"Wicked masster!" it hissed. "Wicked masster cheats us; cheats Smeagol, gollum. He musstn't go that way. He musstn't hurt Precious. Give it to Smeagol, yess, give it to us! Give it to uss!" [Return of the King, Book 6, Chapter 3, "Mount Doom"]

To answer your main question, Frodo and Sam do not trust Gollum. In the books they take it in turns to sleep, with one remaining awake to watch:

"Look here!" Sam whispered to Frodo, not too softly; he did not really care whether Gollum heard him or not. "We've got to get some sleep; but not both together with that hungry villain nigh, promise or no promise." [The Two Towers, Book 4, Chapter 2, "The Passage of the Marshes"]

If Gollum attacked the hobbit on guard, he would wake the other; and Gollum is not strong enough to defeat both the hobbits at once, especially since they are armed and he is not. This is hinted at in the films as well, along with the fact that Frodo hardly sleeps at all.

Finally, it is suggested that possession of the Ring gives Frodo a power to command Smeagol/Gollum:

For a moment it appeared to Sam that his master had grown and Gollum had shrunk; a tall stern shadow, a mighty lord who hid his brightness in grey cloud, and at his feet a little whining dog. Yet the two were in some way akin and not alien; they could reach one another's minds. [The Two Towers, Book 4, Chapter 1, "The Taming of Smeagol"]

As Frodo says in the film, "The Ring is treacherous. It will hold you to your word." Of course, Gollum himself is treacherous and will take any opportunity to seize the Ring for himself, but he still finds it difficult to defy Frodo.