Why was this major Star Wars character killed off in The Force Awakens?

Towards the end of the new Star Wars film The Force Awakens,

Kylo Ren kills his father, Han Solo.

This event caused a lot of grief among fans who were devastated to see this character go, and the filmmakers surely anticipated such a reaction.

Why, out of universe, was the decision made to kill off this character?

Answer

To establish Kylo Ren firmly as a villain.

JJ Abrams himself said this is what it was all about:

Star Wars had the greatest villain in cinema history. So, how you bring a new villain into that world is a very tricky thing. We knew we needed to do something f—king bold. The only reason why Kylo Ren has any hope of being a worthy successor is because we lose one of the most beloved characters.

(Source: this interview with Abrams and his co-writers Lawrence Kasdan and Michael Arndt.)

Kasdan elaborated on this point, saying the story could be seen as Kylo Ren's coming-of-age saga: the coming of age of a villain.

People always say, ‘Why do you think this saga is so popular?’ I really do believe the underlying theme is recognizing your potential and understanding what you’re capable of. It’s a quest we’re all on our entire lives. It doesn’t end. To understand what you’ve inherited, and what you like about that and what you don’t like about that. Have you fulfilled yourself completely — or is it too late? What is dormant? That’s a very real and tangible thing for people every day.

And Abrams again:

Long before we had this title, the idea of The Force Awakens was that this would become the evolution of not just a hero, but a villain. And not a villain who was the finished, ready-made villain, but someone who was in process.

[Han's death] is this massive tradeoff. How can we possibly do that!? But… if we hadn’t done that, the movie wouldn’t have any guts at all. It felt very dangerous.”

To give Han Solo more pathos as a character.

This time quoting Arndt from the same interview:

I had thought Han’s story and Leia’s story was just about them coming back together. At the end of the movie they would have reconciled and gotten over their differences. And you would have said, ‘Okay, bad stuff happened, but at least they’re back together again'. J.J. rightly asked, ‘What is Han doing in this movie?’ If we’re not going to have something important and irreversible happen to him, then he kind of feels like luggage. He feels like this great, sexy piece of luggage you have in your movie. But he’s not really evolving. He’s not really pushing the story forward.

---

How did the various people involved in making the film take it?

Harrison Ford himself doesn't seem to have any problems. He's been campaigning to get Han Solo killed off since Return of the Jedi:

And in the above-referenced interview, Abrams confirmed, with a dismissive hand-wave:

Nah, he was fine.

In fact, Abrams himself was somewhat nervous about this bold move, but Kasdan's support steeled his resolve:

You [Kasdan] wrote some of the greatest lines that Han ever spoke, so there was a level of comfort in the danger. You were willing to go there, which made me feel like it wasn’t necessarily the worst idea.