Yes it is undeniable the deamnd will not go away. But demand is not sufficient to convince say a company to build lamp posts since as mentioned, they won't have a viable way to charge for the lamp posts usage. It is simply not profitable, hence the State has to do it.
Remember, profit is a measure of efficiency, even if there are free riders.
But the private sector will not delve into such ventures simply because it is impossible to work out a system to charge consumers of the goods and services.
This is not necessarily true. The government say's it's impossible, so they take our money at gunpoint and pay for these goods and services. Without the gun to our heads, someone will find a way to provide these services efficiently despite free riders. Of course, they may not always be successful, because some goods and services are not as valuable as alternatives.
For example, let's look at the lamp post. Let's say one person was willing to pay for lamp post. However, no one is willing to help him pay for the lamp post and he doesn't want to pay so much for a lamp post by himself. Now, the man can try to find ways to get others to help pay, which is great. However, if he fails, the electricity from the lamp will be cut off.
Now, either people will start for the lamp to go back on, or they won't. If they don't, then alternative light sources pop up. Instead of people wandering by street light, people are now wondering around using flash lights.
The free rider solution of government purchase with money they have stolen does not take into consideration alternatives, nor does it realize that some goods and services are overall unpopular and a waste of money.
Let's look at trains as public transportation. The government can pay these trains, because they know people wouldn't want to pay for them anyway. The government takes more money from the people they are serving to pay for these trains and now we have government funded trains to take people where they need to go. However, there's a problem. The trains were created to hold 50 passengers a car, but they are averaging only 8 passengers a car. The majority of people paying for the trains aren't even using them! This is what we call government waste.
So how do we differentiate government waste from public "needs"? When a government pays for a "need", then how do we know if there's a more efficient way to provide that need, or if the need really need be provided in the first place?