The following are good arguments for what should be taken care of by the public sector instead of the private sector. And they are from a capitalist economist who has militanted his whole life to spread liberal market mechanisms worldwide, as an efficient way to fight against poverty (Jeffrey Sachs, "The end of poverty, economic possibilities for our time", chapter "Making the investments needed to end poverty"). It is quite long so I will split it into 5 parts, following the 5 points argumentation of the author.
"Why should government finance schools, clinics, and roads, rather than leave those to the private sector ? There are five kinds of reasons, all compelling in the proper context. First, there are many kinds of infrastructure, especially networks like power grids, roads, and other transport facilities - airports and seaports - which are characterized by increasing returns to scale. If left to private markets, these sectors would tend to be monopolized, so they are called natural monopolies. If such capital investments are left to the private sector, the privately owned monopolies would overcharge for their use, and the result would be too little utilization of this kind of capital. Potential users would be rationed out of the market. It is more efficient, therefore, for a public monopoly to provide network infrastructure and set an efficient price below the one that would be set by a private monopolist."