Matt Yglesias has an excellent post here about the lovely city of Santa Monica, which prides itself on its greenness yet has seen barely any population growth in recent decades despite sky-high house prices.
For Santa Monica you can subsitute any number of nice, exclusive places in the US and across Europe. Here in London we have an excellent example in the Royal Borough of Kensington and Chelsea, where the average house price is over a million pounds (pdf) but where the population and the number of households actually fell, albeit only slightly, between the 2001 and 2011 censuses compared to an increase of 8.3% across London as a whole (xls).
Demand for housing in K&C is clearly through the roof, so how come its population isn't growing? The reason is that planning policy, backed up with overwhelming political support from borough residents, makes it very difficult to build any new housing in the area. So very little does get built, so rising demand just pushes up prices.
K&C is a somewhat extreme example (70% of it is in a conservation area!) but the general pattern is widespread. More people want to move to urban areas to be closer to job opportunities or to avail of improvign urban environments, but the people who are already there don't want their neighbourhood to change so they campaign against new housing developments, and politicians dutifully vote those developments down. The supply shortfall pushes up prices, so only the richest can afford to move in and those less well-off either stay put or move to somewhere less nice and/or further out.
I think this is quite a bad situation, because it results in affordability problems, greater segregation by income and environmentally damaging urban sprawl as housing supply is diverted to places further away from jobs and services. And indeed, you hear a lot of concern about these issues in urban political discourse - but what you generally don't hear is any connection being made with all those anti-development crusades that contribute so much to the problems.
Of course there may be downsides to densification of existing urban neighbourhoods in response to rising demand. A lovely neighbourhood may not look quite as lovely when it's had some buildings replaced or some blocks of flats added. Personally I think these things can go either way and some areas would be much improved by densification, but I also think these basically aesthetic considerations are just not that important compared to the effect on people's lives of making more housing available in places where they want to live.
At its root, this is an issue of social justice and of personal ethics. People who care about social justice should be worried about a political process which gives all the power to people who already inhabit a neighbourhood and none to the people who would like to (see Matt Yglesias again). And people who think of themselves as ethical should ask themselves whether they can justify excluding others from enjoying the same neighbourhood they like so much, while adding to our already serious problems of affordability and car-dependent development.