The other day I agreed with noted urban thinker EconomistHulk that we should build more homes by densifying existing urban areas where there is high demand, and also that this densification is frequently blocked by existing property owners for reasons of self-interest.
But Matt Yglesias (who has written an excellent short book on the housing problem) disagrees, arguing that housing scarcity is bad for homeowners, at least financially. The gist of his argument is that if you own a piece of land, then the ability to build more homes on that land is going to make you richer, even if the effect of supply on prices means the price per home is lower. Your neighbours might lose out because the increase in market supply makes their property cheaper, but that's their lookout.
As Matt notes, this begs the question of why we have so much NIMBYism in practice. He suggest it's partly because many people just don't understand how these things work, and partly because property owners attach a large use value to their property which can outweigh the exchange value, i.e. they want to live in the neighbourhood they chose rather than redevelop their land at higher density.
These are good points, but I think there's more to it. For one thing, density restrictions may serve several purposes, and in countries with highly decentralised political systems like the US where many services are funded by local property taxes one purpose is to minimise the level of fiscal 'free riding'. A rule stipulating maximum housing density effectively creates a floor price for housing and a minimum property tax contribution, ensuring that nobody can take advantage of local services without paying roughly the same entry price as everyone else. 'Fiscal zoning' of this kind is basically about keeping the poor out of your neighbourhood, but when key services like schools are locally funded there's a solid financial logic behind it which makes it very popular.
But even leaving these aspects of density restrictions aside, I don't think the home owner's decision is quite as straightforward as Matt suggests. He's right that land is a speculative asset, but as he says it's an unusual one in that home owners don't just own land but live on it too. And, very importantly, each property's value (in use or exchange) depends to a great extent on what does or doesn't happen on neighbouring properties.
The most important fact about urban development is that it creates various spill-over effects, and if you're a homeowner who doesn't plan on moving any time soon you're probably going to want to minimise the possibility of your neighbours redeveloping their property in a way that reduces the value of yours or makes it less enjoyable to live on through effects such as loss of light, traffic congestion and so on*. Rules against densification minimise the risk of this happening, and by buying into a neighbourhood with these rules home owners voluntarily accept a constraint on their own speculative activity in exchange for the knowledge that their neighbours are constrained too. Sure, when they're about to sell up they might be sorry that they can't just build a tower block on their plot of land, but the rest of the time they're probably grateful that nobody can.
To put it another way, Matt is correct that a property owner can increase that property's value by building as much housing on it as the market can bear, even if this extra density damages the value of neighbouring property. But the problem is that your neighbours can do exactly the same thing to you. And if everyone goes hell for leather in redeveloping their own land then you can end up with lower average land values and a less pleasant area than you would if everyone voluntarily agreed to constrain their redeveloping impulses**.
However, the average value of each housing unit would also be considerably lower, so from the point of view of a social planner interested in making housing more affordable this may still be a good deal, especially if you don't think the environmental or amenity costs of density outweigh the benefits. And in many cases there will be an intermediate range where increasing average densities does increase land values. But given the risks involved, the logic of the argument above is that if local planning / zoning rules are basically set by home owners acting in their own self interest then we shouldn't be surprised to see widespread NIMBYism.
* Matt mentions such 'non-pecuniary' factors but doesn't seem to accord them much importance.
** There's a proof of the potential negative impact on land values in DiPasquale and Wheaton's urban economics textbook, which is unfortunately out of print.