This idea is hilarious here in Iowa, where we wonder if some city folk have ever seen real farms. “The reality is that farming is an inherently space-intensive enterprise,” as even Manhattan native Matt Yglesias can recognize. 86 percent of Iowa is farmland (down from over 90 percent just a decade ago). That's a bit shy of 30 million acres. That's about 2000 Manhattans. The mind-blowing productivity growth in agriculture over the 20th century stands as one of the great achievements of human history. It involved immense strides in pest and weed control, farm machinery, bioengineering, and economies of scale. All this has made it possible to feed a rapidly increasing population with decreasing amounts of land and labor.
(Aside… No doubt Happy Planet Index-type people in 1909 were pointing out the physical impossibility of our one finite planet supporting 7 billion people.)
It's nice to have a garden. But didn't seem nice when I was a kid, when my family had a huge garden plot (a quarter acre, maybe?) on the property of a farmer who went to our church. That much garden in a city would seem like some pretty serious urban farming. Set in the vast scale of cultivated central Iowa, it seemed like what it was: almost nothing. Later, as a teenager, I detasseled corn and walked beans. If a field is big enough, and the curve is right, from the middle you can't see anything else.
If not for the massive subsidies it receives, the percentage of land under cultivation in Iowa would decline even more rapidly than it has. But it would remain one of the best places in the world for growing stuff. An unsubsidized Iowa would grow a different mix of stuff, and would traffic in a different mix of animals. Greater heterogeneity would reduce some economies of scale, but the scale of the actual farming–the kind that keeps humanity fed–will probably remain inconceivable to many rooftop basil growers.
UPDATE: People we're justly complaining about the junk chart. It's the only one I could quickly find that had the time-scale I wanted. Here's a better one that goes back just 60 years.