No Manhattan isnt mostly granite, at least on the upper levels. Some of it was actually "made" by adding fill to the shore line over several different time periods.
http://bklyn-genealogy-info.com/Map/NY.1729.html
But also, you have Brooklyn and Queens which are part of Long Island whohc has always had a very strong agriculture base.