A new paper was accepted for journal publication:-)
Song Gao, Linna Li, Wenwen Li, Krzysztof Janowicz, Yue Zhang. (2014) Constructing Gazetteers from Volunteered Big Geo-Data based on Hadoop . Computers, Environment and Urban Systems ,DOI: 10.1016/j.compenvurbsys.2014.02.004, (in press).
Traditional gazetteers are built and maintained by authoritative mapping agencies. In the age of Big Data, it is possible to construct gazetteers in a data-driven approach by mining rich volunteered geographic information (VGI) from the Web. However, the mining, harvesting, and geoprocessing of voluminous Big Geo-Data are very computationally intensive. To address this issue, in this research, we build a scalable distributed platform and a high-performance geoprocessing workflow based on the Hadoop ecosystem to harvest crowd-sourced gazetteer entries. Using experiments based on geotagged datasets in Flickr, we find that the MapReduce-based workflow running on the spatially enabled Hadoop cluster can reduce the processing time compared with traditional desktop-based operations by an order of magnitude. We demonstrate how to use such a novel spatial-computing infrastructure to facilitate gazetteer research. In addition, we introduce a provenance-based trust model for quality assurance. This work offers new insights on enriching future gazetteers with the use of Hadoop clusters, and makes contributions in connecting GIS to the cloud computing environment for the next frontier of Big Geo-Data Analytics.
We integrated the timely released Esri Geometry APIs and GIS Tools For Hadoop to spatially enable the Hadoop cluster for scalable processing of geotagged data from VGI sites and automatically link the results to an ArcGIS Desktop for geovisualization.