Laying the foundations for benchmarking open data automatically: a method for surveying data portals from the whole web

Author(s)

Correa, Andreiwid Sheffer ; Correa da Silva, Flavio Soares ; Chen, YC ; Salem, F ; Zuiderwijk, A

• Document type: Journal article
• Source: PROCEEDINGS OF THE 20TH ANNUAL INTERNATIONAL CONFERENCE ON DIGITAL GOVERNMENT RESEARCH (DGO2019): GOVERNANCE IN THE AGE OF ARTIFICIAL INTELLIGENCE; v. N/A, p. 10-pg., 2019-01-01.

Abstract

The widespread of open data portals in the global scenario makes automated methods for data gathering and assessment a landmark in benchmarking. Some practitioners have already introduced automated approaches focused on mainly quality assessment of selected lists of data portals, but little has been studied about how to ensure exercises happen on the widest range of initiatives across the world, even incipient ones. The purpose of this paper is to provide a method for surveying data portals from the whole web, aiming to produce a whitelist of URLs that point to healthy data portals on the internet. The method was tested on 3.3 billion web addresses from which we found 1,339 open data portals worldwide using the main software platforms in the market CKAN, Socrata, OpenDataSoft and ArcGIS Open Data. Findings showed the choice of the whole web approach increased the number of data portals found, besides offering a workaround for redundancy, discoverability and traceability issues of current sparse and manual-based repositories. This work contributes to development of a fully automated method towards building an independent, reliable and up to date repository as a single source of open data portals operated around the world as well as provide insights about dataset estimation and geographic localization, from which benchmarking exercises may benefit to happen on a larger scale, at higher frequency and with lower costs. (AU)

• FAPESP's process: 15/24485-9 - Future internet for smart cities
• Grantee: Fabio Kon
• Support Opportunities: Research Projects - Thematic Grants
• FAPESP's process: 14/50937-1 - INCT 2014: on the Internet of the Future
• Grantee: Fabio Kon
• Support Opportunities: Research Projects - Thematic Grants