Advanced search
Start date
Betweenand
(Reference retrieved automatically from Web of Science through information on FAPESP grant and its corresponding number as mentioned in the publication by the authors.)

Probing the Statistical Properties of Unknown Texts: Application to the Voynich Manuscript

Full text
Author(s):
Amancio, Diego R. [1] ; Altmann, Eduardo G. [2] ; Rybski, Diego [3] ; Oliveira, Jr., Osvaldo N. [1] ; Costa, Luciano da F. [1]
Total Authors: 5
Affiliation:
[1] Univ Sao Paulo, Inst Phys Sao Carlos, Sao Paulo - Brazil
[2] Max Planck Inst Phys Complex Syst MPIPKS, Dresden - Germany
[3] Potsdam Inst Climate Impact Res PIK, Potsdam - Germany
Total Affiliations: 3
Document type: Journal article
Source: PLoS One; v. 8, n. 7 JUL 2 2013.
Web of Science Citations: 26
Abstract

While the use of statistical physics methods to analyze large corpora has been useful to unveil many patterns in texts, no comprehensive investigation has been performed on the interdependence between syntactic and semantic factors. In this study we propose a framework for determining whether a text (e.g., written in an unknown alphabet) is compatible with a natural language and to which language it could belong. The approach is based on three types of statistical measurements, i.e. obtained from first-order statistics of word properties in a text, from the topology of complex networks representing texts, and from intermittency concepts where text is treated as a time series. Comparative experiments were performed with the New Testament in 15 different languages and with distinct books in English and Portuguese in order to quantify the dependency of the different measurements on the language and on the story being told in the book. The metrics found to be informative in distinguishing real texts from their shuffled versions include assortativity, degree and selectivity of words. As an illustration, we analyze an undeciphered medieval manuscript known as the Voynich Manuscript. We show that it is mostly compatible with natural languages and incompatible with random texts. We also obtain candidates for keywords of the Voynich Manuscript which could be helpful in the effort of deciphering it. Because we were able to identify statistical measurements that are more dependent on the syntax than on the semantics, the framework may also serve for text analysis in language-dependent applications. (AU)

FAPESP's process: 11/50761-2 - Models and methods of e-Science for life and agricultural sciences
Grantee:Roberto Marcondes Cesar Junior
Support type: Research Projects - Thematic Grants
FAPESP's process: 10/00927-9 - Using complex networks to classify texts
Grantee:Diego Raphael Amancio
Support type: Scholarships in Brazil - Doctorate (Direct)