A carregar...
Publicação
Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 1.63 MB | Adobe PDF |
Orientador(es)
Resumo(s)
One of the main objectives of the scientific enterprise
is the development of well-performing yet parsimonious
models for all natural phenomena and systems. In the
21st century, scientists usually represent their models, hypotheses,
and experimental observations using digital computers.
Measuring performance and parsimony of computer
models is therefore a key theoretical and practical challenge
for 21st century science. “Performance” here refers to a
model’s ability to reduce predictive uncertainty about an object
of interest. “Parsimony” (or complexity) comprises two
aspects: descriptive complexity – the size of the model itself
which can be measured by the disk space it occupies –
and computational complexity – the model’s effort to provide
output. Descriptive complexity is related to inference quality
and generality; computational complexity is often a practical
and economic concern for limited computing resources.
In this context, this paper has two distinct but related goals.
The first is to propose a practical method of measuring computational
complexity by utility software “Strace”, which
counts the total number of memory visits while running a
model on a computer. The second goal is to propose the
“bit by bit” method, which combines measuring computational
complexity by “Strace” and measuring model performance
by information loss relative to observations, both in
bit. For demonstration, we apply the “bit by bit” method to
watershed models representing a wide diversity of modelling
strategies (artificial neural network, auto-regressive, processbased,
and others). We demonstrate that computational complexity
as measured by “Strace” is sensitive to all aspects of
a model, such as the size of the model itself, the input data
it reads, its numerical scheme, and time stepping. We further
demonstrate that for each model, the bit counts for computational
complexity exceed those for performance by several
orders of magnitude and that the differences among the models
for both computational complexity and performance can
be explained by their setup and are in accordance with expectations.
We conclude that measuring computational complexity by
“Strace” is practical, and it is also general in the sense that
it can be applied to any model that can be run on a digital
computer. We further conclude that the “bit by bit” approach
is general in the sense that it measures two key aspects of a
model in the single unit of bit. We suggest that it can be enhanced
by additionally measuring a model’s descriptive complexity
– also in bit.
Descrição
Palavras-chave
Contexto Educativo
Citação
Azmi, E., Ehret, U., Weijs, S. V., Ruddell, B. L., and Perdigão, R. A. P.: Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance, Hydrol. Earth Syst. Sci., 25, 1103–1115, https://doi.org/10.5194/hess-25-1103-2021, 2021.
Editora
Copernicus Publications