Reproducible Signal Processing

Visual Information Processing and Protection Group

Reproducible Signal Processing

M. Barni, F. Perez-Gonzalez,

Pushing Science into Signal Processing, IEEE Signal Processing Magazine 22(4):120 - 119, August 2005, DOI: 10.1109/MSP.2005.1458324

Many image processing projects lack meaningful experimental validation. Despite the promising results achieved so far, the application of AI-based methods for MMF is seriously hindered by many shortcomings:

Theoretical papers are validated by a single experiment on a single image
Results need meaningful comparison with competing schemes
Results are justified with only one similar algorithm performing worse
Vaguely described experimental conditions make replicability impossible
Readers must simply trust the authors!

Strong experimental analysis and setup allow to achieve high reproducibility, thus algorithms and experimental setup should avoid ambiguity. So, What is the problem?

Lack of time to implement someone else's algorithms
Lack of researchers to re-implement useful algorithms
Intellectual properties of the proposed algorithms

Reproducible Signal Processing is the solution: share the software and the data set!

Many image processing projects lack meaningful experimental validation. Despite the promising results achieved so far, the application of AI-based methods for MMF is seriously hindered by many shortcomings:

Software usability: which format? Use standardized solutions
Software readability: can other people understand what you are doing? Document your code thoroughly
Licensing problems? Spread your knowledge in an open source format when possible otherwise share your binaries
Algorithms and experiments? Describe them with the right level of detail
Algorithm description? Provide a block diagram or a pseudo-code
Algorithm parameters? Provide a summary table or a detailed list
Datasets? Provide all the source data needed to obtain the same results of the original paper