V.I.P.P. Group

Reproducible Signal Processing

E-mail Print PDF

Current research status:

  • Many image processing projects lack meaningful experimental validation
    • Theoretical papers are validated by a single experiment carried out on a single image
    • Results need meaningful comparison with competing schemes
    • Results are justified with only one similar algorithm performing worse
    • Vaguely described experimental conditions --> impossible replicability
  • Readers must simply trust the authors!

A new paradigm:

  • Strong experimental analysis and setup --> high reproducibility
    • A thorny question: algorithm and experimental setup should avoid ambiguity
  • What is the problem?
    • Lack of time
    • Lack of research groups devoted to experimental signal processing or implementing someone else’s algorithms
    • Intellectual property issues

The RSP framework:

  • A straightforward approach… share the software and the data set!
  • Primary issues:
    • Software usability (which format should we use?) --> standardized solutions
    • Software readability (can other people understand what are doing?) --> highly commented code
    • Licensing problems (open source or binary format?) --> encourage knowledge spreading
  • Algorithms and experiments have to be described --> we have to find the right level of detail:
    • Description (a block diagram or a pseudo-code)
    • Parameters (a summary table or a detailed list)
    • Dataset (all the source data needed to obtain the same results of the original paper)

References:

Links: