Evaluation Campaigns


One important activity of PROMISE is the organization of regular evaluation activities for multilingual and multimedia information systems at an international level and on an annual basis.
CLEF is a renowned evaluation framework, almost a brand, which over the past decade has been running with the support of major players in Europe, many of which are part of the PROMISE consortium. PROMISE aims at providing a unified evaluation environment together with the data gathered in it and the community using it. From the PROMISE perspective, CLEF is both a means to attract people in its evaluation tasks and feeding the gathered knowledge-base and a powerful exploitation channel for transferring and fostering the adoption of the proposed approaches and techniques. Therefore, PROMISE will embed CLEF as one of its activities. Moreover, as of 2010, CLEF is an independent organization, consisting of an annual conference plus a set of focused benchmarking activities and adopting an open and call-based nature to tasks definition, which puts it in line with the PROMISE approach based on use cases.
PROMISE will advance the traditional way of conducting evaluation campaigns and it will exploit the complementary expertise of the partners to set up:
- Evaluation tasks based on use cases: To move from abstract benchmarking to more user-sensitive evaluation schemes, PROMISE will in its requirement analysis phase formulate a set of use cases based on scenarios of use for multimedia and multilingual information access;
- New evaluation methods and metrics for more realistic evaluation according to use cases: PROMISE will advance the current evaluation practices by proposing and testing new evaluation methods and metrics better suited for studying and assessing the interaction among multimedia content, users, languages, and tasks envisioned above;
- Open evaluation infrastructure: PROMISE will develop and provide an open evaluation infrastructure for carrying out the experimentation: it will support all the different steps involved in an evaluation activity;