D2.7.3 - Conclusions from experimental data and combinatorics analysis, M42
Contents
Original goal and task
Goal
We can define optimal combinations of the various methods for different application scenarios, and therefore we produce a combinatorics plan to test hybrid solutions in the remainder of the project.
Original task description
Task 2.7: Measurement, combinatorics and export
In this task we will perform:
- Quantitative comparative evaluation of selection methods relative to selection task parameters. The selection task parameters are initially defined in Task 2.1 and are subject to addition and alteration in Tasks 2.2 through 2.6. They include elements relating to cost/benefit, provenance and context, as discussed in the WP2 description (in previous deliverables).
- Derivation of an empirically supported methodology for selection choice.
- Investigation of combinatorics of different methods
- Export of optimal component combinations for integration, functional testing and case studies.
Guiding questions
Optimization & Tuning: Which decisions were made with respect to data selection and which parameters were varied (optimized?) to obtain good results?
Metrics: Which measures were used to assess quantity and quality of selection results?
Transfer: Can employed data selection methods (or plugins developed) be used for other LarKC tasks? (If so, which ones? If not, why not?)
Proposed structure
Chapter 1: Introduction (MPG)
Chapter 2: Conclusion drawn from LarKC (by each WP2 partner) - Overview over the data selection evaluation practices:
Each of the WP2 partners should provide a brief overview of their data selection procedure and evaluation practices (on 3-5 pages).
The following questions may helpful to organize your ideas into a coherent text.
Please view them as a suggestion and modify them as you see fit to describe your evaluation efforts:For which tasks / workflows / plugins have you selected data?
Data selection: How did you select which data?
- Which selection methods have been used so far?
- Which data sources and queries have been used?
Evaluation procedure: Which procedure did you use to evaluate the success of your data selection?
- Which metrics have you used?
- Did you vary any selection parameters? Which ones? How?
- Which standards or competing methods did you use to compare your results to? (If there are no competing models: Why not?)
What were the results of your evaluation?
- Which conclusions do you draw from the your results?
- What are the next steps?
Has this work resulted in the development of a selection plugin?
- If yes:
- Which one(s)?
- Can this plugin be used for other tasks? Which one(s)?
- If not:
- What is the intended use of your method?
- Which specific aspects of the task do not generalize to others?
- If yes:
Lessons learned: Does your method or result allow to draw conclusions for the evaluation of data selection (within LarKC and/or within semantic web data)?
Chapter 3: Focus on data quality – Improving the quality of RDF retrievals (MPG)
Efforts to improve retrieval results require a way of evaluating the quality of answers and a systematic collection and use of user feedback.
Two sections on:- Improving retrieved results by a simple heuristic based on measures of semantic similarity (MPG/VUA)
- Experiment on user feedback for better gold standards (MPG)
Chapter 4: Summary (MPG?)
Status of Inputs by WP2 partners
- Ontotext
- USFD
- VUA
- WICI
Current status
- 2011 08 15: Structuring and preparing content (MPG)
- 2011 08 16: Requesting input from WP2 partners (MPG)
- (...)