Performance and Scalability assessment Workshop
01-02 December 2009, Amsterdam
Time:
Start: Tuesday 01.12.2009, 9:30h (CET)
End: Wednesday 02.12.2009, 15:00h (CET)
Mission and expected output
- Definition of metrics and validation goals for the assessment of Performance and Scalability of the LarKC Platform.
- How to measure performance and scalability (instrumentation).
- Discussion on some of the current strategies to achieve improvements on performance and scalability. Presentation of current implementation status and next steps. How they contribute to the improvement of performance and scalability:
- Distribution Model
- Distributed Data Layer
Meeting minutes
Summary of meeting results + Next Steps: LarKC_PerformanceWorkshop_WrapUp_V0_2.ppt
When
- From 1st December 2009 (9:30h) to 2nd December 2009 (15:30h) (CET)
Where
VUA, Amsterdam
The meeting will take place at the rooms of the Medical Faculty, located at van der Boechorstraat 7-9.
Tuesday, 1 dec 9:30 , room B541.
Wednesday, 2 dec 9:00, room J182.
A map that also includes the meeting rooms/times can be found here.
The VU is reachable by tram 5, 16, 24 and 51. The best way to plan your trip is through http://journeyplanner.9292.nl/.
In case you need help, Spyros' mobile number is +31617630074
Expected Attendees
CycEur (Luka)
- VUA (Spyros, Annette, Frank)
- UIBK (Barry, Florian)
- Ontotext (Vassil)
- HLRS (Alexey, Matthias, Georgina)
Possible connection via phone call with CEFRIEL.
Agenda
The workshop will start at 9:30h (CET) on Tuesday 01.12.2009 and finalize at 15:00h (CET) on Wednesday 02.12.2009.
Tuesday 01.12.2009
- 09:30 – 09:40 – Welcome and approval of agenda
- 09:40 – 10:00 - Distribution model
- Current status (Alexey)
- 10:00 – 12:30 - Distribution model (cont.) (coffee break at 11:00-11:15)
- Practical session: “trivial” parallelization across several instances of the same plug-in (over partitioned data)
Pre-meeting input:
Plug-in (simple algorithm, e.g. reading RDF triples) (Spyros)
Data (Spyros)
Remote plug-in manager source code (Alexey)
Remote access to HLRS cluster (Alexey)
- Practical session: “trivial” parallelization across several instances of the same plug-in (over partitioned data)
- 12:30 – 13:30 - Lunch
- 13:30 – 14:30 – Distribution model practical session (cont.)
- 14:30 – 16:00 - Assessment of Performance and Scalability (Matthias - intro ppt to trigger discussion)
- Validation goals and metrics. How to measure performance and scalability (instrumentation = tools?).
- Strategies to achieve improvements on performance and scalability (plug-ins distribution, parallelization, data layer distribution).
- How they contribute to the improvement of performance and scalability.
- Presentation of current implementation status and next steps.
- (Distribution model - presented in the morning)
- Distributed data layer, data partitioning, load balancing,… (Matthias, Vassil)
- Current status of Data Layer (Vassil)
- Initial discussion on D5.5.3 structure and content.
Pre-meeting input:
- WP6 input (Daniele, CEFRIEL): In order to improve the time performance of the alpha Urban LarKC, Florian Steinke proposed a possible solution:
- some workflows running at batch time moving the data that alpha Urban LarKC will need (monument and event information) from the Web to the LarKC data layer;
- workflows invoked by users when using the alpha Urban LarKC doesn't interact with external sources but it queries directly the data layer.
- We know how to realize the second kind of workflows (the "reading" ones), but there are different ways to realize the first ones (the "writing" ones): they can be realized as normal plug-ins invoked every period of time by an application, or they can be realized with a "special" decider that works without input executing periodically the workflow. We think that the second solution (a decider that can schedule the execution of workflows) is a more general and it that can be reused easily in different scenarios. What do you think about it?
- WP6 + PO input (reported by Emanuele, CEFRIEL):
- Stefano Bertolo tried out the alpha Urban LarKC [1] and he is worried by the 10 seconds it takes to get an answer. He wanted me and Frank to comment on the plans to match the goal of LarKC to reason on the order of 200ms over billions of triples.
- More precisely he asked Frank and me for plans about and means of achievement for:
- having such queries go from 10 seconds to 5 seconds
- having such queries go from 5 seconds to 1 second
- having such queries go from 1 second to 200 milliseconds
- I answered that I believe 1 is possible by moving the prototype to a better hardware, 2 may be possible if some “smart” caching would be implemented, while I have no idea about how to reach 200 ms. However, WP6 alone cannot make plans.
Wednesday 02.12.2009
- 09:00 – 11:00 - Distribution model (cont.) / Assessment of Performance and Scalability (cont.) - depending to the progress of the first day
- 11:00 – 11:15 – coffee break
- 11:15 – 12:30 - Morning session (cont.)
- 12:30 – 13:00 – Short Lunch
- 13:00 – 14:00 - Morning session (cont.)
- 14:00 – 15:00 - Wrap-up and Next steps towards
- next meeting (Jan 2010)
- next release (March 2010)
If times allows it, and/or discussion is feasible in smaller groups, other topics for discussion:
- Plug-in API redesign
- Proposal for redesign (only RDF types, no Java types), including applicable example (Vassil) (if time, practical session on testing of redesigned API)
- Plug-in annotation language
- Current status (Barry)
Improvement needed after: distribution model, performance&scalability analysis, plug-in redesign (All)
- Other Platform support features for next release
- Multiplexers (Luka)
- Event handling: general approach, considering distribution model (Alexey)
- Plug-in state: general recommendations for plug-in and workflow developers (Barry)
WP6 feedback/requests: assignment of responsibilities and timeline (Features Requests)
Pre-meeting input
http://wiki.larkc.eu/LarkcProject/WP5/PerformanceScalabilityAssessment
So far the Data Layer has been validated with regards to its performance and scalability. Results are documented in LarKC deliverables D5.5.1 and D5.5.2
A definition of scalability within LarKC and a first analysis of scalability dimensions is presented in deliverable D5.3.2
Urban Computing use case evaluation is documented in deliverable http://www.larkc.eu/wp-content/uploads/2008/01/larkc_d66-2nd-periodic-report-on-data-and-performances_final.pdf. Some of the measurements presented here are independent from the use case and depend on the current platform implementation. This will be used as input for the platform evaluation.
D1.4.1 Initial framework for measuring and evaluating heuristic problem solving
http://cacm.acm.org/magazines/2009/8/34493-the-pathologies-of-big-data/fulltext
- FvH: useful observations that could be directly relevant for design/performance of LarKC the data-layer:
as a rule, the largest cardinalities of most datasets—specifically, the number of distinct entities about which observations are made—are small compared with the total number of observations.
What makes most big data big is repeated observations over time and/or space.
As dataset sizes grow, it becomes increasingly important to choose algorithms that exploit the efficiency of sequential access as much as possible at all stages of processing
It would seem intuitively obvious that data with a time dimension, for example, should in most cases be stored and processed with at least a partial temporal ordering to preserve locality of reference as much as possible when data is consumed in time order
The prevailing database model today, however, is the relational database, and this model explicitly ignores the ordering of rows in tables
[this is disastrous because] completely random disk access can be five orders of magnitude slower than sequential access [and even] random access to memory is typically slower than sequential disk.
distributed computing is the most successful strategy known for analyzing very large datasets.
- [followed by a useful discussion of locality of reference in distributed commputing and the dangers of skewed data distributions]
- I'd be interested to hear opinions from those who know more about these issues (Vassil, Spyros, Jacopo) on this article? - Frank.
- FvH: useful observations that could be directly relevant for design/performance of LarKC the data-layer:
Presentations
Distributed model: Distribution_LarKC_WP5_Amsterdam_2009.12.01.ppt
An experiment to deploy plug-ins as RESTful services (tomcat)
Performance and Scalability analysis: PerfScal_Amsterdam_V0_2.pptx
Summary of meeting results + Next Steps: LarKC_PerformanceWorkshop_WrapUp_V0_2.ppt
Accommodation and Travel
For you travel plans within The Netherlands, it is recommended to use http://journeyplanner.9292.nl/. It includes all public transport connections.
The VU is easily accesible from Amsterdam Airport Schiphol. It is about 10 minutes by train (station Amsterdam Zuid/ Amsterdam South). From station Zuid, the VU can be reached by tram 5 or metro/tram 51. Total travel time from Schiphol airport to the VU is approximately 30 minutes.
There is a plethora of hotels in Amsterdam. It is recommended that you chose one in the Museumplein/Museum square area, due to its proximity to the center and the VU (tram 5, 16 and 24, travel time is approx. 20 minutes). The VU is also easily reachable from the center of amsterdam with trams 5, 16, 24 and metro/tram 51 (travel time approx. 30 minutes).