Building LarKC in steps
Response from Barry to Frank/Annette proposal on architectural support feature set
My personal view at the moment, is that the feature list looks a little daunting. I would prefer to explore groups of features, gain some experience, understand requirements and then start making decisions about whether we need a particular feature or where it should be implemented.
Step 0: usage scenario's for different types of users
I think in order to put all these features in context, we should start thinking about what a larkc platform actually is. I believe it is timely to start thinking about how we will deploy larkc, how the various types of user will interact with larkc and how plug-ins will interface to the larkc platform. Some early decisions will help us to focus.
Feb 23: Frank and Annette have described a first usage scenario, and are solliciting others
step 1: Build a simple distributed platform
This means we will have to address remote execution of plug-ins (protocols, start-up, configuration, failure/recovery modes, etc), sharing of data (over a network).
- Learn something about efficient presentation/marshalling of data in comparison to extra processing power from using more machines.
- Understand the pros/cons of different protocols.
step 2: parallelisation inside a plug-in
Attempt the first parallel execution implementation, possibly reasoning or simple search over partitioned data. This will be a "inside a plug-in" parallelism so it's easier for the platform.
- Learn about cluster issues - security, online/batch interaction, upload of input data, firewall/protocols.
- Generate guidelines for plug-in developers deploying on clusters.
step 3: dynamically configurable pipelines
Build a platform that allows dynamic registration (even delivery) of plug-ins. A whole new interface that allows users to dynamically 'upload' new plug-ins, build custom pipelines and execute them. An 'experimentation' platform.
- Understand moving computation rather than data.
step 4: Extend platform to include 'computing@home'.
Many ways to do this, but could add something to manage dynamic registration of remote computing resources.
- Address delivering computation tasks and their remote execution.
- Address unreliable/heterogeneous computation.
- Understand the cost/benefit when large datasets involved.
- Generate guidelines for plug-ins deployed this way.
- Parallelism "outside of plug-ins" - toolkit/framework for this?