ScalableServices

From CEDPS

Jump to: navigation, search

Contents

Documents

Work-in-progress

  • gRAVi:

Grid Remote Application Virtualization Interfaces

Introduction

The need to make application code accessible as a Web Service arises frequently in scientific applications. Depending on context, this apparently simple task can introduce a wide range of requirements, including interface generation, authorization of requests, generation of code to dispatch calls to application code, monitoring and management of tasks, data management, and dynamic mapping of application tasks to processors in respond to changing workloads. The Globus Alliance aims to provide solutions to the most persistent and vexing problems that come up in Grid projects and applications. Our solutions to date are collected in the Globus Toolkit and these solutions are used in many Grid applications and systems.

Driven by the need to deliver analysis to expanding nonexpert user communities and a desire to incorporate analysis components as black boxes into complex analysis chains, we see increasing adoption of an approach in which programs are packaged as scalable science services that process (potentially many) requests to access and analyze data subsets via well-defined service interfaces.


Goals:

Make an executable program accessible as a grid service: We want to make an executable program accessible over the network via a Web Services interface, but we don’t want to have to do any work! Thus, we want tools that can generate the WSDL, provide de-serialization and dispatch code (including the code to call the application program), define and invoke appropriate authorization logic, and so on. Depending on context, we may want the application WSDL to include operations for monitoring and controlling the application. This tool provides a framework that will let users to rapidly wrap their application code as Globus toolkit services and deploy them in a hosting environment. These scenarios involve the following issues:

• Need to integrate existing code into a services framework, to enable sharing of the code across community members, composition of analysis capability into end–to-end analysis chains, and the isolation of clients from details concerning the location and implementation of analysis functions.

• Ability to dynamically and reliably configure and deploy new analysis services and to dynamically vary the computational resources used by analysis services, in response to changing community load.


  • Workspace:
  • Partition and Cache Management: VMs can be quite large and a lot of their disk content can be divided into partitions that are either (1) empty, (2) read-only and shareable by multiple running VMs, and/or (3) have been seen before and are thus cacheable. The Workspace Service can currently handle multiple partitions, some decompression, and also blankspace creation. But future work entails a full partition management solution including more intelligent caching, security, and transfer functionality.
  • Local Resource Management: The Workspace Service currently contains a simple local resource management solution for managing a pool of VMM nodes. We are working on expanding and maturing the local resource situation, experimenting with both (a) the proper way to schedule VMs with overhead awareness and (b) ways to integrate with current local resource management systems running regular jobs.
  • Virtual Clusters: Deploying multiple workspaces at once that depend on each other has both dependency and configuration management issues. We are working on scalable techniques for deploying, managing, and migrating virtual clusters.
  • Configuration Management: Deploying one or many workspaces often means that configuration information only available at deploy-time must be made available. Doing this in a thoughtful manner involves both configuration management systems and having a set of standard virtual appliance components to piece together in predictable ways.
  • Security/Trust Management: Resource provider policies may only accept certain VMs on their sites and developing an assertion/attestation framework for VMs is necessary for this to not be ad hoc and error prone (this includes not only assertions, but the signing and validation mechanisms for both particular VM files and workspace metadata).

An ongoing effort to make the service and clients easier and more effective for both programmatic integration and human use. A C based client will help with integration and performance. Recent examples of this priority are the new User Quickstart Guide, binary installation options, and pre-packaged sample VMs.

Related work

Happy Things

The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources.

The 100 node cluster was composed of a headnode and workernodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology.

Retrieved from "http://www.cedps.net/index.php/ScalableServices"
Personal tools