Volpex@Home

Volpex@Home (originally Volpex@UH) was a volunteer computing project run by the Department of Computer Science at the University of Houston that used the BOINC middleware to execute communicating parallel programs on ordinary, Internet-connected personal computers. The name Volpex stands for Parallel Execution on Volatile Nodes, and the project's central goal was to make the idle time of everyday "volatile" desktop machines usable as a virtual cluster for message passing parallel applications that the standard BOINC model could not run.

Volpex@Home
Volpex@Home logo
Project
Status	Completed
Category	Computer science
Compute	CPU
Requires	BOINC client 6.12 or later
Development
Developer	University of Houston Department of Computer Science; Parallel Software Technologies Laboratory (PSTL)
Author	Jaspal Subhlok; Edgar Gabriel
Sponsor	National Science Foundation (awards CNS-0834750 and MCB-0919974)
Maintainer	Volpex team
Initial release	December 1, 2011  (15 years ago)
Completed	2017
Repository	https://github.com/PSTL-UH/VolpexMPI
Software
Written in	C, Python, C++
Operating system	Windows, Linux
BOINC statistics
Stats as of	January 5, 2012  (14 years ago)
Performance	17.2 GFLOPS
Active users	135
Total users	292
Active hosts	404
Total hosts	824
Analytics
RAC	3,449
GPU performance	None (CPU-only)
Metadata
Website	http://volpexathome.cs.uh.edu/VolPEx/
License	Source-available (source code available on request)

The project also hosted a biological application called inCell@Home, which ran replica-exchange molecular dynamics simulations of protein behavior inside a crowded, cell-like environment in support of drug-design research. The framework research was funded by two National Science Foundation grants, and the project reached about 1,700 volunteers before it was completed and retired in 2017.

A diagram captioned VOLPEX showing a scientific application in the center linked by arrows to a volunteer environment, parallel execution, and heterogeneous devices

The Volpex concept: a scientific application runs on a heterogeneous, volunteer, parallel computing environment.

Overview

Most desktop computers sit idle for much of the day, representing a large pool of unused processing, storage, and communication capacity that grows with the spread of multi-core machines. The difficulty is that these machines are volatile: their owners can switch them off, disconnect them, or reclaim them at any time and without warning. Platforms such as BOINC and Condor handle this volatility well for independent, non-communicating tasks (the so-called "bag-of-tasks" or master-slave model), but they do not natively support tightly coupled, message-passing parallel programs in which tasks must exchange data as they run.

Volpex was designed to remove exactly that limitation, so that a broader class of scientific applications could run on volunteer hardware and many more research projects could benefit from free, donated compute time. Rather than relying only on checkpointing, which the researchers argued is inadequate in environments with very high failure rates, Volpex used managed redundancy: it ran two or more replicas of every parallel process and regenerated failed replicas on demand from healthy ones, so that the application progressed at the rate of the fastest surviving replicas and continued seamlessly as long as at least one replica of each process remained alive.

History

The Volpex framework began as academic computer-science research. A collaborative National Science Foundation grant, "VOLPEX: A Framework for Parallel Execution on Volatile Nodes" (award CNS-0834750), started on 1 September 2008 with Jaspal Subhlok as principal investigator and Rong Zheng and Edgar Gabriel as co-principal investigators at the University of Houston. The associated MPI library, VolpexMPI, was first described in a 2009 paper by Troy LeBlanc, Rakhi Anand, Edgar Gabriel, and Jaspal Subhlok.

The public BOINC project, initially named Volpex@UH, was online by late 2011, when volunteer teams began joining, and it was announced on the official BOINC news page on 22 December 2012 as a University of Houston project seeking alpha testers for parallel computing across multiple hosts. The project later moved to the address volpexathome.cs.uh.edu and became known as Volpex@Home.

The team upgraded their BOINC server to a newer release in March 2013 and recovered from a full server disk in April 2015; new work jobs were still being posted as late as June 2015. The biological side of the work was supported by a second NSF award to Margaret S. Cheung (MCB-0919974, "Effects of macromolecular crowding on protein structure, folding, and interactions"), which began 1 July 2009 and used the distributed simulations to study proteins in cell-like conditions. The project was completed and retired around 2017, and it is listed among finished BOINC projects.

Technical design

Volpex provided two complementary programming interfaces for application developers, plus a simulation effort to predict performance.

A network diagram titled Volpex Testbeds showing the Volpex server connected through the UH network and Internet to a Shark cluster, Condor clusters, a NASA virtualized cluster, corporate networks, and home PCs

The Volpex testbeds: campus clusters, a NASA virtualized cluster, and home PCs linked through the University of Houston network and the public Internet.

Volpex Dataspace

The central component was the Volpex Dataspace, an abstract global data layer that let tasks exchange data objects asynchronously through anonymous Put/Get operations. Instead of volunteer machines contacting one another directly, tasks read and wrote data objects through the project's central server, which acted as an intermediary; the project noted that this design meant there was no direct peer-to-peer security exposure for volunteers. The Dataspace extended BOINC's independent-task model to allow inter-task communication while preserving consistent results even when redundancy or checkpointing caused a single operation to be invoked several times. The team described this, integrated with BOINC, as the first comprehensive solution for reliably running communicating parallel codes on volunteer nodes.

VolpexMPI

VolpexMPI was a subset implementation of the Message Passing Interface (MPI) standard tailored to volatile nodes. Ordinary message-passing exchanges were converted into asynchronous Put/Get operations, and process replication provided robustness. The key fault-tolerance technique was fully distributed, sender-based message logging, which let a restarted or replacement replica reconstruct the messages it needed without a central checkpoint server. A later, portable variant called VolpexPyMPI was written in Python, ran on both Linux and Windows, and accepted user-level MPI programs written in C or Fortran, trading some performance for ease of use and portability across heterogeneous hardware and operating systems.

Because each process had several replicas, a receiving process had to decide which sender replica to contact first. The group developed communication target-selection algorithms for this; a hybrid strategy gave performance close to that of an all-fast-machine configuration, and evaluations used the NAS Parallel Benchmarks across heterogeneous network and processor configurations.

Volpex Simulation

A third thrust, Volpex Simulation, built a virtual model of a real-world desktop grid, complete with realistic node and network characteristics, to estimate how parallel applications would perform under different network parameters and configurations before deployment. Using a measurement and simulation tool chain on the NAS benchmarks, the researchers found that running on a realistic volunteer campus pool produced slowdowns by factors of roughly $2$ to $10$ compared with a dedicated cluster, depending on the benchmark code.

Reliability model

The central trade-off in the Volpex design is between the cost of redundancy and the probability that a parallel job survives. If each process is replicated $r$ times and an individual volunteer node remains available for the duration of a job with probability $p$, then a given process group survives only as long as at least one of its replicas is alive. With independent failures, the probability that a job requiring $n$ distinct processes completes is

$P_{\text{job}}=(1-(1-p{)}^r{)}^n.$

Increasing the replica count $r$ sharply raises survival probability, but it also inflates the total compute demand: the aggregate redundant work relative to a single execution scales approximately as

$W_{\text{total}}\approx r\cdot n\cdot W_{\text{process}}.$

Much of the project's research concerned managing this tension efficiently, for example through on-demand regeneration of lost replicas, sender-based message logging, and optimized checkpointing intervals for inter-dependent replicated processes.

How volunteers participated

Volunteers attached their BOINC client to the project's URL, and the BOINC server recruited hosts for each parallel job. Because a communicating job needs a fixed number of processes running at the same time, the server waited until enough hosts had been recruited and then started them together; recruited hosts that were waiting their turn temporarily idled and released the CPU rather than running the task. Some hosts were designated as spare nodes that stood by to replace a failed node mid-job.

This model produced a few unusual behaviors that the project documented for volunteers: tasks could appear as "Waiting to run (Scheduler wait)" during recruitment, a task could be terminated deliberately once the overall job had finished, and the CPU could sit idle while a process waited for data from another process. Tasks were also given short deadlines and run at high BOINC priority so that all processes of a coordinated job stayed reachable, since one unavailable process could stall a computation involving hundreds of processes. Uniquely for a BOINC project, credit was granted periodically in proportion to the time a host spent on a task, rather than only when a task completed successfully.

Work was available for Windows and Linux machines on x86 and x86-64 CPUs; there was no GPU application, and clients older than BOINC 6.12 were not supported because the project relied on the client reporting its task name for identification.

The BOINC client-server architecture that Volpex extended with its Dataspace communication layer.

Applications

By 2014 the project served two CPU applications, each available for Windows and Linux on Intel x86 (and Linux on x86-64):

• REMD Protein Folding (version 1.35), the inCell@Home replica-exchange molecular dynamics application; and
• Sieve (version 1.08), a lightweight test and benchmarking application used to exercise and validate the Volpex framework.

inCell@Home

Molecular visualization of several protein replicas (red, white, and blue structures) surrounded by a dense field of grey spheres representing crowding macromolecules

Replica-exchange molecular dynamics of proteins (colored structures) inside a densely crowded, cell-like environment of macromolecules (grey spheres), as simulated by inCell@Home.

inCell@Home was the flagship scientific application of Volpex and a collaboration with the research group of Margaret S. Cheung at the University of Houston. It simulated how proteins behave and function inside a realistic, crowded cell-like environment rather than in dilute solution, with the results intended to inform biological drug design. The application ran replica-exchange molecular dynamics (REMD), a method in which many copies (replicas) of a system are simulated at different temperatures and periodically exchange configurations to overcome the kinetic trapping that makes protein folding hard to sample; the loosely coupled, communication-light nature of REMD made it well suited to a replicated, volatile environment and to the Volpex Dataspace.

This tied the computer-science framework directly to a biophysics question: experimental and computational work by the Cheung group showed that macromolecular crowding dramatically affects protein structure, folding, and activity, for example compacting the enzyme phosphoglycerate kinase and increasing its activity under crowded conditions that mimic the inside of a cell. The crowding research combined coarse-grained and atomistic molecular simulation in a multi-scale approach and was supported by NSF award MCB-0919974.

The crowded cytosol of a cell: macromolecular crowding alters protein folding and behavior, the phenomenon studied by inCell@Home.

Funding and statistics

The framework research was supported by the National Science Foundation's Computer Systems Research program under award CNS-0834750 (approximately $296,000 awarded to the University of Houston), and the biological crowding research by award MCB-0919974 (approximately $232,000 awarded).

Volpex remained a small research project throughout its life. In early January 2012 it counted 292 registered users (135 active), 824 hosts (404 active), and 64 teams across 36 countries, with about 1.1 million total credits, a recent average credit of roughly 3,449, and an average throughput of about 17.2 GFLOPS. By the time the project was retired, the French Wikipedia's BOINC project table recorded 1,746 users, 6,688 hosts, 252 teams, and 65,282,403 total credits as of 15 October 2017.

Scientific publications

The University of California, Berkeley list of publications arising from BOINC projects catalogs nine Volpex@Home papers, spanning the framework, its MPI and Dataspace components, message logging, performance estimation, checkpointing, and a summary journal article.

1. Subhlok, Jaspal; Hien Nguyen; Edgar Gabriel; Mohammad Tanvir Rahman (2018). "Resilient parallel computing on volunteer PC grids". Concurrency and Computation: Practice and Experience. DOI: 10.1002/cpe.4478.
2. Rahman, Mohammad Tanvir; Hien Nguyen; Jaspal Subhlok; Gopal Pandurangan (2016). "Checkpointing to Minimize Completion Time for Inter-Dependent Parallel Processes on Volunteer Grids". 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). DOI: 10.1109/CCGrid.2016.78.
3. Islam, Md Tarikul; Hien Nguyen; Jaspal Subhlok; Edgar Gabriel (2015). "Efficient Message Logging to Support Process Replicas in a Volunteer Computing Environment". 2015 IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW). DOI: 10.1109/IPDPSW.2015.91.
4. Nguyen, Hien; Eshwar Pedamallu; Jaspal Subhlok; Edgar Gabriel; Qian Wang; Margaret S. Cheung; David Anderson (2012). "An Execution Environment for Robust Parallel Computing on Volunteer PC Grids". 2012 41st International Conference on Parallel Processing (ICPP). DOI: 10.1109/ICPP.2012.18.
5. Rohit, Eshwar; Hien Nguyen; Nagarajan Kanna; Jaspal Subhlok; Edgar Gabriel; Qian Wang; Margaret S. Cheung; David Anderson (2011). "A Robust Communication Framework for Parallel Execution on Volunteer PC Grids". 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). DOI: 10.1109/CCGrid.2011.72.
6. Anand, Rakhi; Edgar Gabriel; Jaspal Subhlok (2010). "Communication Target Selection for Replicated MPI Processes". Recent Advances in the Message Passing Interface (EuroMPI 2010), Lecture Notes in Computer Science, vol. 6305, pp. 181-190. Springer. DOI: 10.1007/978-3-642-15646-5_21.
7. LeBlanc, Troy P.; Jaspal Subhlok; Edgar Gabriel (2010). "A High-Level Interpreted MPI Library for Parallel Computing in Volunteer Environments". 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid), pp. 405-414. DOI: 10.1109/CCGRID.2010.85.
8. Kanna, Nagarajan; Jaspal Subhlok; Edgar Gabriel; Eshwar Rohit; David Anderson (2010). "A Communication Framework for Fault-Tolerant Parallel Execution". Languages and Compilers for Parallel Computing (LCPC), Lecture Notes in Computer Science. Springer. DOI: 10.1007/978-3-642-13374-9_1.
9. LeBlanc, Troy; Rakhi Anand; Edgar Gabriel; Jaspal Subhlok (2009). "VolpexMPI: An MPI Library for Execution of Parallel Applications on Volatile Nodes". Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM/MPI 2009), Lecture Notes in Computer Science, vol. 5759, pp. 124-133. Springer. DOI: 10.1007/978-3-642-03770-2_19.

Related publications

Additional Volpex-related papers not listed in the BOINC publications index include a performance-estimation study and the journal version of the VolpexMPI work reported on the project's NSF award page:

• Nandagudi, Girish; Jaspal Subhlok; Edgar Gabriel; Judit Gimenez (2012). "Estimation of MPI Application Performance on Volunteer Environments". Euro-Par 2011: Parallel Processing Workshops, Lecture Notes in Computer Science, vol. 7155, pp. 511-520. Springer. DOI: 10.1007/978-3-642-29737-3_56.
• Anand, Rakhi; Troy LeBlanc; Edgar Gabriel; Jaspal Subhlok (2011). "A Robust and Efficient Message Passing Library for Volunteer Computing Environments". Journal of Grid Computing, vol. 9, no. 3, p. 325. DOI: 10.1007/s10723-010-9172-x.

See also

• BOINC (Berkeley Open Infrastructure for Network Computing)
• Volunteer computing
• Distributed computing
• Grid computing
• Desktop grid
• Message Passing Interface
• Rosetta@home