DepSpid

DepSpid
Project
Status	Completed
Category	Computer Science
Compute	CPU
Development
Developer	Bjoern Henke
Author	Bjoern Henke
Sponsor	None (independent hobby project)
Maintainer	None (discontinued)
Initial release	January 1, 2007  (19 years ago) - 1st news item
Discontinued	October 26, 2008  (18 years ago)
Repository	None known
Software
Written in	Perl
Operating system	Windows
Size	Unknown
BOINC statistics
Stats as of	October 26, 2008  (18 years ago)
Active users	0
Total users	2,643
Active hosts	0
Total hosts	9,819
Metadata
Website	http://www.depspid.net/
License	Unknown

DepSpid (short for "Dependency Spider") was a BOINC-based volunteer computing project that used donated CPU time to crawl the World Wide Web and build a database describing how individual web servers, and groups of web servers, depend upon and link to one another. The project was developed and administered by Bjoern Henke and ran on the alternative Perl-based BOINC server implementation known as PerlBOINC. DepSpid remained in an alpha development stage throughout its existence and was formally closed on 26 October 2008.

A generalized web crawler architecture. DepSpid's PerlBOINC-based crawling clients performed a similar fetch-and-parse cycle to build its dependency database.

Purpose and methodology

DepSpid's stated aim was to construct a database containing the dependencies between individual websites and groups of websites, and to collect statistical data describing the structure of the World Wide Web as a whole. An AnandTech community reference guide to distributed computing projects similarly described the project's purpose as being to "create a database of web servers and calculate the dependencies between various servers on different domains."

Conceptually, the crawl results could be modeled as a directed graph

$G=(D,E)$

where $D$ is the set of discovered domains (the nodes of the graph) and $E$ is the set of directed edges representing an observed hyperlink or dependency relationship from one domain to another. By repeatedly traversing outbound links from a set of seed domains, DepSpid's volunteer clients incrementally expanded both $D$ and $E$, in the same general manner used by conventional web crawlers and search engine indexers, but with the crawling workload distributed across volunteered CPU time rather than centralized server infrastructure.

A node-and-edge network visualization. DepSpid's end goal was a similar large-scale graph of dependency relationships between crawled domains, rather than between individuals.

By the time the project was taken offline, its crawler database recorded 755,516,850 crawled pages, having examined 23,620,224 of an estimated 42,578,804 known domains, giving a completion ratio of approximately

$\frac{23,620,224}{42,578,804}\times 100\mathrm{\%}\approx 55.474\mathrm{\%}$

at the time of the final recorded server status.

Technical implementation

Unlike most BOINC projects, which run the standard C++-based BOINC server software on Linux, DepSpid was built on PerlBOINC, an alternate reimplementation of the BOINC scheduler written entirely in Perl scripts and backed by a MySQL database. PerlBOINC was begun in mid-2005 with the specific goal of allowing a BOINC-style project server to run on Microsoft Windows rather than requiring Unix-family server infrastructure. PerlBOINC was also used, in its early form, by PrimeGrid during that project's own development.

Consistent with its PerlBOINC foundation, DepSpid's server ran on Windows, and community records from BOINC statistics forums confirm that, unlike most CPU-based BOINC projects, DepSpid supported only the Windows platform for its crawling clients.

Project statistics

The following figures represent the project's final recorded status before its server was taken permanently offline:

Statistic	Value
Total users	2,643
Total hosts	9,819
Hosts per user	3.71
Domains crawled	23,620,224 of 42,578,804 ({{Str ok|}}approx. 55.474%)
Pages crawled	755,516,850
Transfer rate (final)	0.008 GB/day

Closure

DepSpid's administrator posted a series of status updates through 2008 documenting the project's decline. A message dated 22 June 2008 noted the project had run out of tasks while a "phase-2" application was in development. On 31 August 2008, the project server went offline following a power supply fault that caused voltage fluctuations, raising concerns about possible damage to the server hardware and MySQL tables. The project's message boards were restored briefly on 27 August 2008 after "damaged mysql key files" were repaired, but on 26 October 2008 the project was formally closed. The closure announcement cited an ever-growing backlog of outstanding technical, organizational, and personal issues, and asked volunteers to detach their BOINC clients from the project. As of the closure announcement, it was left open whether the project might resume in some form in the future; it did not do so, and the project has since been listed by community-maintained BOINC project directories as closed, with its server offline.

See also

• BOINC projects
• PrimeGrid