DepSpid - Revision history

Al Piskun at 22:05, 17 July 2026

2026-07-17T22:05:20Z

← Older revision		Revision as of 22:05, 17 July 2026
Line 9:		Line 9:
	\| category = Computer Science		\| category = Computer Science
	\| compute = CPU		\| compute = CPU
	\| dependencies = ~~None known~~		\| dependencies =
	\| developer = Bjoern Henke		\| developer = Bjoern Henke
	\| author = Bjoern Henke		\| author = Bjoern Henke

Al Piskun: Initial release

2026-07-17T21:48:25Z

Initial release

← Older revision		Revision as of 21:48, 17 July 2026
Line 14:		Line 14:
	\| sponsor = None (independent hobby project)		\| sponsor = None (independent hobby project)
	\| maintainer = None (discontinued)		\| maintainer = None (discontinued)
	\| released = ~~Unknown (confirmed operating by~~ 2007)		\| released = {{Start date and age\|2007\|01\|01}} - 1st news item
	\| completed = No		\| completed =
	\| discontinued = {{Start date and age\|2008\|10\|26}}		\| discontinued = {{Start date and age\|2008\|10\|26}}
	\| repository = None known		\| repository = None known

Al Piskun: add archive link

2026-07-17T21:08:30Z

add archive link

← Older revision		Revision as of 21:08, 17 July 2026
Line 35:		Line 35:
	}}		}}

	'''DepSpid''' (short for "'''Dep'''endency '''Spid'''er") was a [[BOINC projects\|BOINC]]-based [[wikipedia:volunteer computing\|volunteer computing]] project that used donated CPU time to crawl the [[wikipedia:World Wide Web\|World Wide Web]] and build a database describing how individual web servers, and groups of web servers, depend upon and link to one another.<ref name="wikilist">{{Cite web \|title=List of distributed computing projects \|url=https://en-academic.com/dic.nsf/enwiki/663034 \|website=En-Academic (archival mirror of a former Wikipedia article revision) \|access-date=2026-07-17}}</ref> The project was developed and administered by Bjoern Henke and ran on the alternative [[wikipedia:Perl\|Perl]]-based BOINC server implementation known as PerlBOINC.<ref name="depspidsite">{{Cite web \|title=DepSpid — the dependency spider (project news archive) \|url=http://depspid.bjoernhenke.de/ \|website=depspid.bjoernhenke.de \|access-date=2026-07-17}}</ref> DepSpid remained in an alpha development stage throughout its existence and was formally closed on 26 October 2008.<ref name="depspidsite" />		'''[https://web.archive.org/web/20100309071056/http://www.depspid.net/ DepSpid]''' (short for "'''Dep'''endency '''Spid'''er") was a [[BOINC projects\|BOINC]]-based [[wikipedia:volunteer computing\|volunteer computing]] project that used donated CPU time to crawl the [[wikipedia:World Wide Web\|World Wide Web]] and build a database describing how individual web servers, and groups of web servers, depend upon and link to one another.<ref name="wikilist">{{Cite web \|title=List of distributed computing projects \|url=https://en-academic.com/dic.nsf/enwiki/663034 \|website=En-Academic (archival mirror of a former Wikipedia article revision) \|access-date=2026-07-17}}</ref> The project was developed and administered by Bjoern Henke and ran on the alternative [[wikipedia:Perl\|Perl]]-based BOINC server implementation known as PerlBOINC.<ref name="depspidsite">{{Cite web \|title=DepSpid — the dependency spider (project news archive) \|url=http://depspid.bjoernhenke.de/ \|website=depspid.bjoernhenke.de \|access-date=2026-07-17}}</ref> DepSpid remained in an alpha development stage throughout its existence and was formally closed on 26 October 2008.<ref name="depspidsite" />

	[[File:WebCrawlerArchitecture.png\|thumb\|300px\|A generalized web crawler architecture. DepSpid's PerlBOINC-based crawling clients performed a similar fetch-and-parse cycle to build its dependency database.]]		[[File:WebCrawlerArchitecture.png\|thumb\|300px\|A generalized web crawler architecture. DepSpid's PerlBOINC-based crawling clients performed a similar fetch-and-parse cycle to build its dependency database.]]

Al Piskun: first light

2026-07-17T21:07:42Z

first light

New page

{{Infobox software
| name = DepSpid
| logo = Depspid.png
| logo caption =
| screenshot =
| caption =
| description = DepSpid, also called the Dependency Spider, was a completed Computer Science BOINC project developed by Bjoern Henke that crawled the World Wide Web to build a database of dependency relationships between web servers and domains.
| status = Completed
| category = Computer Science
| compute = CPU
| dependencies = None known
| developer = Bjoern Henke
| author = Bjoern Henke
| sponsor = None (independent hobby project)
| maintainer = None (discontinued)
| released = Unknown (confirmed operating by 2007)
| completed = No
| discontinued = {{Start date and age|2008|10|26}}
| repository = None known
| programming language = Perl
| operating system = Windows
| size = Unknown
| stats as of = {{Start date and age|2008|10|26}}
| average performance =
| active users = 0
| total users = 2643
| active hosts = 0
| total hosts = 9819
| rac =
| credit per day =
| gpu performance =
| cpu performance =
| website = {{URL|http://www.depspid.net/}}
| license = Unknown
}}

'''DepSpid''' (short for "'''Dep'''endency '''Spid'''er") was a [[BOINC projects|BOINC]]-based [[wikipedia:volunteer computing|volunteer computing]] project that used donated CPU time to crawl the [[wikipedia:World Wide Web|World Wide Web]] and build a database describing how individual web servers, and groups of web servers, depend upon and link to one another.<ref name="wikilist">{{Cite web |title=List of distributed computing projects |url=https://en-academic.com/dic.nsf/enwiki/663034 |website=En-Academic (archival mirror of a former Wikipedia article revision) |access-date=2026-07-17}}</ref> The project was developed and administered by Bjoern Henke and ran on the alternative [[wikipedia:Perl|Perl]]-based BOINC server implementation known as PerlBOINC.<ref name="depspidsite">{{Cite web |title=DepSpid — the dependency spider (project news archive) |url=http://depspid.bjoernhenke.de/ |website=depspid.bjoernhenke.de |access-date=2026-07-17}}</ref> DepSpid remained in an alpha development stage throughout its existence and was formally closed on 26 October 2008.<ref name="depspidsite" />

[[File:WebCrawlerArchitecture.png|thumb|300px|A generalized web crawler architecture. DepSpid's PerlBOINC-based crawling clients performed a similar fetch-and-parse cycle to build its dependency database.]]

== Purpose and methodology ==
DepSpid's stated aim was to construct a database containing the dependencies between individual websites and groups of websites, and to collect statistical data describing the structure of the World Wide Web as a whole.<ref name="wikilist" /> An [[wikipedia:AnandTech|AnandTech]] community reference guide to distributed computing projects similarly described the project's purpose as being to "create a database of web servers and calculate the dependencies between various servers on different domains."<ref name="anandtech">{{Cite web |title=Comprehensive BOINC project list |url=https://forums.anandtech.com/threads/comprehensive-boinc-project-list.2637/ |website=AnandTech Forums |access-date=2026-07-17}}</ref>

Conceptually, the crawl results could be modeled as a directed graph

: <math>G = (D, E)</math>

where <math>D</math> is the set of discovered domains (the nodes of the graph) and <math>E</math> is the set of directed edges representing an observed hyperlink or dependency relationship from one domain to another. By repeatedly traversing outbound links from a set of seed domains, DepSpid's volunteer clients incrementally expanded both <math>D</math> and <math>E</math>, in the same general manner used by conventional [[wikipedia:web crawler|web crawlers]] and [[wikipedia:search engine|search engine]] indexers, but with the crawling workload distributed across volunteered CPU time rather than centralized server infrastructure.<ref name="depspidsite" />

[[File:Social Network Analysis Visualization.png|thumb|300px|A node-and-edge network visualization. DepSpid's end goal was a similar large-scale graph of dependency relationships between crawled domains, rather than between individuals.]]

By the time the project was taken offline, its crawler database recorded 755,516,850 crawled pages, having examined 23,620,224 of an estimated 42,578,804 known domains, giving a completion ratio of approximately

: <math>\frac{23{,}620{,}224}{42{,}578{,}804} \times 100\% \approx 55.474\%</math>

at the time of the final recorded server status.<ref name="depspidsite" />

== Technical implementation ==
Unlike most BOINC projects, which run the standard [[wikipedia:C++|C++]]-based BOINC server software on [[wikipedia:Linux|Linux]], DepSpid was built on '''PerlBOINC''', an alternate reimplementation of the BOINC scheduler written entirely in [[wikipedia:Perl|Perl]] scripts and backed by a [[wikipedia:MySQL|MySQL]] database. PerlBOINC was begun in mid-2005 with the specific goal of allowing a BOINC-style project server to run on [[wikipedia:Microsoft Windows|Microsoft Windows]] rather than requiring Unix-family server infrastructure.<ref name="rechenkraft">{{Cite web |title=PerlBOINC |url=https://www.rechenkraft.net/wiki/PerlBOINC |website=Rechenkraft.net wiki |access-date=2026-07-17}}</ref> PerlBOINC was also used, in its early form, by [[PrimeGrid]] during that project's own development.<ref name="rechenkraft" />

Consistent with its PerlBOINC foundation, DepSpid's server ran on Windows, and community records from BOINC statistics forums confirm that, unlike most CPU-based BOINC projects, DepSpid supported only the Windows platform for its crawling clients.<ref name="bam">{{Cite web |title=Forum::BOINC::AMD64 Client |url=https://www.boincstats.com/forum/9/1774,3 |website=BOINCstats/BAM! |access-date=2026-07-17}}</ref>

== Project statistics ==
The following figures represent the project's final recorded status before its server was taken permanently offline:<ref name="depspidsite" />

{| class="wikitable"
! Statistic !! Value
|-
| Total users || 2,643
|-
| Total hosts || 9,819
|-
| Hosts per user || 3.71
|-
| Domains crawled || 23,620,224 of 42,578,804 ({{Str
ok|}}approx. 55.474%)
|-
| Pages crawled || 755,516,850
|-
| Transfer rate (final) || 0.008 GB/day
|}

== Closure ==
DepSpid's administrator posted a series of status updates through 2008 documenting the project's decline. A message dated 22 June 2008 noted the project had run out of tasks while a "phase-2" application was in development.<ref name="depspidsite" /> On 31 August 2008, the project server went offline following a power supply fault that caused voltage fluctuations, raising concerns about possible damage to the server hardware and MySQL tables.<ref name="depspidsite" /> The project's message boards were restored briefly on 27 August 2008 after "damaged mysql key files" were repaired, but on 26 October 2008 the project was formally closed. The closure announcement cited an ever-growing backlog of outstanding technical, organizational, and personal issues, and asked volunteers to detach their BOINC clients from the project.<ref name="depspidsite" /> As of the closure announcement, it was left open whether the project might resume in some form in the future; it did not do so, and the project has since been listed by community-maintained BOINC project directories as closed, with its server offline.<ref name="anandtech" />

== See also ==
* [[BOINC projects]]
* [[PrimeGrid]]

== References ==
{{Reflist}}

[[Category:Completed BOINC projects]]
[[Category:Computer science BOINC projects]]