SPDR
From Intimation
Contents |
SPDR?
Hi, I'm Anaerin, creator of the SPDR, and I'll be taking you on this minor history lesson of the tool I created, now affectionately referred to as the SPDR.
What does SPDR mean?
SPDR stands for Spidering Paths for Data Recursively. It's name is an homage to the "System Peril Distributed Reflex", a character from the first ARG I played, and one that's related to this very game, ILoveBees.
So, what is it?
Essentially, the SPDR is a recursively exploring script, that walks the various nodes within the "Maze" on the site at http://thewaytheirworldended.com/Q7J8R04J87FFJ983Q2/
To put it another way, SPDR is a program that pretends to be a web client, "Clicking" on the links and exploring the maze, noting any and/or all data it finds on the way.
Technically speaking, what is it?
Technically, it's a <700LOC C#3.5/WPF program knocked up and modified over the course of the game as new types of data become available within the "Maze". If you're burning to see the code, you can download a copy here. I will warn you, it's not at all "best practise" code, and is in fact my first attempt to code anything meaningful in .net. Feel free to do what you like with it, but please respect the PMs resources and don't hammer the server constantly with it.
Where can I get it?
If you look in the section above, there's a link to the source code. There are pre-compiled versions floating around, and while you can use them, I would highly recommend you don't. If everyone's running this tool, which is bandwidth and resource intensive, we may cause serious billing and availability issues for the PMs.
What's the history of this little tool?
Back when I first came across that maze network, I thought it would be a good idea to spider the nodes and try and build a map. SPDR was originally meant to map this network, but this task has been set aside as too technically complex (for now).
- Version 0.1 was a simple "Classic" VBScript ASP page. This is my coding comfort zone. However, unfortunately, this was also FAR too restricting.
- Version 0.5 was first built in VS2010 and .net 4.0. It worked with recursive function calls and synchronous web calls, passing forward an incrementing counter to indicate it's current depth. When it hit the (hard-coded) depth limit, it simply returned. It didn't try and find pathlocks, it didn't decode found data, and it kept everything in RAM. As a direct result of this, it ate RAM hungrily, and output plain text.
- Version 0.6 was re-written to use object-oriented approaches, with asynchronous calls and an object-per-node architecture. It devoured RAM in mighty gulps, without mercy for swapfiles or anything else running at the time. It outputted formatted HTML, to be posted to Google Sites.
- Version 0.7 was modified to follow pathlocks, even if they were deeper than it's hard-coded depth limit.
- Version 0.8 was modified to decode the Base64-encoded data at the end of pathlocks. It's depth limiter was also made user-modifiable.
- Version 0.9 was modified (heavily) to use passed references rather than full objects. It was also made multi-threaded, to try and increase efficiency. The Base64 decoding code was also fixed to work in more than 50% of times (oops!)
- Version 1.0 was modified further, slimming down the passed objects. However, the processing was all still done by the asynchronous fetch handler, making a huge "Waiting list" of stalled threads. Memory was still being eaten. Pathlock following was removed, as there seemed to be no new pathlocks to find.
- Version 1.1 was modified to store the bare minimum required for each node's fetch, storing them in a separate stack and ensuring that only 2 tasks were ever active at once. Memory usage was tamed, but still a large issue.
- Version 3.5 was backported to .net 3.5 (as the Windows 7 install that VS2010 was in had expired). Pathlock following was re-instated, after being cleaned up and re-written.
- Version 3.6 was modified to store it's values in a database. This takes the load off of RAM, finally taming the usage, and enabled faster and easier querying of data. It also allows for off-line querying of data (not yet implemented), as well as greater extensibility for adding new "types" of data (Just a line in the DB enables new a new Data type to be created). It now, also, outputs in WikiText, for posting to this very Wiki.
This is, pretty much, where the code stands at the moment.
So, what does it make?
It makes the page SPDR_Results. It doesn't post this page automatically, it just creates the Wikitext for the page, but the whole thing can be copy-pasted into this wiki in a single step, without need for reformatting.
