Assimilator Web Crawler

Assimilator is a handy tool for automatic search and download of files from the web. The approach is like to drift net fishing for files. AWC will extract all the links from a web page and then parse them for files. Any files matching the type and name specifications will then be queued for downloading. Any links that aren’t files will be queued for parsing and then the process repeats with the next page found. This idea is based on a sort of six degrees of separation theory, i.e. If you know site that contains information related to what your looking for, chances are it will have a link to another related site that my have more information or files that you are searching for. This works well with using a search engine results page.

Future areas of development

• More intelligent searching
• file size criterea
• regex usage
• modification date
• anon FTP socket

Bugs and Issues

• menu shortcut keys don't work unless a window is open
• no new advanced search features work yet (regEX, fileSize and modified)
• no time-out on filesize thread
• file size is not saved
• sorting URL's has no effect on search/download order
• possible error when redirecting to different host
• interface tends to "Stick" with large queues
• analysis thread hogs CPU
• listbox flickers alot due to large number of refreshes
• moving errors back to search causes out of bounds in searchThread
• file:// resolution of networked drives faulty
• closing download window causes an error (use ignore errors as workaround) or stop search before closing.
• search thread crashes on bad URL/HTTP error
• No support for access via proxy servers

Current Status

Completely re-written with support for Mac. Although the Mac version is not posted here due to size restraints, it is available on request. The crawler is now HTTP/1.1 compliant and supports transparent redirections and resuming of downloads. HTML parsing is greatly improved but support for java script parsing for links has yet to be added. New version will search images, hyperlinks and frames for links and can search any sort of file as specified by the user.

• + Finally added custom Tag support
• + Added "as <img>" option to export to view found tags as images
• ~ Export now shows links in a nice table including the refering page
• * Altered close code to (try and) prevent crashes