Good way to perform searches on remote documents?

Hey, folks!

As I continue to work on my project where I'm developing a search tool that searches my company's product documentation, I am beginning to think a little more broadly in its scope.

Presently I'm using the Search object to create an index, then search within that index for search entries. All the documents which the Search object searches are local on that computer. However, I have started wondering how it may be possible to develop the means of searching remote documents, Web sites, forums, and/or WIKIs.

The inspiration for this particular project is Adobe's product Help application. For those of you who are familiar with Adobe Creative Suite 4 or later, you'll know that Adobe (using their technology/platform Adobe AIR) can perform searches of their installed product documentation as well as their online forum, support incidents, etc from the Help application. I'm trying to create something that mimics that.

So, now I have a working (though sluggish) application that can perform local document searches. How would it be possible to search remote documents in which I have access?

I suppose that if I had a back-door to documents which are available off the company Web site but which are stored on a network location in which I have access, I could easily create indexes and search against those documents. But I don't think that method would work in searching forums or WIKIs, nor for other online documents in which I don't have direct access other than the Web. Or is there some other way to access forum and WIKI content from a friendly Web site (meaning, one which I have permission to access its content)?

Thoughts and ideas are welcome.

Most graciously...

RGBreality