[mmasters]

Is there any way to download all of the site message board content? I would consider buying a 10-20-60-100 gig storage drive or whatever's needed for this purpose.

I think there's a treasure trove of information residing here that should be archived by many and often. It would be horrible if the site just went down some day (due to circumstances out of our control) and all this great information was lost.

If money is needed for a hosting site, say for monthly archives, I would be more than happy to donate towards this purpose.

[Jack]

I suspect Adobe Acrobat could do it - it's fairly good at capturing parts of sites, or even entire sites.

It occurs to me that Aaron could generate a set of DVDs and list them on CafePress.com (or other, similar site) at a reasonable but profitable price. Interested persons could then purchase same...

CafePress gladly handles small, specialized customer bases. No setup costs, no inventory - just automatic order fulfillment.

[JPL]

Give me a day or two and I could write some to code to data-mine the forums for you. Presentation could be a problem, though - a lot of it's all over the place (read more $$$ for a decent-paying job - grin). Probably send Aaron into an outside loop though (double-grin). Why not ask him first?

JPL

[Loki]

Are you talking about a website downloader/copier? I tried to download the Planning forum with HTTrack but with only partial success. It'll probably work with the right parameters.

[Aaron]

I'm exporting the entire forum in SQL format right now... will try & post here when fnished.

[grink1tt3n]

I'm exporting the entire forum in SQL format right now... will try & post here when fnished.

Aaron, this may not be such a good idea, as you'll probably be exporting user account information along with it.

I could probably write some nifty SQL to just extract the posts in a nice format, PM me if you want me to do this.

[davep]

I don't think using a web crawler would be a good ploy on this site (I've tried it). Each page is referenced in many ways, so you get a lot of duplication.

You'll have to wait 'til Aaron gets his database dump out.

On a similar note, what would people recommend as a preferred local hard drive search engine (to index all the crap I do download from various sites)? I haven't looked at this kind of thing for a while and I'm sure there must be some good free ones around now.

[Aaron]

So how big ya think the file would be that contains every post on this forum to date?

I'm creating it now...

[mmasters]

So how big ya think the file would be that contains every post on this forum to date?

I'll take a stab at 30 gigs

[Aaron]

{link removed by Shannymara}

141 MB

[mmasters]

141 MB

Damn I was way off

[mmasters]

Is there any kind of little front end application that could dig around a big text file like this and translate it into a virtual enviornment to navigate around and search on?

That would rock.

Could have all of PO.com stored and easily viewable off of a tiny little flash drive...

[JPL]

Is there any kind of little front end application that could dig around a big text file like this and translate it into a virtual enviornment to navigate around and search on?

That would rock.

Could have all of PO.com stored and easily viewable off of a tiny little flash drive...

I've downloaded Aaron's file and have it safely tucked away so I have something to play with in a few years time when the lights are flickering on and off ;o)

To be honest, the easiest way in, now we have the data, would be a local MySQL DB and a copy of Aaron's SQL & code (grin). I would guess he has the forums data stored in 2 or 3 core tables but it would be in-comprehensible without the JOIN-statements. A text-search program really wouldn't be able to dig this out in any comprehensible form...

Or you persuade him to dump it into a Windows-friendly DB format like Access (best of luck!) Or else some-one data-mines it (grin).

JPL

[Aaron]

I can make a CSV copy.

And you can load the sql file above into a MySQL database and do all kinds of things with it.

I'll post the CSV...