Some notes on how I am using crawlers as I’m collecting links.
I’ve started dabbling in crawlers with two simple prototypes—these may not
even be considered crawlers, but simple web fetchers or something like
that—but I think of them as being (or becoming) fill crawlers. Most
crawlers are out exploring the Web, discovering material and often categorizing
them, given some kind of algorithm that determines relevancy. Here, I’m the one
discovering and categorizing; the fill crawler only does the work of watching
those pages, keeping me aware of other possibly relevant sites and notifying me
when I need to update that link.
So, these crawlers are filling in the blanks for certain links. Filling in
missing parts that aren’t editorial. This isn’t a crawler that is feeding the
site’s visitors—it’s there for my utility.
For href.cool, the crawler isn’t really a crawler, given that it doesn’t do
any exploring yet. It just updates screenshots, lets me know when links are
broken and tracks changes over time. Eventually, I hope that it will keep
snapshots of some of those pages and help me find neighboring links.
Anyway, I’ve had that crawler since the beginning and it will stay rather
limited since it’s for personal use.
For indieweb.xyz, I’ve started a crawler that’s also for keeping the links
updated. Yeah, I want to know when something is 404 and keep the comment counts
updated. But I also want to get better comment counts by spidering out to see
the links that are in the chain. This crawler allows indieweb.xyz to stay
updated even if Webmentions don’t continue to come in from that link.
I think the thing that excites me the most about this crawler is that I’d like
it to start understanding hypertext beyond the Indieweb. I’m hoping it can begin
to index TiddlyWikis or dat:// links, so that they can participate. I’d really
like TiddlyWiki users to have more options to broadcast that doesn’t require
plugins or much effort—they should remain focused on writing.
Both of these projects are focused on trying to help the remaining denizens of
straight-up Web hypertext find each other, without it functioning like another
social network that becomes the center of attention. To me, rather than giving
the crawler the power to filter and sort all these writings, it simply acts as a
voracious reader that looks for key signifier that all of normal readers/linkers
are looking for anyway. (Such as links in a comment chain or tags that reveal
categories.)
That’s all I have to say at the moment. I mostly put this out here so that
people out there will know how these sites work—and to connect with other
people (like Brad Enslen and Joe Jennett) who are doing cataloging work, to keep
that discussion going.
Reply: Supposedly Unreadable Tripe
h0p3’s wife does a mic check.
(This is sooo cool—to get a response from h0p3’s wife on her own personal wiki. I just can’t believe we’re having these conversations. This was not what I intended to do on this blog. I actually didn’t have any intentions really—I just wanted to mess with hypertext again—which I guess opened me up to reading random TiddlyWikis and having these delightful, possibly pointless, just-for-funsies conversations. It’s better than anything that I could have intended to do.)
k0sh3k! First off, I love ILL, too. I am a massive cheapskate and I try to avoid clutter—but mostly I just like the weird editions that show up. And I like to see where the books come from. (I give a shoutout to this in my Stories/Novels page.)
My favorite was when Denton Welch’s Maiden Voyage came in. It was an ancient hardback from the 1950s. (It was the first book I read by him—I love him now.) As I read, I began to realize that this edition had been published right after he died (at age 33) and it really transported me to that age. I had a hard time giving that one back.
I actually should read The Educated Mind again before I recommend it. I went back and read my review—and some of my perspectives have changed since then. A lot has happened in four years. I still think I would love that the book bows before the visage of Socrates… (I am not a fast reader.)
I loved him in high school—I guess I have forgotten so much about him. I think I liked him at the time for gimmicky reasons. I know I saw past the mere shape of his poems. I thought he was funny. But to hear about ‘anti-industrialist poems’—you shouldn’t have lost that paper.
I think it’s charming. Your worries about organization or curating—sure, it’s fun to spend time on that stuff—but you’ve put a lot of work into what you’ve got already and it’s already very amusing and interesting to idly search and click around. I like that it’s informal. I like that it’s off-the-cuff.
I feel I should apologize for reading. It feels voyeuristic. Or like a robot eating up feelings. (CAN DESPISING AYN RAND REALLY FEEL THIS GOOD.) And maybe I am just scoping up anecdotes and recommendations in slapdash—this is just my own librarian way. It is shameful, it is noble—it is just a way to pass the time.
Most of the teachers I’ve met and worked with are aware of this and frustrated by it, too. It’s strange to me that this awareness has been around since at least the 1970s—yet it’s only gotten worse, I’d hazard.
There was a conversation between Seymour Papert and Paulo Freire back then that really—well, it might have gone too far in places, but I think it’s mostly right on:
To me, I agree that the scaffolding is important—but I think we tend to make the whole thing about scaffolding and public school tends to be all scaffolding all the time. But I think of scaffolding as being rough-shod. You hammer together a few planks and then get back to the building itself. The scaffolding goes away with time. You forget it was ever there.
(In case this is too vague—I tend to make ‘scaffolding’ synonymous with ‘adult assistance’, Vygotsky’s meaning, rather than the other meanings that float about from time to time.)
Of course, I think the above goes wrong a bit because I view reading as experiential and driven from inside—and I think even “telling” can be this way. Teaching can be very immersive and very improvisational. It’s difficult to know if it can ever be prescribed. (I don’t often watch television, but I think this is one thing that has kept me watching The Good Place—the main character is provided with a personal philosopher, a man who finds himself given an Herculean chore to try to prescribe his wisdom to her, even though it all is completely applicable. It simply cannot be told I think.)
Thank you for all the books and links—I will always be on the lookout for more and I am glad to know you and your family. While I’m interesting in the pioneering work you all are doing with wikis and such, I think it’s eclipsed by the effort you make among your two children. These words might be, at their height, a ‘model’ of us.
But they are only artifacts compared to the humans behind them. This j3d1h and kokonut seem like great additions to our reality. (Just from things they pop off with in h0p3’s writings.)