Some notes on how I am using crawlers as I’m collecting links.
I’ve started dabbling in crawlers with two simple prototypes—these may not
even be considered crawlers, but simple web fetchers or something like
that—but I think of them as being (or becoming) fill crawlers. Most
crawlers are out exploring the Web, discovering material and often categorizing
them, given some kind of algorithm that determines relevancy. Here, I’m the one
discovering and categorizing; the fill crawler only does the work of watching
those pages, keeping me aware of other possibly relevant sites and notifying me
when I need to update that link.
So, these crawlers are filling in the blanks for certain links. Filling in
missing parts that aren’t editorial. This isn’t a crawler that is feeding the
site’s visitors—it’s there for my utility.
For href.cool, the crawler isn’t really a crawler, given that it doesn’t do
any exploring yet. It just updates screenshots, lets me know when links are
broken and tracks changes over time. Eventually, I hope that it will keep
snapshots of some of those pages and help me find neighboring links.
Anyway, I’ve had that crawler since the beginning and it will stay rather
limited since it’s for personal use.
For indieweb.xyz, I’ve started a crawler that’s also for keeping the links
updated. Yeah, I want to know when something is 404 and keep the comment counts
updated. But I also want to get better comment counts by spidering out to see
the links that are in the chain. This crawler allows indieweb.xyz to stay
updated even if Webmentions don’t continue to come in from that link.
I think the thing that excites me the most about this crawler is that I’d like
it to start understanding hypertext beyond the Indieweb. I’m hoping it can begin
to index TiddlyWikis or dat:// links, so that they can participate. I’d really
like TiddlyWiki users to have more options to broadcast that doesn’t require
plugins or much effort—they should remain focused on writing.
Both of these projects are focused on trying to help the remaining denizens of
straight-up Web hypertext find each other, without it functioning like another
social network that becomes the center of attention. To me, rather than giving
the crawler the power to filter and sort all these writings, it simply acts as a
voracious reader that looks for key signifier that all of normal readers/linkers
are looking for anyway. (Such as links in a comment chain or tags that reveal
categories.)
That’s all I have to say at the moment. I mostly put this out here so that
people out there will know how these sites work—and to connect with other
people (like Brad Enslen and Joe Jennett) who are doing cataloging work, to keep
that discussion going.
Reply: Flooding the Culture
Averting our gaze from mainstream culture—cAN It bE DoNE?
Hahah, wow—it’s funny because I find this article to be a similar kind of frustrating read. A good read—perhaps like the Times article was for her—but very frustrating. I wonder: is acceptance by mainstream culture really seen as the ultimate, final, crucial reward?
(Particularly now that we live in an age where it’s clear that the previous generation of cultural winners—be it Jimi Hendrix or Harper Lee—is rapidly fading away, to be replaced by YouTubers, video game streamers, YA writers, reality stars. Isn’t the mainstream culture going to be very ruthless in its war for canonization?)
I mean I love the author’s ultimate point: here, I won’t summarize it, let’s just get into it.
As h0p3 would say: preach it! Tap into it.
But the author spends the entire piece looking away from the underground—scrutinizing the fucking New York Times to show us the way, looking at the top 20 shows on Netflix, stats on buying habits on Amazon. If the concern is that our culture is spending all of their time on Netflix, Amazon and the Times—well, so is this article.
So when we go to ‘tap into it’—what is it? Where is this ‘parallel universe’ we’re looking for? Where does this culture go to look for it? Is it on Amazon and Twitter somewhere? Are we supposed to continue using Netflix and Google—but somehow spend our time on the back alleys of those services?
Is this a request to leave alone the front page of the New York Times and start with the back page? (So much simpler to turn to the back page of the corporeal printed Times than to do so online.)
Clearly, the article decries the entire makeup of these systems:
Yesss! So go outside those neatly ordered corporate-approved spaces, yeah?
Let’s return to that final tap into it! paragraph. The phrase I want to look at is here: “the acknowledgement by powerful organizations.” Wait—so the tap into it! is meant… for them??
Are you asking the powerful organizations to—go outside themselves? Why? So they can continue to show us what’s legitimate? Because they are the authorities on what shit is actually cool?
I mean, yes, I’m not dense—the ‘powerful organizations’ are a massive pipeline of fame and currency—and this stuff can be gasoline to an artist. (Lord knows I want Boots Riley to keep it up—dammit, give the man what he needs!) But all of us out here, all us commoners, put together—we’re pure fuel, too. There was a time when it seemed that those very organizations were at the mercy of the buying public, earlier in this century when the entire system shook in fear of ‘disruption’.
And so, it feels like the article is just asking the mainstream to open a little wider, to give out a few more awards here and there, in lip service to the world of underappreciated, wonderful, unknown artists. (Black artists, in her case—but also in mine, because I want my mind blown by cool shit as much as any of you.) And, yeah, okay, maybe the ‘corpypastas’ might just throw us a bone.
However, I love the ‘parallel universe’ she refers to—that’s our unruly, unpredictable Web—an extension of the underground scenes, of the avant-garde, the mixtape traders, the world of the only critics that matter: our little group of friends. Those mixtapes blow up out here first. Out in our parallel universe: all of you out on your little blogs and wikis that I tap into each day. This world exists. It’s here, even if it faces its own doom on some days, in the face of resurgent mainstream culture.
Fuck the NYT, fuck Netflix—I’m reading you folks.