[SATLUG] Academic web crawler?

Don Davis dondavis at reglue.org
Sun Jul 8 18:43:18 CDT 2012


If one wants to use a web crawler to create social network maps of blogs
and other sites (and related e.g. twitter), it seems pretty straight
forward. (I can find the tools for that pretty easy.)

Google Scholar and whatnot have created their lists of citations etc.

What about social network analyses of works cited?
Unfortunately, not everybody has the decency to use one format (e.g.
APA) so writing a crawler and parser would be a non-negligible use of my
time. Though some journals are kind enough to provide links and other
easily parsed bit for works cited and citing, they are by far in the
minority.


Surely, someone with the ability and motivation has written
tools/libraries/APIs to crawl academic refs or parse refs (from various
formats) nicely?

While on the subject - is there someway to take advantage of DOI #s for
crawling/parsing?



More information about the SATLUG mailing list