Repositories list

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  javaopen-sourcelibrary+ 4

  javaopen-sourcelibraryweb-crawlerrobots-txtsitemapsrobotstxt

  Java

  •

  Apache License 2.0

  •9090 forks•256256 stars•3030 issues•44 pull requests•Updated Feb 26, 2026Feb 26, 2026

• url-frontier

  Public

  crawler-commons/url-frontier’s past year of commit activity
  API definition, resources and reference implementation of URL Frontiers

  grpcwebcrawlingweb-crawlers+ 2

  grpcwebcrawlingweb-crawlersurl-frontierurlfrontier

  Java

  •

  Apache License 2.0

  •1212 forks•6060 stars•22 issues•22 pull requests•Updated Jan 23, 2026Jan 23, 2026

• http-fetcher

  Public
  Wrapper code for Apache HttpClient that provides common page fetching functionality

  Java

  •

  Apache License 2.0

  •44 forks•77 stars•66 issues•55 pull requests•Updated Feb 5, 2024Feb 5, 2024