14 Commits

Author	SHA1	Message	Date
Jan Halfar	e2aeca122d	added ScrapeResultModifierFunc	2020-06-23 17:05:00 +02:00
Jan Halfar	4c5d090358	added normalized links and a links report	2020-06-23 13:22:27 +02:00
Jan Halfar	46aa4ee113	extracted reports, added filters for reports web interface, proper redirects support	2020-06-19 12:15:48 +02:00
Jan Halfar	5a4c391930	hello html schema	2020-04-30 14:34:50 +02:00
Jan Halfar	f6ba72b512	initial support for validations	2020-03-31 13:57:03 +02:00
Jan Halfar	597427303e	starting to use loop completion channel	2020-03-30 12:18:36 +02:00
Jan Halfar	7c26402adf	external scraping and filtering support	2020-03-26 16:18:32 +01:00
Jan Halfar	e7859938dd	chore: README update	2020-03-23 14:56:10 +01:00
Jan Halfar	91dc1fca07	initial robots.txt support	2020-03-11 14:36:55 +01:00
Jan Halfar	257db98e00	added metrics, client pool and improved stats	2020-03-10 17:23:45 +01:00
Jan Halfar	09cb6cb47b	feature: introduced grouping with a header	2019-03-11 17:05:21 +01:00
Jan Halfar	67d9e64529	refactored target to an object with a baseurl and multiple paths	2019-02-25 17:22:58 +01:00
Jan Halfar	cd770df1dc	added config support to ignore robots meta	2019-02-20 16:43:35 +01:00
Jan Halfar	5f1ac35dfa	extracted scrape loop to a file	2019-02-18 11:51:18 +01:00