A self-maintaining data pipeline that runs 24 scheduled jobs across 12 data categories, tracks market opportunities, cross-matches against products, and delivers actionable intelligence via Telegram and Todoist — all without manual intervention.
Real-time status of all data scrapers
How data flows from web to actionable intelligence
Async Python scrapers with Firecrawl fallback
Continuous loop with cron-based scheduling
12 data categories, 2,400+ data points
Health tracking + 105+ alert generation
Morning digest + task push
Automated cron jobs running on VPS
Built to survive the real web
Every scraper has a universal fallback via Firecrawl search API. When direct scraping fails (DNS, 429, JS-rendered sites), data still flows.
Network errors are detected immediately (name resolution, DNS, unreachable) instead of wasting time on retries. Fallback triggers in <1s.
Scraped opportunities are automatically matched against product catalog keywords, surfacing actionable trends for existing shops.
This page is powered by the same automation it describes. 24 scheduler jobs run continuously and refresh data via static JSON.