Tags: shurco/goClone
Tags
β»οΈ refactor(crawler): harden mirror pipeline and modernize backend Changes made in this commit: - Modified: README.md, cmd/main.go, go.mod, go.sum - Modified: pkg/crawler/base.go, concurrency_test.go, crawler.go, crawler_test.go, css.go, init_download.go, integration_test.go - Modified: pkg/fsutil/file.go, folder.go, fsutil_test.go - Modified: pkg/netutil/netutil.go, netutil_test.go - Added: pkg/crawler/asset.go - Deleted: pkg/crawler/img.go, pkg/crawler/js.go Key changes: - Serialize access to discovered assets; fix O(n^2) page scheduling - Replace log.Fatal in parsers with logging; wait for download goroutines - Merge saveJS/saveIMG into saveAsset; package-level CSS url regexp - netutil: context-aware Extractor, strip query in urlExtension, check size before open - fsutil: filepath.Dir/Join, WriteOSFile returns error, truncate on mirror writes - Serve via stdlib http.Server with graceful shutdown; update README and --serve help