From browser automation to plain HTTP
The project started as a Playwright scraper. Reverse-engineering the LTI handshake and the internal APIs replaced the browser with a pure HTTP collector, then widened the scope from one material list to six academic domains without giving the cost back.
- 514 → 24 requests for a materials sweep — 95.3% fewer, with identical output.
- 13.06 s → 0.83 s wall time (15.7× faster); CPU per full six-domain sweep is 0.69 s.
- 785 MiB → 119 MiB peak memory — 84.8% less than the browser version.
-
129.5 KB per sweep across six domains, using ETag
conditional requests (27
304responses in a typical run). - Hourly tracking costs about 2.07 MB per day and 11 seconds of CPU.
Shape
The collector runs on the user's device and keeps normalized state in local SQLite. A local stdio MCP server exposes that state to on-device agents; a cloud worker holds the synchronized academic state for AI applications and third-party clients. Collection never happens in the cloud.
- Device-local credentials — university passwords, Canvas tokens, and session cookies stay in the OS credential store (Keychain / Credential Manager).
- Idempotent collection — collecting the same state twice produces no duplicate downloads or records.
- Event boundary — consumers subscribe to collector events instead of polling the store.
- Source-native identifiers — original LMS ids are preserved; display names are treated as presentation data.
- Package
- pnpm workspace:
core,adapter-cau,collector,mcp-local,worker - Collector
- TypeScript, feature-frozen
- Cloud
- Cloudflare Workers · D1 · R2 (sync and analysis, not collection)
- Status
- Local collector and MCP surface implemented; cloud sync in progress
- License
- Proprietary