Reliable Teachable Imports with Built-in Rate Limiting
Reliable Teachable Imports with Built-in Rate Limiting
Version: 1.0.66
Overview
The Teachable scraping pipeline now includes built-in request throttling, making course imports more reliable and resilient against IP blocks, CAPTCHA challenges, and server-side rate limiting on Teachable's infrastructure.
What Changed
Previous versions of the import engine issued requests to Teachable without any deliberate pacing. On large courses with many lessons and assets, this could generate a high volume of rapid, concurrent requests — behaviour that Teachable's servers may interpret as abusive, resulting in:
- Temporary IP blocks
- CAPTCHA interstitials interrupting the scrape
- Incomplete or failed imports
Version 1.0.66 addresses this by introducing two throttle controls inside the Inngest-powered scraping pipeline:
1. Configurable Inter-Request Delay
A configurable delay is enforced between successive requests during the crawl phase. This paces the scraper and keeps request rates within safe thresholds.
2. Maximum Concurrency for Asset Downloads
When the pipeline fans out to download assets (images, embedded video links, document attachments), it now respects a maximum concurrency cap. Rather than spawning an unbounded number of parallel downloads, the pipeline processes asset batches in controlled bursts.
Configuration
Both the inter-request delay and the maximum concurrency for asset downloads are configurable. Operators can tune these values to match the size and sensitivity of the target Teachable school.
| Setting | Description | Default |
|---|---|---|
| Inter-request delay | Minimum wait time (ms) between sequential page requests | Configurable |
| Max asset concurrency | Maximum number of simultaneous asset download tasks | Configurable |
Larger courses with many attachments may benefit from more conservative (lower) concurrency settings to further reduce the risk of triggering rate limits.
Effect on Import Duration
Imports will take slightly longer to complete on large courses due to the enforced pacing. This is expected behaviour — the delays are intentional and represent a trade-off between speed and reliability. For most imports the added time is negligible; for very large schools with hundreds of lessons and assets, the difference will be more noticeable.
No Action Required
This change is transparent to end users and administrators. Existing import jobs will automatically use the new throttling behaviour on their next run. No changes to import configuration or workflows are needed.