feat: FF HTML scraper for upcoming weeks with forecasts

- scrape_upcoming(weeks_ahead=5) in ff_calendar.py:
  fetches forexfactory.com/calendar?week=... HTML for N weeks ahead,
  parses calendar__table (date/time/currency/impact/event/forecast/previous),
  converts ET times to UTC, upserts into ff_calendar
- Daily scheduler in main.py: runs scrape_upcoming at startup (after 30s delay)
  then every 24h — no manual action needed
- New endpoints: POST /api/eco/ff-scrape?weeks=5, GET /api/eco/ff-scrape/status
- CalendarPage: "Scrape Upcoming (5w)" button (indigo) with polling + result

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OpenSquared
2026-06-26 17:14:00 +02:00
parent a69ced8fff
commit c36b2b2198
4 changed files with 305 additions and 5 deletions

View File

@@ -95,6 +95,22 @@ def startup():
from services.institutional_scheduler import start_institutional_scheduler
start_institutional_scheduler()
# Daily FF scraper — runs once at startup, then every 24h
import threading, time as _time
def _ff_scrape_loop():
_time.sleep(30) # let server fully boot first
while True:
try:
print("[Startup] FF scrape upcoming weeks …", flush=True)
from services.ff_calendar import scrape_upcoming
result = scrape_upcoming(weeks_ahead=5)
print(f"[FF scrape] done: {result.get('total_upserted', 0)} upserted", flush=True)
except Exception as e:
print(f"[FF scrape] loop error: {e}", flush=True)
_time.sleep(86400) # 24h
threading.Thread(target=_ff_scrape_loop, daemon=True).start()
# Auto-import Forex Factory CSV if file is present (force, then delete CSV)
import threading
def _auto_import_ff():