# Movie Scheduler

Automated movie scheduler that reads from NocoDB, generates subtitles with whisper.cpp, burns them into videos with VAAPI encoding, and streams at scheduled times.

## Quick Links

- **[Production Deployment Guide](PRODUCTION.md)** - Complete guide for production setup
- **[Testing Guide](TESTING.md)** - Docker-based test environment
- **[Development Setup](#installation)** - Local development setup

## Features

- **NocoDB Integration**: Syncs job schedules from NocoDB database
- **Automatic Subtitle Generation**: Uses whisper.cpp to generate SRT subtitles
- **Hardware-Accelerated Encoding**: Burns subtitles into videos using VAAPI (h264_vaapi)
- **Scheduled Streaming**: Streams videos to RTMP server at scheduled times
- **Robust Error Handling**: Automatic retries with exponential backoff
- **Comprehensive Logging**: Tracks all operations in database log column
- **Process Management**: Properly tracks and manages streaming processes

## Requirements

- Python 3.7+
- `requests` library
- `whisper.cpp` installed and available in PATH
- `ffmpeg` with VAAPI support
- VAAPI-capable hardware (AMD/Intel GPU)
- NocoDB instance with scheduled jobs table

## Installation

1. Clone or download this repository

2. Install Python dependencies:
   ```bash
   pip install requests
   ```

3. Ensure whisper.cpp is installed:
   ```bash
   # Follow whisper.cpp installation instructions
   # Make sure the binary is in your PATH
   which whisper.cpp
   ```

4. Verify FFmpeg has VAAPI support:
   ```bash
   ffmpeg -hwaccels | grep vaapi
   ```

## Configuration

1. Copy the example environment file:
   ```bash
   cp .env.example .env
   ```

2. Edit `.env` and set the required variables:
   ```bash
   # Required
   export NOCODB_URL="https://nocodb/api/v2/tables/XXXX/records"
   export NOCODB_TOKEN="your_token_here"
   export RTMP_SERVER="rtmp://your_server/live"

   # Optional (defaults provided)
   export RAW_DIR="/root/surowe_filmy"
   export FINAL_DIR="/root/przygotowane_filmy"
   export WHISPER_MODEL="/root/models/ggml-base.bin"
   export VAAPI_DEVICE="/dev/dri/renderD128"
   export STREAM_GRACE_PERIOD_MINUTES="15"
   export NOCODB_SYNC_INTERVAL_SECONDS="60"
   export WATCHDOG_CHECK_INTERVAL_SECONDS="10"
   ```

3. Load the environment variables:
   ```bash
   source .env
   ```

## NocoDB Table Structure

Your NocoDB table should have the following fields:

- `Id` - Unique identifier (Text/Primary Key)
- `Title` - Movie title matching filename (Text)
- `Date` - Scheduled run time (DateTime in ISO format)

The scheduler will automatically create additional columns in the local SQLite database:
- `prep_at` - Preparation time (6 hours before run_at)
- `prep_status` - Preparation status (pending/done/failed)
- `play_status` - Streaming status (pending/done/failed)
- `raw_path` - Path to source video file
- `final_path` - Path to converted video with subtitles
- `log` - Detailed operation log

## Usage

### Running the Scheduler

```bash
# Ensure environment variables are set
source .env

# Run the scheduler
python agent.py
```

The scheduler will:
1. Sync jobs from NocoDB every 60 seconds (configurable)
2. Check stream health every 10 seconds (configurable)
3. Prepare videos 6 hours before their scheduled time:
   - Find matching video file in RAW_DIR
   - Generate subtitles with whisper.cpp
   - Encode video with burned subtitles using VAAPI
4. Stream videos at their scheduled time to RTMP server

### Stopping the Scheduler

Press `Ctrl+C` to gracefully stop the scheduler. It will:
- Stop any active streaming process
- Close the database connection
- Exit cleanly

### Restart & Recovery Behavior

The scheduler handles restarts, power outages, and downtime gracefully:

**On Startup:**
- ✅ Checks for overdue prep jobs and processes them immediately
- ⚠️ Skips jobs where the streaming time has already passed
- 📊 Logs recovery status (overdue/skipped jobs found)

**Grace Period for Late Streaming:**
- ⏰ If prep is done and streaming time is overdue by **up to 15 minutes**, will still start streaming
- ⚠️ Jobs more than 15 minutes late are marked as 'skipped'
- 📝 Late starts are logged with exact delay time

**Recovery Scenarios:**

1. **Short Outage (prep time missed, but streaming time not reached)**
   - Movie scheduled for 21:00 (prep at 15:00)
   - System down from 14:00-16:00
   - ✅ Restarts at 16:00: immediately preps the movie
   - ✅ Streams normally at 21:00

2. **Late Start Within Grace Period (< 15 minutes)**
   - Movie scheduled for 21:00 (prep completed at 15:00)
   - System down from 20:00-21:10
   - ✅ Restarts at 21:10: starts streaming immediately (10 minutes late)
   - 📝 Logged: "Starting stream 10.0 minutes late"

3. **Too Late to Stream (> 15 minutes)**
   - Movie scheduled for 21:00 (prep completed)
   - System down from 20:00-21:20
   - ⚠️ Restarts at 21:20: marks streaming as 'skipped' (>15 min late)
   - 📝 Logged: "Streaming skipped - more than 15 minutes late"

4. **Long Outage (both prep and streaming times passed, >15 min)**
   - Movie scheduled for 21:00 (prep at 15:00)
   - System down from 14:00-22:00
   - ⚠️ Restarts at 22:00: marks entire job as 'skipped'
   - 📝 Logged: "Job skipped - streaming time already passed"

5. **Crash During Processing**
   - System crashes during subtitle generation or encoding
   - ✅ On restart: retries from the beginning with full retry logic
   - All operations are idempotent (safe to re-run)

**Database Persistence:**
- Job status stored in `scheduler.db` survives restarts
- Completed preps are never re-done
- Failed jobs stay marked as 'failed' (can be reset manually to 'pending' if needed)

## File Naming

Place your raw video files in `RAW_DIR` with names that contain the title from NocoDB.

Example:
- NocoDB Title: `"The Matrix"`
- Valid filenames: `The Matrix.mkv`, `the-matrix-1999.mp4`, `[1080p] The Matrix.avi`

The scheduler uses glob matching to find files containing the title.

## Streaming Watchdog

The scheduler includes an active watchdog that monitors streaming:

**Stream Monitoring (checked every 10 seconds):**
- ✅ Detects if ffmpeg stream crashes or exits unexpectedly
- ✅ Automatically restarts stream at the correct playback position
- ✅ Calculates elapsed time since stream start
- ✅ Seeks to correct position using `ffmpeg -ss` flag
- ✅ Limits restarts to 10 attempts to prevent infinite loops
- ✅ Marks stream as 'done' when video completes normally
- ✅ Marks stream as 'failed' after 10 failed restart attempts

**How It Works:**
1. Stream starts and status set to 'streaming' (not 'done')
2. Watchdog checks every 10 seconds if stream process is running
3. If crashed: calculates elapsed time and restarts with seek
4. If completed normally (exit code 0): marks as 'done'
5. If video duration exceeded: marks as 'done'

**Example:**
```
Stream starts at 21:00:00
Stream crashes at 21:05:30 (5.5 minutes in)
Watchdog detects crash within 10 seconds
Restarts stream with: ffmpeg -ss 330 -i video.mp4
Stream resumes from 5:30 mark
```

## Error Handling

The scheduler includes robust error handling:

- **Automatic Retries**: Failed operations are retried up to 3 times with exponential backoff
- **Database Logging**: All operations and errors are logged to the `log` column
- **Status Tracking**: Jobs are marked as 'failed' if all retries are exhausted
- **File Verification**: Checks that subtitle files and encoded videos were created successfully

## Monitoring

### Check Job Status

Query the SQLite database:
```bash
sqlite3 scheduler.db "SELECT nocodb_id, title, prep_status, play_status FROM jobs;"
```

### View Job Logs

```bash
sqlite3 scheduler.db "SELECT nocodb_id, title, log FROM jobs WHERE nocodb_id='YOUR_JOB_ID';"
```

### Console Output

The scheduler logs to stdout with timestamps:
- INFO: Normal operations
- WARNING: Retry attempts
- ERROR: Failures

## Troubleshooting

### Configuration Validation Failed
- Ensure all required environment variables are set
- Check that NOCODB_URL is accessible
- Verify NOCODB_TOKEN has proper permissions

### No Files Found Matching Title
- Check that video files exist in RAW_DIR
- Ensure filenames contain the NocoDB title (case-insensitive glob matching)
- Review the log column in the database for exact error messages

### Subtitle Generation Failed
- Verify whisper.cpp is installed: `which whisper.cpp`
- Check WHISPER_MODEL path is correct
- Ensure the model file exists and is readable

### Video Encoding Failed
- Confirm VAAPI is available: `ffmpeg -hwaccels | grep vaapi`
- Check VAAPI device exists: `ls -l /dev/dri/renderD128`
- Verify subtitle file was created successfully
- Review FFmpeg error output in the log column

### Streaming Failed
- Test RTMP server connectivity: `ffmpeg -re -i test.mp4 -c copy -f flv rtmp://your_server/live`
- Ensure final video file exists
- Check network connectivity to RTMP server

## Development

### Database Schema

```sql
CREATE TABLE jobs (
    nocodb_id TEXT PRIMARY KEY,
    title TEXT,
    run_at TIMESTAMP,
    prep_at TIMESTAMP,
    raw_path TEXT,
    final_path TEXT,
    prep_status TEXT,  -- pending/done/failed/skipped
    play_status TEXT,  -- pending/streaming/done/failed/skipped
    log TEXT,
    stream_start_time TIMESTAMP,  -- When streaming started (for restart seek)
    stream_retry_count INTEGER    -- Number of stream restart attempts
);
```

**Status Values:**
- `play_status='streaming'`: Stream is actively running
- `play_status='done'`: Stream completed successfully
- `play_status='failed'`: Stream failed after 10 restart attempts
- `play_status='skipped'`: Too late to stream (>15 min past scheduled time)

### Customization

Use environment variables to customize:
- `NOCODB_SYNC_INTERVAL_SECONDS` - How often to check NocoDB for new jobs (default: 60)
- `WATCHDOG_CHECK_INTERVAL_SECONDS` - How often to monitor streams (default: 10)
- `STREAM_GRACE_PERIOD_MINUTES` - Late start tolerance (default: 15)

Edit `agent.py` to customize:
- Retry attempts and delay (agent.py:99, 203, 232)
- Preparation time offset (currently 6 hours, see sync function)
- Stream restart limit (agent.py:572)
- Subtitle styling (agent.py:246)
- Video encoding parameters (agent.py:248-250)

## License

This project is provided as-is without warranty.