autobroadcaster/TESTING.md

# Movie Scheduler - Testing Environment

Complete Docker-based testing environment for the movie scheduler. Fire and forget - just run one command and everything is set up automatically!

## Quick Start

```bash
./start-test-environment.sh
```

That's it! The script will:
1. Build all Docker containers
2. Start PostgreSQL, NocoDB, and RTMP server
3. Initialize NocoDB with test data
4. Download a test video (Big Buck Bunny)
5. Start the scheduler

## Services

After running the startup script, you'll have:

### NocoDB Dashboard
- **URL**: http://localhost:8080
- **Email**: admin@test.com
- **Password**: admin123

Login and view the "Movies" table to see scheduled movies and their status.

### RTMP Streaming Server
- **Stream URL**: rtmp://localhost:1935/live/stream
- **Statistics**: http://localhost:8081/stat

This is where processed movies will be streamed.

### Movie Scheduler
Runs in the background, automatically:
- Syncs movies from NocoDB every 60 seconds (configurable)
- Monitors stream health every 10 seconds (configurable)
- Prepares videos 6 hours before scheduled time
- Streams at the scheduled time

## Testing the Workflow

### 1. View Initial Data

The test environment comes with a pre-loaded movie:
- **Title**: BigBuckBunny
- **Scheduled**: 5 minutes from startup

Open NocoDB to see this entry.

### 2. Monitor the Scheduler

Watch the scheduler logs:
```bash
docker compose logs -f scheduler
```

You'll see:
- Syncing from NocoDB
- Preparation jobs (if within 6 hours of scheduled time)
- Subtitle generation
- Video encoding
- Streaming start

### 3. Check the Database

Query the local SQLite database:
```bash
sqlite3 scheduler.db "SELECT nocodb_id, title, prep_status, play_status FROM jobs;"
```

View detailed logs:
```bash
sqlite3 scheduler.db "SELECT title, log FROM jobs WHERE title='BigBuckBunny';"
```

### 4. Watch the Stream

When a movie starts streaming, watch it with VLC:
```bash
vlc rtmp://localhost:1935/live/stream
```

Or use ffplay:
```bash
ffplay rtmp://localhost:1935/live/stream
```

### 5. Add Your Own Movies

#### Option A: Through NocoDB UI

1. Open http://localhost:8080
2. Navigate to the Movies table
3. Click "+ Add New Row"
4. Fill in:
   - **Title**: Name matching your video file
   - **Date**: Schedule time (ISO format: 2024-01-21T20:00:00)
   - Other fields are optional
5. Place video file in `raw_movies` volume

#### Option B: Add Video File Manually

Copy a video to the raw movies volume:
```bash
# Find the volume name
docker volume ls | grep raw_movies

# Copy your video
docker run --rm -v scheduler_raw_movies:/data -v $(pwd):/host alpine \
  cp /host/your-movie.mp4 /data/
```

Then add the entry in NocoDB with matching Title.

## Testing Restart & Recovery

### Test Overdue Job Processing

1. Add a movie scheduled for 15 minutes from now
2. Stop the scheduler:
   ```bash
   docker compose stop scheduler
   ```
3. Wait 20 minutes (past the prep time)
4. Restart the scheduler:
   ```bash
   docker compose start scheduler
   docker compose logs -f scheduler
   ```
5. Watch it immediately process the overdue prep job

### Test 15-Minute Grace Period for Streaming

1. Add a movie scheduled for 5 minutes from now (with prep already done)
2. Stop the scheduler after prep completes:
   ```bash
   docker compose stop scheduler
   ```
3. Wait 10 minutes (past streaming time but within 15-min grace period)
4. Restart and watch logs:
   ```bash
   docker compose start scheduler
   docker compose logs -f scheduler
   ```
5. Stream should start immediately with "Starting stream X.X minutes late" message

### Test Skipping Late Streams (>15 minutes)

1. Add a movie scheduled for 5 minutes from now
2. Let prep complete, then stop the scheduler:
   ```bash
   docker compose stop scheduler
   ```
3. Wait 20 minutes (past the 15-minute grace period)
4. Restart and watch logs:
   ```bash
   docker compose start scheduler
   docker compose logs -f scheduler
   ```
5. Job should be marked as 'skipped' with "more than 15 minutes late"

### Test Skipping Unprepared Expired Jobs

1. Add a movie scheduled for 5 minutes from now
2. Stop the scheduler before prep starts:
   ```bash
   docker compose stop scheduler
   ```
3. Wait 10 minutes (past both prep and streaming time)
4. Restart and watch logs:
   ```bash
   docker compose start scheduler
   docker compose logs -f scheduler
   ```
5. Job should be marked as 'skipped' (too late to prep)

### Test Recovery from Crash During Processing

1. Start processing a large video file
2. Kill the scheduler during encoding:
   ```bash
   docker compose kill scheduler
   ```
3. Restart:
   ```bash
   docker compose start scheduler
   ```
4. Watch it retry the entire operation from the beginning

## Testing Stream Watchdog

### Test Stream Crash and Auto-Restart

1. Start a stream:
   ```bash
   docker compose logs -f scheduler
   ```

2. Find the streaming ffmpeg process and kill it:
   ```bash
   # In another terminal
   docker compose exec scheduler ps aux | grep ffmpeg
   docker compose exec scheduler kill -9 <PID>
   ```

3. Watch the scheduler logs - within 10 seconds it should:
   - Detect the crash
   - Calculate elapsed playback time
   - Restart with seek to correct position
   - Log: "Restarting stream at position X.Xs"

### Test Stream Restart with Correct Position

1. Start a test stream with a longer video
2. Let it run for 2-3 minutes
3. Kill the ffmpeg process
4. Watch logs confirm restart with seek:
   ```
   Restarting stream for job XXX at position 180.0s (attempt 2)
   ```
5. Use VLC to confirm playback resumed at correct position

### Test Stream Completion Detection

1. Use a short test video (1-2 minutes)
2. Watch stream until completion
3. Check logs - should show:
   ```
   Stream completed successfully for job XXX
   ```
4. Check database - play_status should be 'done'

### Test Retry Limit

1. Break the RTMP server or use invalid RTMP URL
2. Start a stream - it will crash immediately
3. Watch the scheduler attempt 10 restarts
4. After 10 attempts, should mark as 'failed':
   ```
   Stream retry limit exceeded for job XXX
   ERROR: Stream failed after 10 restart attempts
   ```

### Test Network Interruption Recovery

1. Start a stream
2. Stop the RTMP server:
   ```bash
   docker compose stop rtmp
   ```
3. Watch stream fail and retry
4. Restart RTMP:
   ```bash
   docker compose start rtmp
   ```
5. Stream should resume at correct position

## Testing Error Handling

### Test Retry Logic

1. Stop the RTMP server:
   ```bash
   docker compose stop rtmp
   ```

2. Watch the scheduler handle the failure and retry:
   ```bash
   docker compose logs -f scheduler
   ```

3. Restart RTMP:
   ```bash
   docker compose start rtmp
   ```

### Test Missing File

1. Add a movie entry in NocoDB with a title that doesn't match any file
2. Set the Date to trigger immediately
3. Watch the scheduler log the error and mark it as failed

### Test Subtitle Generation Failure

1. Add an invalid video file (corrupted or wrong format)
2. Watch the scheduler attempt retries with exponential backoff

## Viewing Results

### Processed Videos

List final videos with burned-in subtitles:
```bash
docker run --rm -v scheduler_final_movies:/data alpine ls -lh /data
```

### Raw Videos

List source videos:
```bash
docker run --rm -v scheduler_raw_movies:/data alpine ls -lh /data
```

### RTMP Recordings (if enabled)

Recordings are saved to the rtmp_recordings volume:
```bash
docker run --rm -v scheduler_rtmp_recordings:/data alpine ls -lh /data/recordings
```

## Customization

### Change Test Video Scheduling

Edit `init-data.sh` before starting, around line 130:
```python
"Date": (datetime.now() + timedelta(minutes=5)).isoformat(),
```

Change `minutes=5` to adjust when the test movie is scheduled.

### Use Different NocoDB Credentials

Edit `docker-compose.yml`:
```yaml
environment:
  NOCODB_EMAIL: "your@email.com"
  NOCODB_PASSWORD: "yourpassword"
```

### Test with Real NocoDB Instance

Comment out the `nocodb` and `postgres` services in `docker-compose.yml` and update the `init` service environment:
```yaml
environment:
  NOCODB_URL: "https://your-nocodb-instance.com"
  NOCODB_EMAIL: "your@email.com"
  NOCODB_PASSWORD: "yourpassword"
```

## Troubleshooting

### Services Won't Start

Check if ports are already in use:
```bash
lsof -i :8080  # NocoDB
lsof -i :1935  # RTMP
lsof -i :8081  # RTMP stats
```

### Initialization Fails

View init logs:
```bash
docker compose logs init
```

Common issues:
- NocoDB not ready yet (increase wait time in init-data.sh)
- Network connectivity issues
- Insufficient disk space

### Scheduler Not Processing

Check if it's running:
```bash
docker compose ps scheduler
```

View logs for errors:
```bash
docker compose logs scheduler
```

Restart if needed:
```bash
docker compose restart scheduler
```

### Video Download Fails

The init script downloads Big Buck Bunny from Google's test video bucket. If this fails:
1. Check internet connectivity
2. Manually download and copy to raw_movies volume
3. Or use your own test video

### VAAPI Not Available

The Docker container runs without GPU access by default (CPU encoding fallback). To enable VAAPI:

Edit `docker-compose.yml` scheduler service:
```yaml
scheduler:
  devices:
    - /dev/dri:/dev/dri
  group_add:
    - video
```

## Cleanup

### Stop Everything
```bash
docker compose down
```

### Remove All Data (including volumes)
```bash
docker compose down -v
```

This removes:
- All containers
- All volumes (database, videos, recordings)
- Network

### Keep Database, Remove Containers
```bash
docker compose down
# Volumes persist - next startup will reuse existing data
```

## Architecture

```
┌─────────────┐
│  PostgreSQL │
└──────┬──────┘
       │
┌──────▼──────┐      ┌─────────────┐
│   NocoDB    │◄─────┤    Init     │  Downloads test video
└──────┬──────┘      │  Container  │  Creates test data
       │             └─────────────┘
       │
┌──────▼──────┐
│  Scheduler  │  Reads from NocoDB
└──────┬──────┘  Generates subtitles
       │         Encodes videos
       │         Manages streaming
       │
┌──────▼──────┐
│ RTMP Server │  Streams processed videos
└─────────────┘
```

## Production Considerations

This test environment uses:
- Embedded PostgreSQL (fine for testing)
- Local volumes (fine for testing)
- Simple authentication (change for production)
- No SSL/TLS (add for production)
- CPU-only encoding (enable VAAPI for production)

For production deployment:
1. Use external PostgreSQL database
2. Configure proper authentication and secrets
3. Enable SSL/TLS for NocoDB
4. Mount persistent storage for videos
5. Enable GPU acceleration (VAAPI)
6. Set up proper monitoring and logging
7. Configure backup for NocoDB database

## Next Steps

Once you've tested the environment:
1. Review the logs to understand the workflow
2. Try adding your own movies
3. Test error scenarios
4. Adjust timing and configuration
5. Deploy to production with proper setup

## Support

For issues with:
- **Scheduler code**: Check `agent.py` and logs
- **Docker setup**: Check `docker-compose.yml` and service logs
- **NocoDB**: Visit https://docs.nocodb.com
- **RTMP streaming**: Check nginx-rtmp documentation

## License

This test environment is provided as-is for development and testing purposes.