feat: multi-GPU support for 4x RTX 5090 (PCIe DDP, BF16)

Hardware analysis:
  4x RTX 5090 32GB without NVLink is fully sufficient.
  PCIe 5.0 all-reduce overhead <1% of step time for MacBERT-large (340M params).
  BF16 mixed precision gives ~2x throughput vs FP32 on 5090.

Module B (Detector) — full 4-GPU DDP via Accelerate:
  - DistributedSampler with per-epoch shuffling (correct DDP data split)
  - BF16 autocast via accelerator.mixed_precision
  - Gradient accumulation handled by accelerator.accumulate()
  - Only rank-0 saves checkpoints and logs to wandb
  - accelerator.gather_for_metrics() for correct multi-GPU validation
  - per_gpu_batch_size=32, effective_batch = 32×4 = 128

Module C (Intervention) — hybrid parallel strategy:
  - Stage 1 (BC warm-up): all 4 GPUs via Accelerate DDP
    TensorDataset broadcast from rank-0 to all processes
  - Stage 2 (PPO): GPU-0 only — env-agent loop is inherently sequential
  - Detector preprocessing: distributed across all 4 GPUs via shard split
    + all_gather_object to collect results on rank-0

Configs updated:
  detector_config.yaml:    per_gpu_batch_size=32, gradient_accumulation_steps=1,
                           mixed_precision=bf16, num_workers=4
  intervention_config.yaml: BC per_gpu_batch_size=256, PPO batch_size=256

Launch scripts added:
  scripts/run_detector.sh         — single command: 4-GPU detector training
  scripts/run_intervention.sh     — single command: hybrid BC+PPO training
  scripts/run_full_pipeline.sh    — end-to-end pipeline steps 1-5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-09 17:56:13 +08:00
parent 4a0e71fb23
commit b4be3983b7
7 changed files with 637 additions and 184 deletions

37
scripts/run_intervention.sh Executable file
View File

@@ -0,0 +1,37 @@
#!/bin/bash
# Train Module C (Intervention Policy) on 4x RTX 5090.
#
# Stage 1 — Behavior Cloning: all 4 GPUs (DDP, BF16)
# Stage 2 — PPO fine-tuning: GPU-0 only (inherently sequential)
#
# Usage:
# bash scripts/run_intervention.sh
# bash scripts/run_intervention.sh data/processed/train.jsonl
set -e
TRAIN_DATA="${1:-data/processed/train.jsonl}"
CONFIG="configs/intervention_config.yaml"
NUM_GPUS=4
echo "=============================================="
echo " CompanionGuard-RL — Module C: Intervention"
echo " Stage 1 (BC) : ${NUM_GPUS}x GPU (DDP, BF16)"
echo " Stage 2 (PPO) : GPU-0 only"
echo " Config : ${CONFIG}"
echo " Train data : ${TRAIN_DATA}"
echo "=============================================="
ACTUAL_GPUS=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
if [ "$ACTUAL_GPUS" -lt "$NUM_GPUS" ]; then
echo "[WARN] Expected ${NUM_GPUS} GPUs, found ${ACTUAL_GPUS}. Adjusting."
NUM_GPUS=$ACTUAL_GPUS
fi
accelerate launch \
--num_processes=${NUM_GPUS} \
--mixed_precision=bf16 \
--multi_gpu \
scripts/train_intervention.py \
--config ${CONFIG} \
--train-data ${TRAIN_DATA}