Reducing Abuse in Voice Chat
ML-powered content moderation at scale
Challenge
Open voice lobbies experienced a 45% rate of toxic incidents, with traditional text-based moderation tools unable to process real-time audio content. User reports were reactive, often coming hours after incidents.
Process
We developed a hybrid approach combining automated detection with human-in-the-loop validation, rolling out in three phases over 6 months.
Research & Discovery
Analyzed 10,000 hours of reported voice incidents to identify patterns and build training data
ML Model Development
Built custom speech-to-text pipeline with toxicity classification using transformer models
Human Validation Layer
Created review queue for borderline cases with expert moderators
Gradual Rollout
A/B tested with 10% of users, then expanded based on performance metrics
Solution
A multi-stage pipeline that converts voice to text, analyzes sentiment and toxicity, and triggers appropriate interventions based on confidence scores.
Technologies Used
Outcome
The system now processes over 1M minutes of voice chat daily, creating safer spaces for community interaction while maintaining the spontaneity of voice communication.
Reflections
Context matters more than individual words - slang and gaming terminology required specialized training data
User education campaigns were as important as the technical solution in changing behavior
Transparency in moderation decisions increased user trust, even when actions were taken
Next Steps
Expanding to cover non-English languages and exploring real-time intervention techniques that don't disrupt gameplay flow.
Supporting Materials
Voice moderation pipeline architecture
Incident reduction over 6-month rollout