Jerry Wu

The bar to build great RL envs is growing rapidly. A great environment needs to: - be safe and aligned - challenge and expose realistic error modes - have no anonymization or data leakage - verification that doesn’t have false positives or negatives - well engineered tools - delivered with great research taste and service - and more Really hard and fun challenges. Join us if you’re interested in solving these problems in the future!

When building RL Environments the natural order of prioritization with app/tool building is: 1. Great feature rich open source app 2. Licensable and hosted closed source app (ex MSFT Office) 3. Cloned version of closed source apps 80% of the time we never make it to 2 or 3 because a great feature rich open source app has many of the capabilities needed to teach generalized understanding for RL (ex LibreOffice vs Excel). In a world of background agents doing work, the user doesn’t care what app (closed vs open) is being used for the action space. What they care about is that the final output is economically valuable. I fully expect to see this trend continue for many software verticals (ex. Medplum for EHRs, LibreOffice for Office Suite). Huge and unintended win for open source apps! More companies should lean in and actively make their software RL-consumable for this reason.

Jev (Noul) replacing LLM as a Judge functions in our Data Quality Pipelines for RL Environment creation. Still collecting results on overall alignment compared to prior methods but overall seems quite strong (and blazing fast). Easy to see the potential gains this primitive can have in the entire data industry. Hats off @CompleteSkeptic & team! The copy and paste instructions for Claude Code in the docs were also quite neat.

Although there may be some value in training on alignment specific RL Envs, my intuition is that the vast majority of "alignment" gains will come from hardening and increasing QA/QC across all RL Envs horizontally. Catching things like: - Impossible Tasks (ex. CyberGym Attack) - Escapable Sandboxes - Unintended code execution - Broken verifiers / non-comprehensive verification Will do more in aggregate than specifically trying to train on alignment RL Envs. Its like plugging one hole while many others exist. To put simply: this is a horizontal problem across the whole data/RL Env industry that every player needs to take super seriously. We all have a responsibility to play.