Senior Software Engineer – Open Source & SWE-Bench Evaluation

The role

You will review and evaluate real-world software engineering tasks derived from open-source repositories. This involves assessing coding tasks based on GitHub issues and pull requests for technical soundness and reproducibility.

What you will do

• Reviewing coding tasks derived from real GitHub issues and pull requests
• Assessing whether problem statements and success criteria are clear and complete
• Evaluating unit tests for correctness, coverage, and robustness
• Identifying flaky tests, missing dependencies, version conflicts, and environment issues
• Determining whether tasks can be reliably reproduced across environments
• Assessing the real-world difficulty and complexity of each task
• Providing clear recommendations on whether tasks should be accepted, improved, or excluded

Required qualifications

• 3+ years of professional software engineering experience
• Strong experience working with large, multi-file codebases
• Experience reviewing pull requests, debugging issues, and maintaining production code
• Strong understanding of unit testing and test coverage
• Ability to evaluate whether tests correctly validate a solution without unnecessarily restricting implementation approaches
• Experience with dependency management, environment setup, and reproducibility
• Strong understanding of Git and GitHub-based development workflows
• Ability to analyze complex technical problems and provide clear written feedback

Nice to have

• Contributions to or maintenance of open-source projects
• Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks
• Experience with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest
• Experience with Docker, CI/CD, pip, conda, or dependency pinning
• Knowledge of test fixtures, test isolation, or property-based testing
• Experience designing technical assessments or reviewing coding challenges
• Experience with AI/ML evaluation, data curation, RLHF, or benchmark development

About the company

Anyone AI is recruiting experienced Software Engineers for a specialized project focused on reviewing and evaluating real-world software engineering tasks derived from open-source repositories.