Remote
Founding Engineer (AI + Backend)
- Salary
- $4,000,000–4,500,000
- Eligible batches
- 2024 · 2025 · 2026
- Location
- Remote
About the role
Openbenchmarks is seeking a founding engineer to shape its AI‑driven benchmarking platform. Based remotely in India, you will design domain‑specific evaluations, build large‑scale autonomous systems, and research model behaviour to help agents choose the right tools.
Responsibilities
- Design and implement end‑to‑end benchmarks that measure agent‑relevant metrics.
- Develop infrastructure for fully autonomous evaluations, including vendor API orchestration and LLM‑as‑judge pipelines.
- Create scoring, metric computation, and drift‑detection mechanisms for continuous model monitoring.
- Generate synthetic data and conduct research on model decision‑making.
- Collaborate with a distributed team of researchers and engineers across SF and India.
Free interview prep
Prepare for the interview
Skills
- Strong experience with APIs and backend services.
- Familiarity with large language models and LLM‑based evaluation pipelines.
- Expertise in building scalable data pipelines and evaluation infrastructure.
- Knowledge of scoring, metric computation, and drift detection techniques.
- Ability to design domain‑specific benchmarks and conduct model‑behavior research.
Resume keywords
- API
- LLM
- Benchmarking
- Evaluation
- Data pipelines
- Scoring
- Metric computation
- Drift detection
You'll apply on ycombinator.com
Apply link checked by CareersAt.Tech and re-tested nightly. You never pay to apply. Report listing
About Openbenchmarks
Openbenchmarks provides independent benchmarking services that evaluate how APIs, tools and AI agents perform specific tasks. It helps both humans and autonomous agents choose the right software by publishing open datasets and evaluation harnesses. The company works with fast‑growing AI‑first firms and is backed by Y Combinator.
1-10 employees · HQ San Francisco, United States
More about Openbenchmarks
