Ruby Engineer – LLM Evaluation (Remote, $15-$20/hr)

About the role

We are seeking a senior Ruby engineer to evaluate large language model (LLM) performance on real‑world coding tasks. This short‑term contract role involves analyzing open‑source Ruby projects, running tests, and collaborating with researchers.

Key responsibilities

• Analyze and triage GitHub issues across well‑maintained, widely used open‑source repositories.
• Set up and configure repositories, including Docker‑based development environments.
• Evaluate unit test coverage, testing practices, and overall code quality.
• Modify, build, run, and test real‑world Ruby codebases locally.
• Assess LLM performance in bug‑fixing and code modification scenarios.
• Collaborate with researchers to identify repositories and issues that present challenging software engineering tasks.

Required profile

• Minimum 2 years of professional experience with Ruby.
• Strong proficiency with Ruby, Git, and Docker.
• Ability to understand and navigate complex codebases.
• Experience contributing to well‑maintained, widely used open‑source repositories, preferably with 5,000+ GitHub stars.
• Comfortable running, modifying, debugging, and testing software projects locally.

Required skills

• Ruby
• Git
• Docker