Your Growth starts here!

Software Engineer AI Evaluation & Automation : 26-02760

San Diego, CA

Posted: 10/05/2026 Job Category: Software Engineering & Development Job Number: 26-02760 Country: US

Job Description

Primary Skills: Python (advanced), Git (advanced), AI coding tools (advanced), API Integration (intermediate), Docker (Proficient)
Contract Type: W2
Duration: 6 months (Possible extension)
Location: San Diego, CA (#LI-Remote PST)
Pay Range: $65.00- $73.00 per hour
#LP

Job Summary:
This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks.

Key Responsibilities:
  • Develop and integrate evaluation systems for AI development tools.
  • Automate benchmark runs with versioned, reproducible environments.
  • Calibrate evaluation metrics against human judgment for accuracy.
  • Support performance benchmarking across quality and efficiency.
  • Analyze and improve the reliability of evaluation workflows.
Must-Have Skills:
  • Experience with AI coding tools and harnesses.
  • Proven evaluation and benchmarking experience.
  • Strong proficiency in Git and containerization with Docker.
  • Python, Java or JavaScript
Industry Experience:
Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.

ABOUT AKRAYA
Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)!

Industry Leaders in Tech Staffing
As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!

Join Akraya Today!
Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!
Apply Online Apply with LinkedIn Apply with Facebook Apply with X

Send an email reminder to:

Share This Job:

Related Jobs:

Login to save this search and get notified of similar positions.

About San Diego, CA

Ready to embark on an exciting career journey in sunny Southern California? Explore our job opportunities in the thriving city of San Diego! Known for its stunning beaches, perfect weather, and vibrant culture, San Diego offers a unique combination of career growth and quality of life. With renowned attractions like Balboa Park, the San Diego Zoo, and the USS Midway Museum, along with a diverse culinary scene and a plethora of art galleries and theaters like the La Jolla Playhouse, there's always something new to experience. Whether you're a sports enthusiast looking to cheer for the Padres at Petco Park or seeking outdoor adventures at Torrey Pines State Natural Reserve, San Diego has something for everyone. Begin your job search today and discover why this picturesque city is the perfect place to elevate your career!