AI Quality and Automation Specialist

About the role

Patrianna, a fast-scaling product development company in Gibraltar working at the intersection of technology and entertainment, is looking for an AI Quality and Automation Specialist. You will make sure its AI agents and critical workflows work by building automated testing and evaluation, and using AI to test AI.

Key responsibilities

• Build and maintain API, integration and end-to-end regression tests that keep critical workflows stable.
• Evaluate AI agents on task completion, response quality, tool use and permissions, and measure how model, prompt and context changes affect results.
• Build quality checks into CI/CD pipelines and turn every production defect into a regression case.
• Use agents to explore workflows, generate tests and reproduce bugs, then rigorously check what they find.
• Share the tools and methods that work with the wider AI team and the central QA department.

Required profile

• Solid test automation and programming skills in TypeScript or Python.
• Hands-on experience with web apps, APIs, Git and CI/CD.
• A structured approach to judging AI output against clear, measurable criteria.
• A knack for quickly reproducing, isolating and pinning down bugs, with a sharp eye for root cause.
• A direct communicator who documents decisions and shares what works.

Required skills

• TypeScript or Python
• Git and CI/CD
• Bonus: agentic browser testing, LLM evaluation frameworks, Supabase, Vercel or GCP

What we offer

• Work at the cutting edge of AI agent evaluation, helping define how it is done.
• Real ownership within a team of three, with accountable, trusted work close to the impact.
• Progress over perfectionism, with achievable deadlines and a focus on shipping, learning and improving.
• An open culture with open Slack channels, outcomes tracked in Monday.com and video calls on Google Meet.
• A strict no-nonsense workplace and equal opportunities hiring based on skills, drive and ideas.