English-Speaking AI Data Evaluation Specialist | Large Language Models
At a glance
Mid-level AI Engineering role at Workster Jobs. Lisbon.
Pay not stated AI Engineering salary
aimlroles summary, based on the employer's posting.
What you'll do
- Review model-generated answers against detailed rules for quality, relevance, accuracy, safety and policy compliance
- Label outputs and apply annotation instructions consistently across different topics and content types
- Spot edge cases or model problems, escalating issues when needed and using feedback to support model training
- Work with structured text and datasets while meeting accuracy, quality and productivity targets
- Protect confidential information and follow project security standards and internal procedures
What you bring
- Native-level English fluency, with at least B2 command of the language
- A strong grasp of generative AI, large language models and evaluation methods
- Analytical judgment and care with detail, plus the ability to follow complex written policies consistently
- Reading and writing skills suited to assessing structured text, datasets and AI-generated material
Who this fits
This role suits someone focused on careful, objective evaluation who can handle confidential material and work to performance targets. It is office-based in Lisbon, with rotating shifts across the full week and two rotating days off.
From the employer
Workster is partnering with a leading global customer experience company to recruit English-speaking AI Data Evaluation Specialists for an innovative artificial intelligence project based in Lisbon, Portugal.
If you’re passionate about AI, language technology, and data quality, this is a unique opportunity to contribute to the development of next-generation Large Language Models (LLMs). You’ll work with cutting-edge AI systems by reviewing, evaluating, and improving AI-generated responses, helping create safer, more accurate, and more reliable conversational experiences for millions of users worldwide.
Your Role
Review and evaluate AI-generated responses using detailed annotation and quality guidelines
Label and annotate language model outputs with accuracy and consistency
Assess AI-generated content for quality, relevance, factual accuracy, safety, and policy compliance
Apply complex annotation guidelines across a wide variety of scenarios and content types
Identify inconsistencies, edge cases, and potential model issues, escalating when appropriate
Analyze structured text, datasets, and AI-generated content to support model improvement
Deliver high-quality work while consistently meeting productivity, quality, and accuracy targets
Maintain strict confidentiality and comply with all project standards, security requirements, and internal procedures
Contribute directly to the ongoing training and improvement of advanced AI systems through detailed evaluations and feedback
24/7 rotating schedule (Monday to Sunday) with 2 rotational days off
Your Qualifications
Native-level fluency in English
Good command of English (B2 level or above)
Strong understanding of Generative AI, Large Language Models (LLMs), and AI evaluation methodologies
Previous experience in AI data annotation, LLM evaluation, Reinforcement Learning from Human Feedback (RLHF), Supervised Fine-Tuning (SFT), Quality Assurance, Content Labeling, Trust & Safety, or similar analytical roles is highly desirable
Excellent analytical thinking and attention to detail
Ability to interpret and consistently apply complex written guidelines and policies
Strong reading comprehension and written communication skills
Comfortable working with structured text, datasets, and AI-generated content
Ability to work efficiently in a fast-paced, quality-driven environment while meeting performance targets
Professional, objective, and capable of handling confidential information with discretion
The Offer
Office-based position in Lisbon, Portugal
Comprehensive onboarding and specialized training in AI evaluation processes
Opportunity to work on one of the world’s most advanced artificial intelligence projects
International and multicultural working environment
Career development opportunities within the rapidly growing AI industry
Private health insurance
Meal allowance
Ongoing learning and professional development
Modern office facilities with a collaborative team culture
Unique opportunity to shape the future of generative AI and language technologies