Functie
Data Scientist (Responsible GenAI)
Level: Medior–Senior
Working arrangement: Hybrid, with office attendance on Thursdays in Utrecht.
The Role
As a Data Scientist in the Responsible GenAI team, you will build evaluation capabilities and guardrails that help teams understand, measure and improve the quality and safety of their GenAI solutions.
You will work at the intersection of data science, software engineering and responsible AI. You’ll translate real-world use cases and risks into robust evaluation methods, reusable tools and practical controls. Your work may span assistants, report generation, document processing and agentic applications.
This is a hands-on development role requiring strong statistical judgement and production experience. You’ll design sound experiments, analyse data and metrics critically, and make clear what the evidence does—and does not—support. You’ll turn experiments into maintainable, tested, production-ready code and work with engineers to integrate it into deployed solutions. The focus is on putting evaluation and safeguards into practice, rather than training foundation models or conducting standalone research.
Your Responsibilities
Build reusable evaluation frameworks and tools for repeatable testing, comparison and regression analysis across GenAI use cases.
Develop and validate evaluation metrics using deterministic checks, model-based evaluation and human assessments. Assess metric reliability, uncertainty and whether metrics measure what they are intended to measure.
Design statistically sound experiments, test datasets and evaluation scenarios with use-case teams and domain experts. Consider sampling strategies, representativeness, ground truth, edge cases and challenging inputs.
Develop and assess guardrails using appropriate methods, including rule-based controls, LLM-as-a-judge approaches and trained ML classifiers. Address risks such as sensitive-data exposure, unsupported outputs, prompt injection and unintended system behaviour.
Measure guardrail effectiveness and trade-offs, including precision and recall, detection thresholds, class imbalance, missed detections, false positives, robustness, drift, latency and cost.
Deliver maintainable, tested, production-ready code and work with engineers to integrate evaluations and controls into CI/CD, APIs, pipelines, deployment, monitoring and production workflows.
Translate use-case needs into actionable evaluation criteria, explain findings and limitations, and turn delivery experience into reusable methods, documentation and tooling.
What You’ll Bring
GenAI systems
Practical experience with LLM-based applications and an understanding of their limitations and failure modes.
Familiarity with retrieval-augmented generation, prompting, grounding and context management.
Ability to assess GenAI outputs against explicit criteria and understand how changes to models, prompts, data or application logic can affect quality and safety.
Production-ready development and integration
Strong Python skills and experience turning experiments and prototypes into maintainable, tested, production-ready code—not just exploratory notebooks.
Experience integrating data-science components with APIs, structured and unstructured data, and automated data-processing or evaluation pipelines.
Hands-on experience with version control, automated testing, CI/CD and reproducible experimentation.
Production experience that enables you to understand deployment, monitoring and operational workflows, diagnose issues and deliver code engineers can integrate and operate. ML or platform engineering specialisation is not required.
Statistics, measurement and experimentation
Experience designing sound evaluations and experiments, and critically analysing metrics and datasets.
Applied statistical knowledge of sampling, dataset representativeness, uncertainty, metric reliability and experimental design, including the limitations of aggregate scores.
Experience validating automated evaluations against human assessments, assessing reliability and investigating disagreement.
Ability to distinguish between weaknesses in a GenAI solution, its evaluation dataset and the evaluator, and explain what conclusions the data supports.
Guardrail development and validation
Experience assessing and selecting guardrail methods, including rule-based controls, LLM-as-a-judge approaches and trained ML classifiers, with a clear understanding of their suitability and limitations.
Ability to turn risks or expected behaviours into testable criteria and measurable controls, and to train or evaluate a classifier when appropriate.
Applied knowledge of precision and recall, detection thresholds, false positives, false negatives and class imbalance, and the ability to balance safety, usability and operational constraints.
Experience testing controls in realistic scenarios and assessing how data or behaviour drift affects their performance over time.
Communication and collaboration
Ability to explain evaluation results, methodological choices and limitations to engineers, product managers and other stakeholders.
Experience working with use-case teams and domain experts to define what good performance means in context.
Ability to write clear technical proposals, evaluation reports and practical documentation.
Ability to turn ambiguous requirements into concrete experiments and implementation steps.
Nice to Have
Experience with agentic evaluation, including tool selection, multi-step workflows and task completion.
Experience testing privacy controls, prompt-injection defences or other GenAI safeguards.
Experience with controlled adversarial testing or robustness assessment.
Experience with evaluation tooling, observability or continuous evaluation in production.
Experience working in a regulated environment and producing traceable validation evidence.
Who Will Thrive Here
You enjoy turning “Can we trust this result?” into a concrete test, a measurable criterion and a useful tool. You’re curious about why AI systems—and their evaluators—behave the way they do.
You combine statistical rigour with a builder’s mindset: you enjoy designing methods, analysing data and metrics critically, and turning experiments into maintainable, tested code that holds up in production. You value evidence over impressive-looking scores, make limitations explicit and work closely with others to solve real problems. You want your work to help multiple teams build better GenAI solutions.
Eisen
- Practical experience with LLM-based applications and an understanding of their limitations and failure modes
- Familiarity with retrieval-augmented generation, prompting, grounding and context management
- Ability to assess Gen
- AI outputs against explicit criteria rather than relying only on intuition
- Understanding of how changes to models, prompts, data or application logic can affect quality and safety
- Strong Python skills and demonstrated experience turning experiments and prototypes into maintainable, tested, production-ready code—not only exploratory notebooks
- Practical experience integrating data-science components with APIs, structured and unstructured data, and automated data-processing or evaluation pipelines
- Hands-on experience with version control, automated testing, CI/CD and reproducible experimentation
- Sufficient production experience to understand deployment, monitoring and production workflows, diagnose issues and deliver code that engineers can integrate and operate. ML or platform engineering specialisation is not required
- Demonstrated experience designing sound evaluations and experiments, and critically analysing metrics and datasets—not just applying evaluation frameworks
- Strong applied statistical knowledge of sampling, dataset representativeness, uncertainty, metric reliability and experimental design, including the limitations of aggregate scores
- Demonstrated experience validating automated evaluations against human assessments, assessing reliability and investigating disagreement
- Ability to distinguish between weaknesses in the Gen
- AI solution, the evaluation dataset and the evaluator itself, and explain which conclusions the data does—and does not—support
- Practical experience assessing and selecting guardrail methods, including rule-based controls, LLM-as-a-judge approaches and trained ML classifiers, with clear reasoning about their suitability and limitations in context
- Ability to translate a risk or expected behaviour into testable criteria and measurable controls, and to train or evaluate a classifier when that is the appropriate approach
- Applied knowledge of precision and recall, detection thresholds, false positives, false negatives and class imbalance, and the ability to balance safety, usability and operational constraints
- Experience testing controls across realistic scenarios, critically checking assumptions and assessing how data or behaviour drift affects guardrail performance over time
Gunningscriteria
- Experience with agentic evaluation, including tool selection, multi-step workflows and task completion
- Experience testing privacy controls, prompt-injection defences or other Gen
- AI safeguards
- Experience with controlled adversarial testing or robustness assessment
- Experience with evaluation tooling, observability or continuous evaluation in production
- Experience working in a regulated environment and producing traceable validation evidence
Extra informatie
Competenties
Explain evaluation results, methodological choices and limitations to engineers, product managers and stakeholders
Work with use-case teams and domain experts to define what good performance means in context
Write clear technical proposals, evaluation reports and practical documentation
Turn ambiguous requirements into concrete experiments and implementation steps
You enjoy turning the question “Can we trust this result?” into a concrete test, a measurable criterion and a useful tool
You are curious about why AI systems—and their evaluators—behave the way they do
You combine statistical rigour with a builder’s mindset
You enjoy designing methods, critically analysing data and metrics, and turning experiments into maintainable, tested code that holds up in production
You value evidence over impressive-looking scores, make limitations explicit and work closely with others to solve real problems
You want your work to help multiple teams build better Gen
AI solutions, rather than improve a single chatbot
Je sollicitatie dient uiterlijk 20 november 2026 om 12:00 uur door ons te zijn ingediend, voorzien van een motivatie (waarin wordt toegelicht hoe aan de eisen en gunningscriteria wordt voldaan) en twee referenties. Wij ontvangen je gegevens graag minimaal één werkdag vóór de sluitingsdatum.
