Functie
Data Scientist (Responsible GenAI)
Location: Utrecht, hybrid working
Assignment duration: 6 months, with possible extension
Working language: English
About the role
As a Data Scientist in the Responsible GenAI team, you will build evaluation capabilities and safeguards that help teams understand, measure and improve the quality and safety of their GenAI solutions. You will work at the intersection of data science, software engineering and responsible AI, turning real-world use cases and risks into robust evaluation methods, reusable tools and practical controls.
Your work may cover assistants, report generation, document processing and emerging agentic applications. This is a hands-on development role requiring sound statistical judgement and production experience. You will design experiments, critically analyse data and metrics, and turn your findings into maintainable, tested code that engineers can integrate into deployed solutions. The focus is on making evaluation and safeguards work in practice—not on training foundation models or conducting standalone research.
Your responsibilities
Build reusable evaluation frameworks and tools for repeatable testing, comparison and regression analysis across GenAI use cases.
Develop and validate evaluation metrics, combining deterministic checks, model-based evaluation and human assessment. Assess reliability, uncertainty and whether metrics measure what they are intended to measure.
Design statistically sound experiments, datasets and evaluation scenarios with use-case teams and domain experts. Consider sampling, representativeness, ground truth, edge cases and challenging inputs.
Develop and assess guardrails using suitable methods, such as rule-based controls, LLM-as-a-judge approaches and trained ML classifiers.
Measure guardrail effectiveness and trade-offs, including precision and recall, thresholds, class imbalance, false positives, missed detections, robustness, drift, latency and cost.
Deliver maintainable, tested, production-ready code and work with engineers to integrate evaluations and controls into CI/CD, APIs, pipelines, deployment, monitoring and production workflows.
Translate use-case needs into actionable evaluation criteria, explain findings and limitations, and turn delivery experience into reusable methods, documentation and tooling.
Required skills and experience
GenAI systems: Practical experience with LLM-based applications and an understanding of their limitations and failure modes. Familiarity with retrieval-augmented generation, prompting, grounding and context management. Ability to assess outputs against explicit criteria and understand how changes to models, prompts, data or application logic affect quality and safety.
Production-ready development: Strong Python skills and experience turning experiments and prototypes into maintainable, tested, production-ready code. Practical experience integrating data-science components with APIs, structured and unstructured data, and automated processing or evaluation pipelines. Hands-on experience with version control, automated testing, CI/CD and reproducible experimentation.
Statistics and experimentation: Experience designing sound evaluations and experiments and critically analysing metrics and datasets. Strong applied knowledge of sampling, representativeness, uncertainty, metric reliability and experimental design, including the limitations of aggregate scores. Experience validating automated evaluations against human assessments and investigating disagreements.
Evaluation judgement: Ability to distinguish between weaknesses in a GenAI solution, its evaluation dataset and the evaluator, and to explain what conclusions the evidence does and does not support.
Guardrails: Practical experience selecting and assessing rule-based controls, LLM-as-a-judge approaches and trained ML classifiers. Ability to turn risks or expected behaviours into testable criteria, and to train or evaluate a classifier when appropriate. Applied knowledge of precision and recall, thresholds, false positives, false negatives and class imbalance.
Testing and communication: Experience testing controls across realistic scenarios and assessing the effects of data or behaviour drift. Ability to explain results, methodological choices and limitations to engineers, product managers and other stakeholders; write clear technical proposals, reports and documentation; and turn ambiguous requirements into concrete experiments and implementation steps.
Collaboration: Able to work with use-case teams and domain experts to define what good performance means in context.
Nice to have
Experience with agentic evaluation, including tool selection, multi-step workflows and task completion.
Experience testing privacy controls, prompt-injection defences or other GenAI safeguards.
Experience with controlled adversarial testing or robustness assessment.
Experience with evaluation tooling, observability or continuous evaluation in production.
Experience working in a regulated environment and producing traceable validation evidence.
You may thrive in this role if
You enjoy turning “Can we trust this result?” into a concrete test, a measurable criterion and a useful tool. You are curious about how AI systems and their evaluators behave. You combine statistical rigour with a builder’s mindset, value evidence over impressive-looking scores, make limitations explicit and enjoy working with others to solve real problems. You want your work to help multiple teams build better GenAI solutions.
Eisen
- The candidate must have practical experience with LLM-based applications and understand their limitations and failure modes
- The candidate must be familiar with retrieval-augmented generation, prompting, grounding and context management
- The candidate must be able to assess Gen
- AI outputs against explicit criteria rather than relying only on intuition
- The candidate must understand how changes to models, prompts, data or application logic can affect quality and safety
- The candidate must have strong Python skills and demonstrated experience turning experiments and prototypes into maintainable, tested, production-ready code, not only exploratory notebooks
- The candidate must have practical experience integrating data-science components with APIs, structured and unstructured data, and automated data-processing or evaluation pipelines
- The candidate must have hands-on experience with version control, automated testing, CI/CD and reproducible experimentation
- The candidate must have sufficient production experience to understand deployment, monitoring and production workflows, diagnose issues and deliver code that engineers can integrate and operate
- The candidate must have demonstrated experience designing sound evaluations and experiments, and critically analysing metrics and datasets, not just applying evaluation frameworks
- The candidate must have strong applied statistical knowledge of sampling, dataset representativeness, uncertainty, metric reliability and experimental design, including the limitations of aggregate scores
- The candidate must have demonstrated experience validating automated evaluations against human assessments, assessing reliability and investigating disagreement
- The candidate must be able to distinguish between weaknesses in the Gen
- AI solution, the evaluation dataset and the evaluator itself, and explain which conclusions the data does and does not support
- The candidate must have practical experience assessing and selecting guardrail methods, including rule-based controls, LLM-as-a-judge approaches and trained ML classifiers, with clear reasoning about their suitability and limitations in context
- The candidate must be able to translate a risk or expected behaviour into testable criteria and measurable controls, and to train or evaluate a classifier when that is the appropriate approach
- The candidate must have applied knowledge of precision and recall, detection thresholds, false positives, false negatives and class imbalance, and be able to balance safety, usability and operational constraints
- The candidate must have experience testing controls across realistic scenarios, critically checking assumptions and assessing how data or behaviour drift affects guardrail performance over time
- The candidate must be able to explain evaluation results, methodological choices and limitations to engineers, product managers and stakeholders
- The candidate must be able to work with use-case teams and domain experts to define what good performance means in context
- The candidate must be able to write clear technical proposals, evaluation reports and practical documentation
- The candidate must be able to turn ambiguous requirements into concrete experiments and implementation steps
- The candidate must be able to work in English
- The candidate must be available for the entire duration of the assignment
- The candidate must successfully complete the pre-employment screening before starting with the client
- The candidate must possess a valid passport or identity card that remains valid during both the intake and on the contract start date and can be presented for verification
- The candidate must not be submitted through a midlance employment construction
- The candidate may only be submitted to one request at a time and, during the four-working-day exclusivity period, may only be submitted to Magnit - Rabobank
- The candidate's CV and personal motivation must be in English
- The personal motivation must describe how the candidate meets the required competencies
Gunningscriteria
- Experience with agentic evaluation, including tool selection, multi-step workflows and task completion is an advantage
- Experience testing privacy controls, prompt-injection defences or other Gen
- AI safeguards is an advantage
- Experience with controlled adversarial testing or robustness assessment is an advantage
- Experience with evaluation tooling, observability or continuous evaluation in production is an advantage
- Experience working in a regulated environment and producing traceable validation evidence is an advantage
Extra informatie
Je sollicitatie dient uiterlijk 19 november 2026 om 08:00 uur door ons te zijn ingediend, voorzien van een motivatie (waarin wordt toegelicht hoe aan de eisen en gunningscriteria wordt voldaan) en twee referenties. Wij ontvangen je gegevens graag minimaal één werkdag vóór de sluitingsdatum.
