Echkhard, S., Jankauskas, V. and Leuschner, E.
Institutional design and biases in evaluation reports by international organizations - Public Administration Review - 84(3), May-Jun, 2024: p.560-573
Governments spend hundreds of millions on evaluations to assess the performance of public organizations. In this article, we scrutinize whether variation in the institutional design of evaluation systems leads to biases in evaluation findings. Biases may emerge because influence over evaluation processes could enable the bureaucracy to present its work in a more positive way. We study evaluation reports published by nine international organizations (IOs) of the United Nations system. We use deep learning to measure the share of positive assessments at the sentence level per evaluation report as a proxy for the positivity of evaluation results. Analyzing 1082 evaluation reports, we find that reports commissioned by operative units, as compared to central evaluation units, systematically contain more positive assessments. Theoretically, this link between institutional design choices and evaluation outcomes may explain why policymakers perceive similar tools for evidence-based policymaking as functional in some organizations, and politicized in others.- Reproduced
https://onlinelibrary.wiley.com/doi/10.1111/puar.13705
Institutional design of evaluation systems, Evaluation bias, Bureaucratic influence, Positive assessments, Deep learning analysis, United Nations organizations, Evidence-based policymaking, Operative units, Central evaluation units, Report commissioning, Public sector evaluation, Organizational performance, Evaluation system design, Political perception, Evaluation outcomes, International organizations, Sentence-level analysis, Evaluation report content, Functional vs politicized tools, Institutional accountability
Institutional design and biases in evaluation reports by international organizations - Public Administration Review - 84(3), May-Jun, 2024: p.560-573
Governments spend hundreds of millions on evaluations to assess the performance of public organizations. In this article, we scrutinize whether variation in the institutional design of evaluation systems leads to biases in evaluation findings. Biases may emerge because influence over evaluation processes could enable the bureaucracy to present its work in a more positive way. We study evaluation reports published by nine international organizations (IOs) of the United Nations system. We use deep learning to measure the share of positive assessments at the sentence level per evaluation report as a proxy for the positivity of evaluation results. Analyzing 1082 evaluation reports, we find that reports commissioned by operative units, as compared to central evaluation units, systematically contain more positive assessments. Theoretically, this link between institutional design choices and evaluation outcomes may explain why policymakers perceive similar tools for evidence-based policymaking as functional in some organizations, and politicized in others.- Reproduced
https://onlinelibrary.wiley.com/doi/10.1111/puar.13705
Institutional design of evaluation systems, Evaluation bias, Bureaucratic influence, Positive assessments, Deep learning analysis, United Nations organizations, Evidence-based policymaking, Operative units, Central evaluation units, Report commissioning, Public sector evaluation, Organizational performance, Evaluation system design, Political perception, Evaluation outcomes, International organizations, Sentence-level analysis, Evaluation report content, Functional vs politicized tools, Institutional accountability