This study aimed to develop a questionnaire that is suitable for the resident-participatory ODA evaluation to minimize the bias and insufficient scientific rigor in the self-made questionnaire developed by evaluators. A modified Delphi survey was conducted twice with 16 ODA experts who had experience in international cooperative development in the fields of rural development, education, health, and poverty. Sixty-two questions were finally selected for resident-participatory ODA evaluation: 7 questions on relevance, 8 questions on efficiency, 9 questions on effectiveness, 11 questions on impact, questions on for sustainability, 4 questions on gender mainstreaming and environmental impact, and 17 questions on spillover effect on the community. A resident survey with the newly developed questionnaire will generate high-quality data that can empirically analyze the causal-effect relationship between ODA activities and performance, and community changes, and it can also contribute to the development of a quantitative ODA evaluation tool at a local level of recipient countries.