Abstract

Conclusion. This study concluded that: 1) The developed HOTS assessment instrument consisted of 10 essay questions that were found to be valid and appropriate in terms of construct, language, and material aspects; 2) Reliability of the HOTS assessment instrument was categorized as high, with a coefficient of .7420.

Keywords: HOTS, Borg and Gall, validity, RnD

Abidin, A., Istiyono, E., Fadilah, N., & Dwandaru, W. (2019). A computerized adaptive test for measuring the physics critical thinking skills. International Journal of Evaluation and Research in Education, 8(3), 376-383. http://doi.org/10.11591/ijere.v8i3.19642

Abosalem, Y. (2016). Assessment Techniques and Students’ Higher-Order Thinking Skills. International Journal of Secondary Education, 4(1), 1-11. http://doi.org/10.11648/j.ijsedu.20160401.11

Afflerbach, P., Cho, B.-Y., & Kim, J.-Y. (2015). Conceptualizing and Assessing Higher-Order Thinking in Reading Order Thinking in Reading. Theory Into Practice, 54(3), 203-212. http://dx.doi.org/10.1080/00405841.2015.1044367

Aman. (2019). Final Examination Test Instruments for History Subject in Yogyakarta, Indonesia: A Quality Analysis. Universal Journal of Educational Research, 7(12), 2857-2866. http://doi.org/10.13189/ujer.2019.071237

Arifin, Z. (2016). Evaluasi Pembelajaran [Learning Evaluation]. Remaja Rosdakarya.

Atitullah, K., Fitriati, S., & Rukmini, D. (2019). Using Revised Bloom’s Taxonomy to Evaluate Higher Order Thinking Skills (Hots) in Reading Comprehension Questions of English Textbook for Year X of High School. English Education Journal, 9(4), 428-436. https://doi.org/10.15294/eej.v9i4.31794.

Avargil, S., Herscovitz, O., & Dori, Y. (2012). Teaching Thinking Skills in Context-Based Learning: Teachers’ Challenges and Assessment Knowledge. Journal of Science Education and Technology, 21, 207-225. http://dx.doi.org/10.1007/s10956-011-9302-7

Ayubi, M., Arthamena, V., & Aznam, N. (2023). Development of HOTS (High Order Thinking Skill) Question Instrument on Thermochemistry Material. Jurnal Penelitian Pendidikan IPA, 9(11), 9945-9954. https://doi.org/10.29303/jppipa.v9i11.5162

Azwar, S. (2016). Reliabilitas dan Validitas Aitem [Reliability and Validity of Aitems]. Buletin Psikologi, 3(1), 19-26. https://jurnal.ugm.ac.id/buletinpsikologi/article/view/13381

Barbara, L., & Wendy, W. (2010). Developing Higher Level Thinking. Journal of Instructional Pedagogies, 3(1), 1-9. https://www.aabri.com/manuscripts/09423.pdf

Barnett, J., & Francis, A. (2011). Using higher order thinking questions to foster critical thinking: a classroom study. Educational Psychology, 32(2), 201-211. https://doi.org/10.1080/01443410.2011.638619

Boone, W., Staver, J., & Yale, M. (2014). Rasch Analysis in the Human Sciences. SpringerLink.

Boopathiraj, C., & Chellamani, K. (2013). Analysis of Test Items on Difficulty Level and Discrimination Index in the Test for Research in Education. International Journal of Social Science & Interdisciplinary Research, 2(2), 189-193.

Brown, G., Irving, S., & Keegen, P. (2018). An Introduction to Educational Assessment, Measurement, and Evaluation: Improving the Quality of Teacher-Based Assessment. Pearson Education.

Cavallo, M., Potter, W., & Rozman, M. (2010). Gender Differences in Learning Constructs, Shifts in Learning Constructs, and Their Relationship to Course Achievement in a Structured Inquiry, Yearlong College Physics Course for Life Science Majors. School Science and Mathematics, 104(6), 288-300. https://doi.org/10.1111/j.1949-8594.2004.tb18000.x

Chalkiadaki, A. (2018). A Systematic Literature Review of 21st Century Skills and Competencies in Primary Education. International Journal of Instruction, 11(3), 1-16. https://doi.org/10.12973/iji.2018.1131a

Chinedu, C., Kamin, Y., & Olabiyi, O. (2015). Strategies for Improving Higher Order Thinking Skills in Teaching and Learning of Design and Technology Education. Journal of Technical Education and Training, 7(2), 35-43. https://publisher.uthm.edu.my/ojs/index.php/JTET/article/view/1081

Clotfelter, C., Ladd, H., & Vigdor, J. (2005). Who teaches whom? Race and the distribution of novice teachers. Economics of Education Review, 24(4), 377-392. https://doi.org/10.1016/j.econedurev.2004.06.008

Conklin, W. (2011). Higher-Order Thinking Skills to Develop 21st Century Learners. Shell Education.

Cooper, C. (2023). An Introduction to Psychometrics and Psychological Assessment. Routledge.

D’Alessio, F., Avolio, B., & Charles, V. (2019). Studying the impact of critical thinking on the academic performance of executive MBA students. Thinking Skills and Creativity, 31, 275-283. https://doi.org/10.1016/j.tsc.2019.02.002

Demirhan, E., & Koklukaya, A. (2014). The Critical Thinking Dispositions of Prospective Science Teachers Procedia-Social and Behavioral Sciences, 116, 1551-1555. https://doi.org/10.1016/j.sbspro.2014.01.433

Drake, F., & Nelson, L. (2008). Engagement in Teaching History: Theory and Practices for Middle and Secondary Teachers (2nd ed.). Pearson.

Febrina, F., Usman, B., & Muslem, A. (2019). Analysis of Reading Comprehension Questions by Using Revised Bloom’s Taxonomy on Higher Order Thinking Skill (HOTS). English Education Journal, 10(1), 1-15. https://jurnal.usk.ac.id/EEJ/article/view/13253/10123

Febriyani, R., Yunita, W., & Damayanti, I. (2020). An Analysis on Higher Order Thinking Skill (HOTS) in Compulsory English Textbook for the Twelfth Grade of Indonesian Senior High Schools. Journal of English Education and Teaching, 4(2), 170-183. https://doi.org/10.33369/jeet.4.2.170-183

Fischer, C., Bol, L., & Pribesh, S. (2011). An Investigation of Higher-Order Thinking Skills in Smaller Learning Community Social Studies Classrooms. American Secondary Education, 39(2), 5-26.

Gall, M. D., Gall J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Allyn, & Bacon/Pearson.

Gojkov, G., Stojanovic, A., & Rajic, A. (2015). Critical Thinking Of Students – Indicator Of Quality In Higher Education. Procedia-Social and Behavioral Sciences, 191, 591-596. https://doi.org/10.1016/j.sbspro.2015.04.501

Guskey, T. (2007). Closing Achievement Gaps: Revisiting Benjamin S. Bloom’s “Learning for Mastery”. Journal of Advanced Academics, 19(1), 8-31. https://doi.org/10.4219/jaa-2007-704

Haley, S., Coster, W., Dumas, H., Fragala-Pinkham, M., Kramer, J., Ni, P., Tian, F., Kao, Y.-C., Moed, R., & Hudlow, L. (2011). Accuracy and precision of the Pediatric Evaluation of Disability Inventory computer-adaptive tests (PEDI-CAT). Developmental Medicine and Child Neurology, 53(12), 1100-1106. http://doi.org/10.1111/j.1469-8749.2011.04107.x

Halpern, D. (1998). Teaching critical thinking for transfer across domains: Disposition, skills, structure training, and metacognitive monitoring. American Psychologist, 53(4), 449-455. https://psycnet.apa.org/doi/10.1037/0003-066X.53.4.449

Heong, Y., Othman, W., Yunos, J., Kiong, T., Hassan, R., & Mohamad, M. (2011). The Level of Marzano Higher Order Thinking Skills among Technical Education Students. International Journal of Social Science and Humanity, 1(2), 121-125. http://dx.doi.org/10.7763/IJSSH.2011.V1.20

Hmeli, C., & Ferrari, M. (1997). The Problem-Based Learning Tutorial: Cultivating Higher Order Thinking Skills. Journal for the Education of the Gifted, 20(4), 401-422. https://doi.org/10.1177/016235329702000405

Kinay, I., & Bagceci, B. (2016). The Investigation of the Effects of Authentic Assessment Approach on Prospective Teachers’ Problem-Solving Skills. International Education Studies, 9(8), 51-59. http://dx.doi.org/10.5539/ies.v9n8p51

Kusuma, M., Rosidin, U., Abdurrahman, & Suyanta, A. (2017). The Development of Higher Order Thinking Skill (Hots) Instrument Assessment In Physics Study. IOSR Journal of Research & Method in Education, 7(1), 26-32. http://doi.org/10.9790/7388-0701052632

Madhuri, G., Kantamreddi, V., & Goteti, L. (2012). Promoting higher order thinking skills using inquiry-based learning. European Journal of Engineering Education, 37(2), 117-123. https://doi.org/10.1080/03043797.2012.661701

Mahoney, J., & Harris-Reeves, B. (2019). The effects of collaborative testing on higher order thinking: Do the bright get brighter? Active Learning in Higher Education, 20(1), 25-37. https://doi.org/10.1177/1469787417723243

Mardapi, D. (2008). Teknik Penyusunan Instrumen Tes dan Nontes [Techniques for Preparing Test and Non-Test Instruments]. Mitra Cendekia Press.

McCormick, N. (2015). Engaging Students in Critical Thinking and Problem Solving: A Brief Review of the Literature. Journal of Studies in Education, 5(4), 100-113. http://dx.doi.org/10.5296/jse.v5i4.8249

McDonald, G. (2012). Teaching Critical & Analytical Thinking in High School Biology? The American Biology Teacher, 74(3), 178-181. https://doi.org/10.1525/abt.2012.74.3.9

McLaughlin, J., Roth, M., Glatt, D., Gharkholonarehe, N., Davidson, C., Griffin, L., & Mumper, R. (2014). The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Academic Medicine, 89(2), 236-243. https://doi.org/10.1097/acm.0000000000000086

Miterianifa, M. (2016). Evaluasi Pembelajaran Kimia [Evaluation of Chemistry Learning]. Cahaya Firdaus.

Polit, D., & Beck, C. (2006). The content validity index: Are you sure you know what’s being reported? critique and recommendations. Research in Nursing & Health, 29(5), 489-497. https://doi.org/10.1002/nur.20147

Pommerich, M. (2004). Developing Computerized Versions of Paper-and-Pencil Tests. The Journal of Technology, Learning and Assessment, 2(6), 3-44. https://ejournals.bc.edu/index.php/jtla/article/view/1666

Qing, Z., Jing, G., & Yan, W. (2010). Promoting preservice teachers’ critical thinking skills by inquiry-based chemical experiment. Procedia-Social and Behavioral Sciences, 2(2), 4597-4603. https://doi.org/10.1016/j.sbspro.2010.03.737

Quieng, M., Lim, P., & Lucas, M. (2015). 21st Century-based Soft Skills: Spotlight on Non-cognitive Skills in a Cognitive-laden Dentistry Program. European Journal of Contemporary Education, 11(1), 72-81. http://doi.org/10.13187/ejced.2015.11.72

Retnawati, H. (2016). Validitas, Reliabilitas dan Karakteristik item [Validity, Reliability and Characteristics of items]. Parama Publikasi.

Sepriyanti, N., Nelwati, S., Kustati, M., & Afriadi, J. (2022). The Effect of 21st-Century Learning on Higher-Order Thinking Skills (Hots) and Numerical Literacy of Science Students in Indonesia Based on Gender. Jurnal Pendidikan IPA Indonesia, 11(2), 314-321. http://doi.org/10.15294/jpii.v11i2.36384

Setiawan, J., Aman, A., & Wulandari, T. (2020). Understanding Indonesian history, interest in learning history and national insight with nationalism attitude. International Journal of Evaluation and Research in Education, 9(2), 364-373. http://doi.org/10.11591/ijere.v9i2.20474

Sudijono, A. (2012). Pengantar evaluasi pendidikan [Introduction to educational evaluation]. Rajawali Pers.

Sudijono, A. (2013). Pengantar Evaluasi Pendidikan [Introduction to educational evaluation]. Raja Grafindo Persada.

Szilagyi, J., Clements, D., & Sarama, J. (2013). Young Children’s Understandings of Length Measurement: Evaluating a Learning Trajectory. Journal for Research in Mathematics Education, 44(3), 581-620. https://doi.org/10.5951/jresematheduc.44.3.0581

Udompong, L., & Wongwanich, S. (2014). Diagnosis of the Scientific Literacy Characteristics of Primary Students. Procedia-Social and Behavioral Sciences, 116, 5091-5096. https://doi.org/10.1016/j.sbspro.2014.01.1079

Wardany, K., Sajidan, & Ramli, M. (2017). Pengembangan Penilaian Untuk Mengukur Higher Order Thinking Skills Siswa [Development of Assessments to Measure Students' Higher Order Thinking Skills]. Jurnal Inkuiri, 6(2), 1-16. https://jurnal.uns.ac.id/inkuiri/article/view/16214

Weiss, D. (2011). Better Data From Better Measurements Using Computerized Adaptive Testing. Journal of Methods and Measurement in the Social Sciences, 2(1), 1-27. https://doi.org/10.2458/v2i1.12351