ENHANCING WRITING SKILLS OF EFL LEARNERS THROUGH AUTOMATED FEEDBACK: AN EMPIRICAL INVESTIGATION

- This study examines the effectiveness of an automated writing evaluation (AWE) tool in improving the writing skills of lower-proficiency EFL learners. The study involved 34 non-English major university students who received AWE feedback from the WhiteSmoke Writing Assistant during their writing practice sessions. The quality of their writing was evaluated using pre-and post-tests, and their perceptions of the feedback were gathered through a questionnaire. The results showed a significant improvement in the quality of students' writing after using AWE feedback. Additionally, the students perceived the AWE feedback as positively impacting the quality of their writing. These findings suggest that AWE feedback is a promising tool that can be incorporated into the writing class program to enhance the quality of students' writing. Writing teachers can use the tool to evaluate students' compositions and provide targeted feedback to help them improve their writing skills. Overall, the study highlights the potential of technology-based writing tools to facilitate language learning and improve EFL learners' writing proficiency.


INTRODUCTION
In recent years, second language writing has seen a growing interest in using technology-based tools to support writing instruction and assessment.One such tool is automated writing evaluation (AWE), which provides instant feedback on written texts' grammatical, lexical, and organizational features.AWE technology has been increasingly integrated into language teaching and learning contexts, and a growing number of studies have investigated its efficacy in enhancing writing skills and providing more efficient and practical feedback to language learners (Bai & Hu, 2016;Hockly, 2019;Huang & Renandya, 2018;Koltovskaia, 2020;Ranalli et al., 2016;Tian & Zhou, 2020;Wilson et al., 2017;Zhang, 2020).
Many of the previous studies focused more on the validity and reliability of the AWE (e.g., Bai & Hu, 2016;Shermis & Burstein, 2013), and few studies investigated the effect of specific AWE feedback on the quality of EFL learners' writing (e.g., Hegelheimer, Dursun, & Li, 2016;Stevenson, 2016).Understanding the effect of particular AWE feedback on the quality of students' writing is essential since it may inform the writing teachers to consider incorporating AWE feedback into their teaching.
The research gap in previous studies is the need for more investigation into the effect of specific AWE feedback on the quality of EFL learners' writing.While there is substantial literature on the validity and reliability of AWE, there is a lack of research on the specific impact of particular AWE feedback on the quality of EFL learners' writing.
This gap is significant because understanding the effect of particular AWE feedback on the quality of students' writing is crucial for informing writing teachers about the potential benefits of incorporating AWE feedback into their teaching.This gap presents an AWE systems, underpinned by computational linguistic algorithms, can analyze written texts and provide comprehensive feedback on various facets of writing, including grammar, vocabulary, syntax, and coherence.Feedback can encompass error correction, improvement suggestions, or scoring based on predefined criteria (Ducasse, 2023;Hinkel, 2004;Khezrlou, 2023;Kirszner & Mandell, 2009;Lee, 2020).AWE technology is engineered to offer more objective and consistent feedback compared to traditional assessment methods such as teacher feedback or peer review, thereby reducing the time and effort required for grading and providing feedback (Chen & Cheng, 2008;Hegelheimer et al., Z, 2016).
With the continual advancement of technology, particularly in Natural Language Processing (NLP) and Latent Semantic Analysis (LSA), software that automatically analyzes students' writing has gained popularity.This software is referred to as Automated Essay Scoring (AES), Automated Essay Evaluation (AEE), or Automated Writing Evaluation (Hockly, 2019).Although researchers use different terms to denote this software, this study prefers the term AWE as it encompasses more text genres and provides formative evaluation, not merely scoring as suggested in AES.The software compares students' writing to a vast database of similar genres.The analyses conducted by the software include syntax, text complexity, vocabulary range, writing style, spelling, and punctuation.A score and improvement suggestions are also provided.AWE platforms such as Criterion, Write & Improve, MY Access, Grammarly and WhiteSmoke are commonly employed in English language teaching (Elliot et al., 2013 for details).
WhiteSmoke Writing Assistant is an online-based software tool that utilizes NLP technology to check grammar, style, spelling, and punctuation.It also provides a writing score based on several metrics, error explanations, and a thesaurus.This software offers two products, Desktop Premium and White Smoke Web, and three pricing packages: Web -5/month, Premium− 6.66/month, and Business -$11.50/month(see www.whitesmoke.com).Unlike other grammar checker software, such as Grammarly, this software does not provide a free version.However, at the time of this study, the software was freely available for the Android version.Therefore, the WhiteSmoke Writing Assistant utilized in this study was the free Android version.
Student perceptions of automated feedback have been a topic of contention in the literature.Some studies (Parra & Calero, 2019;Fang, 2010;Li et al., 2015;Ma, 2013;Tsuda, 2014) have reported that students have favored opinions toward computergenerated feedback in their writing.However, contrasting findings on this topic are presented by other studies (e.g., Chou et al., 2016;Chen & Cheng, 2008;Grimes & Warschauer, 2010).Chen and Cheng (2008) found that scholars were not inclined toward using automated feedback in writing classes.Although research on students' perceptions of Automated Writing Evaluation (AWE) feedback suggests that students with lower language proficiency levels may find AWE feedback beneficial (e.g., Chen & Cheng 2008), there is a shortage of studies conducted among non-English majors and low proficiency English as a Foreign Language (EFL) learners.
This article aims to bridge this research gap by investigating the impact of AWE feedback on a cohort of lower-proficiency EFL students.It seeks to provide EFL students and educators with insights into the functionality of AWE feedback within their educational context and the perceptions of Indonesian EFL students towards such software.Understanding specific learning and teaching practices is crucial as it contributes to the efficacy and preparation of learning and teaching scenarios.
Specifically, this study explores the extent to which AWE feedback enhances the quality of students' writing and the students' perceptions of the automated feedback in writing class.

Research Design
The research design employed in this study was a mixed-method approach, which was a methodology that combined both quantitative and qualitative research elements to provide a more comprehensive understanding of the research problem (Berg-Schlosser, 2012;Creswell, 2014;Sreejesh & Mohapatra, 2014).This approach is particularly beneficial in educational research as it allows for a more nuanced The participants in this study were 34 university students enrolled in a 16-week English course at Adhi Tama Institute of Technology Surabaya.This course was required for and offered to first-year non-English-major students.It was a test-oriented English course before the students were introduced to the TOEFL preparation course in the following semester.The student's English proficiency was low (below 400 in Paper-Based TOEFL).The average age of the students was 20.Males, who were from various regions of Indonesia's East Java, predominated among the student participants.The students were chosen as the participants of this research as they were readily accessible and consented to participate.
(see Appendix 1 for details) for 30 minutes.The prompt was chosen as more and more students were using smartphones for their assistance, and it was relevant to their context.
As the student participants were low-proficiency EFL learners, only one paragraph (approximately 150 words) was expected for students to write based on the given task prompt.After completing the paragraph, students submitted it on Google Classroom, the online platform they used for activities after the class.This draft was a pre-test to inform students' initial writing performance.
In the following session, the students were asked to download WhiteSmoke Writing Assistant on their smartphones.The instructor conducted a 15-minute demonstration of using WhiteSmoke Writing Assistant to familiarize students with the features.After that, the students were asked to upload or copy the text written in the first session and make necessary revisions on their smartphones based on the feedback provided by the WhiteSmoke Writing Assistant.After 30 minutes of revision time, the students were asked to resubmit their revised draft (used as a post-test) to Google Classroom and to complete the questionnaire on how they perceive the automated feedback given by the WhiteSmoke Writing Assistant.In addition, the qualitative data was collected through a questionnaire.The questionnaire was designed to explore students' perceptions of using WhiteSmoke Writing Assistant for their writing.Exploring students' perceptions is one of the

Data Analysis
To answer research question no. 1, students' pre-test and post-test writing were scored using Jacobs, Zinkgraf, Wormuth, Hartfiel, and Hughey's (1981) ESL Composition Profile rubric.It contains five categories: content, organization, vocabulary, language use, and mechanics.The researcher and his colleague teacher each scored three of the students' written texts at random to ensure that the scoring was accurate.The Cronbach's alpha coefficient obtained was 0.98, a high-reliability coefficient.A pairedsample t-test was conducted to determine whether there was a difference in the pre-test and post-test scores, as suggested by Muijs (2004) and Phakiti (2014).
To answer research question no. 2, a questionnaire containing 5-point Likert Scale questions (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree), dichotomous questions (yes and no), and an open-ended question (to comment; see Appendix 1) was analyzed.While the closed-ended questions were coded using the Likert Scale, the open-ended questions were analyzed using the 'grounded theory' method (see Creswell, 2014 for the grounded theory method in detail).The researcher and his colleague's teacher coded five randomly chosen student comments to establish reliability in coding.97% of the inter-rater agreement was achieved for comment classification.

Finding The quality of students' writing after using AWE feedback
The students' writing scores on the pretest before using AWE feedback and the posttest after using it on students' writing tasks can be seen in Table 2.As can be seen from Table 2, students' writing scores in each category, content, organization, vocabulary, language use, and mechanics on the pretest and the posttest experienced changes.A Paired-samples t-test analysis was conducted to see if the changes were significant.Table 3 shows a Paired sample t-test analysis of the student's writing scores on the pretest and the post-test.As displayed in Table 3, there was a significant difference in the student's writing scores for the pretest (M = 69.70,SD = 5.45) and the posttest (M = 79.70,SD = 6.24), t(33) = -9.03,p <.05.It indicates that the use of WhiteSmoke Writing Assistant improved the quality of students' writing.

Students' opinions of the AWE feedback in a writing class
Table 4 shows students' perceptions of the feedback provided by the WhiteSmoke Writing Assistant.The average response of 4.0 shows that students primarily found the software useful for raising the caliber of their writing.
opportunity for future research to delve into the specific effects of different types of AWE feedback on the writing quality of EFL learners, thereby providing valuable insights for writing instruction and assessment.The present study is aimed at addressing such a research gap by exploring the effect of AWE feedback on the quality of EFL learners' writing and EFL learners' perception.While AWE technology holds promise for various advantages, its efficacy concerning improving writing skills and enhancing feedback quality remains a subject of contention within the academic discourse among researchers and practitioners.Certain investigations have yielded affirmative outcomes, elucidating that AWE feedback facilitates learners in recognizing and rectifying errors, elevates the overall caliber of their writing, and fosters heightened motivation and engagement.Conversely, alternative scholarly inquiries have voiced apprehensions about the precision and dependability of AWE systems, the constraints inherent in their feedback mechanisms, and the potential deleterious consequences associated with an undue reliance on technological interventions in writing instruction.
understanding of complex phenomena by capitalizing on quantitative and qualitative research strengths.The quantitative component of the research primarily dealt with the participants' writing performance.This was measured through pretest and post-test scores, allowing for an objective evaluation of the participants' writing skills before and after the intervention.Using pretest and post-test scores is a common method in educational research for measuring improvement in a particular skill or knowledge area over time.In this case, it provides a quantifiable measure of the improvement in the participants' writing performance due to the intervention.On the other hand, the qualitative component of the research focused on the participants' perception of AWE feedback used in the writing class.This involved gathering data on the participants' thoughts, feelings, and experiences with the AWE feedback.This was done through open-and closed-ended survey questions.The qualitative data provided a deeper understanding of the participants' experiences and perceptions.It could offer valuable insights into how the participants received and interpreted the AWE feedback and how it influenced their learning process.
characteristics of what Creswell (2014) defines as a Case Study.It contained 14 questions, 13 closed-ended and 1 open-ended.A combination of closed-ended and open-ended questionnaire items accommodated more responses that might not be found in a single type of question item.The questionnaire was made using Google Forms and distributed to students via Google Classroom.

Table 1 .
The Experimental Procedure of the Study

Table 2 .
Students' Writing Scores on the Pretest and the Posttest

Table 5 .
Students' Perception in General of WhiteSmoke Writing Assistant Yes No In general, WhiteSmoke Writing Assistant has a positive impact on the quality of my writing