Presenter: Dr. Okim Kang
Summarized by: Dr. Linh Phung
Speaking is a complex phenomenon, and judgement of someone’s speaking is affected by stereotypes, accent familiarity, topic familiarity, and other factors. Automated assessment of speaking encounters its own set of challenges, including accuracy, technological reliance, task variation (controlled, spontaneous, and interactive), background noise, test taker preferences (human or machine evaluation), and washback effects on teaching and learning.
The presentation by Okim Kang, Professor of TESOL and Applied Linguistics at Northern Arizona University, focuses on explaining linguistics features in speaking performance, citing her extensive research and that of others in the field in speech analysis. Many of her studies analyze speaking samples from IELTS, TOEFL, Cambridge Assessment, and the Duolingo English Test.
To start, you may be familiar with features in the TOEFL and IELTS rubrics: pronunciation, fluency, grammar, vocabulary, and topic relevance and coherence.
The presenter then drills down into specific units of analysis, illustrating features that impact speaking assessment. The information is summarized below.
Fluency features: speech rates (syllable per second, mean length run, etc.), silent pause (mean length or number or pauses)
Grammatical accuracy and complexity: number of t-units, number of clauses, error-free t-units, number of verb phrases per c-unit
Lexical resources: type/token measures (work-token, word-type, type-token ratio), K1-words
Pronunciation: sounds (target-like syllables), suprasegmental features (stress, prominence, tone choice), functional load error
One key challenge that the speaker explains in detail is the lack of consistent differences between adjacent levels of proficiency in fluency features (e.g. similar mean length of run between B2, C1, and C2 and similar numbers of syllables per second between B2 and C1). This lack of differentiation applies to grammatical complexity features (e.g. similar numbers of error-free T-units among B1, B2, C1, and C2) and vocabulary use (e.g. the use of the first 1000 words among IELTS test takers with Bands 6.0, 6.5, 7.0, and 7.5).
Dr. Kang (2013) highlights the importance of pronunciation, emphasizing features such as stress and pitch, fluency, segmental features, and tone choice. Specifically, 70% lexical stress accuracy is needed for intelligibility. More than 30% error rate in lexical stress impacts intelligibility significantly. In addition, higher lexical stress accuracy is associated with lower degrees of accentedness. Prominence (or stress of a prominent sound in a phrase or sentence) is also important for discourse. Speech rate is also important, and the 3-4 syllables per second is considered the ideal rate. Pausing is another important feature, and what matters is not only the number of pauses, but also where the pauses are.
In terms of segmental features, even speakers at high proficiency levels (C2) still make errors. However, some sounds carry a higher functional load (greater importance for intelligibility than others). Mispronunciation of these sounds, such as k/h, p/b, p/t, d/z, i/a (bit/bat), iy/I (heat/hit), and o/ow (bought/boat), is more likely to hinder intelligibility.
In terms of intonation or tone choice (falling, rising, level/neutral), as proficiency increases, the use of level tone decreases when the use of falling tone increases. Rhythm has less impact on perception of accentedness and intelligibility.
In automated assessment, testers select specific constructs to measure and choose a certain number of features as seen in TOEFL speaking, which incorporates 11 features. Dr. Kang points out that fluency has been shown to be the top predictor of proficiency and typically carries the most weight in score assignments.
To sum up, the issues related to automated speaking assessment include non-linear relationship, the importance of qualitative occurrences rather than only quantitative counts, functional load-based segmental features, and complexity related to intonation (tone choice) in discourse.
Finally, it’s interesting to learn about characteristics of highly intelligible speakers that the presenter argues can be used in listening tests. The speaker argues that when more varieties of English are used for assessment purposes in the future, students may be open to learning about more varieties.
The final issue the speaker addresses is feedback that helps learners to improve their speaking. Her study found that giving students feedback in terms of their speaking rate and other features didn’t not lead to any significant gains, but when learners are provided more instant feedback while using the speech analysis tool, there were some gains.
The speaker wraps up her lecture by summarizing issues related to automated speaking assessment that future endeavors may be able to address. These include the complexity of features as illustrated throughout the presentation, interaction features, reliable training data (to include highly intelligible speech samples from various varieties), and effective learner feedback and the impact of automated speaking assessment on teaching and learning.
If you'd like to learn more, I highly recommend the webinar on YouTube.
Based on insights from the presentation, here is some advice to those preparing for a speaking exam (IELTS, TOEFL, DET, Cambridge Assessment, and other tests) whether it is graded by humans or machines.
Familiarize Yourself with Assessment Criteria: Get to know the assessment criteria used in exams like TOEFL and IELTS, which typically include pronunciation, fluency, grammar, vocabulary, and topic relevance and coherence. Familiarity with these criteria will help you target specific areas for improvement.
Enhance Fluency: Fluency is often the top predictor of speaking proficiency. Aim for a speech rate of 3-4 syllables per second and consider the placement and frequency of pauses in your speech. Smooth and natural delivery is key.
Focus on Pronunciation: Pay special attention to pronunciation, including stress and pitch. Strive for 70% accuracy in lexical stress, as this is crucial for intelligibility.
Segmental Features Matter: Be aware of sounds that carry a higher functional load and focus on correct pronunciation of these sounds.
Seek Instant Feedback: Consider using speech analysis tools that provide instant feedback on your speaking. This kind of feedback and repeated practice can lead to improvements. However, the overcomplicated feedback that only researchers can understand may not be so helpful.
Explore Diverse English Varieties: Be open to exposure to various English varieties. In the future, assessments may involve different accents and dialects, so adaptability in understanding and producing different English types can be advantageous.
Practice Regularly: Like any skill, speaking improves with practice. Engage in regular speaking practice, focusing on the specific areas mentioned above. Record yourself and review your performance to track your progress. The Eduling Speak app offers you many topics and interesting tasks that you can respond to by recording your answers.
Dr. Linh Phung's bio: Dr. Linh Phung has a BA, MA, and EdD in Teaching English to Speakers of Other Languages. She has been teaching English for 20 years and has published and presented widely in the area of second language learning and teaching. She's passionate about creating opportunities for language use and development inside and outside the classroom. More information about her is HERE.