This article was produced and financed by Oslo Metropolitan University - read more
Grammar tools do not improve readability
Research shows that computer tools that check grammar and readability improve spelling, but they do not make the text easier to read.
A fluent text is important in order to make it accessible to everyone.
Improved vocabulary, grammar and syntax, and the correct verb clause and tense can make the text easier to understand.
It thereby also becomes easier to read, and it will be possible for the writer to express their ideas more clearly and succinctly.
Higher demands for readability
The demands for readability are increasing. The Web Content Accessibility Guidelines, WCAG2.1, recommend that online texts should be possible to read for people with a basic formal education.
The software Grammarly has been developed to help writers improve their spelling, grammar and readability in English, but does not actually improve readability, according to research conducted by Hitesh Kaushik, a master’s student in Universal IT Design at OsloMet – Oslo Metropolitan University.
He brought together 15 students who wanted to improve their texts, and conducted an experiment alongside supervisor, linguist and associate professor Evelyn Eika.
Corrects grammar mistakes, but does not improve readability
Hitesh measured the changes in readability before and after using the grammar tool and interviewed the students.
He registered the number of grammar mistakes from screen shots and calculated the ratio between mistakes and how much text that had been written. The mistakes could then be measured across different texts.
The texts were also assessed manually based on five criteria: style, vocabulary, grammar, use of writing rules and clarity in the text.
These criteria were chosen because Grammarly have stated that these are some of the elements the tool can help to improve.
The academic content of the text was not assessed, as it would mean testing specific academic knowledge that would in any case not be influenced by the tool.
The results showed a significant reduction of grammar mistakes, but no significant effect on readability.
Hitesh therefore believes that other tools should be developed that can help to improve readability. This is a challenge posed to software developers.
Hitesh believes that the fact that the tools contribute to reducing spelling mistakes is to be expected, as the aim of grammar checks is to identify mistakes and help correct them.
Correcting mistakes does not necessarily make the text easier to read
A lack of readability is not about mistakes in the text. Mistakes and readability are two different dimensions.
It is possible to envisage a grammatically correct text free of mistakes that is very hard to read, and a text that is very easy to read, but with many trivial grammar and spelling mistakes.
«Good readability makes that text understandable and achieves the goal of informing and educating the reader,» says Hitesh.
«Correcting grammar is useful if you're writing short sentences or to divide the text into paragraphs for instance, but it does not necessarily make the text more understandable.»
It should also be mentioned that the participants in the experiment only accepted around half of the mistakes pointed out by Grammarly.
Suggested changes of no help
Many of the suggested changes to the text were ignored because they changed the meaning of the text.
«We used text examples of 600 to 1,000 words, and the students used Grammarly to make changes to the text and tried, to the best of their ability, to do so correctly,» says Hitesh.
«We made sure that they had not read any of it beforehand or that they re-read anything.»
Hitesh also noticed that the quality of the text does not have anything to do with its readability. What he means by that is that if the vocabulary used and structure of the text are good, it does not in itself play any role.
Gives a false sense of security
In other words, Hitesh’s results do not support the claim of the Grammarly developers that the tools help to improve readability.
This is also emphasised by the supervisor Evelyn Eika:
«The results show that this product does not live up to its description,» says Evelyn Eika.
«It gives a false sense of security to students and staff who use the tool. There are still no short cuts to readable texts. Creating readable texts requires knowledge, experience, time and, not least, manual effort.»
Machine learning can provide options
With that said, many exciting developments are taking place in machine learning.
«It is not unthinkable that we will soon get better quality-promoting writing-support tools, which will also make the writing process more inspiring. But I have yet to test a product that lives up to my expectations,» says Eika.
Wants to conduct research on what can lead to better readability
«Could the study conclude on what might have an influence on readability?»
«What we believe could have an influence on readability needs more research,» says Hitesh.
It could be the relation between text and context, for instance that there is one sentence and then another sentence, and they have no relation to each other.
«We have identified that this can influence readability, and that’s what we're going to look at in the next phase.»
Poor writing skills means increased trust in grammar software
Hitesh also found that those who were less confident writers exhibited more trust in the software they used. They believed such software would help them improve their writing.
«Those who liked Grammarly also assessed their writing skills as poor. I think they would do just as well with the Word spell check. Grammarly was nonetheless of good use to help with full stops, commas, semi-colons and paragraphs.»
Hitesh wants to continue researching this, and perhaps design a language checker for Norwegian and other languages that do not have as many users. Grammarly is only available in English.
Hitesh has a background in technical software and software development and took his education in India. He will present his research at a digital conference in the summer of 2020.
Reference:
Hitesh Mohan Kaushik et.al: Towards Universal Accessibility on the Web: Do Grammar Checking Tools Improve Text Readability? International Conference on Human-Computer Interaction, HCII 2020: Universal Access in Human-Computer Interaction. Design Approaches and Supporting Technologies. Springer. DOI: 10.1007/978-3-030-49282-3_19 · (Summary)
———
Read the Norwegian version of this article at forskning.no
