Our assumptions about the power of words and data to convey information can lead us to believe that AI improves understanding when it does not.
Information has always depended on words and data. But it’s easy to think that words or data themselves represent information. That’s especially true as generative AI encourages users to think about information in terms of natural language sentences.
Skills, knowledge, and understanding are unique to each individual. Users seek advice from trusted experts. However such technology can be an obstacle to clear communication and interfere with the user’s understanding.
Writers have long faced the challenge of integrating what they know into the way they communicate that knowledge. If done poorly, it creates a gap between the writer’s and the reader’s understanding of the subject. Professionals tend to be blunt speakers.
Readers judge a source’s expertise by its mastery of factual information. If a person or program can answer questions with factual information, then it has expert-level knowledge.
But knowledge is not the standard. The concept of a knowledge base can be deceptive because it presents a one-size-fits-all approach. Knowledge is just an external understanding. We can talk about aggregated information — the sum of available information that everyone has contributed. But the understanding available will depend on the individual.
Clear writing does not translate into solid knowledge. For information to be used, it must be readable. But readability does not guarantee that the information is useful to the individual.
Matthew Crawford, author of Shopcraft as Soulcraftpoints out the difference between thinking-as-to do and thinking-as-to write. The content is often separated from what the user needs to do. He argues:
Modern technical book writers are not mechanics or engineers but technical writers. This is a work founded on the assumption that it has principles that can be managed without the author sinking into any particular problem; it is more global than it is. Technical writers know thatbut they don’t know How.
The problem, as Crawford sees it, is that the writer has no personal understanding of the subject he is writing about. The author has thought about the topic clearly, but without direct experience. If faced with unreasonable explanations, the user should:
give a gibberish encounter, and he could do this only by transferring it to a model he had in his mind. how something works.
The key to developing understanding is the mental model that builds the conversation. It should be accurate and understandable to the user.
Writing creates appropriate understanding. The writing process can help writers understand the topic. It helps them clarify the relationships between stories and encourages them to create precise words to describe concepts.
Writing, without a doubt, is important in developing knowledge about complex issues. But the text will only represent the knowledge of the writer, not necessarily the understanding that another reader can gain. The information mentioned should be updated to match the expectations and understanding of foreign students.
The writer should explain how they reached the conclusion and how they communicate that conclusion to others. The writer will have different background knowledge and motivations than readers who want to understand the topic. Although in some fields (scientific research, for example), both the author and the reader will follow a regular, standard process to reach a conclusion, this pattern is not the norm in most cases. Instead, the writer is considered more knowledgeable than the reader and will be concerned with factual and logical details that would not be relevant to a non-expert.
With LLMs, we find that the responses to the user’s notification often repeat the meaning of the expert’s definition and the terms defined in the source content, although this is not appropriate for the user.
Explanations without justification sound like tyranny. Users are curious: they want expert opinions, yet they don’t want to put up with expert-level explanations. At the same time, many users find explanations without reasons unbelievable. They ask why they should believe what is being said.
Public trust in professional experts has been eroded by the “democratization” of information. People don’t like to be told what to do if they feel they are not allowed to make their own decisions. Users feel empowered and entitled to make their own decisions. However, LLMs do not always provide a comprehensive rationale for their answers. The tendency of chatbots to provide “quick answers” can reduce their credibility.
Sources of information should be transparent and easy to track. Users want to know where the information comes from. AI solutions make tracking sources difficult.
Major AI productivity platforms provide limited links to resources in their responses. The answer may contain 5-8 links, usually to obvious sources like Wikipedia. At best, these links provide a shallow overview of the available information.
Dense information is not necessarily clear information. Knowledge graphs represent another paradigm for answering user questions. They can provide better traceability, but at the cost of poor performance. They do not encourage deep understanding.
Because of their complexity, the use of knowledge graphs increasingly depends on the interaction of the user with a natural language conversation. When you use a chatbot to interact with data graph information, the traceability of the information is lost. Google’s information panels, which combine information from multiple sources, require users to click links to perform new searches to trace the origin of the displayed data.
Just because you can explain something doesn’t mean you understand it. It is easy to make statements and come up with reasons why readers should believe what you said. But it is difficult to justify the statements with a valid reason. Many sayings fail to stand up to scrutiny.
Authors and AI platforms both know that credibility depends on giving a reason. Writers have long used formulas like “six reasons why….” to strengthen their promises. Professional writers may follow a template that describes methods, procedures, and evidence before presenting conclusions. AI platforms reproduce these formulas in their responses.
It is easy for users to recognize formula definitions as valid because they match expectations. But in most cases, especially with LLM responses, justification is developed after the conclusion is produced. Chatbots explain things without understanding them.
A widespread myth about LLMs is that they can’t think. Marketers promote the process of “Thought-Action-Observation Loop” and ReAct (Consultation/Act), suggesting that LLMs think. These strategies build on a rapid engineering approach called Chain-of-Thought (CoT) thinking, which is said to reflect the LLM thinking process. However, more detailed research shows that these measures are more effective. They slow down the response speed without bringing any appreciable improvement in the quality of the response. On the contrary, strategies can make the answers equal.
The main problem is misplaced trust: CoTs can appear to be influential even when they do not faithfully reflect the model’s actual decision process.
– Chain-of-Caught Unexplained
Explanations can be myths. To improve perceived credibility, communication is intended to noise professional and he appeared to be based on a comprehensive review of available information.
But imitating conventional arguments does not make arguments work. Few people use critical reading on the Internet, refuting claims, reasons, and evidence.
One of the most reliable forms of communication is the story. Stories often rely on analogical thinking. A Chatbot has access to many examples of physical objects such as user information. We shouldn’t be surprised if we start seeing chatbot descriptions that include sports or TV show analogies.
– Michael Andrews
