论文标题
你说什么？面向任务的对话框数据集不是对话！
What Did You Say? Task-Oriented Dialog Datasets Are Not Conversational!?
论文作者
论文摘要
面向任务的对话框的高质量数据集对于虚拟助手的发展至关重要。然而，三个最相关的大规模对话框数据集遇到了一个常见的缺陷：可以通过仅考虑当前用户话语，忽略对话框历史记录的模型在很大程度上跟踪对话框状态更新。在这项工作中，我们概述了对话和上下文效应的分类法，我们用来检查多沃兹，SGD和SMCALFLOW，这是最新且以任务为导向的对话框数据集。我们以独立于模型的方式分析数据集，并使用强大的文本对文本基线（T5）在实验中证实了这些发现。我们发现，在当前版本中，Smcalflow少于Multiwoz的转弯中的不到4％，而SGD的10％的回合是对话的，而Smcalflow根本不是对话：对话框状态跟踪任务可以简化为单个交换语义解析。最后，我们概述了Desiderata真正的对话对话数据集。
High-quality datasets for task-oriented dialog are crucial for the development of virtual assistants. Yet three of the most relevant large scale dialog datasets suffer from one common flaw: the dialog state update can be tracked, to a great extent, by a model that only considers the current user utterance, ignoring the dialog history. In this work, we outline a taxonomy of conversational and contextual effects, which we use to examine MultiWOZ, SGD and SMCalFlow, among the most recent and widely used task-oriented dialog datasets. We analyze the datasets in a model-independent fashion and corroborate these findings experimentally using a strong text-to-text baseline (T5). We find that less than 4% of MultiWOZ's turns and 10% of SGD's turns are conversational, while SMCalFlow is not conversational at all in its current release: its dialog state tracking task can be reduced to single exchange semantic parsing. We conclude by outlining desiderata for truly conversational dialog datasets.
