Group-wise Contrastive Learning for Neural Dialogue GenerationGroup-wise Contrastive Learning for...

Group-wise Contrastive Learning for Neural Dialogue Generation

Hengyi Cai†,§∗, Hongshen Chen‡Yonghao Song†, Zhuoye Ding‡, Yongjun Bao‡, Weipeng Yan‡, Xiaofang Zhao††Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

§University of Chinese Academy of Sciences, Beijing, China‡JD.com, China

[email protected], [email protected]{songyonghao, zhaoxf}@ict.ac.cn, {dingzhuoye, baoyongjun, Paul.yan}@jd.com

Abstract

Neural dialogue response generation hasgained much popularity in recent years. Max-imum Likelihood Estimation (MLE) objectiveis widely adopted in existing dialogue modellearning. However, models trained with MLEobjective function are plagued by the low-diversity issue when it comes to the open-domain conversational setting. Inspired by theobservation that humans not only learn fromthe positive signals but also benefit from cor-recting behaviors of undesirable actions, inthis work, we introduce contrastive learninginto dialogue generation, where the model ex-plicitly perceives the difference between thewell-chosen positive and negative utterances.Specifically, we employ a pretrained baselinemodel as a reference. During contrastive learn-ing, the target dialogue model is trained to givehigher conditional probabilities for the posi-tive samples, and lower conditional probabil-ities for those negative samples, compared tothe reference model. To manage the multi-mapping relations prevalent in human conver-sation, we augment contrastive dialogue learn-ing with group-wise dual sampling. Exten-sive experimental results show that the pro-posed group-wise contrastive learning frame-work is suited for training a wide range of neu-ral dialogue generation models with very fa-vorable performance over the baseline trainingapproaches.

1 Introduction

Open-domain human-machine dialogue systems,especially the generation-based genre, have at-tracted extensive attention recently. Typically,following the neural encoder-decoder paradigm,contemporary dialogue generation models (Shanget al., 2015; Serban et al., 2016; Xing et al., 2017;Yan, 2018; Huang et al., 2020; Liu et al., 2020),

∗Work done at JD.com.

more often than not, are trained with MaximumLikelihood Estimation (MLE) principle to mimichuman context-response pairs in the training cor-pus. While notable gains have been achieved underthis learning framework, prior art (Li et al., 2016a,2017; Zhang et al., 2018a) suggests that naive MLEobjective used for training neural dialogue genera-tion models is not that effective enough and tendsto result in issues like dull response generation.By optimizing the likelihood of training dialogues,neural models are inclined to assign high probabili-ties to “safe” responses, due to the fact that vacuousresponses like “I don’t know” are of relatively highfrequencies in conversational datasets (Li et al.,2016a).

One promising training framework for neuraldialogue generation is adversarial learning (Good-fellow et al., 2014; Li et al., 2017), where a dis-criminator provides rewards for the generator bycontrastively distinguishing dialogues as human-generated or machine-generated. However, thelearning ability of GANs in text is drasticallylimited due to training instability and model col-lapse (Nie et al., 2019; Caccia et al., 2020). First,the discriminator is usually unlikely to be fooledvery easily, and the generator can hardly learnfrom those ineffective rewards. Second, the gener-ator is sometimes encouraged to mimic the high-frequency generic responses in the training corpus,because in some cases, the discriminator fails todistinguish a good response from a bad one: it caneasily recognize contentful but less-grammaticalresponses as machine-generated, yet treat thosehuman-generated dull responses as the oracle.

In this paper, we introduce contrastive learn-ing (Hadsell et al., 2006; Gutmann and Hyvarinen,2012) into dialogue generation, where the modelexplicitly perceives the difference between the well-chosen positive and negative utterances. From theperspective of contrastive learning, the discrim-

arX

iv:2

009.

0754

3v2

[cs

.CL

] 1

3 O

ct 2

020

ResponseContext Pull togetherPush apartc<latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit> r

<latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit> Matching Degree Low High

We are both foodies!

He would only lay on the porch at night.

I enjoy swimming and reading.How about you?

I like to read John Grisham books.

I just hang out with my 10 cats.

Okie dokie. Here you go.

I do not like cooking. My hobby is to watch sports.

Thanks for the offer though.

I like it, too!

c<latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit><latexit sha1_base64="+A6VEtos3HL/1r2wKYFonhC20Mo=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhxmaDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHdUJEH</latexit>

What are your hobbies? I love to cook.

r<latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit><latexit sha1_base64="juw6eQUUFM3i37Bw8Ysq5tbV1Ao=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhpmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AH0G5EW</latexit>

Reading is my favorite hobby.

A given training pair

r+<latexit sha1_base64="7qhK8ysrEYp56i4P3aViZbHM5Ok=">AAAB/3icbVC7SgNBFL0bXzG+opY2g0EQhLAbBS2DNpYRzAOSNcxOZpMhM7PLzKwQlhT+gq32dmLrp9j6JU42W2jigQuHc+7lXE4Qc6aN6345hZXVtfWN4mZpa3tnd6+8f9DSUaIIbZKIR6oTYE05k7RpmOG0EyuKRcBpOxjfzPz2I1WaRfLeTGLqCzyULGQEGyt1eoFI1cPZtF+uuFU3A1omXk4qkKPRL3/3BhFJBJWGcKx113Nj46dYGUY4nZZ6iaYxJmM8pF1LJRZU+2n27xSdWGWAwkjZkQZl6u+LFAutJyKwmwKbkV70ZuK/XiAWkk145adMxomhksyDw4QjE6FZGWjAFCWGTyzBRDH7OyIjrDAxtrKSLcVbrGCZtGpV77xau7uo1K/zeopwBMdwCh5cQh1uoQFNIMDhGV7g1Xly3px352O+WnDym0P4A+fzB1vxlpI=</latexit>

r-<latexit sha1_base64="YGhG3kA1sz+93BPKCxvsP/ihBJo=">AAACCHicbVDLSsNAFJ3UV62PRl26GSyCG0tSBV0W3bisYB/QxjKZTtqhM0mYuRFLyA/4C251707c+hdu/RKnbRbaeuDC4Zx7OZfjx4JrcJwvq7Cyura+UdwsbW3v7Jbtvf2WjhJFWZNGIlIdn2gmeMiawEGwTqwYkb5gbX98PfXbD0xpHoV3MImZJ8kw5AGnBIzUt8s9X6Yqu+8Be4T0NOvbFafqzICXiZuTCsrR6NvfvUFEE8lCoIJo3XWdGLyUKOBUsKzUSzSLCR2TIesaGhLJtJfOHs/wsVEGOIiUmRDwTP19kRKp9UT6ZlMSGOlFbyr+6/lyIRmCSy/lYZwAC+k8OEgEhghPW8EDrhgFMTGEUMXN75iOiCIUTHclU4q7WMEyadWq7lm1dnteqV/l9RTRITpCJ8hFF6iOblADNRFFCXpGL+jVerLerHfrY75asPKbA/QH1ucP4G6aJA==</latexit>

c+<latexit sha1_base64="OrOgViwM3qSRRqNvypZDLtnD6fc=">AAACCHicbVDLSsNAFJ3UV62PRl26GSyCIJSkCrosunFZwT6gjWUynbRDZ5IwcyOWkB/wF9zq3p249S/c+iVO2yy09cCFwzn3ci7HjwXX4DhfVmFldW19o7hZ2tre2S3be/stHSWKsiaNRKQ6PtFM8JA1gYNgnVgxIn3B2v74euq3H5jSPArvYBIzT5JhyANOCRipb5d7vkxpdt8D9gjpada3K07VmQEvEzcnFZSj0be/e4OIJpKFQAXRuus6MXgpUcCpYFmpl2gWEzomQ9Y1NCSSaS+dPZ7hY6MMcBApMyHgmfr7IiVS64n0zaYkMNKL3lT81/PlQjIEl17KwzgBFtJ5cJAIDBGetoIHXDEKYmIIoYqb3zEdEUUomO5KphR3sYJl0qpV3bNq7fa8Ur/K6ymiQ3SETpCLLlAd3aAGaiKKEvSMXtCr9WS9We/Wx3y1YOU3B+gPrM8fxQKaEw==</latexit>

c-<latexit sha1_base64="MgSCy0oPvF990YhkTe3LIk0r8Bw=">AAACCHicbVDLSsNAFJ3UV62PRl26GSyCG0tSBV0W3bisYB/QxjKZTtqhM0mYuRFLyA/4C251707c+hdu/RKnbRbaeuDC4Zx7OZfjx4JrcJwvq7Cyura+UdwsbW3v7Jbtvf2WjhJFWZNGIlIdn2gmeMiawEGwTqwYkb5gbX98PfXbD0xpHoV3MImZJ8kw5AGnBIzUt8s9X6Y0u+8Be4T0NOvbFafqzICXiZuTCsrR6NvfvUFEE8lCoIJo3XWdGLyUKOBUsKzUSzSLCR2TIesaGhLJtJfOHs/wsVEGOIiUmRDwTP19kRKp9UT6ZlMSGOlFbyr+6/lyIRmCSy/lYZwAC+k8OEgEhghPW8EDrhgFMTGEUMXN75iOiCIUTHclU4q7WMEyadWq7lm1dnteqV/l9RTRITpCJ8hFF6iOblADNRFFCXpGL+jVerLerHfrY75asPKbA/QH1ucPyCyaFQ==</latexit>

Figure 1: An illustration case of group-wise contrastive learning. For a given training instance, the proposedframework explicitly considers the multi-mapping relations in human conversations, by encouraging the dialoguegeneration model to pull the matched sample pairs together and push the mismatched pairs apart in the latent space.

inator in adversarial learning considers human-generated responses as positive utterances and syn-thetic ones as negative samples. Instead, this workdeems highly-matched context-response pairs aspositive samples and mismatched training pairsas negative samples. In particular, we utilize apretrained baseline model as a reference. Dur-ing contrastive learning, for context c and its re-sponse r, the target dialogue model is trained togive higher conditional probabilities p(r|c) for thepositive samples, and lower conditional probabili-ties for those negative samples, compared to the ref-erence model. This training paradigm encouragesthe model to pull the positive data points togetherand push apart the negative samples, as exempli-fied in Figure 1. As a result, our proposed train-ing scheme explicitly takes the semantic associa-tions and differences among training examples intoaccount for dialogue modeling. Besides, by con-trastively characterizing the distinctions relative toa strong reference, our method implicitly enhancesthe distinctiveness of the generated responses aswell, and ensures that the overall performance ofthe target model is not inferior to the reference.

Contrastively learning from one pair of positiveand negative samples is quite straightforward, how-ever, multi-mapping relations prevail in human-human conversations, where there exist multipleappropriate responses for a given context, and aresponse sometimes fits well to several contexts,known as one-to-many and many-to-one relations.Such complex multi-mapping relations are over-

looked in previous learning framework, which ham-pers effective dialogue response learning. Further-more, if a potential highly-matched utterance pair istreated as the negative sample or an outlier is usedas the positive sample, the model may be confused.Therefore, in order to consider the multi-mappingphenomenon in human conversations and remedythe potential problematic false learning samples,and enhance the training stability, we augment con-trastive learning with group-wise dual sampling,where groups of positive and negative instancesare sampled regarding both the context and theresponse, respectively. To further depict subtle dif-ferences between instances in the group, we adaptthe instance importance with the matching scores,and optimize the weighted loss.

We show an illustration case to understand ourlearning framework in Figure 1. Given a trainingcontext-response pair (c, r), for context “What areyour hobbies? I love to cook”, multiple highly-matched responses are organized as the positivesamples r+, and the mismatched utterances aredeemed as the negatives r-. On the dual direction,regarding the response “Reading is my favoritehobby”, multiple sampled context utterances aresimilarly divided into c+ and c-. Compared withthe reference baseline, the target dialogue model istrained to give higher generation probabilities forpositive instances, (c, r+) and (c+, r), and lowerprobabilities for negatives (c, r-) and (c-, r). Bythis mean, the target model is actually induced topull the positive sample pairs together and push the

mismatched pairs apart, and thus learns from thedistinctions between the positives and negatives.

The proposed group-wise contrastive learningframework is suited for training various neural di-alogue generation models. We conduct extensivestudies on three large-scale conversation datasetsusing four popular dialogue models to assess theproposed approach. The experimental results con-firm the effectiveness of our learning frameworkwith very favorable performance over the baselinetraining approaches1.

2 Contrastive Dialogue Learning

2.1 Dialogue Learning by Comparison

Given training data D containing context-responsepairs {(c, r)i}Ni=1, a dialogue model parameterizedby θ aims to map from the input context c to theoutput response r. To achieve this, conventional di-alogue learning approaches search the parameter θby maximizing the conditional probability pθ(r|c)over the training samples. MLE maximizes the log-likelihood of training pairs while adversarial basedapproaches rely on the discriminator to distinguishbetween good responses and bad ones. To com-bat the aforementioned drawbacks of traditionaltraining approaches in dialogue learning, we ad-vocate the use of contrastive learning to explic-itly perceive the difference between the positiveand negative samples. Inspired by Gutmann andHyvarinen (2012); Dai and Lin (2017), we utilize apretrained baseline model pn(·;φ), to provide thetarget dialogue model pm(·;θ) a strong referencewhen contrasting the positive samples and the neg-atives. Humans not only learn from the positivesignals but also benefit from correcting behaviorsof undesirable actions. Intuitively, the target dia-logue model is expected to give higher conditionalprobabilities p(r|c) for the positive samples, andlower conditional probabilities for those negativesamples, compared to the reference model. To-wards this end, we define the difference betweenpm(r|c,θ) and pn(r|c,φ) as:

D((c, r);θ,φ) = logpm(r|c,θ)pn(r|c,φ)

. (1)

We wish that D((c, r);θ,φ) > 0 for any positivepair and vice versa for any negative pair. Con-cretely speaking, we minimize the following loss

1Code is available at https://github.com/hengyicai/ContrastiveLearning4Dialogue

function:

L′′(θ;D,φ) =

− 1

N

∑(c,r)∈D

log σ(D((c, r)+;θ,φ))

− 1

N

∑(c,r)∈D

log [1− σ(D((c, r)-;θ,φ))]

, (2)

where σ(·) is the sigmoid activation function, thegiven training pair (c, r) can be used as the positivesample (c, r)+ and the negative sample (c, r)- canbe obtained through negative sampling using thegiven instance (c, r).

Optimizing the dialogue model with the aboveobjective function is reminiscent of nonlinear lo-gistic regression in Noise-Contrastive Estimation(NCE) (Gutmann and Hyvarinen, 2012). The un-derlying motivation of our formulation and NCEare essentially different. The reference model inour work is utilized to constrain the behaviors ofthe target model, rather than serve as a noise distri-bution to provide noise data. Another difference isthat, instead of using the log-ratio between pm(·;θ)and pn(·;φ) to compute posterior classificationprobabilities as in NCE, we introduce the func-tion D((c, r);θ,φ) to characterize the distinctionsof intrinsic dialogue properties relative to the refer-ence, and encourage the generation of positive sam-ples as well as penalize the negative ones throughminimizing the loss in Eq.(2). Besides, by con-trastively characterizing the distinctions relative toa strong reference, our method implicitly enhancesthe distinctiveness of the generated response aswell, and ensures that the overall performance ofthe target model is not inferior to the reference.

2.2 Contrastive Dual SamplingNevertheless, in the presence of multi-mapping re-lations in human dialogues, effectively samplingthe positive and negative pairs in conversation isnot that straightforward and even runs the risk ofintroducing false learning samples. To managethe complex multi-mapping phenomenon in humanconversations and enhance the training stability,we augment the contrastive learning with group-wise dual sampling, where groups of positive andnegative instances are sampled regarding both thecontext and the response, respectively. To put itconcretely, for each training instance (c, r), wefind a group of positive examples {(c, r+)i}ki=1

with highest matching degree and a group of nega-tive examples {(c, r-)i}ki=1 with lowest matchingdegree, using an off-the-shelf pretrained matching

https://github.com/hengyicai/ContrastiveLearning4Dialogue

https://github.com/hengyicai/ContrastiveLearning4Dialogue

TrainingData

Matching Model

Context

Response

Contrastive Dual Sampling

-1

+1

MatchingScore Group-wise Contrastive Learning

(c, r+)<latexit sha1_base64="u4sKZJgRUy0RI4hwin8w23HxOdM=">AAACDXicbVDLSsNAFJ3UV62v+Ni5GSxCRSlJFXRZdOOygn1AG8tkOqlDZyZhZiLUkG/wF9zq3p249Rvc+iVO2iy09cDlHs65l3s5fsSo0o7zZRUWFpeWV4qrpbX1jc0te3unpcJYYtLEIQtlx0eKMCpIU1PNSCeSBHGfkbY/usr89gORiobiVo8j4nE0FDSgGGkj9e29Ss/nCU5PYNblXXKcpkd9u+xUnQngPHFzUgY5Gn37uzcIccyJ0JghpbquE2kvQVJTzEha6sWKRAiP0JB0DRWIE+Ulk+9TeGiUAQxCaUpoOFF/bySIKzXmvpnkSN+rWS8T//V8PnNZBxdeQkUUayLw9HAQM6hDmEUDB1QSrNnYEIQlNb9DfI8kwtoEWDKhuLMRzJNWreqeVms3Z+X6ZR5PEeyDA1ABLjgHdXANGqAJMHgEz+AFvFpP1pv1bn1MRwtWvrML/sD6/AFGNZtW</latexit>

(c, r�)<latexit sha1_base64="bm9BiwILY0RusrPL2Y/45TCIUFg=">AAACDXicbVDLSsNAFJ3UV62v+Ni5GSxCBS1JFXRZdOOygn1AG8tkOqlDZyZhZiLUkG/wF9zq3p249Rvc+iVO2iy09cDlHs65l3s5fsSo0o7zZRUWFpeWV4qrpbX1jc0te3unpcJYYtLEIQtlx0eKMCpIU1PNSCeSBHGfkbY/usr89gORiobiVo8j4nE0FDSgGGkj9e29Ss/nCU6PYdblXXKSpkd9u+xUnQngPHFzUgY5Gn37uzcIccyJ0JghpbquE2kvQVJTzEha6sWKRAiP0JB0DRWIE+Ulk+9TeGiUAQxCaUpoOFF/bySIKzXmvpnkSN+rWS8T//V8PnNZBxdeQkUUayLw9HAQM6hDmEUDB1QSrNnYEIQlNb9DfI8kwtoEWDKhuLMRzJNWreqeVms3Z+X6ZR5PEeyDA1ABLjgHdXANGqAJMHgEz+AFvFpP1pv1bn1MRwtWvrML/sD6/AFJY5tY</latexit>

(c+, r)<latexit sha1_base64="umJ09Phfo2CnSY2Ww9RF9+llNQk=">AAACDXicbZDLSsNAFIYn9VbrLV52bgaLUFFKUgVdFt24rGAv0MYymU7q0JlJmJkINeQZfAW3uncnbn0Gtz6JkzYLbf3hwMd/zuEcfj9iVGnH+bIKC4tLyyvF1dLa+sbmlr2901JhLDFp4pCFsuMjRRgVpKmpZqQTSYK4z0jbH11l/fYDkYqG4laPI+JxNBQ0oBhpY/XtvUrP5wlO75Lj9ARmLNOjvl12qs5EcB7cHMogV6Nvf/cGIY45ERozpFTXdSLtJUhqihlJS71YkQjhERqSrkGBOFFeMvk+hYfGGcAglKaEhhP390aCuFJj7ptJjvS9mu1l5r89n89c1sGFl1ARxZoIPD0cxAzqEGbRwAGVBGs2NoCwpOZ3iO+RRFibAEsmFHc2gnlo1aruabV2c1auX+bxFME+OAAV4IJzUAfXoAGaAINH8AxewKv1ZL1Z79bHdLRg5Tu74I+szx9HLJtW</latexit>

(c�, r)<latexit sha1_base64="6kYXDIMYqK74dHTDL8Y/CXX2Xzc=">AAACDXicbZDLSsNAFIYn9VbrLV52bgaLUEFLUgVdFt24rGAv0MYymU7q0JlJmJkINeQZfAW3uncnbn0Gtz6JkzYLbf3hwMd/zuEcfj9iVGnH+bIKC4tLyyvF1dLa+sbmlr2901JhLDFp4pCFsuMjRRgVpKmpZqQTSYK4z0jbH11l/fYDkYqG4laPI+JxNBQ0oBhpY/XtvUrP5wlO75KT9BhmLNOjvl12qs5EcB7cHMogV6Nvf/cGIY45ERozpFTXdSLtJUhqihlJS71YkQjhERqSrkGBOFFeMvk+hYfGGcAglKaEhhP390aCuFJj7ptJjvS9mu1l5r89n89c1sGFl1ARxZoIPD0cxAzqEGbRwAGVBGs2NoCwpOZ3iO+RRFibAEsmFHc2gnlo1aruabV2c1auX+bxFME+OAAV4IJzUAfXoAGaAINH8AxewKv1ZL1Z79bHdLRg5Tu74I+szx9KaJtY</latexit>

(a) (b)

c<latexit sha1_base64="tmp+6tTi87FhxZC7UBjnYGj75Dw=">AAAB/XicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYByRLmJ3MJmPmsczMCmEJ/oKt9nZi67fY+iVOki008cCFwzn3ci4nSjgz1ve/vJXVtfWNzcJWcXtnd2+/dHDYNCrVhDaI4kq3I2woZ5I2LLOcthNNsYg4bUWjm6nfeqTaMCXv7TihocADyWJGsHVSsxuJjEx6pbJf8WdAyyTISRly1Hul725fkVRQaQnHxnQCP7FhhrVlhNNJsZsammAywgPacVRiQU2Yzb6doFOn9FGstBtp0Uz9fZFhYcxYRG5TYDs0i95U/NeLxEKyja/CjMkktVSSeXCccmQVmlaB+kxTYvnYEUw0c78jMsQaE+sKK7pSgsUKlkmzWgnOK9W7i3LtOq+nAMdwAmcQwCXU4Bbq0AACD/AML/DqPXlv3rv3MV9d8fKbI/gD7/MHHN6V5g==</latexit>

r<latexit sha1_base64="H89hYtMN/TRYqF40Omfg0wcgqzo=">AAAB/XicbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgZtLCOYByRLmJ3MJmPmsczMCmEJ/oKt9nZi67fY+iVOki008cCFwzn3ci4nSjgz1ve/vJXVtfWNzcJWcXtnd2+/dHDYNCrVhDaI4kq3I2woZ5I2LLOcthNNsYg4bUWjm6nfeqTaMCXv7TihocADyWJGsHVSsxuJTE96pbJf8WdAyyTISRly1Hul725fkVRQaQnHxnQCP7FhhrVlhNNJsZsammAywgPacVRiQU2Yzb6doFOn9FGstBtp0Uz9fZFhYcxYRG5TYDs0i95U/NeLxEKyja/CjMkktVSSeXCccmQVmlaB+kxTYvnYEUw0c78jMsQaE+sKK7pSgsUKlkmzWgnOK9W7i3LtOq+nAMdwAmcQwCXU4Bbq0AACD/AML/DqPXlv3rv3MV9d8fKbI/gD7/MHNJmV9Q==</latexit>

Figure 2: A demonstration of the proposed group-wise contrastive dialogue learning pipeline. For each trainingpair, it first samples a group of highly-matched examples and another group of most mismatched utterances regard-ing both the context and response to build the contrastive samples, using an off-the-shelf conversation matchingmodel (§2.2). The target dialogue model is then trained with group-wise contrastive learning (§2.3).

model to compute the matching scores between thegiven context and candidate responses. Similarly,{(c+, r)i}ki=1 and {(c-, r)i}ki=1 are also retrievedfrom the training set to serve as the context-sidecontrastive examples, as shown in Figure 2(a). Inthis work, we adopt MSN (Yuan et al., 2019), acontext-response matching network based on multi-hop selection, as the off-the-shelf matching model.Note that other sophisticated matching models canalso be applied, e.g., deep attention matching net-work (Zhou et al., 2018).

2.3 Group-wise Contrastive LearningFor each training instance (c, r), as describe in§2.2, we sample k different positive and negativepairs regarding both the dialogue context and its re-sponse to manage multi-mapping relations in con-versation and stabilize the model training. Theresultant well-chosen samples are composed ofpositive samples, {(c, r+)i}ki=1 and {(c+, r)i}ki=1,and the negatives, {(c, r-)i}ki=1 and {(c-, r)i}ki=1.Then, the loss function is updated as:

L′(θ;D,φ) =

− 1

N

∑(c,r)∈D

1

2k + 1

2k+1∑i=1

log σ(D((c, r)+i ;θ,φ))

− 1

N

∑(c,r)∈D

1

2k

2k∑i=1

log [1− σ(D((c, r)-i;θ,φ))]

. (3)

Given varied matching degrees of the collectedcontext-response pairs in open-domain dialogue,indiscriminately training on such data impedes themodel to perceive intra-group differences of thesesamples. We thus utilize the matching score s at-tached with each sample to adapt its instance effecton the group-wise contrastive dialogue learning.Specifically, for a given training example (c, r),

the matching score s+ of its positive pair lies in(0, 1] and the negative score s- lies in [−1, 0]. Toinduce the model learning from sample pairs withvaried matching degrees discriminately, the lossfunction is finally defined to be:L(θ;D,φ) =

− 1

N

∑(c,r)∈D

1

2k + 1

2k+1∑i=1

log [s+i · σ(D((c, r)+

i ;θ,φ))]

− 1

N

∑(c,r)∈D

1

2k

2k∑i=1

log [1 + s-i · σ(D((c, r)-

i;θ,φ))]

.

(4)

The loss function L(θ) reaches its lower boundwhen the positive and negative pairs can be per-fectly distinguished, i.e., pm(r|c,θ)� pn(r|c,φ)for the positive samples and pm(r|c,θ) �pn(r|c,φ) for the negatives, which indicates thatthe target dialogue model is able to clearly contrasta group of positive candidates from the negativeones and generate highly-distinctive responses forthe given contexts.

2.4 DiscussionNeural sequence-to-sequence models trained withthe MLE objective function are plagued by the low-diversity issue when it comes to the open-domainconversational setting, in which bland and genericutterances usually dominate the data distribution.Since the objective of MLE is to maximize onlythe probabilities of ground-truth context-responsepairs, it fails to capture the multi-mapping nature ofhuman dialogues, not to mention the semantic dif-ferences among various candidates for a given ex-ample. While the proposed group-wise contrastivelearning framework explicitly explores multiplevariants of a given dialogue example by leverag-ing an off-the-shelf matching model, and implicitly

PersonaChat Douban OpenSubtitles

#context-response pairs 140,248 218,039 353,046Avg. #turns per context 2.69 3.94 3.79Avg. #words per utterance 11.96 15.28 6.85Training Pairs 113,558 198,039 293,129Validation Pairs 13,602 10,000 29,960Test Pairs 13,088 10,000 29,957#Tokens 18,029 40,000 40,000

Table 1: Data statistics for PersonaChat, Douban andOpenSubtitles datasets.

guarantees the ground-truth generation probabili-ties through the contrastive constraints in Eq.(4).

Adversarial learning approaches and our pro-posed framework both involve an auxiliary modelduring the training process. However, GANs arelearned via a competition between the target gen-erator and the counteracting discriminator, whichneeds careful tuning to prevent model collapse intext modeling (Caccia et al., 2020), whereas in ourframework, the auxiliary reference system mod-els conversation data in the same direction withthe target dialogue model, and is stable during thelearning procedure.

3 Experiments

3.1 Experiment Settings

Datasets We perform experiments on three con-versation datasets: PersonaChat (Zhang et al.,2018b), Douban Corpus (Wu et al., 2017) andOpenSubtitles (Lison and Tiedemann, 2016). Per-sonaChat, an English-language dataset, containsmulti-turn dialogues between pairs of speakers,collected via Amazon Mechanical Turk. Doubanconsists of daily conversations from a popular so-cial networking service—Douban group2 in China.OpenSubtitles contains human-human conversa-tions converted from movie transcripts in English.Data statistics are listed in Table 1.

Experimental Models We apply the proposedgroup-wise contrastive learning framework toseveral state-of-the-art models, including (i)SEQ2SEQ: a LSTM-based sequence-to-sequencemodel with attention mechanisms (Bahdanau et al.,2015), (ii) HRED: a hierarchical recurrent neu-ral dialogue generation model (Serban et al.,2016), (iii) TRANSFORMER: an encoder-decoderarchitecture relying solely on attention mecha-nisms (Vaswani et al., 2017), (iv) HRAN: a hier-archical recurrent attention network for multi-turn

2https://www.douban.com/group

response generation (Xing et al., 2018). Eachmodel is trained using two protocols: the vanillaMLE training procedure and the proposed group-wise contrastive learning procedure, keeping otherconfigurations the same.

Baselines We compare our group-wise con-trastive learning framework against the followingdialogue learning approaches: (i) ADVERSARIAL:an adversarial training approach for response gen-eration (Li et al., 2017), (ii) MMI: a training ob-jective which maximums the mutual informationbetween the dialogue context and its response (Liet al., 2016a; Zhang et al., 2018c), (iii) DEEPRL:a reinforcement learning framework for neuralresponse generation with heuristic reward func-tions to boost response qualities (Li et al., 2016b),(iv) CVAE: a conditional variational auto-encoderlearning framework to maximize the data like-lihood, augmented with the KL-annealing tech-nique (Bowman et al., 2016) and a BOW loss (Zhaoet al., 2017), and (v) DIALOGWAE: a conditionalWasserstein auto-encoder framework, modeling thedistribution of dialogues by training a GAN withinthe latent variable space (Gu et al., 2019).

Automatic Evaluation Metrics We adopt sev-eral standard metrics widely used in existing worksto measure the performance of dialogue gener-ation models, including BLEU (Papineni et al.,2002), embedding-based metrics (Average, Ex-trema, Greedy and Coherence)3 (Liu et al., 2016;Xu et al., 2018; Sedoc et al., 2019), entropy-basedmetrics4 (Ent-{1,2}) (Serban et al., 2017) and dis-tinct metrics (Dist-{1,2,3}) (Li et al., 2016a).

3.2 Implementation and Reproducibility

We implement our model in ParlAI (Miller et al.,2017) and train them on Nvidia P40 GPUs. All themodels use pretrained word embeddings producedby fastText (Bojanowski et al., 2017), and the di-mensionality of word vectors is 300. For experi-mental models, 2-layer LSTM-based encoder anddecoder with hidden size 256 are used in SEQ2SEQ.We use the base Transformer configuration de-scribed in Vaswani et al. (2017), i.e., 6 blockswith 8 attention heads and 512 hidden units. 2-layer GRU-based RNNs are employed to build theword-level encoder, utterance-level encoder and

3https://chateval.org/4We compute the entropy value for the empirical distribu-

tion of n-grams based on the maximum likelihood estimationon the training data.

https://www.douban.com/group

https://chateval.org/

Models BLEU-1 / 2 / 3 / 4 Dist-1 Dist-2 Dist-3 Avg. Ext. Gre. Coh. Ent-1 / 2

(a)

SEQ2SEQ 12.040 / 3.9950 / 0.8815 / 0.2312 0.4309 2.045 4.303 36.33 28.66 63.64 41.66 6.891 / 10.81SEQ2SEQ (I) 13.660 / 4.9160 / 1.5970 / 0.6122 0.8492 5.093 12.000 39.76 31.74 64.76 49.39 7.016 / 10.90HRED 12.410 / 3.8360 / 0.8455 / 0.2364 0.4744 2.546 6.127 36.52 28.37 64.12 39.08 6.792 / 10.66HRED (I) 13.180 / 4.3220 / 1.0360 / 0.3274 0.7130 4.468 11.220 38.54 29.65 64.26 44.07 6.931 / 10.84TRANSFORMER 11.460 / 3.2080 / 0.5389 / 0.1476 0.4813 2.544 6.146 35.72 27.38 63.61 38.02 6.804 / 10.55TRANSFORMER (I) 12.660 / 3.8920 / 0.8406 / 0.2577 0.7859 4.562 10.950 37.42 28.95 64.02 41.96 6.918 / 10.80HRAN 12.190 / 3.8290 / 0.7752 / 0.2171 0.5074 2.883 7.104 36.53 28.08 63.58 40.22 6.964 / 10.83HRAN (I) 13.430 / 4.5030 / 1.0630 / 0.3513 0.7713 4.974 12.380 39.04 30.08 64.48 46.63 6.942 / 10.87

(b)


(c)


Table 2: Automatic evaluation results (%) on the test set of three datasets: (a) PersonaChat, (b) Douban and(c) OpenSubtitles. “I” denotes that the model is trained using our proposed framework. The metrics Average,Extrema, Greedy and Coherence are abbreviated as Avg., Ext., Gre. and Coh., respectively. The best results ineach group are highlighted with bold.

response decoder for both the HRED and HRAN.The GRU hidden size is set to 256. For both mod-els using different training procedures, we pretrainthem by MLE and the result checkpoint is adoptedas the reference model used in our framework. Weemploy BM25 (Robertson and Zaragoza, 2009) toconstruct the index used during the contrastive dualsampling procedure. The group size k is set to 3.Regarding comparison models, we adopt the de-fault configurations used in the original papers. Weoptimize models by Adam (Kingma and Ba, 2015)with an initial learning rate of 0.001 and the batchsize of 128. All systems are trained until the vali-dation loss fails to decrease for 5 checkpoints. Wecompute the loss on the validation set at every 0.5epochs and save the parameters for the top model.Evaluation scores on the test set from the savedmodel are finally reported.

3.3 Evaluation Results

Performance on Experimental Models We in-stantiate the proposed framework on several state-of-the-art models for dialogue generation. Table 2reports the automatic evaluation results of our learn-ing framework and the conventional MLE trainingprocedure. By training dialogue models using the

proposed learning framework, we witness solid per-formance boosts on three conversation datasets interms of almost all the evaluation metrics, com-pared to the vanilla training. Such improvementsare also consistent across various experimental ar-chitectures, affirming the general applicability andsuperiority of the proposed framework.

Comparison with Baseline Approaches Wecompare our proposed framework with existinglearning approaches designed for dialogue gener-ation task. Table 3 summarizes the evaluation re-sults. We observe that our learning frameworkoutperforms previous approaches regarding the ma-jority of evaluation metrics. It is worth noting thatthe proposed framework brings a relatively largeimprovement regarding both the response diver-sity and conversation coherence, indicating thatour approach helps the dialogue model to gener-ate not only informative but also context-relevantresponses, which confirms our hypothesis that thegroup-wise contrastive learning encourages distinc-tiveness.

Human Evaluation We further conduct humanevaluations to assess the proposed learning frame-work. We choose the PersonaChat as our evaluation

Learning Approaches BLEU-1 / 2 / 3 / 4 Dist-1 Dist-2 Dist-3 Avg. Ext. Gre. Coh. Ent-1 / 2

ADVERSARIAL 12.190 / 4.0060 / 0.8950 / 0.2644 0.6269 3.357 7.374 35.93 29.00 63.65 42.38 6.980 / 10.88MMI 14.030 / 4.6460 / 1.3340 / 0.5022 0.4734 2.443 5.515 39.34 30.92 64.84 45.16 6.874 / 10.65DEEPRL 12.660 / 4.0150 / 1.0140 / 0.3314 0.6838 3.838 8.581 37.23 29.68 64.30 44.13 6.885 / 10.85CVAE 11.570 / 2.8100 / 0.6357 / 0.1714 0.2876 2.326 7.506 39.29 30.61 63.67 41.76 6.869 / 10.82DIALOGWAE 11.430 / 2.9260 / 0.5676 / 0.1436 0.9936 5.080 9.928 38.68 28.70 63.39 41.06 7.009 / 11.09Ours 13.660 / 4.9160 / 1.5970 / 0.6122 0.8492 5.093 12.000 39.76 31.74 64.76 49.39 7.016 / 10.90

Table 3: Performance (%) of our approach instantiated on naive SEQ2SEQ and baseline approaches on Per-sonaChat.

Opponent Win Loss Tie Kappa

Ours vs. VANILLA MLE 53% 10% 37% 0.5750Ours vs. ADVERSARIAL 47% 15% 38% 0.5495Ours vs. MMI 43% 12% 45% 0.5863Ours vs. DEEPRL 40% 22% 38% 0.6036Ours vs. CVAE 40% 15% 45% 0.5510Ours vs. DIALOGWAE 45% 18% 37% 0.4216

Table 4: The results of human evaluation on the test setof PersonaChat.

corpus since its expressions are more close to thestyle of daily chat and are easier for the annotatorsto make judgments. Three crowd-sourced graduatestudents are employed to evaluate the quality ofgenerated responses for 100 randomly sampled in-put contexts. During the evaluation, the annotatorsare requested to select a preferred response, or votea tie, considering the following aspects of responsequality: fluency, informativeness, coherence andengagingness. Table 4 summarizes the evaluationresults and the Cohen’s kappa scores (Cohen, 1960)to measure the intra-rater reliability. We observethat our learning framework brings more prefer-able replies compared with the competitors. Thisindicates that training the dialogue model with theproposed group-wise contrastive learning frame-work does improve the response quality.

3.4 Model Analysis

Effect of the Group-wise Learning StrategyTo manage the multi-mapping relations in human-human conversation and stabilize the model train-ing with noisy data, the dialogue model is inducedto contrast a group of positive samples from thenegative ones, pulling the matched sample pairs to-gether and pushing the mismatched pairs apart. Weablate the group-wise learning from the frameworkby using only one pair of positive and negativesamples to verify its effectiveness. As presentedin Table 5 (a), we can see that disabling group-wise learning hurts performance significantly onall evaluation metrics. Note that ablating either

the group-wise positive sampling (Table 5 (b)) orgroup-wise negative sampling (Table 5 (c)) alsoleads to a performance drop with respect to theevaluation metrics. It demonstrates that the group-wise learning strategy plays a key role in achievingstrong performance.

Effect of the Dual Sampling In our framework,the contrastive samples can be organized regardingeither the dialogue context or response, allowingthe dialogue model to explore both the many-to-one and one-to-many relations in human conversa-tion. We investigate different sampling strategiesin Table 5 (d) and (e). We notice that when boththe response-side and context-side samplings to-gether incorporated into the learning framework,the model achieves its best performance, verifyingthe effectiveness of the contrastive dual sampling.

Impact of Matching Scores To discriminativelyexploit the sampled context-response pairs withvaried matching degrees, we utilize the matchingscore attached with each sample to adapt its in-stance effect on the model training. We conductthe ablation test of this learning strategy by sim-ply discarding the impact of matching scores asin Eq.(3). As shown in Table 5 (f), training with-out considering matching degrees of samples leadsto a consistent performance drop, which suggeststhat the system can benefit from perceiving fine-grained differences within the group during con-trastive learning.

Impact of Group Size We explore the impactof using different group size k in our group-wisecontrastive learning framework in Figure 3. We ob-serve that increasing the group size k leads to con-tinuous improvement on the Distinct metric whileother reference-based metrics achieve the best re-sults at a moderate group size. We conjecture thata larger group size allows the dialogue model tolearn from more diverse expressions, meanwhileit also risks introducing more utterances that are

Framework variants BLEU-1 / 2 / 3 / 4 Dist-1 Dist-2 Dist-3 Avg. Ext. Gre. Coh. Ent-1 / 2

(a) w/o group-wise sampling 12.870 / 4.102 / 0.9564 / 0.2308 0.3965 2.070 4.633 36.52 29.09 64.21 42.40 6.836 / 10.62(b) w/o group-wise positive sampling 13.120 / 4.800 / 1.4180 / 0.5967 0.4632 2.270 5.002 38.26 31.18 64.66 43.03 6.812 / 10.49(c) w/o group-wise negative sampling 13.210 / 4.698 / 1.3970 / 0.5587 0.7175 3.532 7.473 38.23 30.96 64.62 46.27 6.911 / 10.68(d) w/o response-side sampling 13.340 / 4.730 / 1.4820 / 0.5779 0.8487 4.964 11.340 39.31 31.51 64.66 48.35 6.938 / 10.75(e) w/o context-side sampling 13.170 / 4.539 / 1.4160 / 0.5308 0.8455 4.892 11.210 39.57 31.81 64.56 47.19 6.904 / 10.66(f) w/o impact of matching scores 13.560 / 4.359 / 1.1140 / 0.3823 0.6086 3.809 9.037 38.78 30.35 64.44 46.88 6.952 / 10.90Full version 13.660 / 4.916 / 1.5970 / 0.6122 0.8492 5.093 12.000 39.76 31.74 64.76 49.39 7.016 / 10.90

Table 5: Ablation test (%) using SEQ2SEQ with different framework variants on PersonaChat.

12.812.913

13.113.213.313.4

1 2 3 4 5 6 7

BLEU

37.838

38.238.438.638.839

39.2

1 2 3 4 5 6 7

Average

0.8

0.850.9

0.95

1

1 2 3 4 5 6 7

Distinct

29.830

30.230.430.630.831

1 2 3 4 5 6 7

Extrema

6464.164.264.364.464.564.6

1 2 3 4 5 6 7

Greedy

46.547

47.548

48.549

49.5

1 2 3 4 5 6 7

Coherencek<latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit>

k<latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit><latexit sha1_base64="FN4g9GFe+3ziaTgM0DMwdUeUKp0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipORmUK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4A0oWM7w==</latexit>





Figure 3: Evaluation results (%) with different group size k on the validation set of PersonaChat using the proposedframework instantiated on SEQ2SEQ. BLEU-1 and Dist-1 are denoted as “BLEU” and “Distinct”, respectively.

inconsistent with the references.

4 Related Work

Learning Methods for Dialogue GenerationTypically, state-of-the-art neural dialogue genera-tion models adopt Maximum Likelihood Estimation(MLE) as their learning approach, maximizing log-likelihood of model parameters on the training data.Though effective, well-known issues, including thenotorious general dull response problem, are re-ported by prior art (Li et al., 2016a; Zhao et al.,2017) on dialogue models trained with MLE.

Alternative dialogue learning approaches are pro-posed to tackle the issues. Li et al. (2016a); Zhanget al. (2018c) introduce the Maximum Mutual In-formation as the objective function to promoteresponse diversity. Techniques of reinforcementlearning (RL) and adversarial learning have beenintroduced into dialogue generation by Li et al.(2016b, 2017) to better approximate the optimiza-tion goal of dialogue models. Conditional varia-tional framework (Zhao et al., 2017; Shen et al.,2017; Park et al., 2018) has also shown a promisein dialogue generation. Gu et al. (2019) furtherintroduce a conditional Wasserstein autoencoderthat employs GAN (Goodfellow et al., 2014) tomodel the multimodal latent structures. Cai et al.(2020) design a multi-curriculum learning frame-

work to facilitate the dialogue model training. Con-trasted with existing learning methods for dialoguegeneration, the proposed framework in this workencourages the model to learn from the differencebetween well-chosen contrastive pairs, which ex-plicitly models the multi-mapping relations in con-versation and promotes the distinctiveness of thegenerated responses in the meantime.

Contrastive Learning The concept of learn-ing by contrasting positive pairs against nega-tive pairs (Hadsell et al., 2006; Gutmann andHyvarinen, 2012) has been successfully adoptedin many tasks. For example, contrastive learningin language modeling task (Mnih and Teh, 2012;Vaswani et al., 2013; Baltescu and Blunsom, 2015)aims to approximate the negative log-likelihood bytraining the model to correctly classify betweengenerated noise samples and words observed inthe training data. Contrastive visual representationlearning (van den Oord et al., 2018; Chen et al.,2020) trains a generative model to score real datapoints higher than negative samples. Dai and Lin(2017) propose to use contrastive learning for im-age caption. Clark et al. (2020) use contrastivelearning to train a discriminative model for lan-guage representation learning. Compared with ex-isting work, samples used in this paper, instead ofbeing sampled randomly, are carefully chosen to ex-

hibit particular properties of human dialogues. An-other difference is that, we manage multi-mappingrelations prevalent in human conversation usingmany positives and many negatives, which capturesboth the intra-group and inter-group variability.

5 Conclusion

In this work, we propose a group-wise contrastivedialogue learning approach, that explicitly per-ceives the difference between the well-chosen pos-itive and negative utterances, and manages themulti-mapping relations in human conversationssimultaneously. Given a training instance, the pro-posed learning framework first organizes a groupof positive samples and negative samples regard-ing context-response matching degrees, and thentrains the target dialogue model to give higher con-ditional probabilities for positive pairs and lowerprobabilities for the negatives. Extensive exper-imental results show that the proposed learningframework brings a solid favorable performanceboost amongst various strong baseline approaches.

Acknowledgments

We would like to thank all the reviewers for theirinsightful and valuable comments and suggestions.Hongshen Chen and Yonghao Song are the corre-sponding authors.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In ICLR.

Paul Baltescu and Phil Blunsom. 2015. Pragmatic neu-ral language modelling in machine translation. InNAACL-HLT.

Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information. Transactions of the Associa-tion for Computational Linguistics, 5:135–146.

Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, An-drew Dai, Rafal Jozefowicz, and Samy Bengio.2016. Generating sentences from a continuousspace. In SIGNLL.

Massimo Caccia, Lucas Caccia, William Fedus, HugoLarochelle, Joelle Pineau, and Laurent Charlin.2020. Language gans falling short. In ICLR.

Hengyi Cai, Hongshen Chen, Cheng Zhang, YonghaoSong, Xiaofang Zhao, Yangxi Li, Dongsheng Duan,

and Dawei Yin. 2020. Learning from easy to com-plex: Adaptive multi-curricula learning for neural di-alogue generation. In AAAI.

Ting Chen, Simon Kornblith, Mohammad Norouzi,and Geoffrey E. Hinton. 2020. A simple frame-work for contrastive learning of visual representa-tions. CoRR, abs/2002.05709.

Kevin Clark, Minh-Thang Luong, Quoc V. Le, andChristopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather thangenerators. In ICLR.

Jacob Cohen. 1960. A coefficient of agreement fornominal scales. Educational and psychological mea-surement, 20(1):37–46.

Bo Dai and Dahua Lin. 2017. Contrastive learning forimage captioning. In NIPS.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair, AaronCourville, and Yoshua Bengio. 2014. Generative ad-versarial nets. In NIPS.

Xiaodong Gu, Kyunghyun Cho, JungWoo Ha, andSunghun Kim. 2019. Dialogwae: Multimodal re-sponse generation with conditional wasserstein auto-encoder. In ICLR.

Michael Gutmann and Aapo Hyvarinen. 2012. Noise-contrastive estimation of unnormalized statisticalmodels, with applications to natural image statistics.J. Mach. Learn. Res., 13:307–361.

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006.Dimensionality reduction by learning an invariantmapping. In CVPR.

Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020.Challenges in building intelligent open-domain dia-log systems. ACM Trans. Inf. Syst., 38(3).

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In ICLR.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,and Bill Dolan. 2016a. A diversity-promoting ob-jective function for neural conversation models. InNAACL-HLT.

Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky,Michel Galley, and Jianfeng Gao. 2016b. Deep re-inforcement learning for dialogue generation. InEMNLP.

Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean,Alan Ritter, and Dan Jurafsky. 2017. Adversariallearning for neural dialogue generation. In EMNLP.

Pierre Lison and Jorg Tiedemann. 2016. Opensub-titles2016: Extracting large parallel corpora frommovie and TV subtitles. In LREC.

Chia-Wei Liu, Ryan Lowe, Iulian Serban, MichaelNoseworthy, Laurent Charlin, and Joelle Pineau.2016. How NOT to evaluate your dialogue system:An empirical study of unsupervised evaluation met-rics for dialogue response generation. In EMNLP.

Qian Liu, Y. Chen, B. Chen, Jian-Guang Lou, ZixuanChen, Bin Zhou, and D. Zhang. 2020. You impressme: Dialogue generation via mutual persona percep-tion. In ACL.

Alexander Miller, Will Feng, Dhruv Batra, AntoineBordes, Adam Fisch, Jiasen Lu, Devi Parikh, andJason Weston. 2017. ParlAI: A dialog research soft-ware platform. In EMNLP.

Andriy Mnih and Yee Whye Teh. 2012. A fast and sim-ple algorithm for training neural probabilistic lan-guage models. In ICML.

Weili Nie, Nina Narodytska, and Ankit Patel. 2019.Relgan: Relational generative adversarial networksfor text generation. In ICLR.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals.2018. Representation learning with contrastive pre-dictive coding. CoRR, abs/1807.03748.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In ACL.

Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. Ahierarchical latent structure for variational conversa-tion modeling. In NAACL-HLT.

Stephen E. Robertson and Hugo Zaragoza. 2009. Theprobabilistic relevance framework: BM25 and be-yond. Foundations and Trends in Information Re-trieval, 3(4):333–389.

Jo ao Sedoc, Daphne Ippolito, Arun Kirubarajan,Jai Thirani, Lyle Ungar, and Chris Callison-Burch.2019. Chateval: A tool for chatbot evaluation. InNAACL (Demonstrations).

Iulian Vlad Serban, Alessandro Sordoni, Yoshua Ben-gio, Aaron C. Courville, and Joelle Pineau. 2016.Building end-to-end dialogue systems using gener-ative hierarchical neural network models. In AAAI.

Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe,Laurent Charlin, Joelle Pineau, Aaron C. Courville,and Yoshua Bengio. 2017. A hierarchical latentvariable encoder-decoder model for generating dia-logues. In AAAI.

Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neu-ral responding machine for short-text conversation.In ACL.

Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, ShuziNiu, Yang Zhao, Akiko Aizawa, and Guoping Long.2017. A conditional variational framework for dia-log generation. In ACL.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NIPS.

Ashish Vaswani, Yinggong Zhao, Victoria Fossum, andDavid Chiang. 2013. Decoding with large-scaleneural language models improves translation. InEMNLP.

Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhou-jun Li. 2017. Sequential matching network: Anew architecture for multi-turn response selection inretrieval-based chatbots. In ACL.

Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang,Ming Zhou, and Wei-Ying Ma. 2017. Topic awareneural response generation. In AAAI.

Chen Xing, Yu Wu, Wei Wu, Yalou Huang, and MingZhou. 2018. Hierarchical recurrent attention net-work for response generation. In AAAI.

Xinnuo Xu, Ondrej Dusek, Ioannis Konstas, and Ver-ena Rieser. 2018. Better conversations by modeling,filtering, and optimizing for coherence and diversity.In EMNLP.

Rui Yan. 2018. “chitty-chitty-chat bot”: Deep learningfor conversational ai. In IJCAI.

Chunyuan Yuan, Wei Zhou, Mingming Li, ShangwenLv, Fuqing Zhu, Jizhong Han, and Songlin Hu.2019. Multi-hop Selector Network for Multi-turnResponse Selection in Retrieval-based Chatbots. InEMNLP-IJCNLP.

Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, andXueqi Cheng. 2018a. Reinforcing coherence for se-quence to sequence model in dialogue generation. InIJCAI.

Saizheng Zhang, Emily Dinan, Jack Urbanek, ArthurSzlam, Douwe Kiela, and Jason Weston. 2018b. Per-sonalizing dialogue agents: I have a dog, do youhave pets too? In ACL.

Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan,Xiujun Li, Chris Brockett, and Bill Dolan. 2018c.Generating informative and diverse conversationalresponses via adversarial information maximization.In NeurIPS.

Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.2017. Learning discourse-level diversity for neuraldialog models using conditional variational autoen-coders. In ACL.

Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, YingChen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu.2018. Multi-Turn Response Selection for Chatbotswith Deep Attention Matching Network. In ACL.

Group-wise Contrastive Learning for Neural Dialogue GenerationGroup-wise Contrastive Learning for...

Documents

Transcript of Group-wise Contrastive Learning for Neural Dialogue GenerationGroup-wise Contrastive Learning for...