WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	srcb	ja-zh	2019/07/25 11:30:58	2916	-	37.72	-	-	37.52	37.44	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
2	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	36.40	-	-	36.46	36.29	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
3	srcb	ja-zh	2019/07/27 15:34:31	3208	-	38.63	-	-	38.34	38.29	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
4	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	36.65	-	-	36.55	36.40	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
5	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	38.66	-	-	38.56	38.43	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
6	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	38.52	-	-	38.43	38.30	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
7	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	-	27.71	-	-	27.70	27.35	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2014)
8	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	-	27.96	-	-	28.01	27.68	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
9	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	-	28.65	-	-	28.65	28.35	0.00	0.00	0.00	0.00	SMT	No	String-to-Tree SMT (2014)
10	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	-	26.69	-	-	26.48	26.30	0.00	0.00	0.00	0.00	EBMT	No	Our baseline system.
11	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-	9.37	-	-	8.93	8.84	0.00	0.00	0.00	0.00	Other	Yes	Online D (2014)
12	NAIST	ja-zh	2014/08/01 17:18:51	122	-	30.53	-	-	30.46	30.25	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System
13	NAIST	ja-zh	2014/08/01 17:27:20	123	-	29.83	-	-	29.77	29.54	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
14	Sense	ja-zh	2014/08/26 15:19:02	201	-	23.09	-	-	22.94	23.04	0.00	0.00	0.00	0.00	SMT	No	Character based SMT
15	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	-	7.26	-	-	7.01	6.72	0.00	0.00	0.00	0.00	Other	Yes	Online C (2014)
16	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-	24.12	-	-	23.76	23.55	0.00	0.00	0.00	0.00	SMT	No
17	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-	19.28	-	-	18.93	18.82	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT system
18	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	-	27.42	-	-	26.82	26.79	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
19	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-	17.86	-	-	17.75	17.49	0.00	0.00	0.00	0.00	RBMT	No	RBMT B (2014)
20	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	-	9.62	-	-	9.96	9.59	0.00	0.00	0.00	0.00	RBMT	No	RBMT C
21	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-	27.21	-	-	27.02	26.83	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications.
22	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-	27.67	-	-	27.44	27.34	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
23	NICT	ja-zh	2014/09/01 09:23:36	260	-	27.98	-	-	28.18	27.84	0.00	0.00	0.00	0.00	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
24	WASUIPS	ja-zh	2014/09/17 00:47:46	371	-	22.71	-	-	22.49	22.39	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
25	WASUIPS	ja-zh	2014/09/17 00:54:35	373	-	24.70	-	-	24.25	24.28	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
26	WASUIPS	ja-zh	2014/09/17 01:08:33	376	-	25.44	-	-	25.04	24.98	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
27	WASUIPS	ja-zh	2014/09/17 01:11:02	377	-	25.60	-	-	25.10	25.07	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
28	WASUIPS	ja-zh	2014/09/17 10:15:13	381	-	22.01	-	-	21.81	21.61	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
29	WASUIPS	ja-zh	2014/09/17 10:17:52	382	-	22.20	-	-	22.02	21.91	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
30	WASUIPS	ja-zh	2014/09/17 10:29:24	385	-	25.45	-	-	25.10	25.01	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
31	WASUIPS	ja-zh	2014/09/17 10:32:13	386	-	25.68	-	-	25.01	25.11	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
32	WASUIPS	ja-zh	2014/09/17 12:04:30	389	-	25.08	-	-	24.81	24.64	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
33	WASUIPS	ja-zh	2014/09/17 12:07:07	390	-	25.63	-	-	25.30	25.18	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
34	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	-	30.08	-	-	29.94	29.87	0.00	0.00	0.00	0.00	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
35	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	-	29.18	-	-	29.00	28.94	0.00	0.00	0.00	0.00	EBMT	No	Kyoto-U team WAT2015 baseline
36	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	-	30.17	-	-	30.15	29.89	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
37	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	-	30.19	-	-	29.98	29.90	0.00	0.00	0.00	0.00	EBMT	No	added one reordering feature, w/ reranking
38	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	-	30.07	-	-	30.14	29.83	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
39	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	-	28.05	-	-	27.84	27.88	0.00	0.00	0.00	0.00	EBMT	No	escaping w/ reranking
40	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-	10.73	-	-	10.33	10.08	0.00	0.00	0.00	0.00	Other	Yes	Online D (2015)
41	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	-	29.99	-	-	29.76	29.81	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system without reranking
42	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	-	31.40	-	-	31.26	31.23	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
43	NAIST	ja-zh	2015/08/31 15:35:36	838	-	31.61	-	-	31.59	31.42	0.00	0.00	0.00	0.00	SMT	No	Travatar System with NeuralMT Reranking
44	NAIST	ja-zh	2015/08/31 15:38:17	839	-	30.06	-	-	29.92	29.73	0.00	0.00	0.00	0.00	SMT	No	Travatar System Baseline
45	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	-	28.65	-	-	28.65	28.35	0.00	0.00	0.00	0.00	SMT	No	String-to-Tree SMT (2015)
46	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-	17.86	-	-	17.75	17.49	0.00	0.00	0.00	0.00	Other	Yes	RBMT B (2015)
47	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	-	7.44	-	-	7.05	6.75	0.00	0.00	0.00	0.00	Other	Yes	Online C (2015)
48	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	11.16	-	-	10.72	10.54	-	0.00	0.00	0.00	Other	Yes	Online D (2016)
49	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	31.98	-	-	32.08	31.72	-	0.00	0.00	0.00	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
50	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	30.00	-	-	29.97	29.78	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
51	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	30.27	-	-	29.94	29.92	-	0.00	0.00	0.00	EBMT	No	KyotoEBMT 2016 w/o reranking
52	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	30.57	-	-	30.49	30.31	-	0.00	0.00	0.00	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
53	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	15.94	-	-	15.68	15.38	-	0.00	0.00	0.00	NMT	Yes	Online D (2016/11/14)
54	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	33.72	-	-	33.64	33.60	-	0.00	0.00	0.00	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
55	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	35.23	-	-	35.23	35.14	-	0.00	0.00	0.00	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
56	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	35.67	-	-	35.30	35.40	-	0.00	0.00	0.00	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
57	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	35.31	-	-	35.37	35.06	-	0.00	0.00	0.00	NMT	No	Ensemble of 5 shared BPE, averaged
58	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	34.97	-	-	34.96	34.72	-	0.00	0.00	0.00	NMT	No	Google's "Attention Is All You Need"
59	TMU	ja-zh	2017/08/03 01:02:47	1743	-	22.92	-	-	22.86	22.74	-	0.00	0.00	0.00	NMT	No	JP-CN reconstructor baseline
60	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	33.26	-	-	33.33	33.14	-	-	0.00	0.00	NMT	No	NMT with Attention
61	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	35.00	-	-	35.35	34.94	-	-	0.00	0.00	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
62	srcb	ja-zh	2018/08/26 11:37:12	2153	-	35.55	-	-	35.32	35.28	-	-	0.00	0.00	NMT	No	Transformer, average checkpoints.
63	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	35.71	-	-	35.67	35.55	-	-	0.00	0.00	NMT	No	Transformer vanilla model
64	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	35.99	-	-	35.89	35.87	-	-	0.00	0.00	NMT	No	MLNMT
65	srcb	ja-zh	2018/09/16 14:47:09	2473	-	37.60	-	-	37.34	37.35	-	-	0.00	0.00	NMT	No	Transformer with relative position, ensemble of 10 models.
66	TMU	ja-zh	2018/09/19 10:58:57	2505	-	7.73	-	-	7.52	7.22	-	-	0.00	0.00	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	srcb	ja-zh	2019/07/25 11:30:58	2916	-	0.860189	-	-	0.859237	0.859522	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
2	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	0.854030	-	-	0.854085	0.854391	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
3	srcb	ja-zh	2019/07/27 15:34:31	3208	-	0.858506	-	-	0.856864	0.857121	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
4	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	0.853793	-	-	0.852731	0.852979	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
5	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	0.858491	-	-	0.857645	0.858103	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
6	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	0.858229	-	-	0.857229	0.857722	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
7	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	-	0.809128	-	-	0.809561	0.811394	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
8	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	-	0.788961	-	-	0.790263	0.790937	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
9	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	-	0.807606	-	-	0.809457	0.808417	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2014)
10	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	-	0.796402	-	-	0.798084	0.798383	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
11	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-	0.606905	-	-	0.606328	0.604149	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2014)
12	NAIST	ja-zh	2014/08/01 17:18:51	122	-	0.818040	-	-	0.819406	0.819492	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
13	NAIST	ja-zh	2014/08/01 17:27:20	123	-	0.829627	-	-	0.830839	0.830529	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
14	Sense	ja-zh	2014/08/26 15:19:02	201	-	0.779495	-	-	0.779502	0.780262	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
15	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	-	0.612808	-	-	0.613075	0.611563	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2014)
16	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-	0.794834	-	-	0.796186	0.793054	0.000000	0.000000	0.000000	0.000000	SMT	No
17	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-	0.764491	-	-	0.765346	0.763931	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT system
18	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	-	0.804444	-	-	0.803302	0.803980	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
19	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-	0.744818	-	-	0.745885	0.743794	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT B (2014)
20	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	-	0.642278	-	-	0.648758	0.645385	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT C
21	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-	0.791270	-	-	0.792166	0.790743	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
22	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-	0.788321	-	-	0.789069	0.788206	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
23	NICT	ja-zh	2014/09/01 09:23:36	260	-	0.806070	-	-	0.808684	0.807809	0.000000	0.000000	0.000000	0.000000	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
24	WASUIPS	ja-zh	2014/09/17 00:47:46	371	-	0.776323	-	-	0.777615	0.777327	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
25	WASUIPS	ja-zh	2014/09/17 00:54:35	373	-	0.790030	-	-	0.790460	0.790898	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
26	WASUIPS	ja-zh	2014/09/17 01:08:33	376	-	0.794244	-	-	0.793945	0.794823	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
27	WASUIPS	ja-zh	2014/09/17 01:11:02	377	-	0.794716	-	-	0.795786	0.795594	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
28	WASUIPS	ja-zh	2014/09/17 10:15:13	381	-	0.767418	-	-	0.767414	0.766092	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
29	WASUIPS	ja-zh	2014/09/17 10:17:52	382	-	0.771952	-	-	0.773341	0.772107	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
30	WASUIPS	ja-zh	2014/09/17 10:29:24	385	-	0.793819	-	-	0.793308	0.793029	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
31	WASUIPS	ja-zh	2014/09/17 10:32:13	386	-	0.795721	-	-	0.795504	0.795129	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
32	WASUIPS	ja-zh	2014/09/17 12:04:30	389	-	0.790498	-	-	0.791430	0.790142	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
33	WASUIPS	ja-zh	2014/09/17 12:07:07	390	-	0.794646	-	-	0.795307	0.794024	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
34	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	-	0.806771	-	-	0.807596	0.807432	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
35	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	-	0.798663	-	-	0.799864	0.798748	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline
36	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	-	0.813490	-	-	0.813233	0.813441	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
37	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	-	0.810674	-	-	0.812372	0.811316	0.000000	0.000000	0.000000	0.000000	EBMT	No	added one reordering feature, w/ reranking
38	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	-	0.817294	-	-	0.816984	0.816981	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
39	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	-	0.799725	-	-	0.800032	0.800988	0.000000	0.000000	0.000000	0.000000	EBMT	No	escaping w/ reranking
40	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-	0.660484	-	-	0.660847	0.660482	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2015)
41	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	-	0.807083	-	-	0.808275	0.808010	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
42	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	-	0.826986	-	-	0.826919	0.827190	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
43	NAIST	ja-zh	2015/08/31 15:35:36	838	-	0.832765	-	-	0.834245	0.833721	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
44	NAIST	ja-zh	2015/08/31 15:38:17	839	-	0.815084	-	-	0.816624	0.816462	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
45	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	-	0.807606	-	-	0.809457	0.808417	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2015)
46	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-	0.744818	-	-	0.745885	0.743794	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B (2015)
47	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	-	0.611964	-	-	0.615048	0.612158	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2015)
48	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	0.665185	-	-	0.667382	0.666953	-	0.000000	0.000000	0.000000	Other	Yes	Online D (2016)
49	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	0.837579	-	-	0.839354	0.835932	-	0.000000	0.000000	0.000000	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
50	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	0.820891	-	-	0.820069	0.821090	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
51	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	0.813114	-	-	0.813581	0.813054	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
52	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	0.829679	-	-	0.829113	0.827637	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
53	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	0.728453	-	-	0.728270	0.728284	-	0.000000	0.000000	0.000000	NMT	Yes	Online D (2016/11/14)
54	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	0.847223	-	-	0.846578	0.846158	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
55	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	0.852084	-	-	0.851893	0.851548	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
56	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	0.849464	-	-	0.848107	0.848318	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
57	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	0.850103	-	-	0.849168	0.847879	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 shared BPE, averaged
58	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	0.850199	-	-	0.850052	0.848394	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
59	TMU	ja-zh	2017/08/03 01:02:47	1743	-	0.798681	-	-	0.798736	0.797969	-	0.000000	0.000000	0.000000	NMT	No	JP-CN reconstructor baseline
60	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	0.844322	-	-	0.844572	0.844959	-	-	0.000000	0.000000	NMT	No	NMT with Attention
61	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	0.851083	-	-	0.851670	0.850222	-	-	0.000000	0.000000	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
62	srcb	ja-zh	2018/08/26 11:37:12	2153	-	0.851766	-	-	0.850968	0.851032	-	-	0.000000	0.000000	NMT	No	Transformer, average checkpoints.
63	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	0.851890	-	-	0.850699	0.850580	-	-	0.000000	0.000000	NMT	No	Transformer vanilla model
64	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	0.851382	-	-	0.851416	0.850944	-	-	0.000000	0.000000	NMT	No	MLNMT
65	srcb	ja-zh	2018/09/16 14:47:09	2473	-	0.859132	-	-	0.858042	0.858162	-	-	0.000000	0.000000	NMT	No	Transformer with relative position, ensemble of 10 models.
66	TMU	ja-zh	2018/09/19 10:58:57	2505	-	0.621413	-	-	0.623292	0.622094	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	srcb	ja-zh	2019/07/25 11:30:58	2916	-	0.786600	-	-	0.786600	0.786600	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
2	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	0.781350	-	-	0.781350	0.781350	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
3	srcb	ja-zh	2019/07/27 15:34:31	3208	-	0.787220	-	-	0.787220	0.787220	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
4	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	0.783970	-	-	0.783970	0.783970	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
5	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	0.787730	-	-	0.787730	0.787730	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
6	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	0.786870	-	-	0.786870	0.786870	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
7	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	0.000000	0.745100	0.000000	0.000000	0.745100	0.745100	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
8	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	0.000000	0.749450	0.000000	0.000000	0.749450	0.749450	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
9	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	0.000000	0.755230	0.000000	0.000000	0.755230	0.755230	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2014)
10	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	0.000000	0.747090	0.000000	0.000000	0.747090	0.747090	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
11	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	0.000000	0.625430	0.000000	0.000000	0.625430	0.625430	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2014)
12	NAIST	ja-zh	2014/08/01 17:18:51	122	0.000000	0.759740	0.000000	0.000000	0.759740	0.759740	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
13	NAIST	ja-zh	2014/08/01 17:27:20	123	0.000000	0.758480	0.000000	0.000000	0.758480	0.758480	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
14	Sense	ja-zh	2014/08/26 15:19:02	201	0.000000	0.746750	0.000000	0.000000	0.746750	0.746750	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
15	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	0.000000	0.587820	0.000000	0.000000	0.587820	0.587820	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2014)
16	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	0.000000	0.727700	0.000000	0.000000	0.727700	0.727700	0.000000	0.000000	0.000000	0.000000	SMT	No
17	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	0.000000	0.685380	0.000000	0.000000	0.685380	0.685380	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT system
18	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	0.000000	0.746000	0.000000	0.000000	0.746000	0.746000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
19	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	0.000000	0.667960	0.000000	0.000000	0.667960	0.667960	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT B (2014)
20	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	0.000000	0.594900	0.000000	0.000000	0.594900	0.594900	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT C
21	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	0.000000	0.754050	0.000000	0.000000	0.754050	0.754050	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
22	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	0.000000	0.751740	0.000000	0.000000	0.751740	0.751740	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
23	NICT	ja-zh	2014/09/01 09:23:36	260	0.000000	0.745980	0.000000	0.000000	0.745980	0.745980	0.000000	0.000000	0.000000	0.000000	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
24	WASUIPS	ja-zh	2014/09/17 00:47:46	371	0.000000	0.728650	0.000000	0.000000	0.728650	0.728650	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
25	WASUIPS	ja-zh	2014/09/17 00:54:35	373	0.000000	0.744150	0.000000	0.000000	0.744150	0.744150	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
26	WASUIPS	ja-zh	2014/09/17 01:08:33	376	0.000000	0.750240	0.000000	0.000000	0.750240	0.750240	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
27	WASUIPS	ja-zh	2014/09/17 01:11:02	377	0.000000	0.750220	0.000000	0.000000	0.750220	0.750220	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
28	WASUIPS	ja-zh	2014/09/17 10:15:13	381	0.000000	0.727920	0.000000	0.000000	0.727920	0.727920	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
29	WASUIPS	ja-zh	2014/09/17 10:17:52	382	0.000000	0.725500	0.000000	0.000000	0.725500	0.725500	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
30	WASUIPS	ja-zh	2014/09/17 10:29:24	385	0.000000	0.749470	0.000000	0.000000	0.749470	0.749470	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
31	WASUIPS	ja-zh	2014/09/17 10:32:13	386	0.000000	0.748360	0.000000	0.000000	0.748360	0.748360	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
32	WASUIPS	ja-zh	2014/09/17 12:04:30	389	0.000000	0.741490	0.000000	0.000000	0.741490	0.741490	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
33	WASUIPS	ja-zh	2014/09/17 12:07:07	390	0.000000	0.747890	0.000000	0.000000	0.747890	0.747890	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
34	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	0.000000	0.765320	0.000000	0.000000	0.765320	0.765320	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
35	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	0.000000	0.764530	0.000000	0.000000	0.764530	0.764530	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline
36	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	0.000000	0.762060	0.000000	0.000000	0.762060	0.762060	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
37	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	0.000000	0.763020	0.000000	0.000000	0.763020	0.763020	0.000000	0.000000	0.000000	0.000000	EBMT	No	added one reordering feature, w/ reranking
38	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	0.000000	0.762520	0.000000	0.000000	0.762520	0.762520	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
39	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	0.000000	0.757440	0.000000	0.000000	0.757440	0.757440	0.000000	0.000000	0.000000	0.000000	EBMT	No	escaping w/ reranking
40	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	0.000000	0.634090	0.000000	0.000000	0.634090	0.634090	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2015)
41	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	0.000000	0.765440	0.000000	0.000000	0.765440	0.765440	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
42	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	0.000000	0.768470	0.000000	0.000000	0.768470	0.768470	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
43	NAIST	ja-zh	2015/08/31 15:35:36	838	0.000000	0.763390	0.000000	0.000000	0.763390	0.763390	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
44	NAIST	ja-zh	2015/08/31 15:38:17	839	0.000000	0.756990	0.000000	0.000000	0.756990	0.756990	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
45	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	0.000000	0.755230	0.000000	0.000000	0.755230	0.755230	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2015)
46	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	0.000000	0.667960	0.000000	0.000000	0.667960	0.667960	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B (2015)
47	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	0.000000	0.566060	0.000000	0.000000	0.566060	0.566060	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2015)
48	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	0.639440	-	-	0.639440	0.639440	-	0.000000	0.000000	0.000000	Other	Yes	Online D (2016)
49	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	0.763290	-	-	0.763290	0.763290	-	0.000000	0.000000	0.000000	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
50	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	0.759670	-	-	0.759670	0.759670	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
51	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	0.764230	-	-	0.764230	0.764230	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
52	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	0.754690	-	-	0.754690	0.754690	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
53	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	0.673730	-	-	0.673730	0.673730	-	0.000000	0.000000	0.000000	NMT	Yes	Online D (2016/11/14)
54	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	0.779870	-	-	0.779870	0.779870	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
55	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	0.785820	-	-	0.785820	0.785820	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
56	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	0.779400	-	-	0.779400	0.779400	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
57	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	0.785420	-	-	0.785420	0.785420	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 shared BPE, averaged
58	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	0.787250	-	-	0.787250	0.787250	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
59	TMU	ja-zh	2017/08/03 01:02:47	1743	-	0.700030	-	-	0.700030	0.700030	-	0.000000	0.000000	0.000000	NMT	No	JP-CN reconstructor baseline
60	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	0.777600	-	-	0.777600	0.777600	-	-	0.000000	0.000000	NMT	No	NMT with Attention
61	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	0.784340	-	-	0.784340	0.784340	-	-	0.000000	0.000000	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
62	srcb	ja-zh	2018/08/26 11:37:12	2153	-	0.787570	-	-	0.787570	0.787570	-	-	0.000000	0.000000	NMT	No	Transformer, average checkpoints.
63	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	0.785440	-	-	0.785440	0.785440	-	-	0.000000	0.000000	NMT	No	Transformer vanilla model
64	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	0.781410	-	-	0.781410	0.781410	-	-	0.000000	0.000000	NMT	No	MLNMT
65	srcb	ja-zh	2018/09/16 14:47:09	2473	-	0.791120	-	-	0.791120	0.791120	-	-	0.000000	0.000000	NMT	No	Transformer with relative position, ensemble of 10 models.
66	TMU	ja-zh	2018/09/19 10:58:57	2505	-	0.545630	-	-	0.545630	0.545630	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	17.000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
2	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	16.000	EBMT	No	KyotoEBMT system without reranking
3	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	12.500	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
4	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	7.750	SMT	No	String-to-Tree SMT (2015)
5	NAIST	ja-zh	2015/08/31 15:35:36	838	7.000	SMT	No	Travatar System with NeuralMT Reranking
6	NAIST	ja-zh	2015/08/31 15:38:17	839	2.750	SMT	No	Travatar System Baseline
7	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	2.500	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
8	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-11.000	Other	Yes	RBMT B (2015)
9	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-14.750	Other	Yes	Online D (2015)

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	ja-zh	2014/08/01 17:18:51	122	17.750	SMT	No	Travatar-based Forest-to-String SMT System
2	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	14.000	SMT	No	String-to-Tree SMT (2014)
3	Sense	ja-zh	2014/08/26 15:19:02	201	10.000	SMT	No	Character based SMT
4	NICT	ja-zh	2014/09/01 09:23:36	260	6.500	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
5	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	3.750	SMT	No	Hierarchical Phrase-based SMT (2014)
6	NAIST	ja-zh	2014/08/01 17:27:20	123	1.250	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
7	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	0.750	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
8	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-0.750	EBMT	No	Our new baseline system after several modifications.
9	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-3.750	SMT	No
10	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-5.250	RBMT	Yes	RBMT system
11	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-8.750	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
12	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-14.500	Other	Yes	Online D (2014)
13	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-20.000	RBMT	No	RBMT B (2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results