WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	38.66	-	-	38.56	38.43	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
2	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	38.52	-	-	38.43	38.30	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
3	srcb	ja-zh	2019/07/27 15:34:31	3208	-	38.63	-	-	38.34	38.29	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
4	srcb	ja-zh	2019/07/25 11:30:58	2916	-	37.72	-	-	37.52	37.44	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
5	srcb	ja-zh	2018/09/16 14:47:09	2473	-	37.60	-	-	37.34	37.35	-	-	0.00	0.00	NMT	No	Transformer with relative position, ensemble of 10 models.
6	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	36.65	-	-	36.55	36.40	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
7	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	36.40	-	-	36.46	36.29	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
8	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	35.99	-	-	35.89	35.87	-	-	0.00	0.00	NMT	No	MLNMT
9	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	35.71	-	-	35.67	35.55	-	-	0.00	0.00	NMT	No	Transformer vanilla model
10	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	35.31	-	-	35.37	35.06	-	0.00	0.00	0.00	NMT	No	Ensemble of 5 shared BPE, averaged
11	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	35.00	-	-	35.35	34.94	-	-	0.00	0.00	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
12	srcb	ja-zh	2018/08/26 11:37:12	2153	-	35.55	-	-	35.32	35.28	-	-	0.00	0.00	NMT	No	Transformer, average checkpoints.
13	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	35.67	-	-	35.30	35.40	-	0.00	0.00	0.00	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
14	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	35.23	-	-	35.23	35.14	-	0.00	0.00	0.00	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
15	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	34.97	-	-	34.96	34.72	-	0.00	0.00	0.00	NMT	No	Google's "Attention Is All You Need"
16	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	33.72	-	-	33.64	33.60	-	0.00	0.00	0.00	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
17	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	33.26	-	-	33.33	33.14	-	-	0.00	0.00	NMT	No	NMT with Attention
18	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	31.98	-	-	32.08	31.72	-	0.00	0.00	0.00	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
19	NAIST	ja-zh	2015/08/31 15:35:36	838	-	31.61	-	-	31.59	31.42	0.00	0.00	0.00	0.00	SMT	No	Travatar System with NeuralMT Reranking
20	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	-	31.40	-	-	31.26	31.23	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
21	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	30.57	-	-	30.49	30.31	-	0.00	0.00	0.00	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
22	NAIST	ja-zh	2014/08/01 17:18:51	122	-	30.53	-	-	30.46	30.25	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System
23	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	-	30.17	-	-	30.15	29.89	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
24	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	-	30.07	-	-	30.14	29.83	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
25	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	-	30.19	-	-	29.98	29.90	0.00	0.00	0.00	0.00	EBMT	No	added one reordering feature, w/ reranking
26	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	30.00	-	-	29.97	29.78	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
27	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	-	30.08	-	-	29.94	29.87	0.00	0.00	0.00	0.00	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
28	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	30.27	-	-	29.94	29.92	-	0.00	0.00	0.00	EBMT	No	KyotoEBMT 2016 w/o reranking
29	NAIST	ja-zh	2015/08/31 15:38:17	839	-	30.06	-	-	29.92	29.73	0.00	0.00	0.00	0.00	SMT	No	Travatar System Baseline
30	NAIST	ja-zh	2014/08/01 17:27:20	123	-	29.83	-	-	29.77	29.54	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
31	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	-	29.99	-	-	29.76	29.81	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system without reranking
32	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	-	29.18	-	-	29.00	28.94	0.00	0.00	0.00	0.00	EBMT	No	Kyoto-U team WAT2015 baseline
33	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	-	28.65	-	-	28.65	28.35	0.00	0.00	0.00	0.00	SMT	No	String-to-Tree SMT (2014)
34	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	-	28.65	-	-	28.65	28.35	0.00	0.00	0.00	0.00	SMT	No	String-to-Tree SMT (2015)
35	NICT	ja-zh	2014/09/01 09:23:36	260	-	27.98	-	-	28.18	27.84	0.00	0.00	0.00	0.00	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
36	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	-	27.96	-	-	28.01	27.68	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
37	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	-	28.05	-	-	27.84	27.88	0.00	0.00	0.00	0.00	EBMT	No	escaping w/ reranking
38	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	-	27.71	-	-	27.70	27.35	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2014)
39	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-	27.67	-	-	27.44	27.34	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
40	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-	27.21	-	-	27.02	26.83	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications.
41	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	-	27.42	-	-	26.82	26.79	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
42	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	-	26.69	-	-	26.48	26.30	0.00	0.00	0.00	0.00	EBMT	No	Our baseline system.
43	WASUIPS	ja-zh	2014/09/17 12:07:07	390	-	25.63	-	-	25.30	25.18	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
44	WASUIPS	ja-zh	2014/09/17 01:11:02	377	-	25.60	-	-	25.10	25.07	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
45	WASUIPS	ja-zh	2014/09/17 10:29:24	385	-	25.45	-	-	25.10	25.01	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
46	WASUIPS	ja-zh	2014/09/17 01:08:33	376	-	25.44	-	-	25.04	24.98	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
47	WASUIPS	ja-zh	2014/09/17 10:32:13	386	-	25.68	-	-	25.01	25.11	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
48	WASUIPS	ja-zh	2014/09/17 12:04:30	389	-	25.08	-	-	24.81	24.64	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
49	WASUIPS	ja-zh	2014/09/17 00:54:35	373	-	24.70	-	-	24.25	24.28	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
50	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-	24.12	-	-	23.76	23.55	0.00	0.00	0.00	0.00	SMT	No
51	Sense	ja-zh	2014/08/26 15:19:02	201	-	23.09	-	-	22.94	23.04	0.00	0.00	0.00	0.00	SMT	No	Character based SMT
52	TMU	ja-zh	2017/08/03 01:02:47	1743	-	22.92	-	-	22.86	22.74	-	0.00	0.00	0.00	NMT	No	JP-CN reconstructor baseline
53	WASUIPS	ja-zh	2014/09/17 00:47:46	371	-	22.71	-	-	22.49	22.39	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
54	WASUIPS	ja-zh	2014/09/17 10:17:52	382	-	22.20	-	-	22.02	21.91	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
55	WASUIPS	ja-zh	2014/09/17 10:15:13	381	-	22.01	-	-	21.81	21.61	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
56	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-	19.28	-	-	18.93	18.82	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT system
57	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-	17.86	-	-	17.75	17.49	0.00	0.00	0.00	0.00	RBMT	No	RBMT B (2014)
58	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-	17.86	-	-	17.75	17.49	0.00	0.00	0.00	0.00	Other	Yes	RBMT B (2015)
59	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	15.94	-	-	15.68	15.38	-	0.00	0.00	0.00	NMT	Yes	Online D (2016/11/14)
60	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	11.16	-	-	10.72	10.54	-	0.00	0.00	0.00	Other	Yes	Online D (2016)
61	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-	10.73	-	-	10.33	10.08	0.00	0.00	0.00	0.00	Other	Yes	Online D (2015)
62	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	-	9.62	-	-	9.96	9.59	0.00	0.00	0.00	0.00	RBMT	No	RBMT C
63	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-	9.37	-	-	8.93	8.84	0.00	0.00	0.00	0.00	Other	Yes	Online D (2014)
64	TMU	ja-zh	2018/09/19 10:58:57	2505	-	7.73	-	-	7.52	7.22	-	-	0.00	0.00	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.
65	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	-	7.44	-	-	7.05	6.75	0.00	0.00	0.00	0.00	Other	Yes	Online C (2015)
66	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	-	7.26	-	-	7.01	6.72	0.00	0.00	0.00	0.00	Other	Yes	Online C (2014)

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	srcb	ja-zh	2019/07/25 11:30:58	2916	-	0.860189	-	-	0.859237	0.859522	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
2	srcb	ja-zh	2018/09/16 14:47:09	2473	-	0.859132	-	-	0.858042	0.858162	-	-	0.000000	0.000000	NMT	No	Transformer with relative position, ensemble of 10 models.
3	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	0.858491	-	-	0.857645	0.858103	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
4	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	0.858229	-	-	0.857229	0.857722	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
5	srcb	ja-zh	2019/07/27 15:34:31	3208	-	0.858506	-	-	0.856864	0.857121	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
6	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	0.854030	-	-	0.854085	0.854391	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
7	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	0.853793	-	-	0.852731	0.852979	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
8	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	0.852084	-	-	0.851893	0.851548	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
9	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	0.851083	-	-	0.851670	0.850222	-	-	0.000000	0.000000	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
10	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	0.851382	-	-	0.851416	0.850944	-	-	0.000000	0.000000	NMT	No	MLNMT
11	srcb	ja-zh	2018/08/26 11:37:12	2153	-	0.851766	-	-	0.850968	0.851032	-	-	0.000000	0.000000	NMT	No	Transformer, average checkpoints.
12	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	0.851890	-	-	0.850699	0.850580	-	-	0.000000	0.000000	NMT	No	Transformer vanilla model
13	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	0.850199	-	-	0.850052	0.848394	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
14	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	0.850103	-	-	0.849168	0.847879	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 shared BPE, averaged
15	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	0.849464	-	-	0.848107	0.848318	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
16	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	0.847223	-	-	0.846578	0.846158	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
17	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	0.844322	-	-	0.844572	0.844959	-	-	0.000000	0.000000	NMT	No	NMT with Attention
18	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	0.837579	-	-	0.839354	0.835932	-	0.000000	0.000000	0.000000	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
19	NAIST	ja-zh	2015/08/31 15:35:36	838	-	0.832765	-	-	0.834245	0.833721	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
20	NAIST	ja-zh	2014/08/01 17:27:20	123	-	0.829627	-	-	0.830839	0.830529	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
21	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	0.829679	-	-	0.829113	0.827637	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
22	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	-	0.826986	-	-	0.826919	0.827190	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
23	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	0.820891	-	-	0.820069	0.821090	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
24	NAIST	ja-zh	2014/08/01 17:18:51	122	-	0.818040	-	-	0.819406	0.819492	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
25	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	-	0.817294	-	-	0.816984	0.816981	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
26	NAIST	ja-zh	2015/08/31 15:38:17	839	-	0.815084	-	-	0.816624	0.816462	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
27	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	0.813114	-	-	0.813581	0.813054	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
28	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	-	0.813490	-	-	0.813233	0.813441	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
29	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	-	0.810674	-	-	0.812372	0.811316	0.000000	0.000000	0.000000	0.000000	EBMT	No	added one reordering feature, w/ reranking
30	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	-	0.809128	-	-	0.809561	0.811394	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
31	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	-	0.807606	-	-	0.809457	0.808417	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2014)
32	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	-	0.807606	-	-	0.809457	0.808417	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2015)
33	NICT	ja-zh	2014/09/01 09:23:36	260	-	0.806070	-	-	0.808684	0.807809	0.000000	0.000000	0.000000	0.000000	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
34	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	-	0.807083	-	-	0.808275	0.808010	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
35	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	-	0.806771	-	-	0.807596	0.807432	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
36	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	-	0.804444	-	-	0.803302	0.803980	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
37	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	-	0.799725	-	-	0.800032	0.800988	0.000000	0.000000	0.000000	0.000000	EBMT	No	escaping w/ reranking
38	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	-	0.798663	-	-	0.799864	0.798748	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline
39	TMU	ja-zh	2017/08/03 01:02:47	1743	-	0.798681	-	-	0.798736	0.797969	-	0.000000	0.000000	0.000000	NMT	No	JP-CN reconstructor baseline
40	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	-	0.796402	-	-	0.798084	0.798383	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
41	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-	0.794834	-	-	0.796186	0.793054	0.000000	0.000000	0.000000	0.000000	SMT	No
42	WASUIPS	ja-zh	2014/09/17 01:11:02	377	-	0.794716	-	-	0.795786	0.795594	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
43	WASUIPS	ja-zh	2014/09/17 10:32:13	386	-	0.795721	-	-	0.795504	0.795129	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
44	WASUIPS	ja-zh	2014/09/17 12:07:07	390	-	0.794646	-	-	0.795307	0.794024	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
45	WASUIPS	ja-zh	2014/09/17 01:08:33	376	-	0.794244	-	-	0.793945	0.794823	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
46	WASUIPS	ja-zh	2014/09/17 10:29:24	385	-	0.793819	-	-	0.793308	0.793029	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
47	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-	0.791270	-	-	0.792166	0.790743	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
48	WASUIPS	ja-zh	2014/09/17 12:04:30	389	-	0.790498	-	-	0.791430	0.790142	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
49	WASUIPS	ja-zh	2014/09/17 00:54:35	373	-	0.790030	-	-	0.790460	0.790898	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
50	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	-	0.788961	-	-	0.790263	0.790937	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
51	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-	0.788321	-	-	0.789069	0.788206	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
52	Sense	ja-zh	2014/08/26 15:19:02	201	-	0.779495	-	-	0.779502	0.780262	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
53	WASUIPS	ja-zh	2014/09/17 00:47:46	371	-	0.776323	-	-	0.777615	0.777327	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
54	WASUIPS	ja-zh	2014/09/17 10:17:52	382	-	0.771952	-	-	0.773341	0.772107	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
55	WASUIPS	ja-zh	2014/09/17 10:15:13	381	-	0.767418	-	-	0.767414	0.766092	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
56	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-	0.764491	-	-	0.765346	0.763931	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT system
57	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-	0.744818	-	-	0.745885	0.743794	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT B (2014)
58	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-	0.744818	-	-	0.745885	0.743794	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B (2015)
59	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	0.728453	-	-	0.728270	0.728284	-	0.000000	0.000000	0.000000	NMT	Yes	Online D (2016/11/14)
60	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	0.665185	-	-	0.667382	0.666953	-	0.000000	0.000000	0.000000	Other	Yes	Online D (2016)
61	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-	0.660484	-	-	0.660847	0.660482	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2015)
62	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	-	0.642278	-	-	0.648758	0.645385	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT C
63	TMU	ja-zh	2018/09/19 10:58:57	2505	-	0.621413	-	-	0.623292	0.622094	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.
64	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	-	0.611964	-	-	0.615048	0.612158	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2015)
65	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	-	0.612808	-	-	0.613075	0.611563	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2014)
66	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-	0.606905	-	-	0.606328	0.604149	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2014)

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	srcb	ja-zh	2018/09/16 14:47:09	2473	-	0.791120	-	-	0.791120	0.791120	-	-	0.000000	0.000000	NMT	No	Transformer with relative position, ensemble of 10 models.
2	Kyoto-U+ECNU	ja-zh	2020/09/17 18:43:01	3814	-	0.787730	-	-	0.787730	0.787730	-	-	-	-	NMT	Yes	ensemble 8 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
3	srcb	ja-zh	2018/08/26 11:37:12	2153	-	0.787570	-	-	0.787570	0.787570	-	-	0.000000	0.000000	NMT	No	Transformer, average checkpoints.
4	ORGANIZER	ja-zh	2017/08/02 01:06:05	1738	-	0.787250	-	-	0.787250	0.787250	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
5	srcb	ja-zh	2019/07/27 15:34:31	3208	-	0.787220	-	-	0.787220	0.787220	-	-	-	-	NMT	No	Transformer with num_units=768, relative position, sentence-wise smooth, encoding side word drop, norm-based batch filtering, residual connection norm, ensemble of 8 models.
6	Kyoto-U+ECNU	ja-zh	2020/09/19 16:56:52	4053	-	0.786870	-	-	0.786870	0.786870	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3814
7	srcb	ja-zh	2019/07/25 11:30:58	2916	-	0.786600	-	-	0.786600	0.786600	-	-	-	-	NMT	No	Transformer with relative position, sentence-wise smooth, encoder side word drop.
8	NICT-2	ja-zh	2017/07/26 14:11:42	1483	-	0.785820	-	-	0.785820	0.785820	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
9	NICT-5	ja-zh	2018/08/27 15:00:25	2175	-	0.785440	-	-	0.785440	0.785440	-	-	0.000000	0.000000	NMT	No	Transformer vanilla model
10	Kyoto-U	ja-zh	2017/08/01 14:17:43	1722	-	0.785420	-	-	0.785420	0.785420	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 shared BPE, averaged
11	NICT-5	ja-zh	2018/08/22 18:56:02	2055	-	0.784340	-	-	0.784340	0.784340	-	-	0.000000	0.000000	NMT	No	Multi-layer-softmax for vanilla transformer. Train 6-layer model. Decode only using 3 layers. 2x faster than 6 layers.
12	Kyoto-U+ECNU	ja-zh	2020/09/10 23:54:57	3676	-	0.783970	-	-	0.783970	0.783970	-	-	-	-	NMT	No	forward-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
13	NICT-5	ja-zh	2018/09/10 14:09:18	2266	-	0.781410	-	-	0.781410	0.781410	-	-	0.000000	0.000000	NMT	No	MLNMT
14	KNU_Hyundai	ja-zh	2019/07/27 08:54:03	3170	-	0.781350	-	-	0.781350	0.781350	-	-	-	-	NMT	Yes	Transformer(base) + Used JPC corpus with relative position, bt, r2l rerank, 4-model ensemble
15	NICT-2	ja-zh	2017/07/26 14:00:26	1478	-	0.779870	-	-	0.779870	0.779870	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
16	Kyoto-U	ja-zh	2017/07/31 15:24:48	1642	-	0.779400	-	-	0.779400	0.779400	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
17	ORGANIZER	ja-zh	2018/08/14 11:39:11	1903	-	0.777600	-	-	0.777600	0.777600	-	-	0.000000	0.000000	NMT	No	NMT with Attention
18	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	0.000000	0.768470	0.000000	0.000000	0.768470	0.768470	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
19	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	0.000000	0.765440	0.000000	0.000000	0.765440	0.765440	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
20	Kyoto-U	ja-zh	2015/07/03 11:01:45	457	0.000000	0.765320	0.000000	0.000000	0.765320	0.765320	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline with reranking
21	Kyoto-U	ja-zh	2015/07/03 11:09:12	458	0.000000	0.764530	0.000000	0.000000	0.764530	0.764530	0.000000	0.000000	0.000000	0.000000	EBMT	No	Kyoto-U team WAT2015 baseline
22	Kyoto-U	ja-zh	2016/08/05 23:26:20	1109	-	0.764230	-	-	0.764230	0.764230	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
23	NAIST	ja-zh	2015/08/31 15:35:36	838	0.000000	0.763390	0.000000	0.000000	0.763390	0.763390	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
24	Kyoto-U	ja-zh	2016/08/02 01:25:11	1071	-	0.763290	-	-	0.763290	0.763290	-	0.000000	0.000000	0.000000	NMT	No	2 layer lstm dropout 0.5 200k source voc unk replaced
25	Kyoto-U	ja-zh	2015/07/31 00:35:46	545	0.000000	0.763020	0.000000	0.000000	0.763020	0.763020	0.000000	0.000000	0.000000	0.000000	EBMT	No	added one reordering feature, w/ reranking
26	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	0.000000	0.762520	0.000000	0.000000	0.762520	0.762520	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
27	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	0.000000	0.762060	0.000000	0.000000	0.762060	0.762060	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
28	NAIST	ja-zh	2014/08/01 17:18:51	122	0.000000	0.759740	0.000000	0.000000	0.759740	0.759740	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
29	NICT-2	ja-zh	2016/08/05 18:09:19	1105	-	0.759670	-	-	0.759670	0.759670	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC)
30	NAIST	ja-zh	2014/08/01 17:27:20	123	0.000000	0.758480	0.000000	0.000000	0.758480	0.758480	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
31	Kyoto-U	ja-zh	2015/08/25 12:51:38	765	0.000000	0.757440	0.000000	0.000000	0.757440	0.757440	0.000000	0.000000	0.000000	0.000000	EBMT	No	escaping w/ reranking
32	NAIST	ja-zh	2015/08/31 15:38:17	839	0.000000	0.756990	0.000000	0.000000	0.756990	0.756990	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
33	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	0.000000	0.755230	0.000000	0.000000	0.755230	0.755230	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2014)
34	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	0.000000	0.755230	0.000000	0.000000	0.755230	0.755230	0.000000	0.000000	0.000000	0.000000	SMT	No	String-to-Tree SMT (2015)
35	bjtu_nlp	ja-zh	2016/08/09 14:48:19	1120	-	0.754690	-	-	0.754690	0.754690	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
36	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	0.000000	0.754050	0.000000	0.000000	0.754050	0.754050	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
37	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	0.000000	0.751740	0.000000	0.000000	0.751740	0.751740	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
38	WASUIPS	ja-zh	2014/09/17 01:08:33	376	0.000000	0.750240	0.000000	0.000000	0.750240	0.750240	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
39	WASUIPS	ja-zh	2014/09/17 01:11:02	377	0.000000	0.750220	0.000000	0.000000	0.750220	0.750220	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
40	WASUIPS	ja-zh	2014/09/17 10:29:24	385	0.000000	0.749470	0.000000	0.000000	0.749470	0.749470	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
41	ORGANIZER	ja-zh	2014/07/11 19:50:50	7	0.000000	0.749450	0.000000	0.000000	0.749450	0.749450	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
42	WASUIPS	ja-zh	2014/09/17 10:32:13	386	0.000000	0.748360	0.000000	0.000000	0.748360	0.748360	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
43	WASUIPS	ja-zh	2014/09/17 12:07:07	390	0.000000	0.747890	0.000000	0.000000	0.747890	0.747890	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
44	Kyoto-U	ja-zh	2014/07/14 14:30:39	18	0.000000	0.747090	0.000000	0.000000	0.747090	0.747090	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
45	Sense	ja-zh	2014/08/26 15:19:02	201	0.000000	0.746750	0.000000	0.000000	0.746750	0.746750	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
46	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	0.000000	0.746000	0.000000	0.000000	0.746000	0.746000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
47	NICT	ja-zh	2014/09/01 09:23:36	260	0.000000	0.745980	0.000000	0.000000	0.745980	0.745980	0.000000	0.000000	0.000000	0.000000	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
48	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	0.000000	0.745100	0.000000	0.000000	0.745100	0.745100	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
49	WASUIPS	ja-zh	2014/09/17 00:54:35	373	0.000000	0.744150	0.000000	0.000000	0.744150	0.744150	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
50	WASUIPS	ja-zh	2014/09/17 12:04:30	389	0.000000	0.741490	0.000000	0.000000	0.741490	0.741490	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
51	WASUIPS	ja-zh	2014/09/17 00:47:46	371	0.000000	0.728650	0.000000	0.000000	0.728650	0.728650	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
52	WASUIPS	ja-zh	2014/09/17 10:15:13	381	0.000000	0.727920	0.000000	0.000000	0.727920	0.727920	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
53	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	0.000000	0.727700	0.000000	0.000000	0.727700	0.727700	0.000000	0.000000	0.000000	0.000000	SMT	No
54	WASUIPS	ja-zh	2014/09/17 10:17:52	382	0.000000	0.725500	0.000000	0.000000	0.725500	0.725500	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
55	TMU	ja-zh	2017/08/03 01:02:47	1743	-	0.700030	-	-	0.700030	0.700030	-	0.000000	0.000000	0.000000	NMT	No	JP-CN reconstructor baseline
56	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	0.000000	0.685380	0.000000	0.000000	0.685380	0.685380	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT system
57	ORGANIZER	ja-zh	2016/11/16 10:58:30	1336	-	0.673730	-	-	0.673730	0.673730	-	0.000000	0.000000	0.000000	NMT	Yes	Online D (2016/11/14)
58	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	0.000000	0.667960	0.000000	0.000000	0.667960	0.667960	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT B (2014)
59	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	0.000000	0.667960	0.000000	0.000000	0.667960	0.667960	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B (2015)
60	ORGANIZER	ja-zh	2016/07/26 12:18:34	1045	-	0.639440	-	-	0.639440	0.639440	-	0.000000	0.000000	0.000000	Other	Yes	Online D (2016)
61	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	0.000000	0.634090	0.000000	0.000000	0.634090	0.634090	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2015)
62	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	0.000000	0.625430	0.000000	0.000000	0.625430	0.625430	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online D (2014)
63	ORGANIZER	ja-zh	2014/08/29 18:53:46	244	0.000000	0.594900	0.000000	0.000000	0.594900	0.594900	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT C
64	ORGANIZER	ja-zh	2014/08/28 12:11:11	216	0.000000	0.587820	0.000000	0.000000	0.587820	0.587820	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2014)
65	ORGANIZER	ja-zh	2015/09/11 10:11:23	891	0.000000	0.566060	0.000000	0.000000	0.566060	0.566060	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online C (2015)
66	TMU	ja-zh	2018/09/19 10:58:57	2505	-	0.545630	-	-	0.545630	0.545630	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT using Sub-character level information. JPO patent data was used as monolingual data in the training process.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	TOSHIBA	ja-zh	2015/08/17 16:29:35	676	17.000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model + post-processing
2	Kyoto-U	ja-zh	2015/08/26 02:17:25	778	16.000	EBMT	No	KyotoEBMT system without reranking
3	Kyoto-U	ja-zh	2015/08/27 13:51:08	793	12.500	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
4	ORGANIZER	ja-zh	2015/09/10 14:12:41	881	7.750	SMT	No	String-to-Tree SMT (2015)
5	NAIST	ja-zh	2015/08/31 15:35:36	838	7.000	SMT	No	Travatar System with NeuralMT Reranking
6	NAIST	ja-zh	2015/08/31 15:38:17	839	2.750	SMT	No	Travatar System Baseline
7	TOSHIBA	ja-zh	2015/07/23 14:49:40	505	2.500	SMT and RBMT	Yes	SPE(Statistical Post Editing) System
8	ORGANIZER	ja-zh	2015/09/10 14:32:38	886	-11.000	Other	Yes	RBMT B (2015)
9	ORGANIZER	ja-zh	2015/08/25 18:59:20	777	-14.750	Other	Yes	Online D (2015)

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	ja-zh	2014/08/01 17:18:51	122	17.750	SMT	No	Travatar-based Forest-to-String SMT System
2	ORGANIZER	ja-zh	2014/07/11 20:00:28	10	14.000	SMT	No	String-to-Tree SMT (2014)
3	Sense	ja-zh	2014/08/26 15:19:02	201	10.000	SMT	No	Character based SMT
4	NICT	ja-zh	2014/09/01 09:23:36	260	6.500	SMT	No	Pre-reordering for phrase-based SMT (dependency parsing + manual rules)
5	ORGANIZER	ja-zh	2014/07/11 19:45:54	3	3.750	SMT	No	Hierarchical Phrase-based SMT (2014)
6	NAIST	ja-zh	2014/08/01 17:27:20	123	1.250	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
7	TOSHIBA	ja-zh	2014/08/29 18:06:20	238	0.750	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
8	Kyoto-U	ja-zh	2014/08/31 23:38:07	257	-0.750	EBMT	No	Our new baseline system after several modifications.
9	BJTUNLP	ja-zh	2014/08/28 20:02:56	224	-3.750	SMT	No
10	TOSHIBA	ja-zh	2014/08/29 17:59:06	236	-5.250	RBMT	Yes	RBMT system
11	Kyoto-U	ja-zh	2014/09/01 08:21:59	259	-8.750	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
12	ORGANIZER	ja-zh	2014/07/18 11:10:37	37	-14.500	Other	Yes	Online D (2014)
13	ORGANIZER	ja-zh	2014/08/29 18:51:05	243	-20.000	RBMT	No	RBMT B (2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results