-
Notifications
You must be signed in to change notification settings - Fork 7
/
MANUAL
2925 lines (2117 loc) · 111 KB
/
MANUAL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
MorphoDiTa: Morphological Dictionary and Tagger
Version 1.11.3-dev
Introduction
============
MorphoDiTa: Morphological Dictionary and Tagger is an open-source tool for
morphological analysis of natural language texts. It performs morphological
analysis, morphological generation, tagging and tokenization and is distributed
as a standalone tool or a library, along with trained linguistic models. In the
Czech language, MorphoDiTa achieves state-of-the-art results with a throughput
around 10-200K words per second. MorphoDiTa is a free software under Mozilla
Public License 2.0 (http://www.mozilla.org/MPL/2.0/) and the linguistic models
are free for non-commercial use and distributed under CC BY-NC-SA
(http://creativecommons.org/licenses/by-nc-sa/4.0/) license, although for some
models the original data used to create the model may impose additional
licensing conditions. MorphoDiTa is versioned using Semantic Versioning
(http://semver.org/).
Copyright 2014 by Institute of Formal and Applied Linguistics, Faculty of
Mathematics and Physics, Charles University in Prague, Czech Republic.
Online Web Application and Web Service
======================================
MorphoDiTa Web Application is available at
http://lindat.mff.cuni.cz/services/morphodita/ using LINDAT/CLARIN
infrastructure (http://lindat.cz).
MorphoDiTa REST Web Service is also available, with the API documentation
available at http://lindat.mff.cuni.cz/services/morphodita/api-reference.php.
Release
=======
Download
--------
MorphoDiTa releases are available on GitHub (http://github.com/ufal/morphodita),
both as source code and as a pre-compiled binary package. The binary package
contains Linux, Windows and OS X binaries, Java bindings binary, C# bindings
binary, and source code of MorphoDiTa and all language bindings). While the
binary packages do not contain compiled Python or Perl bindings, packages for
those languages are available in standard package repositories, i.e. on PyPI
(https://pypi.python.org/pypi/ufal.morphodita/) and CPAN
(https://metacpan.org/pod/Ufal::MorphoDiTa).
- Latest release (http://github.com/ufal/morphodita/releases/latest)
- All releases (http://github.com/ufal/morphodita/releases), Changelog
(https://github.com/ufal/morphodita/blob/master/CHANGES)
Language Models
---------------
To use MorphoDiTa, a language model is needed. The language models are available
from LINDAT/CLARIN (http://www.lindat.cz) infrastructure and described further
in the MorphoDiTa User's Manual. Currently the following language models are
available:
- Czech MorfFlex2+PDT-C: czech-morfflex2.0-pdtc1.0-220710
(http://hdl.handle.net/11234/1-4794) (requires MorphoDiTa 1.9, documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#czech-morfflex2-pdtc))
- Czech MorfFlex+PDT: czech-morfflex-pdt-161115
(http://hdl.handle.net/11234/1-1836) (requires MorphoDiTa 1.9, documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#czech-morfflex-pdt)); older
versions: czech-morfflex-pdt-160310 (http://hdl.handle.net/11234/1-1674)
(documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#czech-morfflex-pdt_changes)),
czech-morfflex-pdt-131112
(http://hdl.handle.net/11858/00-097C-0000-0023-68D8-1) (documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#czech-morfflex-pdt_changes))
- Slovak MorfFlex+PDT: slovak-morfflex-pdt-170914
(http://hdl.handle.net/11234/1-3278) (requires MorphoDiTa 1.9, documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#slovak-morfflex-pdt))
- English Morphium+WSJ: english-morphium-wsj-140407
(http://hdl.handle.net/11858/00-097C-0000-0023-68D9-0) (documentation
(http://ufal.mff.cuni.cz/morphodita/users-manual#english-morphium-wsj))
License
-------
MorphoDiTa is an open-source project and is freely available for non-commercial
purposes. The library is distributed under Mozilla Public License 2.0
(http://www.mozilla.org/MPL/2.0/) and the associated models and data under CC
BY-NC-SA (http://creativecommons.org/licenses/by-nc-sa/4.0/), although for some
models the original data used to create the model may impose additional
licensing conditions.
If you use this tool for scientific work, please give credit to us by
referencing MorphoDiTa website (http://ufal.mff.cuni.cz/morphodita) and
Straková et al. 2014 (http://www.aclweb.org/anthology/P/P14/P14-5003.pdf).
MorphoDiTa Installation
=======================
MorphoDiTa releases are available on GitHub (http://github.com/ufal/morphodita),
either as a pre-compiled binary package, or source code only. The binary package
contains Linux, Windows and OS X binaries, Java bindings binary, C# bindings
binary, and source code of MorphoDiTa and all language bindings. While the
binary packages do not contain compiled Python or Perl bindings, packages for
those languages are available in standard package repositories, i.e. on PyPI and
CPAN.
To use MorphoDiTa, a language model is needed. Here is a list of available
language models (http://ufal.mff.cuni.cz/morphodita#language_models).
If you want to compile MorphoDiTa manually, sources are available on on GitHub
(http://github.com/ufal/morphodita), both in the pre-compiled binary package
releases (http://github.com/ufal/morphodita/releases) and in the repository
itself.
Requirements
------------
- G++ 4.9 or newer, clang 3.2 or newer, Visual C++ 2015 or newer
- make
- SWIG or newer for language bindings other than C++
Compilation
-----------
To compile MorphoDiTa, run make in the src directory.
Make targets and options:
- exe: compile the binaries (default)
- server: compile the REST server
- tools: compile various tools
- lib: compile MorphoDiTa library (decoding only)
- BITS=32 or BITS=64: compile for specified 32-bit or 64-bit architecture
instead of the default one
- MODE=release: create release build which statically links the C++ runtime and
uses LTO
- MODE=debug: create debug build
- MODE=profile: create profile build
Platforms
---------
Platform can be selected using one of the following options:
- PLATFORM=linux, PLATFORM=linux-gcc: gcc compiler on Linux operating system,
default on Linux
- PLATFORM=linux-clang: clang compiler on Linux, must be selected manually
- PLATFORM=macos, PLATFORM=macos-clang: clang compiler on OS X, default on OS
X; BITS=32+64 enables multiarch build
- PLATFORM=win, PLATFORM=win-gcc: gcc compiler on Windows (TDM-GCC is well
tested), default on Windows
- PLATFORM=win-vs: Visual C++ 2015 compiler on Windows, must be selected
manually; note that the cl.exe compiler must be already present in PATH and
corresponding BITS=32 or BITS=64 must be specified
Either POSIX shell or Windows CMD can be used as shell, it is detected
automatically.
Further Details
---------------
MorphoDiTa uses C++ BuilTem system (http://github.com/ufal/cpp_builtem), please
refer to its manual if interested in all supported options.
Other language bindings
-----------------------
C#
--
Binary C# bindings are available in MorphoDiTa binary packages.
To compile C# bindings manually, run make in the bindings/csharp directory,
optionally with the options descriged in MorphoDiTa Installation.
Java
----
Binary Java bindings are available in MorphoDiTa binary packages.
To compile Java bindings manually, run make in the bindings/java directory,
optionally with the options descriged in MorphoDiTa Installation. Java 6 and
newer is supported.
The Java installation specified in the environment variable JAVA_HOME is used.
If the environment variable does not exist, the JAVA_HOME can be specified using
make JAVA_HOME=path_to_Java_installation
Perl
----
The Perl bindings are available as Ufal-MorphoDiTa package on CPAN.
To compile Perl bindings manually, run make in the bindings/perl directory,
optionally with the options descriged in MorphoDiTa Installation. Perl 5.10 and
later is supported.
Path to the include headers of the required Perl version must be specified in
the PERL_INCLUDE variable using
make PERL_INCLUDE=path_to_Perl_includes
Python
------
The Python bindings are available as ufal.morphodita package on PyPI.
To compile Python bindings manually, run make in the bindings/python directory,
optionally with options descriged in MorphoDiTa Installation. Python 3+ is
supported.
Path to the include headers of the required Python version must be specified in
the PYTHON_INCLUDE variable using
make PYTHON_INCLUDE=path_to_Python_includes
MorphoDiTa User's Manual
========================
In a natural language text, the task of morphological analysis is to assign for
each token (word) in a sentence its lemma (cannonical form) and a part-of-speech
tag (POS tag). This is usually achieved in two steps: a morphological dictionary
looks up all possible lemmas and POS tags for each word, and subsequently, a
morphological tagger picks for each word the best lemma-POS tag candidate. The
second task is called a disambiguation.
MorphoDiTa also performs these two steps of morphological analysis: It first
outputs all possible pairs of lemma and POS tag for each token. Consequently,
the optimal combination of lemmas and POS tags is selected for the words in a
sentence using an algorithm described in Spoustová et al. 2009
(http://aclweb.org/anthology//E/E09/E09-1087.pdf).
Like any supervised machine learning tool, MorphoDiTa needs a trained linguistic
model. This section describes the available language models and also the
commandline tools and interfaces. The C++ library is described elsewhere, either
in MorphoDiTa API Tutorial or in MorphoDiTa API Reference.
Czech MorfFlex2+PDT-C Models
----------------------------
Czech models are distributed under the CC BY-NC-SA
(http://creativecommons.org/licenses/by-nc-sa/4.0/) licence. The Czech
morphology uses the MorfFlex CZ 2.0 (http://hdl.handle.net/11234/1-3186) Czech
morphological dictionary and the Czech tagger is trained on PDT-C 1.0
(http://hdl.handle.net/11234/1-3185). The morpholodical derivator is uses the
DeriNet 2.1 (http://hdl.handle.net/11234/1-3765). Czech models work in
MorphoDiTa version 1.9 or later.
Apart from MorfFlex CZ dictionary, a prefix guesser and statistical guesser are
implemented and can be optionally used when performing morphological analysis.
Download
--------
The latest version 220710 of the Czech MorfFlex+PDT models can be downloaded
from LINDAT/CLARIN repository (http://hdl.handle.net/11234/1-4794).
Acknowledgements
----------------
This work has been has been supported by the LINDAT/CLARIAH-CZ project funded by
Ministry of Education, Youth and Sports of the Czech Republic (project
LM2018101).
Publications
------------
- (Straková et al., 2014) Straková Jana, Straka Milan and Hajič Jan.
Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity
Recognition (https://www.aclweb.org/anthology/P/P14/P14-5003.pdf). In
Proceedings of 52nd Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 13-18, Baltimore, Maryland, June
2014. Association for Computational Linguistics.
- (Jonáš Vidra et al., 2019) Jonáš Vidra, Zdeněk Žabokrtský, Magda
Ševčíková, Lukáš Kyjánek. Towards an All-in-One Word-Formation
Resource. In Proceedings of the Second Workshop on Resources and Tools for
Derivational Morphology (DeriMo 2019). Prague, 2019, pp. 81-89.
- (Jan Hajič et al., 2020) Jan Hajič, Eduard Bejček, Jaroslava Hlavacova,
Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpánková.
Prague Dependency Treebank - Consolidated 1.0
(https://aclanthology.org/2020.lrec-1.641.pdf). In Proceedings of the 12th
Language Resources and Evaluation Conference, pages 5208–5218, Marseille,
France. European Language Resources Association.
- (Marie Mikulová et al., 2022) Mikulová Marie, Hajič Jan, Hana Jiří,
Hanová Hana, Hlaváčová Jaroslava, Jeřábek Emil, Štěpánková Barbora,
Vidová Hladká Barbora, Zeman Daniel. Manual for Morphological Annotation,
Revision for the Prague Dependency Treebank - Consolidated 2020 release
(https://ufal.mff.cuni.cz/techrep/tr64.pdf). Technical report no. TR-2020-64,
Institute of Formal and Applied Linguistics, Charles University, Prague,
Czechia, 2020.
MorfFlex CZ 2.0 Morphological System
------------------------------------
The MorfFlex CZ 2.0 uses a so-called PDT-C tag set, which is an evolution of the
original PDT tag set devised by Jan Hajič (Hajič 2004
(http://books.google.cz/books?id=sB63AAAACAAJ)). The tags are positional with 15
positions corresponding to part of speech, detailed part of speech, gender,
number, case, etc. (e.g. NNFS1-----A----). Different meanings of same lemmas are
distinguished and additional comments can be provided for every lemma meaning.
The lemma itself without the comments and meaning specification is called a raw
lemma. The following examples illustrate this:
- Japonsko_;G (raw lemma: Japonsko)
- se_^(zvr._zájmeno/částice) (raw lemma: se)
- tvořit_:T (raw lemma: tvořit)
The complete reference can be found in the Manual for Morphological Annotation,
Revision for the Prague Dependency Treebank - Consolidated 2020 release
(https://ufal.mff.cuni.cz/techrep/tr64.pdf).
PDT-C 1.0 Train/Dev/Test Split
------------------------------
The PDT-C corpus consists of four datasets, but some of them do not have an
official train/dev/test split. We therefore used the following split:
- PDT dataset is already split into train, dev (dtest), and test (etest).
- PCEDT dataset is a translated version of the Wall Street Journal, so we used
the usual split into train (sections 0-18), dev (sections 19-21), and test
(sections 22-24).
- PDTSC and FAUST datasets have no split, so we split it into dev (documents
with identifiers ending with 6), test (documents with identifiers ending with
7), and train (all the remaining documents).
Model Variants
--------------
Apart from the primary model, which predicts all the 15 tag positions and
processed texts with diacritics, we also provide several variants:
- pos_only: Instead of all 15 tag positions, the model predicts only the first
2, which contain the coarse and detailed POS, plus the full lemma, while being
circa 15 times faster than the primary model.
- no_dia, no_dia-pos_only: The forms (during morphological analysis,
generation, and tagging) have the diacritical marks stripped; however, the
lemmas do include them. Useful for processing texts without diacritics.
Model Performance
-----------------
|| | Tags ||||| Lemmas ||||| Performance ||
|| Model | PDT | PCEDT | PDTSC | Faust | MacroAvg | PDT | PCEDT | PDTSC | Faust | MacroAvg | Speed | Size |
| ``czech-morfflex2.0-pdtc1.0-220710`` | 96.29 | 97.00 | 96.90 | 94.87 | 96.27 | 98.69 | 98.85 | 98.18 | 97.53 | 98.31 | 19k toks/s | 24.4MB |
| ``czech-morfflex2.0-pdtc1.0-220710-pos_only`` | 98.99 | 99.12 | 98.45 | 97.85 | 98.60 | 98.50 | 98.63 | 98.09 | 97.05 | 98.07 | 253k toks/s | 9.5MB |
| ``czech-morfflex2.0-pdtc1.0-220710-no_dia`` | 95.57 | 96.13 | 96.40 | 93.46 | 95.39 | 97.88 | 98.20 | 97.67 | 96.57 | 97.58 | 11k toks/s | 30.4MB |
| ``czech-morfflex2.0-pdtc1.0-220710-no_dia-pos_only`` | 98.55 | 98.73 | 98.07 | 97.31 | 98.17 | 97.60 | 97.85 | 97.52 | 95.98 | 97.24 | 177k toks/s | 14.5MB |
Czech MorfFlex+PDT Models
-------------------------
Czech models are distributed under the CC BY-NC-SA
(http://creativecommons.org/licenses/by-nc-sa/4.0/) licence. The Czech
morphology uses the MorfFlex CZ 161115 (http://hdl.handle.net/11234/1-1834)
Czech morphological dictionary and the Czech tagger is trained on PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0). The morpholodical derivator is uses the
DeriNet 1.2 (http://ufal.mff.cuni.cz/derinet). Czech models work in MorphoDiTa
version 1.9 or later.
Apart from MorfFlex CZ dictionary, a prefix guesser and statistical guesser are
implemented and can be optionally used when performing morphological analysis.
Czech models are versioned according to the version of the MorfFlex CZ
morphological dictionary used, the version format is YYMMDD, where YY, MM and DD
are two-digit representation of year, month and day, respectively. The latest
version is 161115.
Compared to Featurama http://sourceforge.net/projects/featurama/
(state-of-the-art Czech tagger implementation), the models are 5 times faster
and 10 times smaller.
Download
--------
The latest version 161115 of the Czech MorfFlex+PDT models can be downloaded
from LINDAT/CLARIN repository (http://hdl.handle.net/11234/1-1836).
Previous Versions
-----------------
- Version 160310 of the Czech MorphoDiTa models can be downloaded from
LINDAT/CLARIN repository (http://hdl.handle.net/11234/1-1674).
- Version 131112 of the Czech MorphoDiTa models can be downloaded from
LINDAT/CLARIN repository
(http://hdl.handle.net/11858/00-097C-0000-0023-68D8-1).
Acknowledgements
----------------
This work has been using language resources developed and/or stored and/or
distributed by the LINDAT/CLARIN project of the Ministry of Education of the
Czech Republic (project LM2010013).
The Czech morphological system was devised by Jan Hajič.
The MorfFlex CZ dictionary was created by Jan Hajič and Jaroslava Hlaváčová.
The morphological guesser research was supported by the projects 1ET101120503
and 1ET101120413 of Academy of Sciences of the Czech Republic and 100008/2008 of
Charles University Grant Agency. The research was performed by Jan Hajič,
Jaroslava Hlaváčová and David Kolovratník.
The tagger algorithm and feature set research was supported by the projects
MSM0021620838 and LC536 of Ministry of Education, Youth and Sports of the Czech
Republic, GA405/09/0278 of the Grant Agency of the Czech Republic and
1ET101120503 of Academy of Sciences of the Czech Republic. The research was
performed by Drahomíra "johanka" Spoustová, Jan Hajič, Jan Raab and Miroslav
Spousta.
The tagger is trained on morphological layer of Prague Dependency Treebank PDT
3.0, which was supported by the projects LM2010013, LC536, LN00A063 and
MSM0021620838 of Ministry of Education, Youth and Sports of the Czech Republic,
and developed by Martin Buben, Jan Hajič, Jiří Hana, Hana Hanová, Barbora
Hladká, Emil Jeřábek, Lenka Kebortová, Kristýna Kupková, Pavel Květoň,
Jiří Mírovský, Andrea Pfimpfrová, Jan Štěpánek and Daniel Zeman.
The morphological derivator is based on DeriNet, which was supported by the
Grant No. 16-18177S of the Grant Agency of the Czech Republic and uses language
resources developed, stored, and distributed by the LINDAT/CLARIN project of the
Ministry of Education, Youth and Sports of the Czech Republic (project
LM2015071).
Publications
------------
- (Hajič 2004) Jan Hajič. Disambiguation of Rich Inflection: Computational
Morphology of Czech. (http://books.google.cz/books?id=sB63AAAACAAJ) Karolinum
Press (2004).
- Hlaváčová Jaroslava, Kolovratník David. Morfologie češtiny znovu a
lépe. In Informačné Technológie - Aplikácie a Teória. Zborník
príspevkov, ITAT 2008. Seňa, Slovakia: PONT s.r.o., 2008, pp. 43-47.
- (Spoustová et al. 2009) Drahomíra "johanka" Spoustová, Jan Hajič, Jan
Raab, Miroslav Spousta. 2009. Semi-Supervised Training for the Averaged
Perceptron POS Tagger. (http://aclweb.org/anthology//E/E09/E09-1087.pdf) In
Proceedings of the 12th Conference of the European Chapter of the ACL (EACL
2009), pages 763-771, Athens, Greece, March. Association for Computational
Linguistics.
- (Straková et al. 2014) Straková Jana, Straka Milan and Hajič Jan.
Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity
Recognition. (http://www.aclweb.org/anthology/P/P14/P14-5003.pdf) In
Proceedings of 52nd Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 13-18, Baltimore, Maryland, June
2014. Association for Computational Linguistics.
- (Žabokrtský et al. 2016) Zdeněk Žabokrtský, Magda Ševčíková, Milan
Straka, Jonáš Vidra and Adéla Limburská. Merging Data Resources for
Inflectional and Derivational Morphology in Czech. In Proceedings of the Tenth
International Conference on Language Resources and Evaluation (LREC 2016),
Portorož, Slovenia, May 2016.
Czech Morphological System
--------------------------
In the Czech language, MorphoDiTa uses Czech morphological system by Jan Hajič
(Hajič 2004 (http://books.google.cz/books?id=sB63AAAACAAJ)). In this system,
which we call PDT tag set, the tags are positional with 15 positions
corresponding to part of speech, detailed part of speech, gender, number, case,
etc. (e.g. NNFS1-----A----). Different meanings of same lemmas are
distinguished and additional comments can be provided for every lemma meaning.
The lemma itself without the comments and meaning specification is called a raw
lemma. The following examples illustrate this:
- Japonsko_;G (raw lemma: Japonsko)
- se_^(zvr._zájmeno/částice) (raw lemma: se)
- tvořit_:T (raw lemma: tvořit)
For a more detailed reference about the Czech morphology, please see Lemma and
Tag Structure in PDT 2.0
(http://ufal.mff.cuni.cz/pdt2.0/doc/manuals/en/m-layer/html/ch02.html).
Main Czech Model
----------------
The main Czech model contains the following files:
czech-morfflex-161115.dict
Morphological dictionary based on the Jan Hajič's (Hajič 2004
(http://books.google.cz/books?id=sB63AAAACAAJ)) system with PDT tag set
created from MorfFlex CZ 161115 (http://hdl.handle.net/11234/1-1834)
morphological dictionary and DeriNet 1.2 (http://ufal.mff.cuni.cz/derinet).
czech-morfflex-pdt-161115.tagger
Tagger trained on the training portion of PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) using the neopren feature set. It contains
the czech-morfflex-161115.dict morphological dictionary. and reaches 95.55%
tag accuracy, 97.86% lemma accuracy and 95.06% overall accuracy on PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) etest data (whose morphological tags and
lemmas were remapped using the czech-morfflex-161115.dict dictionary). Model
speed: ~15k words/s, model size: 18MB.
Part of Speech Only Variant
---------------------------
The PDT tag set used by the main Czech model is very fine-grained. In many
situations, only the part of speech tags would be sufficient. Therefore, we
provide a variant of the model, denoted as pos_only, where only the first two
characters of the fifteen-letter tags are used, representing the part of speech
and detailed part of speech, respectively. There are 67 such two-letter tags.
czech-morfflex-161115-pos_only.dict
Morphological dictionary based on the Jan Hajič's (Hajič 2004
(http://books.google.cz/books?id=sB63AAAACAAJ)) system created from MorfFlex
CZ 161115 (http://hdl.handle.net/11234/1-1834) morphological dictionary and
DeriNet 1.2 (http://ufal.mff.cuni.cz/derinet). Only first two tag characters
of PDT tag set are used.
czech-morfflex-pdt-161115-pos_only.tagger
Very fast tagger trained on the training portion of PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) using the neopren feature set. It contains
the czech-morfflex-161115-pos_only.dict morphological dictionary and reaches
99.01% tag accuracy, 97.69% lemma accuracy and 97.66% overall accuracy on PDT
3.0 (http://ufal.mff.cuni.cz/pdt3.0) etest data (which morphological tags and
lemmas were remapped using the czech-morfflex-161115-pos_only.dict
dictionary). Model speed: ~250k words/s, model size: 5MB.
No Diacritical Marks Variant
----------------------------
Sometimes the text to be analyzed does not contain diacritical marks. We
therefore provide variants of the morphological dictionary and tagger for this
purpose - morphological analysis, morphological generation and tagging employ
forms without diacritical marks. Note that the lemmas do have diacritical marks.
We provide the no_dia variants for all four models described above:
czech-morfflex-161115-no_dia.dict
No diacritical marks variant of czech-morfflex-161115.dict.
czech-morfflex-pdt-161115-no_dia.tagger
No diacritical marks variant of czech-morfflex-161115.tagger. It reaches
94.69% tag accuracy, 97.06% lemma accuracy and 93.84% overall accuracy on PDT
3.0 (http://ufal.mff.cuni.cz/pdt3.0) etest data (which morphological tags and
lemmas were remapped using the czech-morfflex-161115-no_dia.dict dictionary)
with diacritical marks removed. Model speed: ~7.5k words/s, model size: 22MB.
czech-morfflex-161115-no_dia-pos_only.dict
No diacritical marks variant of czech-morfflex-161115-pos_only.dict.
czech-morfflex-pdt-161115-no_dia-pos_only.tagger
No diacritical marks variant of czech-morfflex-161115-pos_only.tagger. It
reaches 98.55% tag accuracy, 97.07% lemma accuracy and 97.02% overall accuracy
on PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0) etest data (which morphological
tags and lemmas were remapped using the
czech-morfflex-161115-no_dia-pos_only.dict dictionary) with diacritical marks
removed. Model speed: ~125k words/s, model size: 11MB.
Models with Raw Lemmas
----------------------
The Czech morphological system distinguish different meanings of same lemmas by
numbering the lemmas with multiple meanings and supplying additional comments
for every lemma meaning, as described and demonstrated in Czech Morphological
System. Sometimes this may be undesirable, for example when comparing to systems
which do not use the MorfFlex CZ (http://hdl.handle.net/11234/1-1834)
morphological dictionary.
To obtain lemmas without any additional information (raw lemmas in terms of
MorphoDiTa API), use strip_lemma_id tag set converter. Previously, specific
dictionary and tagger model variants were provided, which is not needed anymore.
Czech Model History
-------------------
czech-morfflex-161115 and czech-morfflex-pdt-161115 (require MorphoDiTa 1.9 or later)
Trained on PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0) using MorfFlex CZ 161115
(http://hdl.handle.net/11234/1-1834) and DeriNet 1.2
(http://ufal.mff.cuni.cz/derinet), variants: Part of Speech Only, No
Diacritical Marks. Download from LINDAT/CLARIN repository
(http://hdl.handle.net/11234/1-1836).
czech-morfflex-160310 and czech-morfflex-pdt-160310 (require MorphoDiTa 1.0 or later)
Trained on PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0) using MorfFlex CZ 160310
(http://hdl.handle.net/11234/1-1673), variants: Part of Speech Only, No
Diacritical Marks. Download from LINDAT/CLARIN repository
(http://hdl.handle.net/11234/1-1674).
czech-morfflex-131112 and czech-morfflex-pdt-131112 (require MorphoDiTa 1.0 or later)
Trained on PDT 2.5 (http://ufal.mff.cuni.cz/pdt2.5) using MorfFlex CZ 131112
(http://hdl.handle.net/11858/00-097C-0000-0015-A780-9), variants Part of
Speech Only, Raw Lemmas. Download from LINDAT/CLARIN repository
(http://hdl.handle.net/11858/00-097C-0000-0023-68D8-1).
Slovak MorfFlex+PDT Models
--------------------------
Slovak models are distributed under the CC BY-NC-SA
(http://creativecommons.org/licenses/by-nc-sa/4.0/) licence. The Slovak
morphology uses the MorfFlex SK 170914 (http://hdl.handle.net/11234/1-3277)
Slovak morphological dictionary and the Slovak tagger is trained on
automatically translated PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0). Slovak models
work in MorphoDiTa version 1.9 or later.
Apart from MorfFlex SK dictionary, a statistical guesser is implemented and can
be optionally used when performing morphological analysis.
Slovak models are versioned according to the version of the MorfFlex SK
morphological dictionary used, the version format is YYMMDD, where YY, MM and DD
are two-digit representation of year, month and day, respectively. The latest
version is 170914.
Download
--------
The latest version 170914 of the Slovak MorfFlex+PDT models can be downloaded
from LINDAT/CLARIN repository (http://hdl.handle.net/11234/1-3278).
Acknowledgements
----------------
This work has also been supported by the LINDAT/CLARIAH-CZ Research
Infrastructure (https://lindat.cz), supported by the Ministry of Education,
Youth and Sports of the Czech Republic (Project No. LM2018101). It has also been
using language resources developed and stored and distributed by the
LINDAT/CLARIN project of the Ministry of Education of the Czech Republic
(project LM2015071).
The Czech morphological system was devised by Jan Hajič.
The MorfFlex SK dictionary was created by Jan Hajič and Jan Hric.
The tagger algorithm and feature set research was supported by the projects
MSM0021620838 and LC536 of Ministry of Education, Youth and Sports of the Czech
Republic, GA405/09/0278 of the Grant Agency of the Czech Republic and
1ET101120503 of Academy of Sciences of the Czech Republic. The research was
performed by Drahomíra "johanka" Spoustová, Jan Hajič, Jan Raab and Miroslav
Spousta.
The tagger is trained on morphological layer of Prague Dependency Treebank PDT
3.0, which was supported by the projects LM2010013, LC536, LN00A063 and
MSM0021620838 of Ministry of Education, Youth and Sports of the Czech Republic,
and developed by Martin Buben, Jan Hajič, Jiří Hana, Hana Hanová, Barbora
Hladká, Emil Jeřábek, Lenka Kebortová, Kristýna Kupková, Pavel Květoň,
Jiří Mírovský, Andrea Pfimpfrová, Jan Štěpánek and Daniel Zeman.
Publications
------------
- (Hajič 2004) Jan Hajič. Disambiguation of Rich Inflection: Computational
Morphology of Czech. (http://books.google.cz/books?id=sB63AAAACAAJ) Karolinum
Press (2004).
- (Spoustová et al. 2009) Drahomíra "johanka" Spoustová, Jan Hajič, Jan
Raab, Miroslav Spousta. 2009. Semi-Supervised Training for the Averaged
Perceptron POS Tagger. (http://aclweb.org/anthology//E/E09/E09-1087.pdf) In
Proceedings of the 12th Conference of the European Chapter of the ACL (EACL
2009), pages 763-771, Athens, Greece, March. Association for Computational
Linguistics.
- (Straková et al. 2014) Straková Jana, Straka Milan and Hajič Jan.
Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity
Recognition. (http://www.aclweb.org/anthology/P/P14/P14-5003.pdf) In
Proceedings of 52nd Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 13-18, Baltimore, Maryland, June
2014. Association for Computational Linguistics.
Slovak Morphological System
---------------------------
In the Slovak language, MorphoDiTa uses the same morphological system as Czech.
Main Slovak Model
-----------------
The main Slovak model contains the following files:
slovak-morfflex-170914.dict
Morphological dictionary based on the Jan Hajič's (Hajič 2004
(http://books.google.cz/books?id=sB63AAAACAAJ)) system with PDT tag set
created from MorfFlex SK 170914 (http://hdl.handle.net/11234/1-3277)
morphological dictionary.
slovak-morfflex-pdt-170914.tagger
Tagger trained on the training portion of automatically translated PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) using the neopren feature set. It contains
the slovak-morfflex-170914.dict morphological dictionary. and reaches 92.8%
tag accuracy, 96.3% lemma accuracy and 92.0% overall accuracy on PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) etest data (whose morphological tags and
lemmas were remapped using the slovak-morfflex-170914.dict dictionary). Model
speed: ~5k words/s, model size: 17MB.
Part of Speech Only Variant
---------------------------
The PDT tag set used by the main Slovak model is very fine-grained. In many
situations, only the part of speech tags would be sufficient. Therefore, we
provide a variant of the model, denoted as pos_only, where only the first two
characters of the fifteen-letter tags are used, representing the part of speech
and detailed part of speech, respectively. There are 67 such two-letter tags.
slovak-morfflex-170914-pos_only.dict
A variant of `slovak-morfflex-170914.dict`, where only the first two tag
characters are used.
slovak-morfflex-pdt-170914-pos_only.tagger
Very fast variant of slovak-morfflex-170914.tagger predicting only
two-character tags. It reaches 98.3% tag accuracy, 97.4% lemma accuracy and
96.8% overall accuracy on PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0) etest data
(which morphological tags and lemmas were remapped using the
slovak-morfflex-170914-pos_only.dict dictionary). Model speed: ~200k words/s,
model size: 4MB.
No Diacritical Marks Variant
----------------------------
Sometimes the text to be analyzed does not contain diacritical marks. We
therefore provide variants of the morphological dictionary and tagger for this
purpose - morphological analysis, morphological generation and tagging employ
forms without diacritical marks. Note that the lemmas do have diacritical marks.
We provide the no_dia variants for all four models described above:
slovak-morfflex-170914-no_dia.dict
No diacritical marks variant of slovak-morfflex-170914.dict.
slovak-morfflex-pdt-170914-no_dia.tagger
No diacritical marks variant of slovak-morfflex-170914.tagger. It reaches
91.4% tag accuracy, 92.8% lemma accuracy and 89.0% overall accuracy on PDT 3.0
(http://ufal.mff.cuni.cz/pdt3.0) etest data (which morphological tags and
lemmas were remapped using the slovak-morfflex-170914-no_dia.dict dictionary)
with diacritical marks removed. Model speed: ~5k words/s, model size: 18MB.
slovak-morfflex-170914-no_dia-pos_only.dict
No diacritical marks variant of slovak-morfflex-170914-pos_only.dict.
slovak-morfflex-pdt-170914-no_dia-pos_only.tagger
No diacritical marks variant of slovak-morfflex-170914-pos_only.tagger. It
reaches 97.5% tag accuracy, 93.9% lemma accuracy and 93.2% overall accuracy on
PDT 3.0 (http://ufal.mff.cuni.cz/pdt3.0) etest data (which morphological tags
and lemmas were remapped using the slovak-morfflex-170914-no_dia-pos_only.dict
dictionary) with diacritical marks removed. Model speed: ~200k words/s, model
size: 7MB.
English Morphium+WSJ Models
---------------------------
English models are created using the following data:
- SCOWL (Spell Checker Oriented Word Lists): This word list is used in
morphological generation to create all possible word forms of a given word.
Copyright: Copyright 2000-2011 by Kevin Atkinson. Permission to use, copy,
modify, distribute and sell these word lists, the associated scripts, the
output created from the scripts, and its documentation for any purpose is
hereby granted without fee, provided that the above copyright notice appears
in all copies and that both that copyright notice and this permission notice
appear in supporting documentation. Kevin Atkinson makes no representations
about the suitability of this array for any purpose. It is provided "as is"
without express or implied warranty.
- Wall Street Journal, part of the Penn Treebank 3: Morphologically annotated
texts which are commonly used to train English POS tagger.
Licensing: Available as LDC99T42 in LDC catalog under LDC User Agreement.
The resulting models are distributed under the CC BY-NC-SA
(http://creativecommons.org/licenses/by-nc-sa/3.0/) licence. English models work
in MorphoDiTa version 1.1 or later.
English models are versioned according to the release date, the version format
is YYMMDD, where YY, MM and DD are two-digit representation of year, month and
day, respectively. The latest version is 140407.
Download
--------
The latest version 140407 of the English Morphium+WSJ models can be downloaded
from LINDAT/CLARIN repository
(http://hdl.handle.net/11858/00-097C-0000-0023-68D9-0).
Acknowledgements
----------------
This work has been using language resources developed and/or stored and/or
distributed by the LINDAT/CLARIN project of the Ministry of Education of the
Czech Republic (project LM2010013).
The morphological POS analyzer development was supported by grant of the
Ministry of Education, Youth and Sports of the Czech Republic No. LC536 "Center
for Computational Linguistics". The morphological POS analyzer research was
performed by Johanka Spoustová (Spoustová 2008; the
Treex::Tool::EnglishMorpho::Analysis Perl module). The lemmatizer was
implemented by Martin Popel (Popel 2009; the
Treex::Tool::EnglishMorpho::Lemmatizer Perl module). The lemmatizer is based on
morpha, which was released under LGPL licence as a part of RASP system
(http://ilexir.co.uk/applications/rasp).
The tagger algorithm and feature set research was supported by the projects
MSM0021620838 and LC536 of Ministry of Education, Youth and Sports of the Czech
Republic, GA405/09/0278 of the Grant Agency of the Czech Republic and
1ET101120503 of Academy of Sciences of the Czech Republic. The research was
performed by Drahomíra "johanka" Spoustová, Jan Hajič, Jan Raab and Miroslav
Spousta.
Publications
------------
- (Popel 2009) Martin Popel. Ways to Improve the Quality of English-Czech
Machine Translation. (http://ufal.mff.cuni.cz/~popel/papers/master_thesis.pdf)
Master Thesis at Institute of Formal and Applied Linguistics, Faculty of
Mathematics and Physics, Charles University in Prague (2009).
- (Spoustová 2008) Drahomíra "johanka" Spoustová. Morphium - morphological
analyser for Penn treebank POS tagset. (http://ufal.mff.cuni.cz/morphium/)
Perl Software developed at Institute of Formal and Applied Linguistics,
Faculty of Mathematics and Physics, Charles University in Prague (2008).
- (Spoustová et al. 2009) Drahomíra "johanka" Spoustová, Jan Hajič, Jan
Raab, Miroslav Spousta. 2009. Semi-Supervised Training for the Averaged
Perceptron POS Tagger. (http://aclweb.org/anthology//E/E09/E09-1087.pdf) In
Proceedings of the 12th Conference of the European Chapter of the ACL (EACL
2009), pages 763-771, Athens, Greece, March. Association for Computational
Linguistics.
- (Straková et al. 2014) Straková Jana, Straka Milan and Hajič Jan.
Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity
Recognition. (http://www.aclweb.org/anthology/P/P14/P14-5003.pdf) In
Proceedings of 52nd Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 13-18, Baltimore, Maryland, June
2014. Association for Computational Linguistics.
English Morphological System
----------------------------
The English morphology uses standard Penn Treebank POS tags. Nevertheless, the
lemma structure is unique:
- The lemmatizer recognizes negative prefixes and removes it from the lemma. In
terms of MorphoDiTa API, raw lemma is the lemma without negative prefix.
- The negative prefix is also stored to allow morphological generation of word
form with the same negative prefix. In terms of MorphoDiTa API, lemma id is
the raw lemma plus the negative prefix.
The negative prefix is separated from the (always nonempty) lemma using a ^
character (able^un). During morphological generation, the negative prefix is
honored. Furthermore, when the lemma ends with ^ (i.e., negative prefix is
empty, as in able^), forms with negative prefixes are generated. It is also
possible to generate all forms without any negative prefix by appending + after
the lemma (for example able+).
English Model
-------------
The English model contains the following files:
english-morphium-<version>.dict
Morphological dictionary. The SCOWL word list has been automatically analyzed
and lemmatized and uses as the dictionary. The guesser performing the
analyzation and lemmatization is available.
english-morphium-wsj-<version>.tagger
Tagger trained on the training portion of Wall Street Journal (Sections 0-18)
and tuned on the development portion (Sections 19-21). Contains the
english-morphium-<version>.dict morphological dictionary.
The latest version english-morphium-wsj-140407.tagger reaches 97.27% tag
accuracy on Wall Street Journal test portion (Section 22-24). Model speed:
~60k words/s, model size: 6MB.
No Negations Variant
--------------------
Stripping of negative prefixes (or handling the lemmas with negative prefixes
stripped) may not be desirable. Therefore, a variant of the English model
denoted by no_negation is provided, which does not strip negative prefixes from
lemmas.
english-morphium-<version>-no_negation.dict
Morphological dictionary which does not strip negative lemma prefixes. The
SCOWL word list has been automatically analyzed and lemmatized and uses as the
dictionary. The guesser performing the analyzation and lemmatization is
available.
english-morphium-wsj-<version>-no_negation.tagger
Tagger which does not strip negative lemma prefixes, trained on the training
portion of Wall Street Journal (Sections 0-18) and tuned on the development
portion (Sections 19-21). Contains the
english-morphium-<version>-no_negation.dict morphological dictionary.
The latest version english-morphium-wsj-140407-no_negation.tagger reaches
97.25% tag accuracy on Wall Street Journal test portion (Section 22-24). Model
speed: ~60k words/s, model size: 6MB.
English Model Changes
---------------------
english-morphium-140407 and english-morphium-wsj-140407 (require MorphoDiTa 1.1 or later)
Recognize also "non-" as a negative prefix. Formerly, only "non" was
recognized.
english-morphium-140304 and english-morphium-wsj-140304 (require MorphoDiTa 1.0 or later)
Initial release.
Running the Tagger
------------------
Probably the most common usage of MorphoDita is running a tagger to tag your
data using
run_tagger tagger_model
The input is assumed to be in UTF-8 encoding and can be either already tokenized
and segmented, or it can be a plain text which is tokenized and segmented
automatically.
Any number of files can be specified after the tagger_model. If an argument
input_file:output_file is used, the given input_file is processed and the result
is saved to output_file. If only input_file is used, the result is saved to
standard output. If no argument is given, input is read from standard input and
written to standard output.
The full command syntax of run_tagger is
run_tagger [options] tagger_file [file[:output_file]]...
Options: --input=untokenized|vertical
--convert_tagset=pdt_to_conll2009|strip_lemma_comment|strip_lemma_id
--derivation=none|root|path|tree
--guesser=0|1 (should morphological guesser be used)
--output=vertical|xml
Input Formats
-------------
The input format is specified using the --input option. Currently supported
input formats are:
- untokenized (default): the input is tokenized and segmented using a tokenizer
defined by the model,
- vertical: the input is in vertical format, every line is considered a word,
with empty line denoting end of sentence.
Tag Set Conversion
------------------
Some tag sets can be converted to different ones. Currently supported tag set
conversions are:
- pdt_to_conll2009: convert Czech PDT tag set to CoNLL 2009 tag set,
- strip_lemma_comment: strip lemma comment (see Lemma Structure in API
Reference),
- strip_lemma_id: strip lemma id (see Lemma Structure in API Reference).
Morphological Derivation
------------------------
If the morphological model includes a morphological derivator, some
morphological derivation operation may be performed on lemmas:
- none (default): no morphological derivation is performed
- root: lemma is replaced by its root in the morphological derivation tree
- path: lemma is replaced by a space separated path to its root in the
morphological derivation tree (the original lemma is first, followed by its
parent, with the root being the last one)