-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Weird issues with short texts in Russian #100
Comments
Pure statistical approaches to language detection are never 100% correct. Look at the confidence values, Russian is only slightly behind Macedonian. Based on the training data I've used, some of the letter sequences are slightly more likely to occur in Macedonian than in Russian.
Feed longer strings into the detector. Then you will get more reliable results. An interesting approach to solve this problem has been proposed in #101. I will investigate whether changing the probabilities in the mentioned way will produce more accurate results. |
Thanks @pemistahl. The issue for me was that this specific string does not appear to be dramatically correct Macedonian. I don't quite understand the algo used to point out why this is happening though |
Based on the training data I've used, the letter sequences in the text I've briefly explained the algorithm in section 5 of the readme. |
Hi team, great library! Wanted to share an example I stumbled upon, when detecting the language of a very short basic Russian text. It comes out as Macedonian, even though as far as I can tell it's not actually correct Macedonian but is correct Russian. It is identified correctly by AWS Comprehend and other APIs:
The text was updated successfully, but these errors were encountered: