Use presage for improving text prediction in Maliit

Great, great work. Thank you very much. Just a thought from someone ignorant in this field. Would it be possible in a way to use the language dictionaries from aspell, unspell, whatever, to generate the ngrams? Should we modify presage to use their database directly?

@Damien Caliste, yes, it could be used as one of three sources to presage predictions but not for ngram generation since that requires words in running order which a wordlist does not provide. The primary source is the generated ngrams file from actual texts, secondly your actal usage which provides additional ngram frequency information which is persisted, thirdly word lists derived from aspell, hunspell etc and recently used words. For the wordlist part, if in required format, this could be given in the presage configuration file, default is /usr/share/dict/words. But for presage to be useful you would require an ngram database generated from mixed sources to be complemented by your actual language usage. Right now I'm experimenting with optimal sizes and configuration settings.

Ah, ok thanks for the detailed explanation. Got it.

First of all - thank you very much for it! This is an exciting project that makes SFOS surely better!

If we start with English, it looks like there is a COCA that can help us: https://www.corpusdata.org/ . I presume we can feed this to the generator. For other languages, we would probably need similar dataset. A good start would be to contact the language institute/lab that works on language aspects. They should know whether such corpus exists for your language. For Estonian, I think I know who to contact. But maybe before we start contacting everyone, it would be great to get few databases sorted, so we would have experience with it and know what to ask for. As English does have such database freely accessible, should we start with that?

There seem to be also freely available n-grams databases for few languages at https://www.ngrams.info/ . Not sure whether its the same format though.

@rinigus, thankyou from all of us. Miklós, the main developer, expressed in github that with this early release it might be a strategic/predagogic point to keep back on the number of languages. (But correct me if my impression is wrong). I am currently working on best practices adding language resources to hopefully be useful when the app/library is ready for more people/languages joining in. For now Hungarian and Swedish is well enough for the initial testing, but I could of course release an English ngram database, which undoubtly would attract a lot more people.