Snowball as a shared library

[Sufrostico]

Hi there,

Is there a way to compile snowball to a shared library?

Generating something like libsnowball.so ? or libsnowball.dll?

[Sufrostico]

Hi there, any news on this?

[ojwb]

Please have patience, it's barely been a week since you opened this.

[Sufrostico]

oks, thanks for the answer

[mitya57]

FWIW Debian also ships a patch to use shared library.

[Sufrostico]

@mitya57 can you provide a link, can't find it in alioth

[mitya57]

@Sufrostico It is here. It doesn't apply cleanly to the latest snowball git, but in September I have made an attempt to rebase it against the current git master. My attempt is here — maybe not thoroughly tested.

[Sufrostico]

@mitya57 Nice!

I also made one a few days ago, but yours looks way better.

[ojwb]

There are really two things that need doing to have snowball as a good quality shared library.

The obvious one is to make the build system build a shared library.

The other (which hasn't been addressed here) is that the C code snowball currently generates contains a lot of relocations due to the many arrays of string literals. These result in a load time penalty (as the dynamic loader has to perform these relocations), and a memory overhead (as the pages containing these relocations can't be shared between different processes which have the snowball library loaded). And the problem gets worse for each new stemmer that is contributed.

For more details of these relocations, see section 2.4.3 of this document:

https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf

I actually solved this in a somewhat primitive but effective way many years ago for the fork of snowball we use in Xapian (the reason why we ended up with a fork was that it was sometimes hard to get Martin to accept patches, especially as he was often very busy with other things).

Since I took over as maintainer I've been gradually merging changes from the fork back in to the main tree, with the hope that eventually Xapian could just use upstream snowball, but this one is still to do.

[emirotin]

Looks like this is related: abiliojr/fts5-snowball#4
I managed to compile it (the downstream sqlite ext which first requires the snowball to be compiled) into an .so which seems to work fine for me

[dvershinin]

FYI, for RedHat Enterprise Linux 7 based systems, there is libstemmer in EPEL repository.
Which installs the shared library /usr/lib64/libstemmer.so.0

[amirouche]

Any update on this?

[amirouche]

a pkg-config .pc would be nice too.

[amirouche]

I packaged snowball stemmer for guix (not yet in official tree). You can find the package definition at https://git.sr.ht/~amz3/guix-amz3-channel/tree/master/amz3/language.scm#L71

• I use http://snowball.tartarus.org/dist/libstemmer_c.tgz
• .. and CFLAGS=-Iinclude -fPIC
• then the following bits will generate the shared library:

            (apply invoke "gcc" "-o" "libstemmer.so"
                    "-shared"
                    "libstemmer/libstemmer.o"
                    (append
                     (find-files "src_c"  ".\\.o$")
                     (find-files "runtime"  ".\\.o$")))