Skip to content

Commit

Permalink
set blf_01 Tokenizer max token count to 0 so entangled tokens are not…
Browse files Browse the repository at this point in the history
… generated (#31)
  • Loading branch information
elliVM authored Apr 19, 2024
1 parent 0207fc8 commit 4ed70f2
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ public class TokenizerUDF implements UDF1<String, List<byte[]>> {
public List<byte[]> call(String s) throws Exception {
if (tokenizer == null) {
// "lazy" init
tokenizer = new Tokenizer(32);
tokenizer = new Tokenizer(0);
}

// create empty Scala immutable List
Expand Down

0 comments on commit 4ed70f2

Please sign in to comment.