How to process a large document which has longer text length for NER? #1028
Unanswered
AayushSameerShah
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
📝 Brief
I am trying to use the NER for healthcare wanting to extract key "disorders" or "diseases" from different articles from the web for my use-case.
🧠 The model
I have used the "huggingface" model and followed the procedure like given here JSL Tutorial to convert the HF model in TF and use in SparkNLP. And now I have the following code:
👩🏻💻 Code
Then I have the text:
🙋🏻♂️ The question:
I can understand that whole article can't be passed at once, but there has to be some smart way. Since I am new in here, I am not sure whether to split the article in 512 chunks and pass them one by one or something else.
Will anyone please help me here?
Thank you,
Aayush 🤗
Beta Was this translation helpful? Give feedback.
All reactions