Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

advice needed: producer under gunicorn never sends messages #983

Open
twillis opened this issue Dec 4, 2019 · 1 comment
Open

advice needed: producer under gunicorn never sends messages #983

twillis opened this issue Dec 4, 2019 · 1 comment

Comments

@twillis
Copy link

twillis commented Dec 4, 2019

TLDR: is there a recommended way to use pykafka in a web app to produce messages?

I need to produce some messages as part of a web application and I can do so by creating the connection when the request comes in, but the overhead is too much with sync=true. messages never arrive when connection/topic/producer are module level.

The app in question is running in gunicorn with preload_app=True and worker_class="gthread"

Right now I'm getting around this by spawning a thread and producing the message for the request there but I have concerns that spawning an additional thread per request /response to write message would cause issues under a production load.

I would be running into the same kinds of problems with the database if I was spawning a connection per thread or even worse, per request, however the author(s) of sqlalchemy gives me a connection pool and it works like a charm even with 48 worker threads per 8 processes.

I'm thinking something similar is needed for kafka but i'm kind of surprised it hasn't been written already.

This leads me to believe that maybe i'm on the wrong track here and there is a way in pykafka for producing messages in the context of a web app without the web developer having to know all the intricacies of threading/processes with this library.

Using something like celery seems redundant here and I would rather not take on the complexity of putting rabbitmq or redis in our infrastructure so I can produce messages into kafka.

@twillis
Copy link
Author

twillis commented Dec 10, 2019

here's source that demonstrates the problem. https://github.com/twillis/kafka_web

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant