marskafka

Marskafka is a Kafka data processing pipeline that uses deep learning with Kafka to colorize & tweet images from NASA’s Mars Rover API

1. Images from our Mars Rovers are consumed from NASA's API and are filtered between two topics:

• queue.bwimg for greyscale images
• stream.colorimg for RGB images

2. An instance of the deoldify deep learning network listens to the greyscale queue, colorizes the images, and forwards them to the colored stream
3. Both topics will them merge at nabokov– a twitter bot that takes 3 images from the current epoch (daily) and tweets them to this account

I built this to learn about distributed computing, publish/subscribe models, and stream processing

Codebase structure

Directory	Description
pushkin	HTTP service that consumes Mars API & pipes to kafka
tolstoy	Deep learning service
nabokov	Twitter Bot

Notes

• i had initially wanted to deploy and expose my own instance of deoldify and use my own API keys but i'm temporarily using this api for my colorization needs, as i don't have the time to dive into MLOps with AWS Lambda, and AWS SageMaker,