Hi Memphis,

Thanks for your contribution and I completely agree that Apache Kafka, Apache Spark and Hadoop are the best solution to stream stock price data in almost real time (micro-batches).

Having said that, this article is meant to focus less on batch vs stream processing and more on creating an introductory tutorial on Airflow as a scheduling tool (particularly how to set DAGs properly) :D

By the way, have you ever used any streaming tool for a project? If yes, how difficult was to implement a working pipeline the very first time?

Thank you :)


Snr BI Engineer @Wise | 🏆 Among Top Writers In Data Engineering 💻 Follow & Contact Me 🤝 https://www.linkedin.com/in/anbento4