In this article, we describe how we orchestrate Kafka, Dataflow and BigQuery together to ingest and transform a large stream of events. When adding scale and latency constraints, reconciling and reordering them becomes a challenge, here is how we tackle it.
The above diagram is a fairly good representation of our architecture at Storyblocks. If you’re not an experienced web developer, you’ll likely find it complicated. The walk through below should make…
I’m sort of obsessed about iteration speed. I’ve written about this in the past and it deserves more posts in the future, but the quick summary is that iteration speed is always going to be the strongest competitive advantage in this industry. There’s of course many ways we can iterate faster, but for today let’s focus on two particular aspects of it: testing and deploying more often.
W. Wang, W. Wang, Q. Li, and F. Yang. Ontology, Epistemology, and Teleology for Modeling and Simulation, volume 44 of Intelligent Systems Reference Library, Springer, Berlin Heidelberg, (2013)
Y. Zhou, J. Pan, X. Ma, B. Luo, X. Tao, and J. Lu. Proceedings of the 2007 ACM symposium on Applied computing, page 97--103. New York, NY, USA, ACM, (2007)
F. Solms. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, page 363--373. New York, NY, USA, ACM, (2012)
S. Zhang, and S. Goddard. International Conference on Information Technology: Coding and Computing, 2005. ITCC 2005, 2, page 443-- 448 Vol. 2. IEEE, (April 2005)
B. Wang, C. Wen, W. Zhu, and J. Sheng. International Symposium on Computer Science and Computational Technology, 2008. ISCSCT '08, 2, page 383--386. IEEE, (December 2008)
Y. Oh, D. Lee, S. Kang, and J. Lee. 5th IEEE/ACM International Conference on Formal Methods and Models for Codesign, 2007. MEMOCODE 2007, page 87--88. IEEE, (June 2007)
D. Menasce, J. Sousa, S. Malek, and H. Gomaa. Proceedings of the 7th international conference on Autonomic computing, page 195--204. New York, NY, USA, ACM, (2010)