In this article, we describe how we orchestrate Kafka, Dataflow and BigQuery together to ingest and transform a large stream of events. When adding scale and latency constraints, reconciling and reordering them becomes a challenge, here is how we tackle it.
The above diagram is a fairly good representation of our architecture at Storyblocks. If you’re not an experienced web developer, you’ll likely find it complicated. The walk through below should make…
I’m sort of obsessed about iteration speed. I’ve written about this in the past and it deserves more posts in the future, but the quick summary is that iteration speed is always going to be the strongest competitive advantage in this industry. There’s of course many ways we can iterate faster, but for today let’s focus on two particular aspects of it: testing and deploying more often.
W. Wang, W. Wang, Q. Li, and F. Yang. Ontology, Epistemology, and Teleology for Modeling and Simulation, volume 44 of Intelligent Systems Reference Library, Springer, Berlin Heidelberg, (2013)