Streaming, Fast and Slow_Caito Scherr
What if you were given 2 weeks to prepare for running your first marathon, and had to be able to keep up with the fastest runner? This is the story of a small team that jumped head first into building a top tier stream processing service on a tight timeline, and how they prepared for being able to keep up with the fastest components. This talk will be of interest to those who want to build larger, more complex stream processing applications (and quickly), but will also benefit anyone who is looking to adopt a new streaming technology (or use it in a way that is new to their company), at any scale. Other takeaways include pragmatic steps to leverage unique features of various stream processing frameworks to more effectively solve common, real world data streaming problems.
Streaming, Fast and Slow_Caito Scherr
- 1. STREAMING, FAST AND SLOW June 2020 Caito Scherr Caito_200_OK
- 2. • Caito Scherr • Portland, OR, USA • Software Engineer • Apache Flink Who am I? Caito_200_OK
- 8. Background Caito_200_OK Product A usage Product B usage Product C usage Subscription Info Account Info Data Store, UI Data pipeline
- 10. STREAMING, FAST AND SLOW Caito_200_OK
- 12. … Like training for your first marathon Caito_200_OK
- 13. Caito_200_OK … Like training for a new marathon
- 14. Caito_200_OK … Like training for a new marathon
- 15. Caito_200_OK … Like training for a new marathon
- 16. Caito_200_OK … Like training for a new marathon Integration
- 17. Caito_200_OK … Like training for a new marathon Integration Operations
- 18. Caito_200_OK … Like training for a new marathon Integration Operations Analytics
- 20. Ecosystem Some of the best features of stream processing for your team can also be the biggest drawbacks for teams that interact with you. Caito_200_OK
- 21. Ecosystem Start by assuming each team may be treating their data differently from the next. Caito_200_OK
- 22. Ecosystem Optimize for: understanding your application’s impact, and allowing that data to drive the architecture and design process. Caito_200_OK
- 24. Streaming Community Caito_200_OK Documentation that is interactive, A safe space for others to experiment, And, take a map!
- 26. Streaming Community Caito_200_OK Don’t underestimate the power of a good map
- 28. Resources Can your resources handle this new challenge? Caito_200_OK
- 35. • Pulsar Summit • Jun, Rosalie & the conference event staff! • Maureen Dugan Thank you! • Ruby Andrews • Eric Shamow • Adam Klein • Matthew Scharr • Max Aguirre Caito_200_OK
- 36. ❖ Squirrel photo: https://www.blipfoto.com/entry/1844394 ❖ Track logo: https://www.wikidata.org/wiki/Q1004435 ❖ shoes: https://www.ems.com/hoka-one-one-mens- speedgoat-2-trail-running-shoes/2043264.html ❖ All other photos of Caito: Hood to Coast team (Van 2) ❖ All other non-Caito photos: Caito ❖ Blog: caito-200-ok.com/blog/streaming-fast-and-slow ❖ “Community” photo models: Zack Hobson, Cory Johannsen Image Credits Caito_200_OK