Pulsar Functions Deep Dive_Sanjeev kulkarni
•
4 likes•2,667 views
Pulsar Functions provide a simple yet powerful way of interacting with Pulsar topics, transforming, enriching and analyzing data contained in the streams. And with pluggable runtime environments, one can run Pulsar functions as threads/processes managed by Pulsar, or as containers/pods managed by external schedulers like Kubernetes. This talk does into the deep weeds of the underlying concepts in its implementation. In particular we will talk about the concepts of Runtime and scheduler that manages Pulsar managed functions. We will also delve into current pitfalls and areas of improvement.
Report
Share
![© 2020 SPLUNK INC.
Pulsar Functions:- Future Work
Discover/Collect Cycle
Repeating Cycle
Don’t drop discovered tasks on
failures
BatchSource
public interface BatchSource<T> {
void open(final Map<String, Object> config,
SourceContext context);
void discover(Consumer<byte[]> taskEater);
void prepare(byte[] task);
Record<T> readNext();
}
Batch Sources](https://cdn.statically.io/img/image.slidesharecdn.com/sanjeevkulkarni-200625023630/85/Pulsar-Functions-Deep-Dive_Sanjeev-kulkarni-45-320.jpg)
Pulsar Functions Deep Dive_Sanjeev kulkarni
- 1. © 2020 SPLUNK INC. Pulsar Functions A Deep Dive | Pulsar Summit 2020 Sanjeev Kulkarni sanjeevk@splunk.com
- 2. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 3. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 4. © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Bringing Serverless concepts to the streaming world. Execute processing logic per message on input topic Function output goes to an output topic • Optional Abstract View Core Concept
- 5. © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Emphasis on simplicity Great for 90% use-cases on streams • Filtering • Routing • Enrichment Not meant to replace Spark/Flink SDK-less API import java.util.function.Function; public class ExclamationFunction implements Function<String, String> { @Override public String apply(String input) { return input + "!"; } } Simple API
- 6. © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Flexible execution environments • Pulsar managed – Thread – Process • Externally managed – Kubernetes CRUD based Rest API Function lifecycle
- 7. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 8. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Submit to any worker Json repr of FunctionConfig • tenant/namespace/name • Input/Output • configs • lot more knobs …. Function Code • jars/.py/zip/etc FunctionConfig public class FunctionConfig { private String tenant; private String namespace; private String name; private String className; private Collection<String> inputs; private String output; private ProcessingGuarantees processingGuarantees; private Map<String, Object> userConfig; private Map<String, Object> secrets; private Integer parallelism; private Resources resources; ... } Function Representation
- 9. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow AuthN/AuthZ checks FunctionConfig validation • missing parameters • Incorrect parameters • Local Configs Function Code Validation • class presence, etc Copy Code to Bookeeper FunctionMetaData message FunctionMetaData { FunctionDetails functionDetails; PackageLocationMetaData packageLocation; uint64 version; uint64 createTime; map<int32, FunctionState> instanceStates; FunctionAuthenticationSpec functionAuthSpec; } Submission Checks
- 10. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow System of record Stores all Functions • map from <FQFN, FunctionMetaData> FQFN:- Fully Qualified Function Name Backed by Pulsar Topic • Function MetaData Topic Contains a MetaData Topic Tailer Function MetaData Manager Function MetaData Manager MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …}
- 11. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Just before Function creation/update/delete Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …}
- 12. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …} foo -> {functionDetails : {...}, version: 2, …}
- 13. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Merge the updates Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …} foo -> {functionDetails : {......}, version: 2, …}
- 14. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Merge the updates Increment the version Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …} foo -> {functionDetails : {......}, version: 3, …}
- 15. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Merge the updates Increment the version Write to MetaData Topic Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …}
- 16. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Merge the updates Increment the version Write to MetaData Topic Tailer reads and verifies Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …}
- 17. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Make a copy of the current state Merge the updates Increment the version Write to MetaData Topic Tailer reads and verifies Upon no conflict, tailer updates Function MetaData Manager:- Update State Machine MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {.....}, version: 3, …}
- 18. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Multiple Workers Function MetaData Manager:- When do conflicts occur? MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 2, …} Worker 2
- 19. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Multiple Workers Concurrent updates to same function Function MetaData Manager:- When do conflicts occur? MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 2, …} Worker 2
- 20. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Multiple Workers Concurrent updates to same function First Writer Wins Function MetaData Manager:- When do conflicts occur? MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 3, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 3, …} Worker 2
- 21. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Multiple Workers Concurrent updates to same function First Writer Wins Others are rejected Function MetaData Manager:- When do conflicts occur? MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 3, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 3, …} Worker 2
- 22. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Submit to any worker Validation load scales linearly Deterministic State Machine MetaData Topic is audit log Advantages MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 3, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 3, …} Worker 2
- 23. © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow MetaData topic topic growth MetaData Topic compaction non-trivial Worker Start time All Workers know everything Pitfalls MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 3, …} Worker 1 MetaData Topic Tailer foo -> {functionDetails : {...}, version: 3, …} Worker 2
- 24. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 25. © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Abstracts out Scheduler Executed only on a Leader Invoked when • Function CRUD operations �� create/update – delete • Worker Changes – Unresponsive/dead workers – New workers – Periodic – Leadership changes IScheduler Interface public interface IScheduler { List<Assignment> schedule(<List<Instance> unassigned, List<Instance> current, Set<String> workers); } Pluggable Scheduler
- 26. © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Empty Coordination Topic Failover Subscription Active Consumer is the Leader Leader Election Leader Election Coordination Topic Worker 1Worker 2Worker 3
- 27. © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Assignment Topic Written by the Leader Compacted based on key(FQFN + Instance Id) All workers know about all assignments Function Assignments Assignment Topic Worker 1Worker 2Worker 3 {foo, 1} : worker-1, ... {foo, 1} : worker-1, ... {foo, 1} : worker-1, ... Assignment Tailer Assignment Tailer Assignment Tailer
- 28. © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Stores Assignment Compacted Key -> (FQFN + InstanceId) Assignment message Instance { FunctionMetaData functionMetaData = 1; int32 instanceId = 2; } message Assignment { Instance instance = 1; string workerId = 2; } Assignment Topic
- 29. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 30. © 2020 SPLUNK INC. Pulsar Functions:- Execution Workflow Triggered by Changes to Assignment Table Takes care of the worker’s specific assignments Function lifecycle management via Spawner Function RunTime Manager Assignment Topic Worker {foo, 1} : worker-1, ... Assignment Tailer RunTime Manager Spawner Spawner
- 31. © 2020 SPLUNK INC. Pulsar Functions:- Execution Workflow Abstracts out execution environments using Runtime Factory Manages Function lifecycle Maintains grpc connection with Function instance Spawner GRPC Channel Spawner Function
- 32. © 2020 SPLUNK INC. Pulsar Functions:- Execution Workflow Short-circuit MetaData Manager and Runtime Manager Directly use Spawner Local Runner GRPC Channel Spawner Function Local Runner
- 33. © 2020 SPLUNK INC. Pulsar Functions:- Execution Workflow Simple interface for creating execution environments Creates Runtimes Runtime Factory public interface RuntimeFactory { void initialize(WorkerConfig workerConfig); Runtime createContainer(InstanceConfig instanceConfig, String codeFile); void close(); } Runtime Factory
- 34. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 35. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Java Instance is (source, function, sink) ensemble. Source abstracts reading from input topics Sink abstracts writing to output topic Java Instance Source -> Process -> Sink Source Sink f
- 36. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Pulsar Source implements the Source interface to read from Pulsar topics Pulsar Sink implements Sink interface to write to Pulsar topic Java Instance Regular Pulsar Functions Pulsar Source Pulsar Sink f
- 37. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Java Instance What if we have non-Pulsar Source? Non Pulsar Source Pulsar Sink f
- 38. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Java Instance Pulsar IO Non Pulsar Source Pulsar Sink f
- 39. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Non Pulsar Source reads from external system Identity Function lets the data pass thru Pulsar Sink writes to Pulsar Java Instance Pulsar IO Source Non Pulsar Source SinkIdentity
- 40. © 2020 SPLUNK INC. Pulsar Functions:- Java Instance Pulsar Source reads from Pulsar topics Identity Function lets the data pass thru Non Pulsar Sink writes to an external system Java Instance Pulsar IO Sink Pulsar Source Non Pulsar Sink Identity
- 41. © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work Agenda
- 42. © 2020 SPLUNK INC. Pulsar Functions:- Future Work Each setup only supports a static Runtime(Process/Thread/Pods) Change it to be dynamically specified during submission Function RunTime Manager Changes Dynamic Runtime Selection Assignment Topic Worker {foo, 1} : worker-1, ... Assignment Tailer RunTime Manager Spawner Function-1 Spawner Function-2 Thread Process
- 43. © 2020 SPLUNK INC. Pulsar Functions:- Future Work MetaData Topic not compacted Stores all function change requests Worker needs to read from beginning upon startup Function MetaData Topic Compaction MetaData Topic Tailer MetaData Topic foo -> {functionDetails : {...}, version: 2, …}
- 44. © 2020 SPLUNK INC. Pulsar Functions:- Future Work Chaining Functions Output of one going as input of others A simple workflow API Function Mesh f1 f2 f3 f4
- 45. © 2020 SPLUNK INC. Pulsar Functions:- Future Work Discover/Collect Cycle Repeating Cycle Don’t drop discovered tasks on failures BatchSource public interface BatchSource<T> { void open(final Map<String, Object> config, SourceContext context); void discover(Consumer<byte[]> taskEater); void prepare(byte[] task); Record<T> readNext(); } Batch Sources
- 46. Thank You © 2020 SPLUNK INC.