jbonofre's blog

As part of the Google Cloud ecosystem, Google created Dataflow SDK. Now, as a Google, Talend, Cask, data Artisans, PayPal, and Cloudera join effort, we are proposing Apache Dataflow to the Apache Incubator. I’m proud, glad and excited to be the champion on the Apache Dataflow proposal. But first, I would like to thank James Malone and Frances Perry from Google for their help, always open minded and interesting discussion. It’s really great to work with them ! Let’s take a quick tour on what will be Apache Dataflow. Architecture and Programming Model Imagine, you have a Hadoop cluster where you used MapReduce jobs. Now, you want to “migrate” these jobs to Spark: you have to refactore all your jobs which requires lot of works and cost a lot. And after that, see the effort and cost if you want to change for a new platform like Flink: you have to refactore your jobs again. Dataflow aims to provide an abstraction layer between your code and the execution runtime. The SDK allows you to use a

In the first part of the “Talend ESB Continuous Integration” serie, we saw how to test the Camel routes created by the studio, by leveraging Camel Test Kit. We saw how to have automatic testing using Jenkins. The Maven POM that we did assumes that the route has been deployed (on the local repository or on a remote repository like Apache Archiva). But, it’s not so elegant that a Studio directly publish to the Archiva repository, especially from a continuous integration perspective. In this second article, I will show how to use the Talend commandline with Maven, and do nightly builds using Jenkins. Talend CommandLine CommandLine introduction The Talend commandline is the Talend Studio without the GUI. Thanks to the commandline, you can do a lot of actions, like checkout, export route, publish route, execute route. Actually, you can do all actions except the design itself 😉 You can find commandline*.sh scripts directly in your Talend Studio installation, or you can launch the commandlin