Docker machine is a great tool to easily start a Docker host on most public Cloud providers out there. Very handy as a replacement to Vagrant if all you want is a Docker host in the Cloud.

It automatically installs the Docker daemon and sets up the TLS authentication so that you can communicate with it using your local Docker client. It also has some early features to start a Swarm cluster (i.e Multiple Docker hosts).

Since I have been playing with Kubernetes lately, and that there is a single node install available, all based on Docker images...

I thought, let's hack Docker machine a bit so that in addition to installing the docker daemon it also pulls a few images and starts a few containers on boot.

The result is kmachine. The usage is exactly the same as Docker machine. It goes something like this on exoscale (did not take time to open 6443 on all providers...PR :)):

$ kmachine create -d exoscale foobar
$ kmachine env foobar
kubectl config set-cluster kmachine --server=https://185.19.29.23:6443 --insecure-skip-tls-verify=true
kubectl config set-credentials kuser --token=abcdefghijkl
kubectl config set-context kmachine --user=kuser --cluster=kmachine
kubectl config use-context kmachine
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://185.19.29.23:2376"
export DOCKER_CERT_PATH="/Users/sebastiengoasguen/.docker/machine/machines/foobar5"
export DOCKER_MACHINE_NAME="foobar5"
# Run this command to configure your shell: 
# eval "$(kmachine_darwin-amd64 env foobar5)"

You see that I used kubectl, the Kubernetes client to automatically setup the endpoint created by machine. The only gotcha right now is that I hard coded the token...Easily fixed by a friendly PR. We could also setup proper certificates and TLS authentication. But I opted for the easy route for now. If you setup your env. You will have access to Kubernetes, and Docker of course, the original docker-machine functionality is not broken.

$ eval "$(kmachine env foobar)"
$ kubectl get pods
POD                    IP        CONTAINER(S)         IMAGE(S)                                     
kubernetes-127.0.0.1             controller-manager   gcr.io/google_containers/hyperkube:v0.17.0 
                                 apiserver            gcr.io/google_containers/hyperkube:v0.17.0                                             
                                 scheduler            gcr.io/google_containers/hyperkube:v0.17.0                                             
$ kubectl get nodes
NAME        LABELS        STATUS
127.0.0.1   Schedulable   <none>    Ready

Since all Kubernetes components are started as containers, you will see all of them running from the start. etcd, the kubelet, the controller, proxy etc.

$ docker ps
CONTAINER ID        IMAGE                                        COMMAND               
7e5d356d31d7        gcr.io/google_containers/hyperkube:v0.17.0   "/hyperkube controll     
9cc05adf2b27        gcr.io/google_containers/hyperkube:v0.17.0   "/hyperkube schedule               
7a0e490a44e1        gcr.io/google_containers/hyperkube:v0.17.0   "/hyperkube apiserve               
6d2d743172c6        gcr.io/google_containers/pause:0.8.0         "/pause"                            
7950a0d14608        gcr.io/google_containers/hyperkube:v0.17.0   "/hyperkube proxy --                                                                                               
55fc22c508a9        gcr.io/google_containers/hyperkube:v0.17.0   "/hyperkube kubelet                                                                                                
c67496a47bf3        kubernetes/etcd:2.0.5.1                      "/usr/local/bin/etcd                                                                                                       

Have fun ! I think it is very handy to get started with Kubernetes and still have the Docker machine setup working. You get the benefit of both, easy provisioning of a Docker host in the Cloud and a fully working Kubernetes setup to experience with. If we could couple it with Weave of Flannel, we could setup a full Kubernetes cluster in the Cloud, just like Swarm.

Couple weeks ago Google finally published a technical paper describing Borg, their cluster management system that they built over the last ten years or more and that runs all Google services.

There are several interesting concepts in the paper, one of them of course being that they run everything in containers. Whether they use Docker or not is unknown. Some parts of their workloads probably still use LMCTFY, - Let Me Contain That For You-. What struck me is that they say to not be using full virtualization. It makes sense in terms of timeline, considering that Borg started before the advent of hardware virtualization. However, their Google Compute Engine offers VM as a Service, so it is fair to wonder how they are running their VMs. This reminded me of John Wilkes talk at MesosCon 2014. He discussed scheduling in Borg (without mentioning it) and at 23 minutes in his talk, mentions that they run VMs in containers.

Running VM in containers does make sense when you think in terms of a cluster management system that deals with multiple type of workloads. You treat your IaaS (e.g GCE) as a workload, and contain it so that you can pack all your servers and maximize utilization. It also allows you to run some workloads on bare-metal for performance.

Therefore let's assume that GCE is just another workload for Google and that it runs through Borg.

Since Borg laid out the principles for Kubernetes, the cluster management system designed for containerized workloads and open sourced by Google in June 2014. You are left asking:

"How can we run VMs in Kubernetes ?"

This is where Rancher comes to our help to help us prototype a little some-some. Two weeks ago, Rancher announced RancherVM, basically a startup script that creates KVM VMs inside Docker containers (not really doing it justice calling it a script...). It is available on GitHub and super easy to try. I will spare you the details and tell you to go to GitHub instead. The result is that you can build a Docker image that contains a KVM qcow image, and that running the container starts the VM with the proper networking.

Privilege gotcha

With a Docker image now handy to run a KVM instance in it, using Kubernetes to start this container is straightforward. Create a Pod that launches this container. The only caveat is that the Docker host(s) that you use and that form your Kubernetes cluster need to have KVM installed and that your containers will need to have some level of privileges to access the KVM devices. While this can be tweaked with Docker run parameters like --device and --cap-add, you can brute force it in a very unsecure manner with --privilege. However Kubernetes does not accept to run privileged containers by default (rightfully so). Therefore you need to start you Kubernetes cluster (i.e API server and Kubelet with the --allow_privilege=true option).

If you are new to Kubernetes, check out my previous post where I show you how to start a one node Kubernetes "cluster" with Docker compose. The only modification that I did from that post, is that I am running this on a Docker host that also has KVM installed, that the compose manifest specifies --allow_pivileged=true in the kubelet startup command, and that I modify the /etc/kubernetes/manifests/master.json by specifiying a volume. This allows me not to tamper with the images from Google.

Let's try it out

Build your RancherVM images:

$ git clone https://github.com/rancherio/vm.git
$ cd vm
$ make all

You will now have several RancherVM images:

$ sudo docker images
REPOSITORY                           TAG                 ...
rancher/vm-android                   4.4                 ...
rancher/vm-android                   latest              ...
rancher/ranchervm                    0.0.1               ...
rancher/ranchervm                    latest              ...
rancher/vm-centos                    7.1                 ...
rancher/vm-centos                    latest              ...
rancher/vm-ubuntu                    14.04               ...
rancher/vm-ubuntu                    latest              ...
rancher/vm-rancheros                 0.3.0               ...
rancher/vm-rancheros                 latest              ...
rancher/vm-base                      0.0.1               ...
rancher/vm-base                      latest              ...

Starting one of those will give you access to a KVM instance running in the container.

I will skip the startup of the Kubernetes components. Check my previous post. Once you have Kubernetes running you can list the pods (i.e group of containers/volumes). You will see that the Kubernetes master itself is running as a Pod.

$ ./kubectl get pods
POD         IP        CONTAINER(S)         IMAGE(S)                                     ...
nginx-127             controller-manager   gcr.io/google_containers/hyperkube:v0.14.1   ...
                      apiserver            gcr.io/google_containers/hyperkube:v0.14.1                                             
                      scheduler            gcr.io/google_containers/hyperkube:v0.14.1

Now let's define a RancherVM as a Kubernetes Pod. We do this in a YAML file

apiVersion: v1beta2
kind: Pod
id: ranchervm
labels:
  name: vm
desiredState:
  manifest:
    version: v1beta2
    containers:
      - name: master
        image: rancher/vm-rancheros
        privileged: true
        volumeMounts:
          - name: ranchervm
            mountPath: /ranchervm
        env:
         - name: RANCHER_VM
           value: "true"
    volumes:
      - name: ranchervm
        source:
          hostDir: 
            path: /tmp/ranchervm

To create the Pod use the kubectl CLI:

$ ./kubectl create -f vm.yaml 
pods/ranchervm
$ ./kubectl get pods
POD         IP            CONTAINER(S)         IMAGE(S)                                     ....
nginx-127                 controller-manager   gcr.io/google_containers/hyperkube:v0.14.1   ....
                          apiserver            gcr.io/google_containers/hyperkube:v0.14.1                                             
                          scheduler            gcr.io/google_containers/hyperkube:v0.14.1                                             
ranchervm   172.17.0.10   master               rancher/vm-rancheros                         ....

The RancherVM image specified contains RancherOS. The container will start automatically but of course the actual VM will take couple more seconds to start. Once it's up, you can ping it and you can ssh to the VM instance.

$ ping -c 1 172.17.0.10
PING 172.17.0.10 (172.17.0.10) 56(84) bytes of data.
64 bytes from 172.17.0.10: icmp_seq=1 ttl=64 time=0.725 ms

$ ssh rancher@172.17.0.10 
...
[rancher@ranchervm ~]$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[rancher@ranchervm ~]$ sudo system-docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
229a22962a4d        console:latest      "/usr/sbin/entry.sh    2 minutes ago       Up 2 minutes                            console             
cfd06aa73192        userdocker:latest   "/usr/sbin/entry.sh    2 minutes ago       Up 2 minutes                            userdocker          
448e03b18f93        udev:latest         "/usr/sbin/entry.sh    2 minutes ago       Up 2 minutes                            udev                
ff929cddeda9        syslog:latest       "/usr/sbin/entry.sh    2 minutes ago       Up 2 minutes                            syslog              

Amazing ! I can feel that you are just wondering what the heck is going on:)

You want to kill the VM ? Just kill the pod:

$ ./kubectl delete pod ranchervm

Remember that a Pod is not a single container but could contain several ones as well as volumes.

Let's go a step further, and scale the number of VMs by using a replication controller.

Using a Replication Controller to scale the VM

Kubernetes is quite nice, it builds on years of experience with fault-tolerance at Google and provides mechanism for keeping your services up, scaling them and rolling new versions. The replication Controller is a primitive to manage the scale of your services.

So say you would like to automatically increase or decrease the number of VMs running in your datacenter. Start them with a replication controller. This is defined in a YAML manifest like so:

id: ranchervm
kind: ReplicationController
apiVersion: v1beta2
desiredState:
  replicas: 1
  replicaSelector:
    name: ranchervm
  podTemplate:
    desiredState:
      manifest:
        version: v1beta2
        id: vm 
        containers:
          - name: vm
            image: rancher/vm-rancheros
            privileged: true
            volumeMounts:
              - name: ranchervm
                mountPath: /ranchervm
            env:
              - name: RANCHER_VM
                value: "true"
        volumes:
          - name: ranchervm
            source:
              hostDir:
                path: /tmp/ranchervm
    labels:
      name: ranchervm
  

This manifest defines a Pod template (the one that we created earlier), and set a number of replicas. Here we start with one. To launch it, use the kubectl binary again:

$ ./kubectl create -f vmrc.yaml 
replicationControllers/ranchervm
$ ./kubectl get rc
CONTROLLER   CONTAINER(S)   IMAGE(S)               SELECTOR         REPLICAS
ranchervm    vm             rancher/vm-rancheros   name=ranchervm   1

If you list the pods, you will see that your container is running and hence your VM will start shortly.

$ ./kubectl get pods
POD               IP            CONTAINER(S)         IMAGE(S)                                     ...
nginx-127                       controller-manager   gcr.io/google_containers/hyperkube:v0.14.1   ...
                                apiserver            gcr.io/google_containers/hyperkube:v0.14.1                                                    
                                scheduler            gcr.io/google_containers/hyperkube:v0.14.1                                                    
ranchervm-16ncs   172.17.0.11   vm                   rancher/vm-rancheros                         ...

Why is this awesome ? Because you can scale easily:

$ ./kubectl resize --replicas=2 rc ranchervm
resized

And Boom, two VMs:

$ ./kubectl get pods -l name=ranchervm
POD               IP            CONTAINER(S)   IMAGE(S)               ...
ranchervm-16ncs   172.17.0.11   vm             rancher/vm-rancheros   ...
ranchervm-279fu   172.17.0.12   vm             rancher/vm-rancheros   ...

Now of course, this little test is done on one node. But if you had a real Kubernetes cluster, it would schedule these pods on available nodes. From a networking standpoint, RancherVM can provide DHCP service or not. That means that you could let Kubernetes assign the IP to the Pod and the VMs would be networked over the overlay in place.

Now imagine that we had security groups via an OVS switch on all nodes in the cluster...we could have multi-tenancy with network isolation and full VM isolation. While being able to run workloads in "traditional" containers. This has some significant impact on the current IaaS space, and even Mesos itself.

Your Cloud as a containerized distributed workload, anyone ???

For more recipes like these, checkout the Docker cookbook.

CloudStack comes with a simulator. It is very handy for testing purposes, we use it to run our smoke tests on TravisCI for each commit to the code base. However if you want to run the simulator, you need to compile from source using some special maven profiles. That requires you to check out the code and setup your working environment with the dependencies for a successfull CloudStack build.

With Docker you can skip all of that and simply download the cloudstack/simulator image from the Docker Hub. Start a container from that image and expose port 8080 where the dashboard is being served. Once the container is running, you can use docker exec to configure a simulated data center. This will allow you to start fake virtual machines, create security groups and so on. You can do all of this through the dashboard or using the CloudStack API.

So you want to give CloudStack a try ? Use Docker :)

$ docker pull cloudstack/simulator

The image is a bit big and we need to work on slimming it down but once the image is pulled, starting the container will be almost instant. If you feel like sending a little PR just the Dockerfile, there might be a few obvious things to slim down the image.

$ docker run -d -p 8080:8080 --name cloudstak cloudstack/simulator

The application needs a few minutes to start however, something that I have not had time to check. Probably we need to give more memory to the container. Once you can access the dashboard at http://localhost:8080/client you can configure the simulated data-center. You can choose between a basic network which gives you L3 network isolation or advanced zone which gives you a VLAN base isolation:

$ docker exec -ti cloudstack python /root/tools/marvin/marvin/deployDataCenter.py -i /root/setup/dev/basic.cfg

Once the configuration completes, head over to the dashboard http://localhost:8080/client and check your simulated infrastructure

Enjoy the CloudStack simulator brought to you by Docker.

Docker is a lot of fun, one of its strength is in the portability of applications. This gave me the idea to package the CloudStack management server as a docker image.

CloudStack has a simulator that can fake a data center infrastructure. It can be used to test some of the basic functionalities. We use it to run our integration tests, like the smoke tests on TravisCI. The simulator allows us to configure an advanced or basic networking zone with fake hypervisors.

So I bootstrapped the CloudStack management server, configured the Mysql database with an advanced zone and created a docker image with Packer. The resulting image is on DockerHub, and I realized after the fact that four other great minds already did something similar :)

On a machine running docker:

docker pull runseb/cloudstack
docker run -t -i -p 8080:8080 runseb/cloudstack:0.1.4 /bin/bash
# service mysql restart
# cd /opt/cloudstack
# mvn -pl client jetty:run -Dsimulator

Then open your browser on http://<IP_of_docker_host>:8080/client and enjoy !

I was at CloudExpo Europe in London last week for the Open Cloud Forum to give a tutorial on CloudStack tools. A decent crowd showed up, all carrying phones. Kind of problematic for a tutorial where I wanted the audience to install python packages and actually work :) Luckily I made it self-paced so you can follow at home. Giles from Shapeblue was there too and he was part of a panel on Open Cloud. He was told once again "But Apache CloudStack is a Citrix project !" This in itself is a paradox and as @jzb told me on twitter yesterday "Citrix donated CloudStack to Apache, the end". Apache projects do not have any company affiliation.

I don't blame folks, with all the vendors seemingly supporting OpenStack, it does seem that CloudStack is a one supporter project. The commit stats are also pretty clear with 39% of commits coming from Citrix. This number is also probably higher since those stats are reporting gmail and apache as domain contributing 20 and 15% respectively, let's say 60% is from Citrix. But nonetheless, this is ignoring and mis-understanding what Apache is and looking at the glass half empty.

When Citrix donated CloudStack to the Apache Software Foundation (ASF) it relinquished control of the software and the brand. This actually put Citrix in a bind, not being able to easily promote the CloudStack project. Indeed, CloudStack is now a trademark of the ASF and Citrix had to rename their own product CloudPlatform (powered by Apache CloudStack). Citrix cannot promote CloudStack directly, it needs to get approval to donate sponsoring and follow the ASF trademark guidelines. Every committer and especially PMC members of Apache CloudStack are now supposed to work and protect the CloudStack brand as part of the ASF and make sure that any confusion is cleared. This is what I am doing here.

Of course when the software was donated, an initial set of committers was defined, all from Citrix and mostly from the former cloud.com startup. Part of the incubating process at the ASF is to make sure that we can add committers from other organization and attract a community. "Community over Code" is the bread and butter of ASF and so this is what we have all been working on, expanding the community outside Citrix, welcoming anyone who thinks CloudStack is interesting enough to contribute a little bit of time and effort. Looking at the glass half empty is saying that CloudStack is a Citrix project "Hey look 60% of their commits is from Citrix", looking at it half full like I do is saying "Oh wow, in a year since graduation, they have diversified the committer based, 40% are not from Citrix". Is 40% enough ? of course not, I wish it were the other way around, I wish Citrix were only a minority in the development of CloudStack.

Couple other numbers: Out of the 26 members of the project management committee (PMC) only seven are from Citrix and looking at mailing lists participation since the beginning of the year, 20% of the folks on the users mailing list and 25% on the developer list are from Citrix. We have diversified the community a great deal but the "hand-over", that moment when new community members are actually writing more code than the folks who started it, has not happened yet. A community is not just about writing code, but I will give it to you that it is not good for a single company to "control" 60% of the development, this is not where we/I want to be.

This whole discussion is actually against Apache's modus operandi. Since one of the biggest tenant of the foundation is non-affiliation. When I participate on the list I am Sebastien, I am not a Citrix employee. Certainly this can put some folks in conflicting situations at times, but the bottom line is that we do not and should not take into account company affiliation when working and making decisions for the project. But if you really want some company name dropping, let's do an ASF faux-pas and let's look at a few features:

The Nicira/NSX and OpenDaylight SDN integration was done by Schuberg Phillis, the OpenContrail plugin was done by Juniper, Midokura created it's own plugin for Midonet and Stratosphere as well, giving us a great SDN coverage. The LXC integration was done by Gilt, Klarna is contributing in the ecosystem with the vagrant and packer plugins, CloudOps has been doing terrific job with Chef recipes, Palo-Alto networks integration and Netscaler support, a google summer of code intern did a brand new LDAP plugin and another GSoC did the GRE support for KVM. RedHat contributed the Gluster plugin and PCExtreme contributed the Ceph interface while Basho of course contributed the S3 plugin for secondary storage as well as major design decisions on the storage refactor. The Solidfire plugin was done by, well Solidfire and Netapp has developed a plugin as well for their virtual storage console. NTT contributed the CloudFoundry interface via BOSH. On the user side, Shapeblue is leading the user support company. So no it's not just Citrix.

Are all these companies members of the CloudStack project ? No. There is no such thing as a company being a member of an ASF project. There is no company affiliation, there is no lock in, just a bunch of guys trying to make good software and build a community. And yes, I work for Citrix and my job here will be done when Citrix only contributes 49% of the commits. Citrix is paying me to make sure they loose control of the software, that a healthy ecosystem develops and that CloudStack keeps on becoming a strong and vibrant Apache project. I hope one day folks will understand what CloudStack has become, an ASF project, like HTTP, Hadoop, Mesos, Ant, Maven, Lucene, Solr and 150 other projects. Come to Denver for #apachecon you will see ! The end.

CloStack

CloStack is a Clojure client for Apache CloudStack. Clojure is a dynamic programming language for the Java Virtual Machine (JVM). It is compiled directly in JVM bytecode but offers a dynamic and interactive nature of an interpreted language like Python. Clojure is a dialect of LISP and as such is mostly a functional programming language.

You can try Clojure in your browser and get familiar with its read eval loop (REPL). To get started, you can follow the tutorial for non-LISP programmers through this web based REPL.

To give you a taste for it, here is how you would 2 and 2:

user=> (+ 2 2)
4

And how you would define a function:

user=> (defn f [x y]
  #_=> (+ x y))
#'user/f
user=> (f 2 3)
5

This should give you a taste of functional programming :)

Install Leinigen

leiningen is a tool for managing Clojure projects easily. With lein you can create the skeleton of clojure project as well as start a read eval loop (REPL) to test your code.

Installing the latest version of leiningen is easy, get the script and set it in your path. Make it executable and your are done.

The first time your run lein repl it will boostrap itself:

$ lein repl
Downloading Leiningen to /Users/sebgoa/.lein/self-installs/leiningen-2.3.4-standalone.jar now...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.0M  100 13.0M    0     0  1574k      0  0:00:08  0:00:08 --:--:-- 2266k
nREPL server started on port 58633 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> exit
Bye for now!

Download CloStack

To install CloStack, clone the github repository and start lein repl:

git clone https://github.com/pyr/clostack.git
$ lein repl
Retrieving codox/codox/0.6.4/codox-0.6.4.pom from clojars
Retrieving codox/codox.leiningen/0.6.4/codox.leiningen-0.6.4.pom from clojars
Retrieving leinjacker/leinjacker/0.4.1/leinjacker-0.4.1.pom from clojars
Retrieving org/clojure/core.contracts/0.0.1/core.contracts-0.0.1.pom from central
Retrieving org/clojure/core.unify/0.5.3/core.unify-0.5.3.pom from central
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from central
Retrieving org/clojure/core.contracts/0.0.1/core.contracts-0.0.1.jar from central
Retrieving org/clojure/core.unify/0.5.3/core.unify-0.5.3.jar from central
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.jar from central
Retrieving codox/codox/0.6.4/codox-0.6.4.jar from clojars
Retrieving codox/codox.leiningen/0.6.4/codox.leiningen-0.6.4.jar from clojars
Retrieving leinjacker/leinjacker/0.4.1/leinjacker-0.4.1.jar from clojars
Retrieving org/clojure/clojure/1.3.0/clojure-1.3.0.pom from central
Retrieving org/clojure/data.json/0.2.2/data.json-0.2.2.pom from central
Retrieving http/async/client/http.async.client/0.5.2/http.async.client-0.5.2.pom from clojars
Retrieving com/ning/async-http-client/1.7.10/async-http-client-1.7.10.pom from central
Retrieving io/netty/netty/3.4.4.Final/netty-3.4.4.Final.pom from central
Retrieving org/clojure/data.json/0.2.2/data.json-0.2.2.jar from central
Retrieving com/ning/async-http-client/1.7.10/async-http-client-1.7.10.jar from central
Retrieving io/netty/netty/3.4.4.Final/netty-3.4.4.Final.jar from central
Retrieving http/async/client/http.async.client/0.5.2/http.async.client-0.5.2.jar from clojars
nREPL server started on port 58655 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> exit

The first time that you start the REPL lein will download all the dependencies of clostack.

Prepare environment variables and make your first clostack call

Export a few environmen variables to define the cloud you will be using, namely:

export CLOUDSTACK_ENDPOINT=http://localhost:8080/client/api
export CLOUDSTACK_API_KEY=HGWEFHWERH8978yg98ysdfghsdfgsagf
export CLOUDSTACK_API_SECRET=fhdsfhdf869guh3guwghseruig

Then relaunch the REPL

$lein repl
nREPL server started on port 59890 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (use 'clostack.client)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
nil

You can safely discard the warning message which only indicates that 'clostack' is meant to be used as a library in a clojure project.
Define a client to your CloudStack endpoint

user=> (def cs (http-client))
#'user/cs

And call an API like so:

user=> (list-zones cs)
{:listzonesresponse {:count 1, :zone [{:id "1128bd56-b4d9-4ac6-a7b9-c715b187ce11", :name "CH-GV2", :networktype "Basic", :securitygroupsenabled true, :allocationstate "Enabled", :zonetoken "ccb0a60c-79c8-3230-ab8b-8bdbe8c45bb7", :dhcpprovider "VirtualRouter", :localstorageenabled true}]}}

To explore the API calls that you can make, the REPL features tab completion. Enter list or de and press the tab key you should see:

user=> (list
list                                list*                               list-accounts                       list-async-jobs                     
list-capabilities                   list-disk-offerings                 list-event-types                    list-events                         
list-firewall-rules                 list-hypervisors                    list-instance-groups                list-ip-forwarding-rules            
list-iso-permissions                list-isos                           list-lb-stickiness-policies         list-load-balancer-rule-instances   
list-load-balancer-rules            list-network-ac-ls                  list-network-offerings              list-networks                       
list-os-categories                  list-os-types                       list-port-forwarding-rules          list-private-gateways               
list-project-accounts               list-project-invitations            list-projects                       list-public-ip-addresses            
list-remote-access-vpns             list-resource-limits                list-security-groups                list-service-offerings              
list-snapshot-policies              list-snapshots                      list-ssh-key-pairs                  list-static-routes                  
list-tags                           list-template-permissions           list-templates                      list-virtual-machines               
list-volumes                        list-vp-cs                          list-vpc-offerings                  list-vpn-connections                
list-vpn-customer-gateways          list-vpn-gateways                   list-vpn-users                      list-zones                          
list?

user=> (de
dec                           dec'                          decimal?                      declare                       def                           
default-data-readers          definline                     definterface                  defmacro                      defmethod                     
defmulti                      defn                          defn-                         defonce                       defprotocol                   
defrecord                     defreq                        defstruct                     deftype                       delay                         
delay?                        delete-account-from-project   delete-firewall-rule          delete-instance-group         delete-ip-forwarding-rule     
delete-iso                    delete-lb-stickiness-policy   delete-load-balancer-rule     delete-network                delete-network-acl            
delete-port-forwarding-rule   delete-project                delete-project-invitation     delete-remote-access-vpn      delete-security-group         
delete-snapshot               delete-snapshot-policies      delete-ssh-key-pair           delete-static-route           delete-tags                   
delete-template               delete-volume                 delete-vpc                    delete-vpn-connection         delete-vpn-customer-gateway   
delete-vpn-gateway            deliver                       denominator                   deploy-virtual-machine        deref                         
derive                        descendants                   destroy-virtual-machine       destructure                   detach-iso                    
detach-volume

To pass arguments to a call follow the syntax:

user=> (list-templates cs :templatefilter "executable")

Start a virtual machine

To deploy a virtual machine you need to get the serviceofferingid or instance type, the templateid also known as the image id and the zoneid, the call is then very similar to CloudMonkey and returns a jobid

user=> (deploy-virtual-machine cs :serviceofferingid "71004023-bb72-4a97-b1e9-bc66dfce9470" :templateid "1d961c82-7c8c-4b84-b61b-601876dab8d0" :zoneid "1128bd56-b4d9-4ac6-a7b9-c715b187ce11")
{:deployvirtualmachineresponse {:id "d0a887d2-e20b-4b25-98b3-c2995e4e428a", :jobid "21d20b5c-ea6e-4881-b0b2-0c2f9f1fb6be"}}

You can pass additional parameters to the deploy-virtual-machine call, such as the keypair and the securitygroupname:

user=> (deploy-virtual-machine cs :serviceofferingid "71004023-bb72-4a97-b1e9-bc66dfce9470" :templateid "1d961c82-7c8c-4b84-b61b-601876dab8d0" :zoneid "1128bd56-b4d9-4ac6-a7b9-c715b187ce11" :keypair "exoscale")
{:deployvirtualmachineresponse {:id "b5fdc41f-e151-43e7-a036-4d87b8536408", :jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4"}}

To query the asynchronous job, you can use the query-async-job api call:

user=> (query-async-job-result cs :jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4")
{:queryasyncjobresultresponse {:jobid "418026fc-1009-4e7a-9721-7c9ad47b49e4", :jobprocstatus 0, :jobinstancetype "VirtualMachine", :accountid "b8c0baab-18a1-44c0-ab67-e24049212925", :jobinstanceid "b5fdc41f-e151-43e7-a036-4d87b8536408", :created "2013-12-16T12:25:21+0100", :jobstatus 0, :jobresultcode 0, :cmd "com.cloud.api.commands.DeployVMCmd", :userid "968f6b4e-b382-4802-afea-dd731d4cf9b9"}}

And finally to destroy the virtual machine you would pass the id of the VM to the destroy-virtual-machine call like so:

user=> (destroy-virtual-machine cs :id "d0a887d2-e20b-4b25-98b3-c2995e4e428a")
{:destroyvirtualmachineresponse {:jobid "8fc8a8cf-9b54-435c-945d-e3ea2f183935"}}

With these simple basics you can keep on exploring clostack and the CloudStack API.

Use CloStack within your own clojure project

Hello World in clojure

To write your own clojure project that makes user of clostack, use leiningen to create a project skeleton

lein new toto

Lein will automatically create a src/toto/core.clj file, edit it to replace the function foo with -main. This dummy function returns Hellow World !. Let's try to execute it. First we will need to define the main namespace in the project.clj file. Edit it like so:

defproject toto "0.1.0-SNAPSHOT" :description "FIXME: write description" :url "http://example.com/FIXME" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :main toto.core :dependencies [[org.clojure/clojure "1.5.1"]])

Note the :main toto.core

You can now execute the code with lein run john . Indeed if you check the -main function in src/toto/core.clj you will see that it takes an argument. Surprisingly you should see the following output:

$ lein run john
john Hello, World!

Let's now add the CloStack dependency and modify the main function to return the zone of the CloudStack cloud.

Adding the Clostack dependency

Edit the project.clj to add a dependency on clostack and a few logging packages:

:dependencies [[org.clojure/clojure "1.5.1"]
               [clostack "0.1.3"]
               [org.clojure/tools.logging "0.2.6"]
               [org.slf4j/slf4j-log4j12   "1.6.4"]
               [log4j/apache-log4j-extras "1.0"]
               [log4j/log4j               "1.2.16"
                :exclusions [javax.mail/mail
                             javax.jms/jms
                             com.sun.jdkmk/jmxtools
                             com.sun.jmx/jmxri]]])
                             

lein should have created a resources directory. In it, create a log4j.properties file like so:

$ more log4j.properties 
# Root logger option
log4j.rootLogger=INFO, stdout

# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

A discussion on logging is beyond the scope of this recipes, we merely add it in the configuration for a complete example.

Now you can edit the code in src/toto/core.clj with some basic calls.

(ns testclostack.core
  (:require [clostack.client :refer [http-client list-zones]]))

(defn foo
  "I don't do a whole lot."
  [x]
  (println x "Hello, World!"))

(def cs (http-client))

(defn -main [args]
  (println (list-zones cs))
  (println args "Hey Wassup")
  (foo args)
)

Simply run this clojure code with lein run joe in the source of your project. And that's it, you have sucessfully discovered the very basics of Clojure and used the CloudStack client clostack to write your first Clojure code. Now for something more significant, look at Pallet

There has been a lot of discussions lately within the OpenStack community on the need for an AWS API interface to OpenStack nova. I followed the discussion from far via a few tweets, but I am of the opinion that any IaaS solution does need to expose an AWS interface. AWS is the leader in Cloud and has been since 2006 -yes that's seven years- Users are accustomed to it and the AWS API is the de-factor standard.

When Eucalyptus started, it's main goal was to become an AWS clone and in 2012 signed an agreement with Amazon to offer seamless AWS support in Eucalyptus. Opennebula has almost always offered an AWS bridge and CloudStack has too, even though in total disclosure the interface was broken in the Apache CloudStack 4.1 release. Thankfully the AWS interface is now fixed in 4.1.2 release and will also be in the upcoming 4.2 release. To avoid breaking this interface we are developing a jenkins pipeline which will test it using the Eucalyptus testing suite.

Opennebula recently ran a survey to determine where to best put its efforts in API development. The results where clear with 47% of respondents asking for better AWS compatibility. There are of course developing official standards from standard organizations, most notably OCCI from OGF and CIMI from DMTF. The opennebula survey seems to indicate a stronger demand for OCCI than CIMI, but IMHO this is due to historical reasons: Opennebula early efforts in being the first OCCI implementation and Opennebula user base especially within projects like HelixNebula.

CIMI was promising and probably still is but it will most likely face an up-hill battle since RedHat announced it's scaling back on supporting Apache DeltaCloud. I recently heard about a new CIMI implementation project for Stratuslab from some of my friends at Sixsq, it is interesting and fun because written in Clojure and I hope to see it used with Clostack to provide a CIMI interface to CloudStack. We may be couple weeks out :)

While AWS is the de-facto standard, I want to make sure that CloudStack offers choices for its users. If someone wants to use OCCI and CIMI or AWS or the native CloudStack API they should be able to. I will be at the CloudPlugfest Interoperability week in Madrid Sept 18-20 and I hope to demonstrate a brand new OCCI interface to CloudStack using rOCCI and CloudStack ruby gem. A CloudStack contributor from Taiwan has been working on it.

The main issue with all these "standard" interfaces is that they will never give you the complete API of a given IaaS implementation. They by nature provide the lowest common denominator. That roughly means that the user facing APIs could be standardized but the administrator API will always remain hidden and non-standard. In CloudStack for instance, there are over 300 API calls. While we can expose a compatible interface, this will always only cover a subset of the overall API. It also brings the question of all the other AWS services: EMR, Beanstalk, CloudFormation... Standardizing on those will be extremely difficult if not impossible.

So yes, we should expose an AWS compatible interface, but we should also have OCCI, CIMI and of course our native API. Making those bridges is not hard, what's hard is the implementation behind it.

All of this would leave us with Google Compute Engine (GCE), and I should be able to bring back some good news by the end of september, stay tuned !!!

In my previous post I showed how to deploy hadoop clusters on-demand using Apache Whirr and CloudStack. Whirr can do much more than hadoop: Cassandra, Ganglia, Solr, Zookeeper etc...and of course Elasticsearch.

This post is really a little wink at a CloudStack PMC member and also a start of investigating if ES would be a better choice than Mongodb for all my email and logs analysis.

Let's get to it. In the Whirr source find the elasticsearch.properties file under /recipes modify it to your liking and to your cloud:

For this test I am using exoscale again, but any CloudStack cloud like ikoula, pcextreme or leaseweb will do (some European chauvinism right there :) ). In a basic zone, specify your ssh keypairs that whirr will use to boostrap ES.

whirr.cloudstack-keypair=exoscale

Set the number of machines in the cluster

# Change the number of machines in the cluster here
whirr.instance-templates=2 elasticsearch

Set the instance type of each cluster instanc (e.g large, extra-large etc..). I have not tried to pass them by name I just use the uuid from CloudStack.

whirr.hardware-id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8

And of course define the endpoint

whirr.provider=cloudstack
whirr.endpoint=https://api.exoscale.ch/compute

And define the template that you want to use (e.g Ubuntu, CentOS etc)

whirr.image-id=1d16c78d-268f-47d0-be0c-b80d31e765d2

Finally, define the ES tarball you want to use. Whirr has not updated this in a long time, so the default is still set at 0.15. Remember to change it.

# You can specify the version by setting the tarball url
whirr.elasticsearch.tarball.url=http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.2.tar.gz

Then launch the cluster

whirr launch-cluster --config elasticsearch.properties

When whirr is done launching and bootstrapping the instance you will get something like:

You can log into instances using the following ssh commands:
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.90
[elasticsearch]: ssh -i /Users/toto/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o 
StrictHostKeyChecking=no toto@185.19.28.92
To destroy cluster, run 'whirr destroy-cluster' with the same options used to launch it.

Don't bother with those IPs, the cluster is already dead when you read this and the security group are set so that I am the only one who can access them. That said you now have a working elasticsearch cluster in the cloud and you can hit it with the API:

$ curl -XGET 'http://185.19.28.90:9200/_cluster/nodes?pretty=true'
{
  "ok" : true,
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "njCzrXYaTnKxtqKsMrV-lA" : {
      "name" : "Powderkeg",
      "transport_address" : "inet[/185.19.28.90:9300]",
      "hostname" : "elasticsearch-f76",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.90:9200]"
    },
    "bdqnkSNfTb63eGMM7CUjNA" : {
      "name" : "USAgent",
      "transport_address" : "inet[/185.19.28.92:9300]",
      "hostname" : "elasticsearch-a28",
      "version" : "0.90.2",
      "http_address" : "inet[/185.19.28.92:9200]"
    }
  }

Really cool, have fun !!!

A vote has started on the libcloud dev list for the 0.13 release. The release notes detail all the new features and fixes. I am stoked about this because a few patches that I submitted are included in this release candidate. I patched the CloudStack driver quite a bit to improve the support for Clouds with a basic zone like Exoscale. There is more work to do on this driver including better support for Advanced zone especially for port forwarding, firewall rules and more unit tests. A the CloudStack hackathon last sunday @pst418 submitted a few patches for unit tests and they made it as well into 0.13 RC, terrific.

If you don't know libcloud, it's a python based API wrapper to abstract the various cloud APIs. With libcloud you can create connections to multiple clouds, potentially using different API. At a high level, it is similar to jclouds for JAVA or deltacloud written in ruby. There was already a CloudStack driver but its functionality was limited. If you grab my quickie libcloud shell, you can try to follow this walkthrough of what you can do with libcloud and a CloudStack basic zone. Of course you will need a CloudStack endpoint.

Start the libshell and check what zone you are on:

$ python ./libshell.py 
Hello LibCloud Shell !!
You are running at: https://api.exoscale.ch/compute
>>> conn.list_locations()
[<NodeLocation: id=1128bd56-b4d9-4ac6-a7b9-c715b187ce11, name=CH-GV2, country=AU, driver=CloudStack>]

You might notice a wrong country code, it is hard-coded in libcloud, I need to file a bug for this. Get the list of templates (or images in libcloud speak):

>>> conn.list_images()
[<NodeImage: id=01df77c3-0150-412a-a580-413a50924a18, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=89ee852c-a5f5-4ab9-a311-89f39d133e88, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=ccd142ec-83e3-4108-b1c5-8e1fdb353ff9, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=f2101a0c-eaf7-4760-a143-0a5c940fd864, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
 <NodeImage: id=77d32782-6866-43d4-9524-6fe346594d09, name=CentOS 5.5(64-bit) no GUI (KVM), driver=CloudStack  ...>,
 <NodeImage: id=29cba09f-4569-4bb3-95e7-71f833876e3e, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=ee241b47-4303-40c8-af58-42ed6bf09f8c, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=754ea486-d649-49e5-a70e-5e5d458f0df0, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=0f9f4f49-afc2-4139-b26b-b05a9f51ea74, name=Windows Server 2012, driver=CloudStack  ...>,
 <NodeImage: id=954752a8-0486-46bb-8e3f-0adb3e01c619, name=Linux CentOS 6.4 64-bit, driver=CloudStack  ...<]

I cut the previous output, but there are also Ubuntu and CentOS images on that cloud...You can then list the various instance types or sizes in libcloud speak.

>>> conn.list_sizes()
[<NodeSize: id=71004023-bb72-4a97-b1e9-bc66dfce9470, name=Micro, ram=512 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8, name=Tiny, ram=1024 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=21624abb-764e-4def-81d7-9fc54b5957fb, name=Small, ram=2048 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=b6e9d1e8-89fc-4db3-aaa4-9b4c5b1d0844, name=Medium, ram=4096 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=c6f99499-7f59-4138-9427-a09db13af2bc, name=Large, ram=8182 disk=0 bandwidth=0 price=0 driver=CloudStack ...>, 
<NodeSize: id=350dc5ea-fe6d-42ba-b6c0-efb8b75617ad, name=Extra-large, ram=16384 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
 <NodeSize: id=a216b0d1-370f-4e21-a0eb-3dfc6302b564, name=Huge, ram=32184 disk=0 bandwidth=0 price=0 driver=CloudStack ...>]

What I added is the management of ssh key pairs and security groups, you can now list, create and delete both keypairs and security groups, and use those when deploying nodes. (Don't bother trying to do anything with the keys below, I deleted everything.)

>>> conn.ex_list_keypairs() 
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'Default 
Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'0.0.0.0/0', u'protocol':
 u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': 
u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test',
 u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

>>> conn.ex_create_keypair('test')
{u'privateKey': u'-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQCAuPufJnbzFkyIUaMymkXwfilA5qS2J9fx2Q3EWNDs1m89FLFq\n19ERjG43ZvRuI/KGwWZwHbhqoqJth7WQYNe
lYCmOUYRaepxTrpU4TGDGuhqMh9D9\
noNMIFx3ktkcTitxkSY/5h/pXqSB2XXURLpZWwZxjEYwWCpWE8i7uVtIR7wIDAQAB\nAoGAOv0Kik9lOVbhsaK/yAO8w039d7l6h+NQaYtPbMhKzg4iofomp9DJBWK2a3sp\
nzoN4s9pTKFPmXC+1gb4sLULD72ENSgyozrPMCJ0tytNa3ebVCCYDvlagEZ83KwGn\nnr2BIRLul1xyHVa+amvTemuxi7OOx0u/0aZ6jPVW9ocPahkCQQDnUACG3Q2p60Mx\
nCttW7Go2wz0BhaI8ibG8rHorhdrFJUrsTI6QrLh340uHCIEAuWUXjYDX2u5MTPBG\njWbL/A/9AkEAjnX8EzO/B/RooFTxhYdxv7D1Dzcx4H59mFzjez6N/mjAHjKL4p66\
nqGFgLa9HMhDPFOs83VetBXcihu1vueffWwJBALQUD2TvAS0wz82FYz8nrITXuE3Q\nCH7Sv8FgEXiCq89hehO+ghrVrIMBPBJzJ2M18iLE8fKaKXzTRRfYC5hwss0CQHY8\
nBdIKCGoZtxwaY7lnCEkIHNtb+9FOKf7iWQpYiJC1b32ghei3xEMrTh+ccYJj4PqD\noigyNC9tCQLi3O92OjECQGYR2z8IDlfnl7G8p7eiyBciuoDyyq5/oJHhkHbkbwzr\
nW0Uun+rEcjRnXbUN8wUQ6FSFrxk2VSajbCBteTOrF24=\n-----END RSA PRIVATE KEY-----\n', u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}

>>> conn.ex_list_keypairs()
[{u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}, {u'name': u'foobar',
 u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_delete_keypair('test')
u'true'

>>> conn.ex_list_keypairs()
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_create_security_group('heyhey')
{u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id':
 u'77ad73b5-a383-4e13-94be-a38ef9877996', u'domain': u'runseb@gmail.com', u'name': u'heyhey'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'
0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description':
 u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}, 
{u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id': u'77ad73b5-a383-4e13-94be-a38ef9877996',
 u'domain': u'foobar@gmail.com', u'name': u'heyhey'}]

>>> conn.ex_delete_security_group('heyhey')
u'true'

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': 
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
 u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
 u'description': u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

With a security group created you can now add an ingress rule:

>>> conn.ex_authorize_security_group_ingress(securitygroupname='toto',protocol='TCP',cidrlist='0.0.0.0./0',
startport=99)
{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'this is a test', u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 99, u'cidr': u'0.0.0.0./0', 
u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', 
u'description': u'Default Security Group', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr':
 u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id':
 u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': u'default'}, {u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': 
u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': 
[{u'startport': 99, u'cidr': u'0.0.0.0./0', u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}], 
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

Great, keypair and security group work fine. Now let's do the basics of starting an instance and let's test the suspend and resume which was not in the Driver until now. Nothing earth shattering but it's an improvement.

>>> size=conn.list_sizes()
>>> image=conn.list_images()

>>> n=conn.create_node(name='yoyo',size=size[0],image=image[0])

>>> n.ex_stop()
u'Stopped'
>>> n.ex_start()
u'Running'

And that's it for now, CloudStack support is getting better, there is still tons of work to do: improve the advanced zone support, check the load-balancer and storage support, add unit-tests and new CloudStack features like auto-scaling. Carry on !!!!

On June 13th we had a Build a Cloud Day workshop in Schuberg Phillis offices in Amsterdam. We had a good crowd with a little over 30 people gathering to learn about CloudStack, its use cases, monitoring, PaaS, SDN and client tools. We had good fun and it was a very productive meeting.

We started off with a talk from Arjan Eriks from Schuberg Phillis, Arjan gave us a terrific overview of the Schuberg's culture and how the Cloud and DevOps processes naturally appeared within Schuberg. Their team oriented approach to customer support evolved to using CloudStack as their core IT solution to provide an agile infrastructure that meets the need of their customers. Adding to the Cloud infrastructure, a DevOps culture permeated Schuberg even before DevOps really existed. His slides below will give you a good idea of how a company moves to the Cloud and uses agile methods to make customers and employees "happy".

Following Arjan we had the chance to hear from Julien Pivoto (@roidelapluie) of Inuits an IT consulting company that helps its customers adopt open source solutions and embrace a DevOps culture. Julien presented the suite of open source monitoring tools that can be used to monitor a Cloud infrastructure at scale. From logstash, elasticsearch and Kibana, to collectd, graphite and riemann. Julien did a great job introducing all these tools and explaining how they stitch together to help automation of a Cloud infrastructure.

Hugo Trippaers (@Spark404) then gave us a very practical talk about SDN (Software Defined Networking). Hugo has developed the Nicira NVP integraton in Apache CloudStack and manages the Cloud at Schuberg Phillis. He reviewed all the SDN support in CloudStack (e.g Nicira, native OVS manager, Big Switch, Midonet) and then dove into a Nicira use case. This really brought home the fact that SDN is not a fiction but a reality.

After these first talks about a company's view of Cloud and DevOps, a review of key monitoring tools and SDN solutions in CloudStack, Giles Sirett from Shapeblue presented us with several CloudStack use cases. Shapeblue is a CloudStack integrator in the UK with offices in India, US and Brazil. Giles insider's look on how company choose a Cloud solution and how CloudStack empowers them is unique. This was a very energizing talk showing use that CloudStack is gaining momentum and being adopted all over the world.

CloudStack is a IaaS solution, but once you have built your IaaS you need to focus on application delivery. Arash Kaffamanesh from Cloudssky talked to us about PaaS and showed a few demos on Appscale and Stackato integration with CloudStack. PaaS integration is set to become a hot topic within the next year and Arash lightning talk certainly set the stage. I know of folks working on better CloudStack support in Cloudfoundry/BOSH as well, so expect to see PaaS/CloudStack integration soon.

After these relatively strategic talk we went back to developer tools with Sander Botman. Sander is currently taking the lead in maintaing the Knife CloudStack plugin. Knife is the command line utility from Opscode, it is a great tool to manage your Chef repository. Written in ruby it is highly customizable and has numerous plugins. Sander demoed the knife-cloudstack plugin which let's you provision instances on your CloudStack based cloud and automatically boostrap these instances using chef recipes. I have used knife-cs recently and even submitted a few patches. It is a terrific tool, easy to use with lots of options. If you are looking for a tool to easily provision, configure and manage instances in your cloud, this is it.

Finally, I wrapped up the talks with demos about several CloudStack clients and tools. I have been testing lots of tools lately to prepare content for CloudStack University. I focused on Apache tools form the ASF such as libcloud and jclouds which just entered the incubator. I also looked at higher level wrappers like Apache whirr to create hadoop-clusters on demand. The slides below are still a work in progress and I will put up screencasts up really soon, Stay tuned.

To end the day, we had a few beers, before heading out to the DevOps days pre-conference party. A few folks sneaked out before we took the picture but we all left energized, having learned a ton. We all look forward to the CloudStack conference November 20-22nd in Amsterdam. Should be a blast.

Over the last couple weeks I have been using exoscale, a swiss public cloud provider based on CloudStack. They just launched after a beta testing phase that I had the chance to be part of. Their offering is primarily aimed at developers. The folks at Exoscale were kind enough to give me couple "gift cards" during the CloudStack Geneva meetup and I was able to get going. Together with PCExtreme, Leaseweb, and iKoula they are one more European public cloud provider in production with Apache CloudStack that I know of.

Their cloud is almost straightforward: two data centers in Geneva and Vernier, with hardware hosted by Equinix. They run Apache CloudStack 4.0.2, the latest release and use KVM hypervisors on Ubuntu based servers. One customization that they made and that I am aware of is that they patched CloudStack to output logs using logstash and use Kibana for visuzalization. They offer CentOS 6.4 and Ubuntu 12.04/13.04 64 bit templates with instance types from 512MB with 1 core to 32 GB with 8 cores. Their development and operations team is relatively small for such an offering but they are backed by the Veltigroup a leading IT provider in Switzerland, which gives them a 20 person team for support. Their developers are seasoned IT infrastructure enthusiasts who participate in the DevOps, openBSD, Clojure and Pallet community. The lead developer, Pierre-Yves Ritschard, formerly with paper.li, recently participated in DevOps Days Paris and has contributed a Clojure client to CloudStack: clostack. They are embracing open source, not only by using it, but also by contributing to the various communities that make up the foundation of Cloud services.

While CloudStack comes with a powerful and efficient Web UI, exoscale decided to create their own UI and integrate it with a ticketing, monitoring and billing system that they developed. It reinforces the fact the CloudStack API is extremely rich and that the default UI was actually designed as a proof of concept rather than something that all users should use. The UI will please developers by its simplicity and straigthforward ease of use. From talking to them, I know they they will soon open source the python client they developed to build the UI backend. Pierre Yves told me it resembled a little bit my toy UI that uses Flask and builds a REST wrapper on top of the CloudStack API. See a snapshot of an instance view below:

They envision an entirely public cloud, meaning that all their instances are on the public network (nb: They also have a premium service for the enterprise with secure VPN access). They use a CloudStack basic zone with security groups ala EC2. Users can get VNC access and of course ssh access to the instance using ssh keys created via the console. Since it is still in beta-testing they have currently turned off some of the features like snapshotting. These feature limitations are well documented and in their roadmap for future releases of their Cloud services. In the meantime they are also working on high level services like Jenkins continuous infrastructure services and most probably big data services. The Big Data service is just a guess on my part but would make sense considering Pierre-Yes involvement with Pallet and a recent demo he did at Eurocloud.

I have used exoscale a lot in those last two weeks and I am extremely pleased with it. It is actually becoming my favorite testbed. The UI is great as I mentioned, but all the CloudStack clients also work (of course :) ), CloudMonkey is easily setup to talk to their endpoint, I submitted patches to libcloud last week and you can now use with exoscale. I even contributed a CloudStack driver for SaltStack (an alternative to Chef and Puppet) based on testing with exoscale. Right away I felt comfortable with exoscale and even gave a lecture at HEPIA where I put 7 students on it in 20 minutes. If you want to learn more, Exoscale will be at Build a Cloud Day Paris on June 19th.

So here is the catchphrase of the blog: Exoscale, it's simple and it works

The summer is going to be full with CloudStack events and talks in Europe.

• Bucharest JUG next week May 30th:
  http://www.bjug.ro/editii/12.html
  
• Open Cloud Day Zurich, June 11th:
  https://www.ch-open.ch/events/aktuelle-events/open-cloud-day-2013/
  
• Build a Cloud Day, Amsterdam June 13th:
  http://bacdamsterdam.eventbrite.com
  
• Build a Cloud Day, Paris, June 19th:
  http://bacdparis.eventbrite.com
  
• London CloudStack meetup, July 4th:
  http://www.eventbrite.com/event/6727182183#
  
• Dublin CloudStack meetup, July 10th
  In the works, stay tuned
  
• RMLL, Brussells, July 6th-11th
  Agenda to be released soon, a CloudStack talk and a tutorial has been accepted.
  http://2013.rmll.info/en/
  

Choose your event, let's meet, brainstorm and build a great community

I am closing in on my first year with Citrix working in the community team of Apache CloudStack, it's about time to describe how I look at Citrix and why I decided to leave a +10 year career in academia behind to work on Open Source...at Citrix. Disclaimer: if you don't want to read about Citrix products, stop right there.

Prior to joining Citrix I only knew it as the company that "brings windows to unix" delivering applications through an equivalent of remote desktop. It's a very unsavory and unfair summary at best and the XenDesktop folks will hate me for it :). What I quickly discovered is that Citrix has embraced the Cloud trying to deliver applications "anywhere, to anyone on any device", it is more than talk as I realized by going through the product list. This was especially intriguing to me because I did not understand why Citrix was getting into the Cloud Infrastructure as a Service (IaaS) layer area by acquiring cloud.com. IaaS software is more the purview of data centers vendors and Citrix struck me as a "top of the stack" company.

To understand why I think Citrix is a true Cloud company you need to get back to the Cloud definition that most of us refer to these days. The NIST definition, it defines Cloud as "...a model for enabling ubiquitous on-demand network access to a shared pool of network resources...that can rapidly be provisioned and released with minimal management effort or service provider interaction " it is the coming of age of utility computing where anyone can access any computing service at anytime from anywhere. Virtualization has been a key enabler of Cloud in the data center and the IaaS layer (or Fabric layer) has become the most mature of the Cloud layers. With Citrix acquiring Cloud.com and donating CloudStack to the Apache Software Foundation (ASF) it told me that Citrix was catching up and complementing its portfolio of products. What I did not expect is to realize that Citrix already had the other pieces of the Cloud puzzle building a suite of products that would enable on-demand/scalable network access to applications autonomically, in short: an app-focused cloud middleware.

At the fabric layer, resource sharing deals with computer, network and storage services. Server virtualization is a key component of IaaS and with XenServer Citrix has the leading hypervisor of the Cloud. To share compute resources, and manage farms of XenServer an orchestrator is needed. That's where CloudPlatform (commercial offering of Apache CloudStack) fits in. CloudPlatform orchestrates compute services and leverages existing storage and networking solutions. But here comes NetScaler, the networking piece, load-balancer, application firewall, branch repeater and cloud connector (via GRE tunnels, IPsec and so on). In storage, Sharefile is there. Unfortunately it is not an enterprise data center storage solution like today's large scale object store or traditional NAS solution but more of a DropBox equivalent (again not doing it justice) which includes corporate data governance.

At the Platform layer or PaaS, Citrix deals more with brokering and management of applications than a PaaS like Microsoft Azure or Google App Engine. CloudPortal is set to become a true Cloud broker offering a marketplace of applications, usage metering and billing making use of enterprise IaaS solutions to provision the applications in the datacenter. XenDesktop, XenApp and VDI-in-a-box are some of the solutions that CloudPlatform can broker. This will bring the traditional Citrix business to the Cloud (i.e Avalon project), providing on-demand, scalable, metered IT services.

At the SaaS layer, Citrix provides true collaboration services with it's GoTo* suite and Podio a social/collaborative platform. Surely these will soon be bundled in CloudPlatform for on-demand access. The nature of these apps speak to Citrix commitment to Clouds and a new way of working "anywhere, with anyone on any device" as they empower remote workers. And despite Yahoo's recent announcement about remote working I am very thankful for these apps and for Citrix walking the talk. To keep enabling mobility Citrix also offers additional middleware which I place in this SaaS layer even though they are not end-user application. XenMobileMDM and CloudGateway solve the problem of access management to application and data from the many devices that are now part of an enterprise IT.

Missing Pieces: This short mapping of Citrix products to the three basic layers of the Cloud highlights couple missing pieces. I am listing them here but I have no visibility and Citrix acquisition strategy so don't read anything else into it :). While Citrix has now a strong offering at the IaaS layer with XenServer, CloudPlatform (based on Apache CloudStack) and NetScaler. It does lack a true Cloud data center storage solution that would open the door to the Big Data market. Sharefile is more tailored for the individual end-user within the enterprise. Maybe packaging Apache Hadoop could fill this gap. In any case Citrix has strategic partnerships with NetApp and EMC as well as an innovative partner InkTank which is behind Ceph. At the PaaS layer, CloudPlatform is really a broker between app consumers and the cloud backends. A PaaS for application developers would be an interesting addition, even though these markets are still in the "trigger". A partnership with EMC on CloudFoundry and RedHat on OpenShift focused on integration with CloudStack/CloudPlatform would be interesting. I might also complement the middleware systems with a Data Analytics solution to mine one's data center and optimize application delivery, but that's my research hat thinking. The SaaS is a huge market and the goal is not to cover it all. I don't see any glaring gaps there, what strikes me the most at this layer is that the type of device we use will evolve quickly (e.g Google Glass), if one day people show up to work with "Glasses" what will Citrix Receiver be on Glass ?

Open Source: With this strong vision of Clouds, Citrix seemed perfect for me. What really made me accept the challenge of a new job was that Citrix open sourced CloudPlatform, donating it to the Apache Software Foundation. I had been an Open Source consumer (mostly) for a long time and it seemed right to get on the producer side. This was a true departure from its Microsoft ties, first by acquiring a data center software but second by reaching out to the Linux community. After the Xen Project this move spoke volume to me and was at the same time intriguing. Citrix has now stepped its Open Source efforts even more, putting the Xen Project under the stewardship of the Linux Foundation and being part of the OpenDayLight Software Defined Networking (SDN) consortium. There again IaaS is leading the way. I don't know yet what's in store for the other layers of the Cloud at Citrix, but if the recent Open Source activities are any signs, we can expect to see some Open Sourcing (in various forms: full software, API, community edition etc) at the other layers. This will further improve Citrix's reputation in the OSS community and help create an ecosystem around the terrific portfolio that we already have. Open Sourcing is key to innovation and we want Citrix to be a leading innovator of the Cloud.

In short, there is more to Clouds than virtual machine management and Citrix tackles the end-to-end cloud spectrum reaching out to the dream of computing as a utility in a mobile world.

Search for cloud computing and you will get approximately 190 million results, search for cloud computing security and you will get 120 million results. This is very rough data of course but it gives us an idea that when talking about Cloud, security is a big concern. Go to a conference and talk about Cloud, and you can be certain that one of the big questions you will get asked is "But what about Security ?"

Disclaimer and bias: This question always leaves me pondering, mostly because my personal background and bias always makes me wonder what people are afraid off in the Cloud and what do they see that Cloud brings to bear that is different from any existing distributed systems running over the internet. I am not an enterprise security expert, I used to teach an introductory course on network security, but I have spent my fair share thinking about Clouds especially at the IaaS layer. There, the new technology that could represent a new attack vector is virtualization and I only read about two non-traditional efforts that really challenged the security of virtualization: the controversial bluepill project in 2006 and the cross-VM side channel attack reported by a research group at MIT in 2009 (there are of course more...). Most problems publicly described with IaaS have been with spam and DDOS. Where on one hand cloud providers are being used to send spam and on the other hand cloud providers are victim of DDOS threatening the availability of services.

However, in the fall I had the chance to participate in the DELL in the Clouds Think Tank in London. It is there that I started to understand that what most people where worried about with the Cloud had more to do with legal issues, governance, compliance and contracts than hardcore attacks. Indeed when dealing with a cloud provider you are exposing your data to new risks for the simple fact that it is not under your total control and you need to manage those risks. Moving your data out of your secured premises and putting them in the hands of another party exposes you to new threats. This is the core of information assurance and risk management. Cloud security is therefore more about updating your security guidelines, making sure that you are compliant with the law and being confident that you can respond appropriately to any attack or business continuity issues. Cloud security is less about the fear of a new technology that exposes new attack vectors. The risks may be new to your enterprise but the attacks and vulnerabilities used are not new to the internet.

To learn more and come up with a plan I now point people to the Cloud Security Alliance (CSA) and their guidelines. It is a 176 pages document which coupled with the ENISA cloud security assessment (125 pages :)) forms the basis of the CSA Certificate of Cloud Security Knowledge (CCSK). I have finished reading the CSA guidelines and once I read the ENISA report I will take the CCSK exam.

The CSA guidelines are a set of reports covering fourteen domains of interest to Cloud security. From Governance and Legal Issues to Incident Response and Virtualization (to name a few). One sentence truly resonated with me due to my personal bias explained earlier. It is in the Application Security domain chapter which states: "Cloud-based software applications require a design rigor similar to an application connecting to the raw internet - the security must be provided by the application without any assumptions being made about the external environment" indeed doing the opposite would be one of the fallacies of distributed systems design enunciated by Peter Deutsch from SUN. There lies in my view the biggest risk, thinking that you can take an application that has been designed in-house assuming a secure local network and wanting to move it to the cloud as-is not managing the risks due to the fact that a) the network is not secure b) bandwidth is not infinite c) latency is not zero d) transport has a cost.

Any service, application, provider, data that is accessible over the public internet is being attacked and is subject to risk of being stolen, tempered with, disrupted and even shutdown. This is not fear mongering, it is just a fact and if you design an application or use a service thinking otherwise you will be exposing yourself and not managing risks properly. Similarly to the new cloud software being developed (e.g Hadoop, Cassandra, CloudStack), that are designed assuming failures of components, when moving to the Cloud one needs to assume attacks and unsecured networks. This is not saying that the Cloud is unsecure, this is saying that you need to adopt the proper security posture, a different one than if you have been operating under the -at least perceived- warmth and coziness of a secured local network.

Getting back to the CSA guidelines, the first domain Cloud Architecture is the perfect introduction to Cloud with reference to the NIST definition of cloud computing. It then follows with what I think is the most important section: Governing in the Cloud, it presents risk management as key to an enterprise governance and introduces legal issues and compliance management as it pertains to Cloud. The chapters help to define the proper security posture, defining or updating security policies that will make sense for Cloud use, understanding the assets that will be at risk and understanding if and how compliance will be enforced. As such it is not specific to Cloud computing, it is really best practices of risk management and understanding the contracts being signed with the cloud providers. Will those contracts expose you ? Do providers follow data protection standards ? Are the providers subject to any laws that may expose you (e.g Patriot Act) ? How can you remedy those risks ? Which providers can give you the compliance you need ? To help with these decisions, CSA created the Security Trust and Assurance Registry (STAR). Cloud providers who participate in the registry submit answers to a questionnaire that lists the standard they follow in 99 categories from audit and compliance to operations, business continuity, human resources, forensics. This registry is key to choose cloud providers that will match your security and governance needs.

The last section of the CSA guidelines is about Operating in the Cloud. From Disaster recovery, data center operations, incident response to encryption, authentication and virtualization. This section is not specific to cloud but comes into play in selecting providers to ensure for example, that the provider data center operations matches your requirements. Or to ensure that in case of incidents you will have access to the logs (defining which ones in a contract). I was pleased to see John Kinsella (@johnlkinsella) from Stratosec as one of the authors of the chapter on Application Security. John is a member of the CloudStack Project Management Committer (PMC). I was also happily surprised to see a chapter on Security as a Service something that Mice Xia from tcloudcomputing and a committer on CloudStack has been working on.

To summarize, knowledge is power (a bit cheezy I know). When moving to the Cloud, an enterprise should engage their security experts from the on-set making sure that risk is managed and that everything is in compliance, this is standard information assurance and part of good enterprise governance. When negotiating (or not negotiating) contracts with Cloud Providers the STAR registry can help choose the providers that will best match the requirements of the enterprise. I am not being paid by CSA or SANS, but I would recommend people to get the CCSK certification and probably also a SANS course on Cloud Security. In my opinion the cloud is not less secure than anything else. The Cloud (at least in its public form) is about accessing resources over the network and locating assets off-premises, this intrinsically presents risks but it is manageable risk that needs to be part of a design. You need to design for attacks, test your designs, monitor, counter-strike. Basic warfare. And remember, the network is secure...Oupss...sorry !