Running a jenkins slave inside docker is more often than not good enough, you may want to run jenkins jobs that requires docker cli. I just release a docker image that does just that.

• Go to Jenkins > manage > node management
• Click Create a node
• Check Permanent Agent
• Set working directory to something like /data/jenkins-slave (if you specify something else, don't forget to also change the command below)
• Save
• Download slave-agent.jnlp, open it to retrieve your secret
• Change the command below with the working directory you specified, your slave-agent URL and jenkins secret

docker run --rm \
-v /data/jenkins-slave:/data/jenkins-slave \
-v /var/run/docker.sock:/var/run/docker.sock \ 
--rm fgribreau/jenkins-slave:latest java -jar /app/agent.jar \ 
-jnlpUrl "http://YOUR_OWN_JENKINS.com/computer/SLAVE_NAME/slave-agent.jnlp" \
-secret "JENKINS_SECRET"

• Run it
• Done 👍

Why is this awesome?

Now that I can use docker directly from Jenkins jobs, it means I can setup cron jobs through jenkins just with configuration. Since one of docker sweet point is the ability to run CLI tools without any more setup that downloading an image, I now can setup nightly elastic/curator cron jobs just from Jenkins interface without the initial overhead of environnement setup.

I get fail build notifications directly from Hipchat (always be proactively alerted when something goes wrong!), traceability (who started the job) and of course I can reuse my Jenkins authorization setup for cron job as well. Thus no need to setup something like Rundeck when Jenkins already handles the feature set I want!

I had to handle a 2 month old issue related to ElasticSearch. A feature I did not know from one of our large monolithic application at iAdvize was not working anymore. Instead of looking directly at thousand lines of code, the fastest way to resolve it was to gather informations related to the issue from outside the app. I had to answer this questions:

• Is the ElasticSearch server remotely accessible from the frontend servers?
• If it does not, the first thing to do is to bring back the connexion between the two.
• Is the ElasticSearch request valid? 
• If it does not, maybe:
• we are doing the request on the wrong indice/alias? 
• the query body is malformed?
• Does it yield results?
• If it does not
• but we should get some, then either the query is invalid or elasticsearch have an issue
• If it does
• but from the app point of view we don't get anything
• it's related to the app itself and we will have to look at the code

To verify the first 3 points in one shot, I started tcpdump on the ElasticSearch server using the command below:

tcpdump -A -nn -s 0 'tcp dst port 9200 and (((ip[2:2] - ((ip[0]&0xf)<<2 -="" tcp="" xf0="">>2)) != 0)' -i eth1

Note: don't forget to change the interface you want to listen on.

Looking at the result I discovered that the app was doing an elasticsearch query on a missing alias. That explains it! Once the alias created, the application feature was working again in production.

The final step was to setup jenkins (or rundeck) to run daily elastic/curator in order to refresh the alias otherwise :

docker run -it --rm bobrik/curator:3.5.1 --host "elasticsearch.domain.com" --port 80 alias --name plugin-xxx-log indices --prefix plugin-xxx-log- --prefix plugin-salesforce-log- --timestring %Y.%m --time-unit months

Now we will be proactively alerted if anything goes wrong. One less thing to worry about!

[Update] I now also use tcpflow to better display content (too bad it's not maintained anymore):

tcpdump -A -l -nn -s 0 'tcp port 9200' -i eth0 | tcpflow -c -e

First thing first: open Jenkins, go to Settings > Build Nodes > Add new node > Select Permanent Agent, then set the distant workspace directory (our HOST_JENKINS_SLAVE_HOME in the script above) and select Launch agent via Java Web Start.

Finally, login into your slave machine, put and edit the script above, run it and job done!

#!/usr/bin/env bash
JENKINS_ENDPOINT="http://my-jenkins.com"
SLAVE_SECRET="YOUR_SLAVE_SECRET_FROM_JENKINS_NODE_SETTINGS"

# where your slave working dir will be, it should be the same as the one you configured in Jenkins master settings
HOST_JENKINS_SLAVE_HOME="/data/jenkins-slave"

# download the slave.jar on the slave machine
curl -s "${JENKINS_ENDPOINT}/jnlpJars/slave.jar" > slave.jar

# run the slave inside a docker
# replace "-d" by "-it --rm" for debugging
docker run -d --name jenkins-slave -v ${PWD}:/app -v ${HOST_JENKINS_SLAVE_HOME}:${HOST_JENKINS_SLAVE_HOME} java:7 java -jar /app/slave.jar -jnlpUrl "${JENKINS_ENDPOINT}/computer/docker-slave/slave-agent.jnlp" -secret $SLAVE_SECRET