This site is a golang application deployed on a Kubernetes cluster running in GKE.

I’ve decide to go with this massive overkill for my site to learn how to manage a Kubernetes cluster and to learn proper deployment techniques and patterns. With that in mind first I decided to implement SSL protection and termination for all apps I was going to run. After some looking around I found the kube-lego project. This uses LetsEncrypt to create certificates for your site. The resulting certs are stored in Kubernetes Secrets and automatically updated on expiry (90 days).

Deploying this service was a great learning experience. I deployed it in both the nginx and gke cloud load balancer configurations. This taught me the importance of Ingress resources and the power of routing in Kubernetes. The ability to manage multiple domains from a single cluster and quickly throw up prototypes on subdomains has already served me well in a few occasions. I ended up going with the nginx implementation because it was cheaper (no need to pay for a cloud LB)!

Next on the list of must-have services was a CI/CD service. After getting this blog and the kube-lego service running I quickly got tired of the manual effort required to deploy. Not to mention that strategy is not maintainable. Container deployment platforms assume you are using some kind of consistent build system to produce your images.

So I went looking! First I tried the jenkins solution helpfully provided by Google. I was able to quickly get it working but I really dislike the UI/API Jenkins offers and it feels old! I decided to go with something shinier: Drone. In addition to being a golang project (gophers!) It has a snazzy UI and all the functionality I need. The bulletproof Github integration they offer and the declarative build file syntax won me over.

Drone Build

After CI was taken care of I wanted to improve my cluster monitoring. I decided to continue running the heapster instance that is provisioned when the cluster is started. The reason being that I really enjoy the UI that ships with the cluster. The ability to visualize and manipulate (killing pods to upgrade them to new latest) the different objects (deployments, pods, services, etc…) in the cluster and quickly see pod logs has been invaluable for debugging and groking the system.

When using kubectl proxy to access the console I found that the server frequently crashed. I’ve found this useful to keep the proxy running through panics:

# add to your .bash_profile
kubeweb () {
  until kubectl proxy; do
    echo "Server crashed"
  done
}

For the improved monitoring I wanted something that would give me pod, deployment, service, and namespace level rollups and some canned dashboards. I also wanted to try prometheus to see what all the buzz was about. So I tried deploying the GiantSwarm example. The only additional work I had to do was tying the ingress they offer into kube-lego. They even had pre-canned dashboards!

Namespace CPU Usage

While I’m quite happy with my current monitoring service I’ve decided to try another option. I work at InfluxData and we have just released a Kubernetes plugin for telegraf. That combined with the other parts of the stack and my experience with the platform made me think it was time to build out that solution. I would like to have a similar drop in experience for the Influx stack so I’m building it myself! There is an existing implementation by Deis however it is intertwined with the rest of their stack. Go check out the repo for the WIP. I’ll do a full blog post once the project is in a more usable state.

After finishing my metrics monitoring project I will try and tackle the logs. I’m planning on using the ElasticSearch stack. At that point I’ll feel comfortable ripping out Heapster.

Look to this space for more articles on these topics! If you have any questions or other inquiries drop me a line!

Jack