Kubernetes on AWS: a developer's perspective

It’s a sad fact that it’s taking more and more computing power to learn new things in programming these days!

I love my ThinkPad X1 Carbon, but it’s not powerful enough to be running workloads worthy of any envy.

Take Kubernetes, for example, or its vendor distributions like OpenShift. There are quite a few moving parts in it. Even a fairly lightweight Kubernetes cluster can require a lot of memory and CPU overhead.¹

Not to mention, Kubernetes requires a fair amount of confidence to deploy and configure it on your own.

This overhead is before I’ve even added any apps. The current trend for architecting new applications seems to be to wire up lots of individual microservices, so it’s becoming increasingly harder to use my lowly developer machine for a bit of hacking here, or a quick demo there.

And then on top of that, there’s the extra overhead of things like a Service Mesh, serverless frameworks, container log aggregators and observability tools. Now you can’t hope to run all of these on your laptop, so you have to find other ways.

This makes me quite sad. It makes the barrier-to-entry very high for new engineers, because you have to either have money for decent computing power, or whip out your credit card and spend on cloud computing.

Anyway this is a rant for another day, perhaps.

In this article I’m talking specifically about one way to get some beefy Kubernetes power on demand when you’re learning or hacking.

Why would I want to run Kubernetes?

First up, I should clarify something. I’m not talking about setting up a working production-like cluster. I’m talking about a cluster for a Kubernetes developer.

As a developer, I would like easy access to a good-sized Kubernetes cluster…so that…. I can learn and play, in a reasonably safe environment, and I don’t need to spend hours troubleshooting SSL certificates or IPv6 addressing or other crap!

When I’m learning or hacking, there’s nothing worse than having to do a whole load of yak shaving. Task after task, just to get going. It really bursts my balloon, and makes me wish I’d never bothered!

(reaches for a strong yet fruity drink 🍹)

I would like an easy way to play with Kubernetes. I can get a Kubernetes developer cluster by using Minishift, or Minikube. But they come with the overhead of running a virtual machine on my laptop. Because of that, I quickly hit a ceiling for the workloads that I can run on my laptop.

I am hungry…. for power.

Some of my wonderful friends run their own OpenShift labs at home (Exhibit 1, Exhibit 2, Exhibit 3) but my flat is pretty small and there’s not enough room for a small server farm just now.

So I thought, why not try running my own pet Kubernetes cluster in the cloud! It could be a good way to learn and run bigger workloads (and grant myself system admin privileges, thankyou very much…)

So today I’ve been checking out AWS’s Kubernetes service, which is called Elastic Kubernetes Service or EKS.

What is EKS?

EKS is Amazon’s Kubernetes offering. Give AWS your credit card and they will run a Kubernetes cluster for you. They will create the cluster, deploy any number of worker nodes that you like, and then you can use it to run your Kubernetes apps.

EKS is not to be confused with ECS, which is Amazon’s non-Kubernetes way of running containers in the cloud.

I’m sure ECS is great and everything, but I’d rather use EKS, so I can be learning my Kubernetes primitives, and having a great time running around the cloud-native sweet shop, and installing the latest trendy software onto my cluster.

EKS effectively offers you a Kubernetes cluster with a single click, with everything configured for you. There is no Kubernetes Dashboard built in, but you can easily install one. It integrates with AWS IAM, so you use your AWS identity to authenticate to the cluster.

There are more details on AWS’s EKS landing page, where my favourite sentence is:

“EKS is the best place to run Kubernetes for several reasons.”

I like this sentence, because it sounds like the kind of sentence I would have written at school when I was 12. Excerpt I would probably have written something like: “Computers are the best for several reasons.”

Creating an EKS cluster on AWS

I was pleased at how easily I was able to set EKS up.

I followed the getting started guide on the AWS docs. I decided to point-and-click my way through the web UI (AWS Management Console). But you can use the eksctl command-line utility if you prefer (more on that below).

Another tip if you’re a beginner and want to avoid unnecessary diversions, is to limit the amount of new tools you first need to understand to do the thing you want to do.

It’s good to know that the control plane, which is where all the Kubernetes infrastructure lives (like the Kubernetes API server, etcd data store, and so on), seems to be hidden away from me, so I can’t break it.

To create my cluster, I did these things:

Create an EKS cluster using the web UI - and you can give it a name, as if it was your own pet! I named mine cillablack of course:
Add worker nodes to the cluster. These are added in things called Node Groups. These will create the nodes that actually run your containers. You need at least 1 of these to do anything useful. The “master” nodes, or “control plane”, where the Kubernetes API server runs, is not configured here.

If you’re creating the EKS cluster in one of the supported regions, you can alternatively use something called Fargate. This is a serverless way of running your Kubernetes Pods, so you don’t even see the worker nodes as EC2 instances either. It’s like magic.
Grab credentials using the AWS CLI, and access the cluster using kubectl. This part involves a couple of fiddly steps, as I needed to upgrade to the latest version of the AWS CLI, and install the aws-iam-authenticator tool, which provides auth to the Kubernetes cluster.

But then, it was working! And without much effort, too.

Spin up a cluster fast with eksctl

If you want to create a cluster even faster, you can use ‘eksctl’. eksctl is an “official” command-line utility for creating AWS EKS clusters. It is excellent. Grab eksctl from here.

Run eksctl create cluster and it will do all the things you need to create an EKS cluster. It creates subnets, a nodegroup, and configures optional things like CloudWatch logs. Actually using eksctl would have made this article a lot shorter, but then what’s the point in blogging? :-)

Then at the end of your day, use eksctl to delete your cluster again, using eksctl delete cluster <cluster name>. Magic.

Where are the worker nodes?

When you create a Kubernetes cluster in AWS, the main thing to note is that the control plane is basically hidden from you. Here’s my list of EC2 instances in the AWS management console:

I added 1 worker node to my EKS cluster. This can be seen here as an EC2 instance. AWS configures this for you and joins it to the EKS cluster.

As you can see, I have only one EC2 instance running. There’s no sign of the instances that are running etcd and the Kubernetes API, or anything like that. They are hidden somewhere where I can’t access them (or at least I’m not aware of how to do that yet)

Testing the cluster

So how to test a fresh Kubernetes install? By running a Pod, of course.

I used the hello-world image from Docker Hub. It just prints out a hello message.

You can run this against the new cluster. This command will create a Pod which runs the hello-world container:

              $ kubectl run hello-world --restart=Never --generator=run-pod/v1  --image=hello-world

            

Then check the logs and there should be a nice message from the container - it ran for me, and produced this output:

              $ kubectl logs pod/hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

It’s “just” Kubernetes… but that’s OK

The downside of EKS is that I’m “just” getting plain Kubernetes, of course. I don’t get any of the nice features that I’m used to with OpenShift, like container image builds, an image registry and so on.

Instead, I have to build my images myself and push them to Amazon ECR or to Docker Hub.

It all takes a bit of getting used to, especially if you’ve mostly just been used to OpenShift.

Open thy wallet…. let’s talk about the cost

If you want to start learning Kubernetes then you could do worse than spin up your own cluster on AWS. It doesn’t cost too much, as long as you delete clusters when you’re not using them.

So you want to know the exact costs. Well, at the time of writing (Jul 2020), I’m paying about $0.10 per hour for my EKS cluster, plus the cost of my nodes. These are charged at Amazon’s usual EC2 instance prices, plus storage fees.

I started with 1 node. So if I forget to spin a cluster down, it will cost $2.40 per day for the EKS cluster plus the cost of 1 x t3.small EC2 instance for 24 hours. This is a drop in the ocean for big companies, but for individual developers it can add up quickly. So definitely delete the cluster when you’re not using it.

These would be my rough estimated costs to run EKS for 1 month with one t3.small node:

1 x node (EC2 t3.small) = $0.022 per hour = $15.84
1 x EKS cluster = $0.10 per hour = $72
20GB EBS storage = $3.30
Total: $91.14

A lot of money for a very small cluster! I wouldn’t leave it running all the time, it’s just not worth it. So I’ll create it and shut it down when needed. My next step will probably be to create some automation to do that. Automation, yaaaassssss!!!

But would I recommend EKS for hacking around on Kubernetes? Yes! And it’s even easier if you use eksctl to create and delete your cluster.

Right, now I’m off to shut down my cluster before my credit card self-destructs….