K8S — Kind -Docker registry for local development environments
One of the most advantages of docker and kubernetes is the velocity to manage applications, the portability and scalability. We can build and running everywhere without any issue with dependencies.
There is one part which can be a little bit “annoying” and it is the time to build the container and have it running into a kubernetes cluster. We can use CI/CD for that task, write some code, push to the repository, the CI/CD server builds the container, tagged it and push it to the regiestry, once there, we have to deploy it to kubernetes.
In order to have a quick way to do that, just form our local development environment, I create a repository with the pourporse to show you how to setup a project, write code in local and use a local cluster with our local registry to have quick feedback loop.
To setup that environment, we need the following tools:
With all the tools installed in your computer, we will create a project structure which will container a simple Golang API, it will be running in a Docker container and we will use Kind as a Kubernetes cluster.
The project will have the following folder structure:
├── cmd
│ ├── bootstrap
│ └── main.go
├── Dockerfile
├── go.mod
├── go.sum
├── infra
└── deployment.yml
└── deployment_blue_green.yml
├── Makefile
├── README.md
└── scripts
└── setup_cluster.sh
└── deploy_blue_gren.shFirst of all, we will start with our main.go file:
The main.go will import the package bootstrap and initializate it using the Run function
package main
import (
"log"
"github.com/jrmanes/kind_golang/cmd/bootstrap"
)
func main() {
if err := bootstrap.Run(); err != nil {
log.Fatal(err)
}
}Let’s go to the bootrap.go file, you will find it in the cmd/bootstrap folder.
The following package, will create our web server and expose a path to the world, running an http server in port 8080(by default).
package bootstrap
import (
"fmt"
"log"
"net/http"
"os"
"os/signal"
"time"
"github.com/gorilla/mux"
)
// Server define an http standard library
type Server struct {
server *http.Server
}
func Root(w http.ResponseWriter, r *http.Request) {
response := "Welcome to Kind Golang"
fmt.Fprintf(w, response)
}
func Run() error {
// check the env var in order to set the port
port := os.Getenv("SERVER_PORT")
if port == "" {
port = "8080"
}
r := mux.NewRouter()
r.HandleFunc("/", Root)
http.Handle("/", r)
// set configuration to our server
serv := &http.Server{
Addr: ":" + port,
Handler: r,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
}
server := Server{server: serv}
// start the server.
go Start(server)
// Wait for an in interrupt panic
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
<-c
// Attempt a graceful shutdown.
Close()
return nil
}
// Start the server.
func Start(srv Server) {
log.Printf("Server running on http://localhost%s", srv.server.Addr)
log.Fatal(srv.server.ListenAndServe())
}
// Close server resources.
func Close() error {
return nil
}At this point, we have our web server ready to use, running in port 8080, we can test it directly in our computer using the command:
go run cmd/main.goIf you go to a webserver and access to http://localhost:8080 you will be able to see something like:
Welcome to Kind GolangNow is time to setup our kubernetes cluster using kind and setup our local docker registry.
For that, we will use the following script:
#!/bin/sh
set -o errexit
# create registry container unless it already exists
reg_name='kind-registry'
reg_port='5000'
running="$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)"
if [ "${running}" != 'true' ]; then
docker run \
-d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \
registry:2
fi
# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
endpoint = ["http://${reg_name}:${reg_port}"]
nodes:
- role: control-plane
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
- role: control-plane
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
- role: control-plane
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
- role: worker
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
- role: worker
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
- role: worker
image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
EOF
# connect the registry to the cluster network
# (the network may already be connected)
docker network connect "kind" "${reg_name}" || true
# Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: local-registry-hosting
namespace: kube-public
data:
localRegistryHosting.v1: |
host: "localhost:${reg_port}"
help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOFTo execute it, we will have to grant execution permissions:
sudo chmod +x <script_name>Once the script finished, you will be able to check the nodes of your kuberentes cluster using the command:
kubectl get nodesAnd the output will be some like:
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane,master 34m v1.21.1
kind-control-plane2 Ready control-plane,master 34m v1.21.1
kind-control-plane3 Ready control-plane,master 33m v1.21.1
kind-worker Ready <none> 33m v1.21.1
kind-worker2 Ready <none> 33m v1.21.1
kind-worker3 Ready <none> 33m v1.21.1At this point, let’s create out Docker container to build out application. With the following Dockerfile, you will be able to build the API server.
We are going to call our service as: kind_golang
FROM golang:1.16.3 as builder
LABEL maintainer="Jose Ramon Mañes - github.com/jrmanes/kind_golang"
ADD . /app
WORKDIR /app
RUN go fmt ./...
RUN go test -v ./... -cover -coverprofile=coverage.out
RUN go tool cover -func=coverage.out
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main ./cmd/main.go
######## Start a new stage from scratch #######
FROM alpine:latest
# Copy the Pre-built binary file from the previous stage
COPY --from=builder /app/main .
CMD ["./main"]To build, tag and push our container we will use the following commands:
docker build . -t kind_golang:latest
docker tag kind_golang:latest localhost:5000/kind_golang:latest
docker push localhost:5000/kind_golang:latestAfter execute them, if everything goes well, the output will be something like:
The push refers to repository [localhost:5000/kind_golang]
77e16dfc38a1: Pushed
8ea3b23f387b: Pushed
latest: digest: sha256:13ca545f809d1449912102734e89064fbb786250427e804a60d3f8d9d560f827 size: 739At this point we have our web server application built in a docker container, tagged and push to our local registry.
Next step is deploy our application to kubernetes, for that, we will use a manifest with a deployment and a service to expose our API.
---
apiVersion: v1
kind: Service
metadata:
name: kind-golang-service
labels:
app: kind-golang
spec:
ports:
- port: 8080
protocol: TCP
selector:
app: kind-golang
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kind-golang
name: kind-golang
spec:
replicas: 1
selector:
matchLabels:
app: kind-golang
template:
metadata:
labels:
app: kind-golang
spec:
containers:
- image: localhost:5000/kind_golang:latest
name: kind-golang
imagePullPolicy: AlwaysTo deploy that, we will execute the following command:
kubectl apply -f ./infraThe output will be something like:
default kind-golang-558cbddf8f-4pr6b 1/1 Running 0 1sAnd if we check all the resources in the default namespace, we will see:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kind-golang-service ClusterIP 10.96.109.96 <none> 8080/TCP 15sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kind-golang 1/1 1 1 15sNAME DESIRED CURRENT READY AGE
replicaset.apps/kind-golang-558cbddf8f 1 1 1 15s
To check against our localhost we can use the following command in order to forward the service that we just create to our localhost:
kubectl port-forward service/kind-golang-service 8080:8080If you go to a webserver and access to http://localhost:8080 you will be able to see something like:
Welcome to Kind GolangNow we can build our container and deploy it to our local kubernetes cluster using the local registry.
I created a tool which will help us to have a very fast and automated feedback loop in our local envs.
There is an script in the folder scripts which is called deploy_blue_green.sh
That tool checks the current version of our service and manage multi version, creating a blue/green deployment against our kubernetes cluster.
The script is:
#!/bin/bash
################
PROJECT_NAME=kind_golang
CURRENT_VERSION=$(kubectl get deployment kind-golang -o yaml | grep " image: localhost" | cut -d':' -f4)
TMP_MANIFEST="_tmp"
DEPLOYMENT_MANIFEST_EXT=".yml"
DEPLOYMENT_MANIFEST_NAME="../infra/deployment_blue_green"
DEPLOYMENT_MANIFEST=$DEPLOYMENT_MANIFEST_NAME$DEPLOYMENT_MANIFEST_EXT
DEPLOYMENT_MANIFEST_TMP=$DEPLOYMENT_MANIFEST_NAME$TMP_MANIFEST$DEPLOYMENT_MANIFEST_EXT
RUN=start
################
# will check the current version and change it to the new one
function checkCurrentVersion() {
if [[ $CURRENT_VERSION == "latest" ]];then
VERSION=v2
build_tag_push
sed "s/VERSION/$VERSION/g" $DEPLOYMENT_MANIFEST > $DEPLOYMENT_MANIFEST_TMP
else
VERSION=latest
sed "s/VERSION/$VERSION/g" $DEPLOYMENT_MANIFEST > $DEPLOYMENT_MANIFEST_TMP
build_tag_push
fi
echo "current version is: $CURRENT_VERSION"
echo "version is: $VERSION"
}
# build_tag_push will tag the current latest version to the atual version and push it to the registry
function build_tag_push(){
cd ..
docker build . -t localhost:5000/${PROJECT_NAME}:$VERSION
docker push localhost:5000/${PROJECT_NAME}:$VERSION
cd -
}
# kube_deploy will deploy using the new version
function kube_deploy(){
kubectl apply -f $DEPLOYMENT_MANIFEST_TMP
}
# clean_files all tmp file
function clean_files(){
rm $DEPLOYMENT_MANIFEST_TMP
}
# The script starts here
function start() {
checkCurrentVersion
kube_deploy
clean_files
}
# We use our var $RUN in order to call the function start
$RUN