Deploying a GO application to Google Cloud via Docker and Semaphore CI
20 Sep 2016Introduction
In this tutorial we will deploy a sample application (written in the GO programming language) to Google Cloud. We will package our application using Docker in order to make the final deployment as close as possible to the development environment.
Prerequisites
For this tutorial we will need
- A Docker installation
- An account for Semaphore CI (free registration required)
- An account for Google cloud (free registration required)
- Our favourite text editor and command line terminal to edit all related files
Note regarding docker on Windows
For this article I am using Docker on Windows. The recommended way to run Docker on Windows (or even Mac) is via the Docker toolbox. The Docker toolbox creates a “normal” VM on Windows and then runs the Docker engine on top, giving you access to containers. This means that things are a bit more complicated as Docker does NOT run on localhost (as it would be expected on a Linux installation).
You might see some slightly different screenshots on your system if you use Linux but the basic concepts are the same.
Source code for this article
You can find the source code mentioned in this article on Github. You can follow along either by creating a repository on your own and typing the source files yourself, or by simply cloning this repository to have everything ready.
Step 1 - Creating the Go application
First we will create our GO application. For the purposes of this tutorial we will start with a really trivial web
server that prints a hello message (plain text) when accessed on port 8080. Here is the complete GO file named trivial-web-server.go
:
For the rest of this tutorial we will mention [WORKDIR] as the root directory of our project. Choose any location that you see fit on your workstation for this directory.
Place the GO source file into [WORKDIR]/src/sample as per GO guidelines. You can choose any other name instead of sample
if you like.
We also create the folder [WORKDIR]/bin
that will hold the compiled binary.
Running the GO application via Docker
We need to run our GO application at least once to verify that it is working correctly. Of course we could install the GO development tools locally and run the application directly on our workstation. But this would beat one of the main objectives of Docker - having the same environment during development and during production deployment.
Thus we will instead run our application in a Docker container that will have the GO runtime installed. At the time of writing the current official Docker image for GO is golang:1.7.1. We will therefore launch it on our Docker engine and compile/run our sample Web application.
This image comes with the directories go/src
and go/bin
that already mirror our source code structure.
The docker command runs a Docker instance of image golang:1.7.1
, exposes port 8080 to the Docker host (or VM if you use Windows/Mac) and gives
us an interactive terminal inside the container. We also mount the folder that contains our source inside the running container. To verify
that the source code is available we run the ls
command.
So now we have a Linux environment with the GO runtime available and with access to the source code. We compile and run our code:
Success! Our code is now compiled and runs as expected. To verify it really runs on port 8080 we can use our browser. If you use Docker on Linux
you can visit the localhost port directly. For Windows we need the VM ip that should be available if you run docker-machine ls
on your terminal.
Here is the application running.
Now we are ready to create a Docker image of our application.
Step 2 - Packaging manually the application into a Docker image
Exit the container by pressing Ctrl-C
and then Ctrl-D
to get back to the Docker host prompt.
Because we mounted the parent directory of bin
we now have the GO binary intact left on the Docker host even though the container that compiled it is no
longer active.
The next step is to package this binary into a Docker image. We will use the same GO image we used for compiling, which will assure the correct functionality of our application when it gets deployed to production.
We create the following Dockerfile at the WORKDIR
directory. As you can see it just takes the GO binary and runs it.
To verify the docker build, we create a sample image and run it:
The application should again be available on our browser as before.
Even though we have successfully packaged our application into a Docker image, we need to automate the process instead of typing manually commands in the terminal. This will allow us to setup Continuous Integration so that each code commit results in an automatic rebuild of the Docker image.
Step 3 - Using Semaphore CI to automate the build process
Semaphore CI is a cloud CI service that can not only automate our build but also comes with Docker support built-in.
For public repositories it is completely free, so if we commit our code on a public Github (or Bitbucket) repository, we can easily setup a build process with only a free registration.
I have chosen Github for this tutorial and have connected Semaphore with my Github account as well, so that it has direct access to all my repositories.
Once we have connected Semaphore to Github we are ready to create a new build project:
Semaphore will briefly analyze the contents of the repository and will automatically suggest the Docker infrastructure after noticing our Docker file in the root directory of the project. Very cool indeed!
The first thing we need to declare is the GO version that will use for compilation. Naturally it should be as close as possible to our Docker image. Semaphore supports the latest version of GO directly from the dropdown menu making this step very easy.
Next we should enter the commands that Semaphore will run after each code commit.
Here we enter two commands, one for compiling the GO source file and one for building the Docker image. (Semaphore has a different GOPATH
than the Docker image and thus we compile the file by directly defining the output file).
Once the configuration is saved, you can run your first successful build!
Speeding the build by caching the Docker image in Semaphore CI
If you run the build more times you will notice that the golang image is downloaded again and again.
Semaphore offers a cache mechanism for Docker images so that each build can skip the download step and find the cache right away.
To enable the cache we add the special SemaphoreCI command docker-cache restore
before our build and the docker-cache snapshot
command after our build.
Now if you run your build multiple times you will notice that SemaphoreCI no longer pulls the GoLang image from docker hub but instead restores it from the build cache.
Step 4 - Pushing the Docker image to a repository
One of the most important tenets of CI is the “build once” rule. The same binary that gets built by our CI server will be sent to the test/staging/production environments. This assures the correct functionality of the system. Docker makes this very easy as we can simply keep the Docker image built by SemaphoreCI into a Docker repository after each build.
There are many options for a Docker repository. Since we work with Google cloud in this article it makes sense to use the Docker repository offered by Google called Google Container Registry. Naturally SemaphoreCI has also built-in support for pushing and pulling images to GCR.
To use the Google Container Engine you need a free Google Account. Specifically for Google Cloud you also need a credit card to enable a trial period. The credit card is used as a verification method and is not charged during or after the trial finishes.
If you don’t enter a credit card most of the functionalities described in the following sections are disabled.
Creating an external auth key for Google Cloud
Once you have billing enabled, locate the credentials section in the Google Console.
Select the “Create Service Account” button and fill-in the details. Make sure that you select the “Owner” role so that the service account has read/write access (which translates to pull/push for a Docker repository).
The browser will save a JSON file locally.
Authorizing SemaphoreCI to push images to Google Container Registry
Now that the JSON auth file is saved locally, open it and copy all contents to clipboard. Go the SemaphoreCI UI and select “add-ons” from the top right corner.
Select the docker registry option. Finally select the GCR entry and paste the contents of your JSON. Make sure that you also enter your Google account email.
Now you are ready to modify the build commands to send the final Docker image to the GCR registry.
Run the Semaphore build again and if everything goes well you should see the image pushed into the GCR repository.
We now have our GO application dockerized and available in the Google Container Registry.
Step 5 - Deploying the Docker image to Google Cloud (Manually)
We now reach the most important step. We have compiled our GO application, created a Docker image and uploaded it to the Docker Registry. We need to deploy it to Google cloud to actually use the application.
There are multiple ways to deploy the application to the Google cloud. These include:
- Using a normal VM with Docker support. This is a bare bones method regarding functionality and without any official support.
- Using a Container Image. This method is more advanced as it supports cloud-init and other features.
- Using a full featured Kubernetes cluster for easy scaling and orchestration. This is the recommended option for real world production deployments.
For brevity reasons we will use the first method. On a real system you should spend some time however to understand how Kubernetes works and use that method instead.
Creating a VM and deploying a Docker image on Google cloud can be easily performed via the Google GUI console or via the integrated gcloud command line tool. Since we want to automate our build process completely we will use the gcloud tool (first locally and then in SemaphoreCI).
Authorizing the gcloud tool for external access
We could install locally the complete Google SDK to get access to the gcloud tool, but that would beat the whole purpose of using Docker in the first place. Thankfully there is already a Docker image that contains the SDK itself.
To authorize gcloud access we will reuse the JSON auth file we saved in Step 4.
First move the JSON file file to [WORKDIR]
so that we can use it via docker. This is only temporary and will not actually commit the file anywhere.
If everything goes well you should see a list of supported VM images by Google cloud. The ones we will use
will be named container-vm-v[date]
(container-vm-v20160321-2
at the time of writing).
Deploying a Docker image on a Google Cloud VM (without using the Google Console)
Before we launch the Google VM we also need to tell it what Docker image it will launch. To set this up
create the metadata manifest called
my-google-cloud-manifest.yml
with the following content:
We also commit this file on the root of the git repository as it will also be used later by SemaphoreCI.
Finally we launch our image with the following command (from within the running Docker instance)
Once you run this command you should wait a bit until the VM is created for the first time. Once that is finished the docker image should be live on Gcloud. You can see it on Google Console:
And also on your browser:
Now we have all the pieces ready to fully automate our build with SemaphoreCI.
Step 5 - Deploying the Docker image to Google Cloud automatically via SemaphoreCI
Remember that we use the gcloud
executable outside Google Console and thus it needs to pass authorization.
We have a JSON keys that holds credentials but we don’t want to commit it to Github as it contains sensitive information.
SemaphoreCI supports these kind of files very well in the project settings as custom configuration files.
Go to your SemaphoreCI project settings and click on Configuration Files
. Paste the contents of the Google Cloud JSON
auth key. Semaphore will make the key available during our build so that we don’t have to commit to our repo.
Then go back to the your project and click the Setup Up Deployment
button. Select Generic deployment for the list (as Semaphore does not have native support for Google Cloud yet). On the next screen you will be asked about manual or automatic deployment.
I chose automatic to simplify things.
Then select the master
branch and we are ready to enter our deployment commands. Since the build server on Semaphore does not have the gcloud
tool built-in we have to install first. Then we re-deploy our Docker image.
Finally name your server with any name you want and start the build.
After it succeeds you will see your application live again (but this time it was deployed by SemaphoreCI)
This concludes the tutorial! You now have a Dockerized GO application automatically built by SemaphoreCI and deployed to Google cloud.
If we change anything in our GO application, SemaphoreCI will instantly rebuild it after we commit our changes completing the CI pipeline. For example, I changed the string returned by the web server and after some minutes after my commit, it was live on Google cloud.
After you finish the tutorial don’t forget to stop the VM in Google Console so that no extra charges happen on your credit card. (You can
also stop the instance via the gcloud
command line tool)
Conclusion
We now have a full Continuous Delivery infrastructure with SemaphoreCI. The whole process is as follows
- We commit our change to Github
- SemaphoreCI picks up the change
- SemaphoreCI builds our GO application and creates a binary
- SemaphoreCI packages the binary to a Docker image
- SemaphoreCI pushes the Docker image to the Google Container Registry
- SemaphoreCI notifies Google cloud that a restart is required
- Google cloud re-downloads the Docker image
- Google cloud stops the VM with the old Docker image
- Google cloud starts the VM with the new Docker image
Here is a graphical visualization of our complete Pipeline.
Where to Go from Here
I hope you found this simple tutorial useful. Some further pointers that you can explore are the following:
- The Golang Docker image we use is very big. We could try to slim it down by finding a smaller compatible image. This would make the push step much faster.
- Alternatively we could compile our GO binary as a static executable so that it doesn’t need the GoLang Docker image at all.
- A real application will have several other services apart from the webserver (for example a DB). This could be achieved with Docker compose.
- Instead of manually downloading the Google Cloud SDK on the SemaphoreCI build machine we could also use the Google SDK Docker image we used locally to make the build environment exactly the same as our local environment.
- As already mentioned you should not use single VM images in Google cloud for real applications. A Kubernetes cluster should be used instead.
The full source code can be found at the Github repository.
Enjoy!
Go back to contents, contact me via email or find me at Twitter if you have a comment.