Understanding Dependency Management in Go
Hi, everyone! As you may have noticed I’m really interested into Go and since I fell in love with this language I’d like to write about it more frequently.
If you don’t know Go yet I really think you should learn it. Go is awesome!
Go is a relatively young language and when it comes to vendoring and dependency management it still isn’t really mature yet. The Go community, however, has been adopting some practices and creating tools to supply this demand. This is what we are going to talk about today.
Understanding The Go Ecosystem
Go aims to use simplicity to achieve complex tasks and therefore make the language more fun and productive. Since the beginning of the language, the Go team chose that they would use string literals to make the import syntax arbitrary to describe what was being imported.
This is what is written in their own docs:
An explicit goal for Go from the beginning was to be able to build Go code using only the information found in the source itself, not needing to write a makefile or one of the many modern replacements for makefiles. If Go needed a configuration file to explain how to build your program, then Go would have failed.
To achieve this goal, the Go team favors conventions over configurations. These are the conventions adopted when it comes to package management:
- The import path is always derived in a known way from the source code’s URL. This is why your imports look like:
github.com/author/pkgname. By doing this we also get an automatically managed namespace, since these online services already manage unique paths for each package.
- The place where we store packages in our file system is derived in a known way from the import path. If you search for where
gostores the packages you download you will be able to relate it to the URL it was downloaded from.
- Each directory in a source tree corresponds to a single package. This makes it easier for you to find source code for certain packages and helps you organize your code in a standardized way. By tying the folder structure to the package structure we don’t need to worry about both at the same time, because file system tools become package management tools.
- The package is built using only information from the source code. This means no
configurationand frees you from having to adopt specific (and probably complicated) toolchains.
Now that we’ve said that, it is easy to understand why the
go get command works the way it does.
Understanding Go Tools’ Commands
Before we get into these commands let’s understand what is the “
$GOPATH is an environment variable that points to a directory which can be considered as a workspace directory. It will hold your source codes, compiled packages and runnable binaries.
go get command is simple and works almost like a
git clone. The argument you must pass to
go get is a simple repository URL. In this example, we will use the command:
go get https://github.com/golang/oauth2.
When running this command,
go will simply fetch the package from the URL you provided and put it into your
$GOPATH directory. If you navigate to your
$GOPATH folder you will now see that you’ve got a folder in
src/github.com/golang/oauth2 which contains the package’s source files and a compiled package in the
pkg directory (alongside its dependencies).
Whenever you run
go get you should have in mind that any downloaded packages will be placed in a directory that corresponds to the URL you used to download it.
It also has a bunch of other flags available, such as
-u which updates a package or
-insecure which allows you to download packages using insecure schemes such as HTTP. You can read more about “advanced” usage the
go get command at this link.
Also, according to
go help gopath, the
go get command also updates submodules of the packages you’re getting.
go install in a package’s source directory it will compile the latest version of that package and all of its dependencies in the
Go build is responsible for compiling the packages and their dependencies, but it does not install the results!
As you might have noticed by the way Go saves its dependencies, this approach to dependency management has a few problems.
First of all, we are not able to determine which version of a package we need unless it is hosted in a completely different repository, otherwise
go get will always fetch the latest version of a package. This means that if someone does a breaking change to their package and don’t put it in another repo you and your team will end up in trouble because you might end up fetching different versions of the same package and this will then lead to “works on my machine” problems.
Another big problem is that, due to the fact that
go get installs packages to the root of your
src directory, you will not be able to have different versions of your dependencies for each one of your projects. This means that you can’t have projects which depend on different versions of the same package, you will either have to have one version or the other at a time.
In order to mitigate these problems, since Go 1.5, the Go team introduced a
vendoring feature. This feature allows you to put your dependency’s code for a package inside its own directory so it will be able to always get the same versions for all builds.
Let’s say you have a project called
awesome-project which depends on
popular-package. In order to guarantee everyone in your team will use the same version of
popular-package you can put its source inside
$GOPATH/src/awesome-project/vendor/popular-package. This will work because
Go will then try to resolve your dependencies’ path starting at its own folder’s
vendor directory (if it has at least one
.go file) instead of
$GOPATH/src. This will also make your builds deterministic (reproducible) since they will always use the same version of
It’s also important to notice that the
go get command won’t put the downloaded packages into the
vendor folder automatically. This is a job for vendoring tools.
When using vendoring you will be able to use the same import paths as if you weren’t, because
Go will always try to find dependencies in the nearest
vendor directory. There’s no need to prepend
vendor/ to any of your import paths.
In order to be able to fully understand how vendoring works we must understand the algorithm used by Go to resolve import paths, which is the following:
- Look for the import at the local
vendordirectory (if any)
- If we can’t find this package in the local
vendordirectory we go up to the parent folder and try to find it in the
vendordirectory there (if any)
- We repeat step 2 until we reach $GOPATH/src
- We look for the imported package at $GOROOT
- If we can’t find this package at $GOROOT we look for it in our $GOPATH/src folder
Basically, this means that each package can have its own dependencies resolved to its own vendor directory. If you depend on package
x and package
y, for example, and package
x also depends on package
y but needs a different version of it, you will still be able to run your code without trouble, because
x will look for
y in its own
vendor folder while your package will look for
y in your project’s vendor folder.
Now, a practical example. Let’s say you’ve got this folder structure:
$GOPATH src/ github.com/user/package-one/ one.go myproject main.go vendor/ github.com/user/package-one/ one.go client/ client.go vendor/ github.com/user/package-one/ server/ server.go vendor/ github.com/user/package-one/ one.go
If we imported
github.com/user/package-one from inside main.go it would resolve to the version of this package in the
vendor directory at the same folder:
$GOPATH src/ github.com/user/package-one/ one.go myproject main.go <-- Importing package-one from here vendor/ github.com/user/package-one/ <-- resolves to here one.go client/ client.go vendor/ github.com/user/package-one/ server/ server.go vendor/ github.com/user/package-one/ one.go
Now if we import the same package in
client.go it will also resolve this package to the
vendor folder in its own directory:
$GOPATH src/ github.com/user/package-one/ one.go myproject main.go vendor/ github.com/user/package-one/ one.go client/ client.go <-- Importing package-one from here vendor/ github.com/user/package-one/ <-- resolves to here server/ server.go vendor/ github.com/user/package-one/ one.go
The same happens when importing this package on the
$GOPATH src/ github.com/user/package-one/ one.go myproject main.go vendor/ github.com/user/package-one/ one.go client/ client.go vendor/ github.com/user/package-one/ server/ server.go <-- Importing package-one from here vendor/ github.com/user/package-one/ <-- resolves to here one.go
Understanding Dependency Management
Now that we’ve learned all of these things about how
Go handles imports and vendoring it’s about time we finally talk about dependency management.
The tool I’m currently using to manage dependencies in my own projects is called
godep. It seems to be very popular and it works fine for me, thus I highly recommend you to use that too.
It is built around the way
vendoring works. All you’ve gotta do to start using it is use the command
godep save whenever you want to save your dependencies to your
When you run
godep will save a list of your current dependencies in
Godeps/Godeps.json and then copy their source code into the
vendor folder. It’s also important to notice that you need to have these dependencies installed on your machine in order for
godep to be able to copy them.
Now you can commit the
vendor folder and its contents to ensure everyone will have the same versions of the same packages whenever running your package.
Another interesting command is
godep restore, which will install the versions specified in your
Godeps/Godeps.json file to your
In order to update a dependency all you’ve gotta do is update it using
go get -u (as we’ve talked about earlier) and then run
godep save in order for
godep to update the
Godeps/Godeps.json file and copy the needed files to the
A Few Thoughts on the Way Go Handles Dependencies
At the end of this blog post, I would also like to add my own opinion about the way Go handles dependencies.
I think that Go’s choice of using external repositories to handle package’s namespaces was great because it makes the whole package resolution much more simple by joining the file system concepts with the namespace concepts. This also makes the whole ecosystem work independently because we’ve now got a decentralized way to fetch packages.
However, decentralized package management comes at a cost, which is not being able to control all the nodes which are part of this “network”. If someone decides they’re going to take their package out of
github, for example, then hundreds, thousands or even millions of builds can start failing all of a sudden. Name changes could also have the same effect.
Considering Go’s main goals this makes total sense and is a totally fair tradeoff. Go is all about convention instead of configuration and there is no simpler way to handle dependencies than the way it currently does.
Of course, a few improvements could be made, such as using git tags to fetch specific versions of packages and allowing users to specify which versions their packages will use. It would also be cool to be able to fetch these dependencies instead of checking them out on source control. This would allow us to avoid dirty diffs containing only changes to code in the
vendor directory and make the whole repository cleaner and more lightweight.
Get in touch!
If you have any doubts, thoughts or if you disagree with anything I’ve written, please share it with me in the comments below or reach me at @thewizardlucas on twitter. I’d love to hear what you have to say and do any corrections if I made any mistakes.
At last, but not least, make sure you take a look at this awesome talk by Wisdom Omuya on GopherCon 2016 in which he explains how Go Vendoring works and also points out some details about its inner workings.
Thanks for reading this!