Rendered at 14:08:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jerf 39 minutes ago [-]
This is hiding in the obfuscation of being overly specific. The problem is general. To name a package, one must be in a namespace of some sort. As the universe does not provide any sort of abstract ambient "namespace" we can all appeal to, all namespaces are human constructs. As human constructs, they may fail.
In this terminology, the article basically amounts to saying "This namespace can fail! The solution is, use another!"
But that doesn't get you anywhere, because the new namespace can fail too. In fact it is almost certain that it will fail sooner than "github.com's DNS and hosting" as a namespace.
There is no solution where you tie your software release to a namespace that can't fail because there is no namespace that can't fail. It doesn't matter if your favorite language uses DNS or has a centrally blessed repository or if it distributes a blessed list of package names with the language itself or anything else. The namespace can fail.
Therefore, the only thing you can really do is be resilient against failures, and in a lot of ways, the only practical solution to resilience is just to assume that if, in the future, someone has problems getting a package due to a namespace failure, they will not helplessly disintegrate into a puddle of tears while your code is lost forever, but that the future person will instead solve this perfectly solvable problem.
Cthulhu_ 14 minutes ago [-]
This tries to solve the hypothetical problem of a library designer moving their codebase to another domain and breaking imports. But, this rarely happens in practice, so unless you plan to move hosting in advance, it may not be necessary.
That said, in the normal run of things you can mark your last release on e.g. github as deprecated, leave a note that it's moved to xyz, and wait for the users to migrate themselves. Migrating is fairly trivial in that it's a find / replace.
thih9 18 hours ago [-]
> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
throwawayffffas 2 hours ago [-]
I don't love that either, having all of your stack point to your own domains for how it can be accessed is nice. But in my opinion, it's desirable to be able to build your whole stack from source code. Of course you download external dependencies, but for code you own you should be able to build it from source.
I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.
bayindirh 2 hours ago [-]
I also think that vendoring or at least caching your dependency tree somewhere local, so you can build everything from ground up to a known state.
It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.
I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.
Joker_vD 9 hours ago [-]
Yeah, the only problem is when your whole company decides to move to a different custom domain entirely, e.g. from contoso.com to xtools.cn (which doesn't even resolves if you're not on the corporate VPN), and half of your friendly teams, whose libraries you depend on, drag out their own migration by fiddling with their local /etc/hosts and .gitconfig and .ssh/config (yes, they also wrote bespoke scripts that do those changes to their CI machines; no, trying to run those same scripts on your CI machines breaks your own shims) instead of updating their source code.
zahlman 16 hours ago [-]
I think TFA is specifically targeted at Go because of how their package system works.
mort96 18 hours ago [-]
I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.
someonebaggy 14 hours ago [-]
Or example.com/yourname if your module is never going to be imported from another project.
mort96 6 hours ago [-]
I do something like this in all my projects now. Except that I don't use example.com, I just use the name of the project. So if I'm working on a program called Frobnicator, I'll just have `module Frobnicator;` in my go.mod and do things like `import "Frobnicator/lib/blah"` in my source files. It works really well to be honest for non-library software.
But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.
dolmen 4 hours ago [-]
If the first part of the module path is without dot, it is assumed to be stdlib.
Many tools will fail if you break this contract.
mort96 3 hours ago [-]
Oh that's horrible. I've somehow never encountered it but damn.
dolmen 4 hours ago [-]
yourname.internal/mymodule is even better. But don't expect to Go toolchain (especially "go get" or "go mod download") to work.
to clarify: I mean literally "example.com", not your own domain name
handoflixue 18 hours ago [-]
I'm confused by the article, and this comment, treating these URLs as difficult to replace.
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
mort96 18 hours ago [-]
I've gone through such a migration. Huge Go code base distributed across a lot of git repositories had to be moved to a different git host.
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
dolmen 4 hours ago [-]
Similar pain as upgrading a module to a new major version (eg: v1 -> v2) [1], but at scale, because of modules dependencies: every module in the domain are affected at once as they must change namespace, like if everyone changed its major version.
> Different projects depended on different versions of the same internal libraries, we had to...
This is one of the main motivations for using a monorepo with all third party dependencies imported all the way in, and only one version for everything.
Of course it causes a whole bunch of other problems and is kinda expensive to scale, so most places won't do it.
dlisboa 16 hours ago [-]
Isn’t this exactly why the Go team recommended vendoring deps for years and years before go modules came up? Even now it takes one simple command to vendor them.
mort96 6 hours ago [-]
Well vendoring is a technical solution of the same caliber as adding module aliases to the go.mod. By that I mean, sure, it keeps the code compiling right now, but all your source files still contain references to abandoned infrastructure. It changes when you have to do the work, but unless you want your code to reference abandoned infrastructure forever, you still need to do the work.
0cf8612b2e1e 8 hours ago [-]
Google runs a monorepo which makes that kind of organization possible. Most companies are going to have a mismatch of teams using incompatible library versions.
Pick your poison. Each approach has some seriously negative trade offs in the extremes.
sjbzbeiks 10 hours ago [-]
I think is usually a symptom of how hard it is for the organization to do things, the technical side of this is not actually that hard.
I say this because I’ve gone through a few of these, some with monorepos (the easiest, search and replace and you’re done), some not (yes the hardest but you can just vendor).
At least in the situations I’ve faced this was genuinely not horrible beyond just the organization itself being complicated about the solution.
mort96 3 hours ago [-]
How do you propose it's solved in an easier way, given the constraints that 1) you don't want references to old infrastructure in your code and 2) you want to do the move without changing how the code works? Is there some secret trick we missed which would've made this much easier?
gnaman 8 hours ago [-]
I don't know enough about git but shouldn't you be able to take all your git data (refs, blobs etc) and import it to your destination host and have this work without issues? It might be a lot of data sure but the infra is surely cheaper than multiple weeks of engineering effort
mort96 7 hours ago [-]
Moving the repositories over was no problem at all! But once all the data is on the new hosts, all the Go source files still reference the old git host, since imports in Go are URLs to a git host.
oefrha 13 hours ago [-]
> And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
That makes no sense? As long as you’re using go modules, you can just host an internal go proxy for internal modules similar to proxy.golang.org to archive the old versions, and they won’t depend on the old git host.
mbreese 17 hours ago [-]
What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
lelandbatey 18 hours ago [-]
Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
quectophoton 12 hours ago [-]
> you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from
Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].
So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.
[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.
[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.
Joker_vD 9 hours ago [-]
> Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).
mort96 18 hours ago [-]
And now all your source files contain URLs to abandoned infrastructure. Is that what you'd want long term?
someonebaggy 3 hours ago [-]
How's that any different from java code using package names starting with com.sun?
mort96 2 hours ago [-]
com.sun is an identifier, nothing more. No tooling sees com.sun and assumes, "oh that means I can make an HTTP request to the IP address which sun.com resolves to".
In Go, the package identifiers are literally URLs. Go's own tooling assumes that it can make an HTTP request to the URL and that the response will be HTML with particular tags which Go's tooling will parse and use to resolve a git repository which can be 'git clone'd. Leaving them as-is when you abandon the infrastructure they reference literally means leaving dead links in your source code.
someonebaggy 2 hours ago [-]
If it's not fetching from the URL then it's not really a URL, just an identifier that looks like a URL.
mort96 2 hours ago [-]
What do you mean? Go's tooling definitely tries to fetch from that URL.
foldr 2 hours ago [-]
That's just the default, which you can easily override if you need to. IMO there is nothing wrong with treating Go package names as arbitrary identifiers like com.sun.* If you're spending a lot of time updating imports because the underlying repo has moved from its original URL, then just don't do that. It's unnecessary.
mort96 2 hours ago [-]
And leave URLs to abandoned infrastructure all over the place? No thank you.
foldr 1 hours ago [-]
Sounds like classic developer OCD to me. Old URLs used as Go package identifiers are completely harmless.
mort96 45 minutes ago [-]
Until your tooling automatically makes a request to a now-untrusted server and trusts its response...
Any tool that automatically assumes that a URL in a Go module import is 'trusted' is just a broken tool.
gravypod 18 hours ago [-]
It's harder to replace in history.
dewey 18 hours ago [-]
> That is, if you move your git hosting to GitLab then you have to change your code!
You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.
furyofantares 16 hours ago [-]
Yeah, I think putting github.com in your repo is a mistake but it seems massively overstated. Seems trivial to fix all the code, or to just use go.mod, or first use go.mod and then fix your code.
kelnos 16 hours ago [-]
What about everyone else using your code? You are going to ask them all to update their code as well? Many of them probably won't realize, and will get stuck on an old version that never gets updated, even to fix bugs and security issues.
Cthulhu_ 11 minutes ago [-]
As library maintainer, you can mark the last old domain version as deprecated:
// Deprecated: use example.com/mod/v2 instead.
module example.com/mod
As library user, it's your responsibility to keep your stuff updated. There may also be utilities in the various automatic dependency updaters that can migrate these things.
quacker 11 hours ago [-]
I think the migration can be made pretty easy for users with deprecation notices and the `go fix` tool. Better explained in [1] (I have not tried it myself).
The idea is: the package maintainer pushes a final version of the old package that is a shim, and which has a deprecation notice. The shim imports the new package, and forwards all calls to the new package (and I suppose type aliases and variable aliases as well). The shim includes `//go:fix inline` annotations on all exported symbols, so that when users run `go fix` it rewrites their code to use the new package, by inlining their usages of the old package (which is now simply a shim that references the new package).
Not perfect. There is still a bit of a discoverability problem. Not all tooling warns on deprecated packages/functions (go toolchain doesn't, but gopls and staticcheck do). And users need to know to run `go fix`.
By this logic, those folks should be pointing to their own hosted mirrors of the project, no?
peesem 16 hours ago [-]
why not just use replacements to begin with if you don't want to bother with setting up domains?
"replace example => github.com/example/example" at first, then change to
"replace example => gitlab.com/example/example" or similar when there's a migration
(i don't use go, but i hope this is possible and i am confused if it's not)
mort96 18 hours ago [-]
And you'll just leave references to abandoned infrastructure in your code forever?
cyberpunk 17 hours ago [-]
cmd+shift+r in goland, push to a branch, open a PR, ask devops to make the old location archived and prevent further writes. After a couple months remove access to the old one and see what breaks.
You know, normal development task, small amount of story points..
mort96 17 hours ago [-]
You're now arguing that it's easy, which is a completely different argument. The comment I responded to argued that it's unnecessary ("just do the rewrite in go.mod, leave source files unchanged").
Yep that sounds pretty bad. Personally, we'd just eat not being able to build old versions, we archive previous builds, and in the case of a hotfix or whatever we'd use the go.mod rewrite trick. Not sure why you needed the commit/branch setup, did you change the history?
mort96 17 hours ago [-]
Have libfoo with a bunch of different versions. Authservice depends on libfoo 1.2.0. APIservice depends on libfoo 1.3.0.
Now you want to move from github.com to git.example.org. You change libfoo, authservice and apiservice to use git.example.org, that part is just simple tedious work. But the v1.2.0 and v1.3.0 tags of libfoo are old commits from before the move, so they still reference github.com! Now you need to branch off of the v1.2.0 and v1.3.0 tags of libfoo and do the same change there.
So we have only 3 repositories with only 2 dependencies and we already have to do the search/replace 5 times.
Imagine now that apiservice depends on authservice v2.3.7 just to include some type definitions. Authservice is currently on version 2.4.0 but the types haven't changed so apiservice hasn't upgraded its dependency. Now you need to branch off of authservice v2.3.7 too and do the job there. Oh and authservice v2.3.7 depends on libfoo v1.2.5, so now you need to make a branch off of libfoo v1.2.5 with the search/replace.
3 repositories with a straightforward dependency relationship, 7 search/replace jobs.
The numbers get terrifying as you scale this up.
rot256 46 minutes ago [-]
Package management in Go always felt pretty hacky. Among the issues is the confusion between "what something is" and "where something is". At the beginning there wasn't even a way to have different versions of a package, with the justification that you should just fix an interface and keep it stable till the end of time, there was also no way to "lock dependencies" so at build time you got whatever that repo served you. It has gotten better since, but still some jankyness as a result of "growing" a package system.
p4bl0 16 hours ago [-]
True, but beware of the domain name you're using. Because VeriSign may unilaterally decide to delete your domain name along with thousands of others [1] and you're back to square one…
Or if your goal is to open source code, think about the lifetime of the entity owning the domain: the source code might be of interest to users beyond the control of the domain name. Domain controled by an individual or a company? What if the domain is not renewed, or taken over by an hostile entity?
Deepcopy [1] is a Go package that is still heavily used despite its creator has disappeared 9 years ago. At least GitHub is a stable and trusted host as a distribution point and communication point for users.
I broadly like Go's "the import is the hosted location (or a pointer to it)" quite a lot, as it largely solves name-squatting and ownership and a lot more (while allowing major risks with domain sales/abandonment), but yeah - I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name. A few languages now have content-addressed imports/packages, instead of just adding hashes as verification, and I hope we see more in the future.
Signed modules / including the signature hash would also solve a lot, e.g. it'd mean domain sales no longer silently inherit full permissions. It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.
hnlmorg 5 hours ago [-]
> I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name.
You already can use SHA in go.mod in exactly the same way you would use a version string.
I don’t know about using multiple proxies in go mod though.
throwaway894345 13 hours ago [-]
> It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.
How many mainstream languages have content addressed imports? I can’t think of any, so I assume I’m misunderstanding your meaning of the term because you seem to be suggesting that it is common and Go is the outlier for lacking it?
Groxx 10 hours ago [-]
Go is not really an outlier for not having signed packages (there are a fair number that have it, but far from most)... but definitely stuck behind common accepted practice. By decades, if comparing against some (e.g. Java).
Which keeps happening with stuff they rebuild from scratch - an excellent and somewhat unique first showing, far beyond what most first attempts manage, but followed by near-complete stagnation while issues that everyone familiar with the field predicted from miles away pile up.
vips7L 1 hours ago [-]
Coming from the Java side this is typical of Google libraries. They start off impressive and gain wide adoption but then they stop supporting changes the community wants, missing basic features.
ablob 13 hours ago [-]
At least the fallout will be less if you have to resolve your custom domain differently in the intranet due to something like this.
If github is "taken down" you can't just resolve the whole domain differently, you need to only resolve the packages differently. If your "Golang domain" is taken down, it should be a lot easier to hotfix until something proper is implemented (the quickest and dirtiest would be a hostfile-entry).
cyh555 9 hours ago [-]
reading the article confuses me though I don't want to do the context switching by googling what 3rd level domain is about
hnarn 3 hours ago [-]
All you need to know is that for some reason you could buy foo.bar.com without owning bar.com directly from the registrar.
It is being retired because very few people use it. It kind of sucks but this is how domains work, it’s not real estate.
alavilli 13 hours ago [-]
[flagged]
3 hours ago [-]
unscaled 16 hours ago [-]
> One of the good features of Go is that you namespace your code with the location to fetch the code.
Then the rest of the article explains why this is NOT a good feature in practice.
I don't think it's unworkable either, but this is one of these little thing that Go decided to do different and convinced its fans that this is a great idea and all the other languages where doing it wrong. After a couple of road bumps appeared, instead of admitting there are some advantages to having official package names, we're now told that everybody should just set up their own custom domain with an nginx server or a Go Vanity URLs forwarder to serve traffic for their GitHub-hosted packages.
12_throw_away 13 hours ago [-]
combine these domain aliases with the caching goproxy you're supposed to use, and they're well on their way to reinventing a traditional package index from first principles
st3fan 15 hours ago [-]
Great when a company goes out of business and the domains are dangling. Next one to scoop it up takes over source code that others new depend on. We've seen this many times already in other ecosystems. It is a very bad situation.
It is bad advice to move your packages under your own domain. You will never be as good as Microsoft to keep paying for the domain. There are no guarantees in life but I do guarantee you that when you go out of business, that domain is the last thing you will think about.
epwr 14 hours ago [-]
Thinking this through, using my domain might actually be much less risky than using a host's. If Gitlab closes down, and someone else buys the domain, then I'm forced to make a massive refactor with little notice.
If I own the domain, I'm likely to continue to owning it until I shut down my company. At which point, no one is likely to be running my code.
Possibly different if I was developing an open source library or something, but either way there's risk and need to be thoughtful of the risk versus impact.
someonebaggy 14 hours ago [-]
Well, actually you don't download from GitHub - you download from the central mirror proxy operated by Google, and that will always return the same package given the same versioned reference, even if the original source differs from it.
SenHeng 20 hours ago [-]
GitHub is almost forever. Your custom domain disappears when you stop paying the bills, which if you’re an open source developer has a higher likelihood than GitHub disappearing.
One day, we’re all going back to vendoring dependencies.
mkj 10 hours ago [-]
Unless the field of software development somehow totally stagnates, I'm confident that at some point I will move to a different hosting service, as github goes the way of sourceforge, travis-ci, and myspace. So my personal domain will continue to be relevant (it's lasted fine 23 years).
Maybe at that point it won't be using git either, or maybe it will - who knows.
otterley 20 hours ago [-]
Why did we stop vendoring dependencies in the first place?
collabs 18 hours ago [-]
Because it sucks. But maybe it should suck? It would make us think twice before adding dependencies.
I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?
Sorry if this feels like a stream of consciousness because it is ↔
otterley 17 hours ago [-]
Vendoring dependencies doesn’t necessarily mean you have to include binary objects in your repo. They could be content-addressable artifacts in your org’s private blob store.
someonebaggy 14 hours ago [-]
i.e. they could be NIH git-lfs.
jeremyjh 17 hours ago [-]
I think its a good question. The short answer is: because tooling doesn't default to this or make it easy.
The answer to the question of why THAT is the case - is not so easy to answer. The main benefit I can see with using lock files instead of vendoring is that it saves a lot of storage and diff history from entering your repository. So clones are much faster, backups smaller etc.
I think Go used to work this way (automated vendoring) but it’s the only language I can think of that ever did in terms of standard tooling. It would be instructive to learn why that changed.
st3fan 14 hours ago [-]
I vendor all my Go dependencies all the time for every project. It is the way.
throwaway894345 13 hours ago [-]
Has this ever been necessary/useful? Genuinely curious because I’ve been using Go since 2012 and can’t recall a time when vendoring solved a problem better than “regular” modules. Like have there been times when the source went down and the module proxy went down (or didn’t have your dependency cached)?
metaltyphoon 18 hours ago [-]
Because its not as convenient
otterley 17 hours ago [-]
Can you elaborate?
thayne 17 hours ago [-]
Updates mean you have to copy over all the code into your repo, which creates a large diff, and hope you aren't overwriting any local changes someone might have made.
It bloats your repo, both with the actual code, and the large diffs when you update it.
You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.
If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.
Etc.
otterley 15 hours ago [-]
Disk is cheap.
Recursive dependencies have the same issues whether you vend them or not.
Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.
So, what’s left?
thayne 14 hours ago [-]
> Disk is cheap
The biggest problem isn't (usually) disk space, or network bandwidth, it is that git operations slow down as the size of the repo grows. And it means that cloning or pulling the repo takes longer, which can be especially problematic for CI.
> Recursive dependencies have the same issues whether you vend them or not.
Package managers usually handle resolving recursive/transitive dependencies for you. Some have support for vendoring dependencies, but not all do. In theory, you could have similar tooling for vendoring dependencies, but in practice that often isn't the case.
otterley 14 hours ago [-]
These are all solvable problems IME. But it does mean you need people who are experienced at solving them or who care enough to learn.
thayne 13 hours ago [-]
> But it does mean you need people who are experienced at solving them or who care enough to learn.
That sounds like it's inconvenient to me.
otterley 13 hours ago [-]
The price of security and business continuity is the occasional inconvenience. Tale as old as time.
thayne 9 hours ago [-]
Vendoring isn't a clear win for security. It may provide some protection against "supply chain" attacks, but makes it more difficult for you to respond to vulnerabilities discovered in the version you vendored. And for business continuity, in many cases depending on an external registry is an acceptable risk.
And both of those concerns can be addressed by using an internal/private registry that mirrors the packages you need.
But in any event, your original question was to elaborate on why vendoring is inconvenient. Whether or not the benefits are worth the inconvenience is a different question, to which IMHO the answer is "it depends". Sometimes it is, and sometimes it isn't.
otterley 9 hours ago [-]
> Vendoring isn't a clear win for security. It may provide some protection against "supply chain" attacks, but makes it more difficult for you to respond to vulnerabilities discovered in the version you vendored.
How so? Whether a dependency is vendored or not, you still have to update it to integrate a security update to that dependency, don't you? The alternative is to not pin your dependencies, but that is far riskier overall.
> Whether or not the benefits are worth the inconvenience is a different question
I would contend that it is the most important question. :-)
throwaway894345 13 hours ago [-]
> Disk is cheap
If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk. If I want to upgrade the disk on my work laptop, I’m SOL.
> Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.
It’s not obvious to me how putting the dependencies in a vendor folder solves the diff problem. Does every code host allow you to hide diffs to certain directories?
And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?
otterley 13 hours ago [-]
> If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk
Is this a significant risk in reality? MBPs today come with a minimum of 1TB of storage. Even 5 years ago I think the minimum was 256 GB. This is more than large enough for all but the most massive repositories, even with vendored dependencies. And you can always plug in external SSDs or HDDs or connect to a network server.
> And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?
This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.
throwaway894345 4 minutes ago [-]
> This is more than large enough for all but the most massive repositories
Consider the possibility that the drive needs to accommodate more than just a single repository?
I have lots of software, entire language toolchains, AI models, Docker images, application volumes, VMs, etc.
And this isn’t a theoretical concern; I have to free up space every few months.
> This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.
So there is no advantage scenario that you’re aware of?
stackedinserter 17 hours ago [-]
We didn't!
JodieBenitez 10 hours ago [-]
go mod vendor... works for me. Also it makes sense for an ecosystem like Go with strong standard lib and strong backward compatibility commitment.
In the case of a typical software enterprise having your domain gone means that you probably don't care about the code anymore anyway.
farthest 49 minutes ago [-]
So what happens when the domain provider gets bought and the domain in your OSS project gets auctioned and compromised? This is turtles all the way down it seems..
motbus3 2 hours ago [-]
Holy cow. When I was learning go I thought I was the only one crazy to think that is not a good default
gumby 18 hours ago [-]
Seems like an error to use a URL. This is perfect for a URN or some other form of URI.
It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.
someonebaggy 14 hours ago [-]
The only useful alternative you're talking about would be a content-addressed one. A "URN" that relies on some kind of mutable registry lookup is actually a URL (both are URIs).
0xCMP 16 hours ago [-]
This is great and I think for any company/individual who is going to ensure their domain is registered and maintained this makes a lot of sense. It would be catastrophic, but there is no reason Github couldn't disappear or otherwise change some policies that require moving away from it. The way Go works makes committing these package names tied to Github so much more weighty than simply the place you pull from.
The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.
dirkc 2 hours ago [-]
I've been saved by the commit hash in the dependency when reviving an old Python project. I could locate a fork of the repo and use the same version to get the code going again. I ended up vendoring the dep.
shieldagent 2 hours ago [-]
We hit this after an org rename. replace covered our modules; transitive imports of the old github.com path still broke builds.
alexfortin 2 hours ago [-]
Thanks for the article, it inspired me to come up with my own solution (and blog post):
Isn't this going to introduce the dependency of domain management? I understand the positive side of it, but put some infra level manamgment layer to individual Open Source devs, just a thought. But yeah, positives vs negatives weigh and pick.
MadVikingGod 14 hours ago [-]
Here's the primer for those who thing find a replace is the answer. Pretend you have a project with A 1.2.3 depends on B 2.4.6 which depends on C 0.1.1. If you are in github you get
module github.com/example/A
requires (
github.com/example/B 2.4.6
github.com/example/C 0.1.1
)
You now have many choices on how to proceed, but none of them will include A) being able to build old releases, or B) doing so without making changes to all dependencies.
One choice is go to the leaves, C in our example, and make a release on gitlab. Then go to B, and have B depend on the gitlab C. This is fine for rolling forward, but if you want to roll back you would have to rewrite all of C to use the gitlab url, and find all of the equivalent tags and repush all of them. Also tell your users that any binaries you've released now have new hashes. Then repeat this for every repo. It's not a small task.
pidgeon_lover 4 hours ago [-]
"Package manager"-like behaviour like this is evil. Importing software/code at runtime from the internet (like Apt, Go, Pacman, Pip, NPM, etc.) is the equivalent of those dreadful online stub installer packages, and it's impressive it's as trusted a mechanism as it is.
Leave the Go code or Linux distro to link-rot for a couple decades and all imports like this will be dead or serving malware, leaving the software unusable.
deanc 2 hours ago [-]
That's all well and good, but what happens if you drop off the face of the earth due to personal issues, health issues or worse and your domain expires?
blackjpi 10 hours ago [-]
I don’t think I would use a 301 permanent redirect for this sort of thing. If you later want to switch from GitHub to some other provider, software that aggressively caches permanent redirects (like browsers) will remember the old redirect to GitHub, especially since you don’t have any cache control headers set. Maybe not as large a concern will CLI tools, but something to be aware of as this has bit me in the past.
nirui 7 hours ago [-]
A GitHub URL at least is still an credible identifier, your custom domains is not, and likely will never be given how the system is accustomed to. Domain is for branding, not identification.
Unless of course, it's special domains that has identification built in, such as .onion which is generated in such way (cryptographic keys) no other people can easily obtain control even after the domain is no longer maintained.
If you really don't want to use GitHub, an alternative is just use .internal suffix (i.e. yourproject.internal/project) in combination with `replace` directives in go.mod. But that require your user to manually download/install your package and then edit their own go.mod.
nicce 7 hours ago [-]
> A GitHub URL at least is still an credible identifier, your custom domains is not, and likely will never be given how the system is accustomed to. Domain is for branding, not identification.
The post is about commercial software. If enterprise’s domain is not credible enough, then there are bigger problems.
hk1337 14 hours ago [-]
It's a rather internet focused language. Even if you don't point it to github, some other packages will and you still have to point it to something (i.e. gitlab, codeberg, or your personal git server). Then what happens if those servers are no longer available?
throwaway894345 13 hours ago [-]
Go defaults to using a caching proxy, but you can use your own too.
okanat 13 hours ago [-]
Just like [^1] says:
> After spending years doing those FFI dances in both directions, I’ve reached the conclusion that the only good boundary with Go is a network boundary.
It looks like this extends into the package manager too. The only good boundary with Go programs is a socket. The only good boundary with the Go package manager (? if we can say it is one) is a domain and a DNS server (your own or somebody else's in the case of gopkg.in).
I think we need better handle to the original location/author/project than a domain. Any of such "go" proxy is yet another SPOF.
We should lean towards IPNS (Name Seever part of IPFS) or some distribited ledger.
gchamonlive 17 hours ago [-]
Thanks for answering my question! So yes, if I intend to use go and study the packages I'd have to rely on beforehand, if they're all hosted on GitHub should make me steer away from Golang. Or at least plan to cache the packages I need locally so CI always have something to work with when building.
Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge
Is it just changing all instances of `github.com` to `proxy.golang.org`? Shouldn't `go build` warn about using third party proxies for dependencies? Should `go get` offer a flag to automatically use the proxy when adding dependencies?
tugback 5 hours ago [-]
Definitely had a private GitHub dependency disappear once. Vendoring saved us a massive headache downstream.
andreashaerter 20 hours ago [-]
There are also a few Hugo templates for vanity import paths.
And yes I'm aware of the irony of hosting this on GitHub... still figuring out a good workflow for maintaining our OSS on Codeberg and GitHub in parallel fed from the internal forge. The dependency on our own domain is real but we favor it.
serbuvlad 15 hours ago [-]
> That is, if you move your git hosting to GitLab then you have to change your code!
Not to be cynical about this but I fail to see how running sed s///g after a very rare event qualifies as a serious problem.
Now, sure, given that the solution is so simple, it's a nice recommendation, but still...
grumpy_coder 12 hours ago [-]
How many other people have hit the pain of injecting ssh keys for gitlab into a docker container building your code in CI?
Compiling code shouldn't hit network by default.
okoddcat 9 hours ago [-]
why you need ssh key for CI? The CI_JOB_TOKEN or scoped PAT also works via http to pull/push the code.
popcornricecake 14 hours ago [-]
I fear one of these days someone will copy a legit package and make it onto your search results, and then the Go tools pull in something extra.
mosselman 17 hours ago [-]
I have not worked with Go, so I thought I'd ask: why wouldn't simple find and replace be able to fix this? And why wouldn't a coding agent be able to do this for you quickly?
Another way to fix this kind of issues would be to fallback on the Software Heritage which archives the whole GitHub.
This is something Nix and Guix are trying to achieve: by default, they fetch source code from the original endpoint, but if it doesn't respond anymore, they would query the Software Heritage archive [1].
Because go mod doesn't use SHA, it makes things a bit more complicated but Nix/Guix encountered the same issue and there are some mechanisms to workaround this issue.
That's a beautiful solution. I love it! The alternative work went with was to just build everything ourselves.
silverwind 6 hours ago [-]
I disagree, prefer to see directly which forge a module is hosted on.
18 hours ago [-]
drunken_thor 14 hours ago [-]
Honestly google should have always had a package index even if it points tooling to the repo, but it would maintain the package name with the ability to change where the repo is.
bionsystem 18 hours ago [-]
braid has been a treat for us to manage external dependencies. Much cleaner than submodules.
globular-toast 8 hours ago [-]
So you need a domain and an nginx server running somewhere? That seems like more of a liability than just controlling a domain. Isn't there a way to configure this this just using DNS and no web server?
verdverm 11 hours ago [-]
fun fact, in the process of designing CUE modules, the Go team said they would use a registry if they could do it over, CUE uses OCI registries
I'm not sure that custom domains are a good idea. It's way too easy to end up with broken links once vanity domains go away.
However, Golang saves the hashes of all the modules in `go.sum` files, so we just need to add a way to do content-addressable fetches. This solves the issue of reproducibility for old builds. As long as you can find the module in some repository somewhere, you'll be able to build it.
The next step is supporting module _evolution_. We need a way to declare: "From this point onward, `github.com/company/someproject` is now `company.com/someproject`", so that all the references are to these packages are identical. This is possible on a per-package basis with `replace` directives, but this doesn't scale.
And this is not easy to solve in general (especially in the age of supply-chain attacks). If the initial project cooperates (or if Github can be convinced to help), perhaps at least a part of this can be solved by adding special "redirecting module" support to Go.
0xbadcafebee 18 hours ago [-]
Even better reason: you can later point this domain at an artifact registry. This not only gives you reliability and flexibility, it also secures your software supply chain. You don't need an SBOM or anything fancy to get started, just pull all your artifacts into a central source and improve it over time. Install an artifact registry anywhere you can run a container, use dumb static shared credentials, and start with "proxy mode". Later on you can pin or restrict versions, verify checksums, implement SSO, etc. This is going to become table stakes in the new security landscape.
prasadvara 17 hours ago [-]
You don't need an SBOM or anything fancy to get started
-- Curious how this avoids SBOM need?
0xbadcafebee 9 hours ago [-]
An SBOM is an inventory; you don't need to make an inventory to pull files through a proxy.
PunchyHamster 4 hours ago [-]
Now you coupled it to 2 services you don't own, congratulations on making it worse.
Bonus points if DNS mafia deems your cool TLD to now cost 10x times more coz it's "premium" and you're stuck paying up or telling everyone to migrate
skybrian 18 hours ago [-]
Not really convinced. I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
racingmars 18 hours ago [-]
> I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
People can delete projects from GitHub. Businesses whose priorities might change may not keep an old project set to public up on GitHub because they won't want people to continue contacting them for support, or they don't want to be responsible for updating security vulnerabilities for projects they are abandoning so they'd rather just pull it offline, etc.
Whether the library you're importing is hosted on GitHub or not, never assume it will be there tomorrow. *Always vendor your dependencies.*
skybrian 17 hours ago [-]
I believe they'd also be cached by proxy.golang.org? But companies should probably run their own proxy.
15 hours ago [-]
bigwhite 16 hours ago [-]
[flagged]
ArgumentMoney81 7 hours ago [-]
[flagged]
GnosiWorks 6 hours ago [-]
[dead]
ambicapter 16 hours ago [-]
[dead]
Meneth 17 hours ago [-]
You shouldn't be using fully qualified domains at all. Use relative paths instead, like sibling-hosted git submodules.
In this terminology, the article basically amounts to saying "This namespace can fail! The solution is, use another!"
But that doesn't get you anywhere, because the new namespace can fail too. In fact it is almost certain that it will fail sooner than "github.com's DNS and hosting" as a namespace.
There is no solution where you tie your software release to a namespace that can't fail because there is no namespace that can't fail. It doesn't matter if your favorite language uses DNS or has a centrally blessed repository or if it distributes a blessed list of package names with the language itself or anything else. The namespace can fail.
Therefore, the only thing you can really do is be resilient against failures, and in a lot of ways, the only practical solution to resilience is just to assume that if, in the future, someone has problems getting a package due to a namespace failure, they will not helplessly disintegrate into a puddle of tears while your code is lost forever, but that the future person will instead solve this perfectly solvable problem.
That said, in the normal run of things you can mark your last release on e.g. github as deprecated, leave a note that it's moved to xyz, and wait for the users to migrate themselves. Migrating is fairly trivial in that it's a find / replace.
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.
It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.
I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.
But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.
Many tools will fail if you break this contract.
https://datatracker.ietf.org/doc/draft-davies-internal-tld/
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
https://go.dev/doc/modules/major-version
This is one of the main motivations for using a monorepo with all third party dependencies imported all the way in, and only one version for everything.
Of course it causes a whole bunch of other problems and is kinda expensive to scale, so most places won't do it.
Pick your poison. Each approach has some seriously negative trade offs in the extremes.
I say this because I’ve gone through a few of these, some with monorepos (the easiest, search and replace and you’re done), some not (yes the hardest but you can just vendor).
At least in the situations I’ve faced this was genuinely not horrible beyond just the organization itself being complicated about the solution.
That makes no sense? As long as you’re using go modules, you can just host an internal go proxy for internal modules similar to proxy.golang.org to archive the old versions, and they won’t depend on the old git host.
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].
So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.
[1] See: https://go.dev/ref/mod#go-mod-file-replace:~:text=replace%20...
[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.
[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.
Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).
In Go, the package identifiers are literally URLs. Go's own tooling assumes that it can make an HTTP request to the URL and that the response will be HTML with particular tags which Go's tooling will parse and use to resolve a git repository which can be 'git clone'd. Leaving them as-is when you abandon the infrastructure they reference literally means leaving dead links in your source code.
Any tool that automatically assumes that a URL in a Go module import is 'trusted' is just a broken tool.
You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.
The idea is: the package maintainer pushes a final version of the old package that is a shim, and which has a deprecation notice. The shim imports the new package, and forwards all calls to the new package (and I suppose type aliases and variable aliases as well). The shim includes `//go:fix inline` annotations on all exported symbols, so that when users run `go fix` it rewrites their code to use the new package, by inlining their usages of the old package (which is now simply a shim that references the new package).
Not perfect. There is still a bit of a discoverability problem. Not all tooling warns on deprecated packages/functions (go toolchain doesn't, but gopls and staticcheck do). And users need to know to run `go fix`.
1: https://go.dev/blog/inliner#example-renaming-ioutilreadfile
"replace example => github.com/example/example" at first, then change to "replace example => gitlab.com/example/example" or similar when there's a migration
(i don't use go, but i hope this is possible and i am confused if it's not)
You know, normal development task, small amount of story points..
For my thoughts about why it's not easy, see this comment: https://news.ycombinator.com/item?id=49870363
Now you want to move from github.com to git.example.org. You change libfoo, authservice and apiservice to use git.example.org, that part is just simple tedious work. But the v1.2.0 and v1.3.0 tags of libfoo are old commits from before the move, so they still reference github.com! Now you need to branch off of the v1.2.0 and v1.3.0 tags of libfoo and do the same change there.
So we have only 3 repositories with only 2 dependencies and we already have to do the search/replace 5 times.
Imagine now that apiservice depends on authservice v2.3.7 just to include some type definitions. Authservice is currently on version 2.4.0 but the types haven't changed so apiservice hasn't upgraded its dependency. Now you need to branch off of authservice v2.3.7 too and do the job there. Oh and authservice v2.3.7 depends on libfoo v1.2.5, so now you need to make a branch off of libfoo v1.2.5 with the search/replace.
3 repositories with a straightforward dependency relationship, 7 search/replace jobs.
The numbers get terrifying as you scale this up.
[1] https://neil.fraser.name/news/2026/09/03/
Deepcopy [1] is a Go package that is still heavily used despite its creator has disappeared 9 years ago. At least GitHub is a stable and trusted host as a distribution point and communication point for users.
[1]: https://pkg.go.dev/github.com/mohae/deepcopy
Signed modules / including the signature hash would also solve a lot, e.g. it'd mean domain sales no longer silently inherit full permissions. It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.
You already can use SHA in go.mod in exactly the same way you would use a version string.
I don’t know about using multiple proxies in go mod though.
How many mainstream languages have content addressed imports? I can’t think of any, so I assume I’m misunderstanding your meaning of the term because you seem to be suggesting that it is common and Go is the outlier for lacking it?
Which keeps happening with stuff they rebuild from scratch - an excellent and somewhat unique first showing, far beyond what most first attempts manage, but followed by near-complete stagnation while issues that everyone familiar with the field predicted from miles away pile up.
If github is "taken down" you can't just resolve the whole domain differently, you need to only resolve the packages differently. If your "Golang domain" is taken down, it should be a lot easier to hotfix until something proper is implemented (the quickest and dirtiest would be a hostfile-entry).
It is being retired because very few people use it. It kind of sucks but this is how domains work, it’s not real estate.
Then the rest of the article explains why this is NOT a good feature in practice.
I don't think it's unworkable either, but this is one of these little thing that Go decided to do different and convinced its fans that this is a great idea and all the other languages where doing it wrong. After a couple of road bumps appeared, instead of admitting there are some advantages to having official package names, we're now told that everybody should just set up their own custom domain with an nginx server or a Go Vanity URLs forwarder to serve traffic for their GitHub-hosted packages.
It is bad advice to move your packages under your own domain. You will never be as good as Microsoft to keep paying for the domain. There are no guarantees in life but I do guarantee you that when you go out of business, that domain is the last thing you will think about.
If I own the domain, I'm likely to continue to owning it until I shut down my company. At which point, no one is likely to be running my code.
Possibly different if I was developing an open source library or something, but either way there's risk and need to be thoughtful of the risk versus impact.
One day, we’re all going back to vendoring dependencies.
Maybe at that point it won't be using git either, or maybe it will - who knows.
I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?
Sorry if this feels like a stream of consciousness because it is ↔
The answer to the question of why THAT is the case - is not so easy to answer. The main benefit I can see with using lock files instead of vendoring is that it saves a lot of storage and diff history from entering your repository. So clones are much faster, backups smaller etc.
I think Go used to work this way (automated vendoring) but it’s the only language I can think of that ever did in terms of standard tooling. It would be instructive to learn why that changed.
It bloats your repo, both with the actual code, and the large diffs when you update it.
You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.
If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.
Etc.
Recursive dependencies have the same issues whether you vend them or not.
Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.
So, what’s left?
The biggest problem isn't (usually) disk space, or network bandwidth, it is that git operations slow down as the size of the repo grows. And it means that cloning or pulling the repo takes longer, which can be especially problematic for CI.
> Recursive dependencies have the same issues whether you vend them or not.
Package managers usually handle resolving recursive/transitive dependencies for you. Some have support for vendoring dependencies, but not all do. In theory, you could have similar tooling for vendoring dependencies, but in practice that often isn't the case.
That sounds like it's inconvenient to me.
And both of those concerns can be addressed by using an internal/private registry that mirrors the packages you need.
But in any event, your original question was to elaborate on why vendoring is inconvenient. Whether or not the benefits are worth the inconvenience is a different question, to which IMHO the answer is "it depends". Sometimes it is, and sometimes it isn't.
How so? Whether a dependency is vendored or not, you still have to update it to integrate a security update to that dependency, don't you? The alternative is to not pin your dependencies, but that is far riskier overall.
> Whether or not the benefits are worth the inconvenience is a different question
I would contend that it is the most important question. :-)
If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk. If I want to upgrade the disk on my work laptop, I’m SOL.
> Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.
It’s not obvious to me how putting the dependencies in a vendor folder solves the diff problem. Does every code host allow you to hide diffs to certain directories?
And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?
Is this a significant risk in reality? MBPs today come with a minimum of 1TB of storage. Even 5 years ago I think the minimum was 256 GB. This is more than large enough for all but the most massive repositories, even with vendored dependencies. And you can always plug in external SSDs or HDDs or connect to a network server.
> And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?
This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.
Consider the possibility that the drive needs to accommodate more than just a single repository?
I have lots of software, entire language toolchains, AI models, Docker images, application volumes, VMs, etc.
And this isn’t a theoretical concern; I have to free up space every few months.
> This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.
So there is no advantage scenario that you’re aware of?
Also: https://proxy.golang.org/
It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.
The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.
https://a.l3x.in/blog/vanity-go-import/
You now have many choices on how to proceed, but none of them will include A) being able to build old releases, or B) doing so without making changes to all dependencies.
One choice is go to the leaves, C in our example, and make a release on gitlab. Then go to B, and have B depend on the gitlab C. This is fine for rolling forward, but if you want to roll back you would have to rewrite all of C to use the gitlab url, and find all of the equivalent tags and repush all of them. Also tell your users that any binaries you've released now have new hashes. Then repeat this for every repo. It's not a small task.
Leave the Go code or Linux distro to link-rot for a couple decades and all imports like this will be dead or serving malware, leaving the software unusable.
Unless of course, it's special domains that has identification built in, such as .onion which is generated in such way (cryptographic keys) no other people can easily obtain control even after the domain is no longer maintained.
If you really don't want to use GitHub, an alternative is just use .internal suffix (i.e. yourproject.internal/project) in combination with `replace` directives in go.mod. But that require your user to manually download/install your package and then edit their own go.mod.
The post is about commercial software. If enterprise’s domain is not credible enough, then there are bigger problems.
> After spending years doing those FFI dances in both directions, I’ve reached the conclusion that the only good boundary with Go is a network boundary.
It looks like this extends into the package manager too. The only good boundary with Go programs is a socket. The only good boundary with the Go package manager (? if we can say it is one) is a domain and a DNS server (your own or somebody else's in the case of gopkg.in).
[^1]: https://fasterthanli.me/articles/lies-we-tell-ourselves-to-k...
We should lean towards IPNS (Name Seever part of IPFS) or some distribited ledger.
Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge
https://news.ycombinator.com/item?id=49434625
Let's take a project I worked on for a couple of years some time ago, it has this go.mod file: https://gitlab.com/nunet/device-management-service/-/blob/ma...
Is it just changing all instances of `github.com` to `proxy.golang.org`? Shouldn't `go build` warn about using third party proxies for dependencies? Should `go get` offer a flag to automatically use the proxy when adding dependencies?
Here's mine: https://github.com/foundata/hugo-theme-govanity (e.g. used at https://golang.foundata.com/ )
And yes I'm aware of the irony of hosting this on GitHub... still figuring out a good workflow for maintaining our OSS on Codeberg and GitHub in parallel fed from the internal forge. The dependency on our own domain is real but we favor it.
Not to be cynical about this but I fail to see how running sed s///g after a very rare event qualifies as a serious problem.
Now, sure, given that the solution is so simple, it's a nice recommendation, but still...
Compiling code shouldn't hit network by default.
This is something Nix and Guix are trying to achieve: by default, they fetch source code from the original endpoint, but if it doesn't respond anymore, they would query the Software Heritage archive [1].
Because go mod doesn't use SHA, it makes things a bit more complicated but Nix/Guix encountered the same issue and there are some mechanisms to workaround this issue.
[1] https://www.softwareheritage.org/2025/05/21/software-heritag...
MVS + OCI is the gold standard imo
CUE module proposal (implemented a while back) - https://github.com/cue-lang/cue/discussions/2939
MVS, the fast and deterministic dependency resolving algorithm Go uses instead of SAT solvers - https://research.swtch.com/vgo-mvs
However, Golang saves the hashes of all the modules in `go.sum` files, so we just need to add a way to do content-addressable fetches. This solves the issue of reproducibility for old builds. As long as you can find the module in some repository somewhere, you'll be able to build it.
The next step is supporting module _evolution_. We need a way to declare: "From this point onward, `github.com/company/someproject` is now `company.com/someproject`", so that all the references are to these packages are identical. This is possible on a per-package basis with `replace` directives, but this doesn't scale.
And this is not easy to solve in general (especially in the age of supply-chain attacks). If the initial project cooperates (or if Github can be convinced to help), perhaps at least a part of this can be solved by adding special "redirecting module" support to Go.
-- Curious how this avoids SBOM need?
Bonus points if DNS mafia deems your cool TLD to now cost 10x times more coz it's "premium" and you're stuck paying up or telling everyone to migrate
People can delete projects from GitHub. Businesses whose priorities might change may not keep an old project set to public up on GitHub because they won't want people to continue contacting them for support, or they don't want to be responsible for updating security vulnerabilities for projects they are abandoning so they'd rather just pull it offline, etc.
Whether the library you're importing is hosted on GitHub or not, never assume it will be there tomorrow. *Always vendor your dependencies.*