The last week or so I've been trying to get an ancient legacy Ruby on Rails app up to date. This was running Ruby 1.9.3 and rails 3.0.6 and the dependencies have not been touched at all in close to 10 years. This project led me to write the article about why I think executable file formats aren't the best way to track dependencies.

A crucial issue with a project like this is that nobody actually remembers any of the intentions of anything at the time the code was written. As a result anything that isn't documented or made explicit in the code now becomes a time consuming process to re-discover the intentions.

It turns out that the packaging on this project was a mess. There's a gemfile but it appears that the production environment didn't have exactly the same packages installed as described in the gemfile. Also recreating the production environment is now impossible since this project never had anything like a build script or containerization and the knowledge of what was installed is now lost. I don't think the lack of containerization is a shortcoming of the people who made this either, Docker came out a few years after this project started. Also back then people just didn't have as many good tools for dependency management. One of the first challenges I ran into was that the version of Ruby was old enough that it was hard to find good docker images for Ruby 1.9.3.

So it turns out that the main issue with this is that the build step wasn't documented, this made it especially difficult to try to figure out exactly what was installed and what was not. Adding to this problem was the fact that some dependencies had been pulled and are now hard to install. Now I understand why this step tends to not be so good in older projects, it was simply far harder to make repeatable builds back then, additionally a lot of people just weren't aware of containerization.

If you are creating a Rails project now I highly recommend using something like Bundler, not having good dependency management is just hell. If there's one take away from this article it's that using the proper dependency management tools, in this case Bundler, saves an enormous amount of hassle.

I kept running into obscure installation issues like the following:

/var/lib/gems/2.3.0/gems/activesupport-3.0.6/lib/active_support/values/time_zone.rb:272: warning: circular argument reference - now
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:94:in `rescue in block (2 levels) in require': There was an error while trying to load the gem 'sorcery'. (Bundler::GemRequireError)
Gem Load Error is: uninitialized constant ActiveSupport::LogSubscriber
Backtrace for gem load error is:
/var/lib/gems/2.3.0/gems/activerecord-3.0.6/lib/active_record/log_subscriber.rb:2:in `<module:ActiveRecord>'
/var/lib/gems/2.3.0/gems/activerecord-3.0.6/lib/active_record/log_subscriber.rb:1:in `<top (required)>'
/var/lib/gems/2.3.0/gems/activerecord-3.0.6/lib/active_record/base.rb:24:in `<top (required)>'
/var/lib/gems/2.3.0/gems/sorcery-0.9.1/lib/sorcery.rb:73:in `<module:Sorcery>'
/var/lib/gems/2.3.0/gems/sorcery-0.9.1/lib/sorcery.rb:3:in `<top (required)>'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:91:in `require'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:91:in `block (2 levels) in require'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:86:in `each'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:86:in `block in require'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:75:in `each'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:75:in `require'
/var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler.rb:106:in `require'
/src/config/application.rb:8:in `<top (required)>'
/var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:28:in `require'
/var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:28:in `block in <top (required)>'
/var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:27:in `tap'
/var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:27:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
Bundler Error Backtrace:
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:90:in `block (2 levels) in require'
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:86:in `each'
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:86:in `block in require'
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:75:in `each'
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/runtime.rb:75:in `require'
    from /var/lib/gems/2.3.0/gems/bundler-1.13.7/lib/bundler.rb:106:in `require'
    from /src/config/application.rb:8:in `<top (required)>'
    from /var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:28:in `require'
    from /var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:28:in `block in <top (required)>'
    from /var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:27:in `tap'
    from /var/lib/gems/2.3.0/gems/railties-3.0.6/lib/rails/commands.rb:27:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'

This is really strange because Rails should install ActiveSupport but yet something hasn't gone right. Searching around for solutions isn't that easy for these problems because the error output is especially hard to search for.

Generally speaking when I run into difficult problems I like to dig into the details a bit more, so I found this post that describes the way in which Gems work and this post about how Rails loads dependencies. My heart sank when I read the following quote from the article:

Most of the time, gems in Ruby Just Work. But there’s a big problem with Ruby magic: when things go wrong, it’s hard to find out why.

You won’t often run into problems with your gems. But when you do, Google is surprisingly unhelpful. The error messages are generic, so they could have one of many different causes. And if you don’t understand how gems actually work with Ruby, you’re going to have a tough time debugging these problems on your own.

This really seemed spot on to me, I'm good at searching up technical problems and I was finding very little help. It was obvious that the cost in terms of time and effort for solving this would be high so I was immediately facing an important choice here, I could try to keep plowing on or I could cut my losses. The 60 code commits I'd already made cast a shadow of the sunk cost fallacy over the whole thing. But now that I'm more mature I realize that sometimes you just have to cut your losses, there's no point in throwing away more time just because you've already thrown away a bunch of time. Though emotionally and organizationally this can be a tough sell even if you know it's the right choice. When I asked what the intentions were with the dependencies and got a "I don't know it was 10 years ago and I don't remember" response any remaining doubts I had were removed, a change in approach was needed.

The moral of the story is that you really need to document your dependencies preferably in some form of well structured file format that can be used with dependency management tooling. In this particular instance there were dependencies for the code that were not present in the Gemfile and nobody really knew what exactly was installed previously. Trying to fix this incrementally might just not be possible, at least not in any sort of cost effective manner. Ever since Joel Spolsky two decades ago wrote about the disaster that was the Netscape re-write a lot of software engineers have been hesitant to engage in large scale re-writes. Part of his reasoning is that there's a lot of value in working code as it embodies business value. In this case nothing worked and it didn't appear to be easy to actually get anything to work in a manner that maintained the integrity of the old code, meaning that less value was going to be lost via the re-write compared to a working product. An incremental improvement would just be getting the app into a working-but-still-legacy state and the effort might have been bigger than a re-write.