Heptapod 0.17.0rc1 released with 3 tech previews

Posted on Mon 09 November 2020 in announcements

Another exciting development cycle is coming to an end with the release of Heptapod 0.17.0rc1 today, featuring GitLab 13.4 and three different technology previews. Many thanks to our contributors!

Heads up: all projects get migrated to the hashed storage. Major changes in installation from source, see INSTALL_HEPTAPOD.

A lot of progress was made in the build and packaging department. There is in particular:

We'll have no less than three different technology previews in Heptapod 0.17: Git projects, Mercurial native projects and Omnibus deployments.

Build and Packaging

Omnibus and Docker

While Heptapod 0.16.0 allowed us to have our own Omnibus build, and hence in particular to ship modified or new static assets (see heptapod#338), we were still building the final Docker image as a modification on top of that.

In the 0.17 cycle, we followed up on that initial effort, to the point where the final Docker image is the one generated by Omnibus. Since that image is essentially a preinstallation of the Debian pacakge generated by Omnibus for Ubuntu 16.04, this means in particular that

  • we finally have a viable Omnibus Debian package for Ubuntu 16.04. This is the missing all-in-one installation option for those of you that would prefer not to use Docker yet won't go all the way to manage an installation from source, but it's hardly tested outside of the quite special Docker environment. That makes our first technology preview.

    We are currently looking at options for proper distribution of the Omnibus Debian package. Meanwhile, please contact us directly if you want to give it a try. Also don't hesitate to tell us if you're interested in other Linux distributions.

  • the configuration of Heptapod specific services (hg-inner-http and hgitaly) can be managed like other GitLab services, with /etc/gitlab/gitlab.rb or Docker environment variables and gitlab-ctl reconfigure

  • our final Docker image is smaller by almost 200MB (uncompressed).

  • the whole process is much more automated. We're close to the point where pushing a tag in the main Heptapod repository (Rails application) would go all the way to produce the Omnibus package and Docker image and run the final functional tests on top of that. This gives us more time to focus on the development itself.

Version numbers policy for components

With Heptapod 0.17, we will be stopping to attribute the general version number to all our components, such as Heptapod Workhorse or py-heptapod. It was convenient for a while, allowing us to move forward without asking ourselves premature questions, but lately it had become mostly a drag.

This will save us lots of time, meaning we'll have more for features and bug fixes.

See heptapod#352 for full details of the new version policy.

Git support

That will be the second technology preview in Heptapod 0.17.0:

New project page, showing the VCS type selector

Users can now select a VCS type on the "New Project" page

Overview of a Git project, showing the VCS type

Projects using a non-default VCS type now display it explicitely, as shown on this Git project.

This was the second most wanted feature in our early user surveys, but we knew it wasn't a low hanging fruit: the tumultuous youth of Heptapod entailed lots of hacking away Git-specific code.

On January 14th, we added a new vcs_type field to Projects in the database schema, and since then, each refactoring, each conflict happening while merging upstream has been an opportunity to introduce conditionals based on that field.

Needless to say, we've also had lots of direct refactorings to detangle the Mercurial and Git repository handling code. As a result, there wasn't so much lacking in Heptapod 0.16 to reinstate Git projects: some UI independence, explicit testing and subsequent debugging.

Of course, this is all very fresh, and will need lots of testing to get past the "technology preview" stage. This is what pre-release versions are for, and we are actively counting on our user community to give feedback.

There is a new application setting in the Administration area to control admissible VCS types for new projects, with Git being allowed by default. What do you think of that choice? Don't hesitate to voice your opinion!

New project page, showing the VCS type selector

In an expected, yet still very exciting way, this "multi VCS" independence will also allow us to introduce incrementally the Mercurial native mode (visible in the screenshot above), and that brings us to our third foundational technology preview.

Mercurial native mode

Heptapod 0.17.0 will also have for the first time the option to create a project in Mercurial native mode. That is our third technology preview.

It is very experimental at this point, we expect many basic features not to work. That's precisely why we are shipping it: to assess what is further needed. People are very welcome to try, just brace yourself, work on a testing server, and read below to get your expectations right and about error reporting.

Refinements on the HGitaly plan

We've had steady progress in the HGitaly project, which is the cornerstone of the native Mercurial mode, i.e., GitLab handling Mercurial repositories without any conversion to Git happening under the hood.

First, we've make the plan more concrete by laying down three milestones for integration of the Mercurial native mode. These are not tied to Heptapod version numbers, because we have to bump the minor version number (the "y" in "x.y") when there are significant changes, and some of these changes are tied to an external schedule, namely upstream GitLab releases.

HGitaly will be shipped for the first time with Heptapod 0.17 and unconditionally serve archive requests (heptapod#204). Since this is quite independent from the rest of the application, this will serve as a general warm-up.

User exposition of the Mercurial native mode

In the user experience, Mercurial native projects will be treated as if they were under different version control system: a new option beside Mercurial and Git at creation time, and explicit display on the Project home page etc.

If you want to try the native mode, you'll have to activate "Mercurial (native)" in the new Application setting for VCS types. Then users can create Mercurial native projects.

Of course, eventually, we'll only have one type of Mercurial project, and that will be the native one. But we still have a long way to go for that.

What is supposed to work in 0.17.0rc1

  • push and pull should be the same as for regular Mercurial projects
  • Mercurial commit hashes only, everywhere in the web application (UI and API). This is the main goal of the HGitaly1 milestone.
  • basic project browsing, with some broken pages. For example, at the time of this writing, file lists ("trees") but commit and file ("blobs") views are still broken.
  • the inner mechanism that notifies the main application of Mercurial changes has not been adapted yet. Therefore, do not expect Merge Request detection and pipeline creation to work. We have a clear plan (heptapod#364) for that, and expect it to be ready for the final 0.17.0 release.

How to report broken pages or requests

First, thanks for wanting to give us feedback on this experimental mode. Too much issue duplication would be counter productive, so here's what you can do about that.

The majority of problems are expected to be missing HGitaly method implementations. They come with explicit logs in the Rails application:

Gitlab::Git::CommandError (12:Not implemented. Tracking issue: https://foss.heptapod.net/heptapod/hgitaly/-/issues/15.):
  lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'

and in the Mercurial log:

[2020-11-09 11:11:58 +0100] [793335] [ERROR] [hgitaly.errors] Not implemented. Tracking issue: https://foss.heptapod.net/heptapod/hgitaly/-/issues/15

You can then add the use case you've stumbled over to the tracking issue if needed, and upvote it if you feel it to be more important that other similar issues.

What will still be missing in the final 0.17.0

  • Migration scripts for existing (hg-git based) projects
  • We will still be using an auxiliary Git repository to present some content. Removing this need will be the goal of the Hgitaly2 milestone.
  • Potentially everything that's not working in rc1 and we didn't had time to fix before the final release.

In conclusion

This has been again a very busy and exciting development cycle.

Many thanks to all the people that helped making this happen, be it through code contributions or financial support. It is much appreciated.