Heptapod 0.29 released and general development news

Posted on Thu 24 February 2022 in announcements by Georges Racinet

The release last week of Heptapod 0.29.0 provides a good opportunity to look back at what happened since the previous of these announcements and point out the subjects that we will focus on in the next development cycles. Since then we also released Heptapod 0.29.1 to include security updates from upstream GitLab.

The major advances made since Heptapod 0.26 are about native Mercurial support and commercial CI/CD with Clever Cloud for heptapod.host. These are explained in details in this post, but first things first…

Heptapod 0.29

Heptapod 0.29 brings in GitLab 14.6, exposes for the first time the migration of Mercurial legacy projects (with Git conversion) to native, and removes the intermediate HGitaly1 running mode of Mercurial native projects (more on the native Mercurial topic below).

Heptapod 0.29.0 was installed on February 22nd on foss.heptapod.net and heptapod.host. The follow-up 0.29.1 was installed on February 28th.

Heptapod 0.29.1 can be installed as a Docker image and from source.

The full changelog is available as usual alongside the sources.

Many thanks to all the people involved to make this happen!

Mercurial native projects

Early Heptapod versions were exposing all Mercurial content to the GitLab web application by converting it internally to Git. This was a great move to quickly evaluate the potential of such a solution, with two obvious drawbacks:

  • most commit identifiers (hashes) in the Web user interface were Git commit hashes, not obviously related to what users can see in their local repositories.
  • the conversion to Git is an unacceptable overhead.

Since then, we've introduced "native" Mercurial projects, which don't have the first problem, because the commit identifiers handled by GitLab are Mercurial changeset hashes, but still rely on the Git conversion for some operations.

To this day, Heptapod still supports the first way of doing things, in the form of Projects with the "Mercurial (legacy with Git conversion)" VCS type.

Much more background on this is given in The road to fully native Mercurial in Heptapod. It boils down to this:

  • our plan is to gradually improve how much native Mercurial projects truly are native, up to the point where Git is not involved at all.
  • The minimal viable support level for native Mercurial projects is called "HGitaly1" (introduced in Heptapod 0.17).
  • the only heavy data migration occurs while making a Project native. In other words, there are no persistent references to Git content in the database.

In the announcement following the 0.26.0rc1 release, we were saying that the internal infrastructure was in place to drop the conversion of Mercurial content to an internal auxiliary Git repository, and therefore that Heptapod 1.0 was in sight.

Here is a summary of what happened on that front in the releases since then:

  • 0.27: this release was focused on unrelated matters (leap to GitLab 14.4 and Mercurial push-mirror feature).
  • 0.28: the HGitaly1 mode was disabled by default.
  • 0.28: exposition of the mode without Git, with explicit activation by feature flags.
  • 0.28: major progress in the migration to native Mercurial (runnable by systems administrators only).
  • 0.28: relabeling of VCS Type labels, so that a "Mercurial Project" means a native Mercurial project.
  • 0.29: removal of the HGitaly1 mode.
  • 0.29: the migration to native Mercurial is now exposed in the REST API, hence available to all Project Maintainers (caution advised).

We also made a few interesting experiments in that time frame. For a very long time, we've been stating that Heptapod was able to host projects in the 100k changesets ballpark. The main reason for that was precisely the conversion to Git. For instance, the import of PyPy (about 100k changesets) into foss.heptapod.net took more than 4 hours.

As soon as there was a way to stop it, we've turned to one of our favorite projects for Mercurial scalability measures: mozilla-central (around 600k changesets). Unsurprisingly, its import to a development Heptapod instance took about half an hour, same time as a hg clone not involving Heptapod. In the same vein, importing the repository used by Octobus for development of Mercurial itself (about 51k changesets) only takes about one minute.

Builtin CI/CD on heptapod.host

In the time frame between Heptapod 0.26 and 0.29, we've devoted some time to work on Heptapod PAAS Runner, and more specifically on its Clever Cloud variant.

The end goal was to provide a builtin ready-made CI/CD as part of our commercial offering on heptapod.host.

Of course, users of heptapod.host have always been able to host their own Heptapod Runners and register them at the Project or Group level, and that is still possible.

But we knew that it wasn't enough. Not everybody has the will to maintain the needed infrastructure, be it made of virtual machines or bare metal. Moreover, in many concrete cases, the fixed costs that are incurred cannot be justified by the actual running time of the CI/CD.

The end result is that today, all that users of heptapod.host have to do to get the CI/CD to work is to activate the shared runners for their Groups or Projects.

Billing happens by the second and the size of the running VMs (also known as "flavor") can be specified through job variables.

More detail (and examples!) are available in our dedicated page, Clever Cloud Runner documentation.

Other changes occurring since Heptapod 0.26

It has become routine, but it wouldn't be fair to the amount of work involved not to mention upstream GitLab version updates: we went from GitLab 14.2 in Heptapod 0.26 to GitLab 14.6 in Heptapod 0.29.

Another interesting feature is the Mercurial push mirror capability, introduced in Heptapod 0.27. It can be used for instance to position Heptapod as an upstream working area, pushing automatically to an existing canonical repository.

Development areas in the next few releases

Of course we will continue to keep Heptapod up to date with respect to its upstream, with GitLab 14.7 required for March 22nd and GitLab 14.8 for April 22nd.

Native Mercurial

The fact that we are still depending on Git is the thing that prevents us from claiming the 1.0 version.

As of Heptapod 0.29, the only feature that we are lacking to stop by default the conversion to Git for native projects is the programming languages statistics. As was already mentioned in the Heptapod 0.26 announcement, we'll have to develop a Mercurial extension providing the same functionality as GitHub Linguist for this.

We will probably keep on opening up the migration of existing projects to native Mercurial, by providing a Web user interface for it in one of the two next releases.

Better support of drive-by contribution

We will start working soon on the plan to allow contribution without being granted the Developer role.

The plan relies on implementing topic namespaces, in the topic extension itself, and mapping them automatically to Heptapod users, thus creating the notion of personal namespaces.

People with the Reporter role will be allowed automatically to push on topics in their personal namespaces, and create Merge Requests from them. By default, all authenticated users have the Reporter role on public projects.

Mercurial 6.1

Including Mercurial 6.1 in Heptapod will have a major positive performance impact on projects with lots of active branches.

Current testing on Mercurial 6.1rc0 indicates that it will work right away: all our internal and end-to-end tests are already passing. Unless there is a nasty surprise, Mercurial 6.1 will be included in Heptapod 0.30, which is due on March 22nd.