Apertis Packaging CI

Apertis stores the source of all the shipped packages in GitLab and uses a set of GitLab CI pipelines to manage the workflows to:

land updated sources to OBS which will then build the binary outputs
pull updates from upstream distributions like Debian 10 Buster

Adding downstream changes

The standard Contribution Process applies unchanged to packaging repositories, pushing changes to wip/ branches and getting them landed to the apertis/* branches via Merge Requests.

The only additional requirement imposed by the Debian packaging format is that changes outside of the debian/ folder are not allowed and would cause the source-building pipeline to fail. Check the Debian Packaging documentation to find how patches affecting code outside of the debian/ folder should be handled.

Updating debian/changelog should be done separately as the last step when issuing a release, generating the changelog entries from the Git commit log, which makes writing good commit log messages even more important.

A merge request should be submitted on GitLab for each bug or tasks. To ease the review process, in particular to avoid churn in case or rebases, it is recommended to leave the editing of debian/changelog to a dedicated merge request once all the other MRs have been landed, see the section about landing downstream changes to the main archive below.

In order to follow a release early, release often philosophy it is also recommended to avoid delaying release commits to include additional features. This gives the possibility to receive early feedback on the downstream changes.

If you still wish to edit debian/changelog for any reason, just make sure that the changelog entry you're writing has the distribution field set to UNRELEASED, using gbp dch --auto --ignore-branch to ensure the formatting is correct.

The CI pipeline will locally generate a source package for each commit pushed to the packaging repositories, which can be retrieved by browsing the pipeline artifacts. The generated sources will be versioned to indicate that they are not yet suitable for release.

With the distribution field set to UNRELEASED package sources get uploaded to the :snapshots OBS project matching the branch (that is, apertis:v2020dev0:target:snaphots when landing changes to the apertis/v2020dev0 branch of a :target package): the following section about landing downstream changes to the main archive below describes in detail how to set the distribution to land the package to the appropriate main OBS project.

Landing downstream changes to the main archive

Once downstream changes to a package are deemed ready to be published in the Apertis main archive, a proper release needs to be issued.

Push a wip/ branch updating debian/changelog
- use GBP_CONF_FILES=/dev/null gbp dch --release -D apertis --debian-branch=apertis/v2020dev0 to generate a release changelog entry summarizing all the changes already landed on the apertis/v2020dev0 branch
- ensure that the distribution field has been changed from UNRELEASED to apertis
Create a Merge Request based on your wip/ branch for the most recent release branch where you want to land your changes:
- for published stable release, the main branch (for instance apertis/v2019) should never be targeted directly but updates and fixes should go through the apertis/v2019-security or apertis/v2019-updates branches, see Process after a product release for more details
- for instance, if you want to land changes to both the development and stable releases, push your wip/ source branch and create a MR for the development one first (for instance, apertis/v2020dev0) and then, once merged, create a MR for the stable one ((for instance, apertis/v2019-updates)
Get the Merge Request reviewed and landed
The CI pipeline will then build-check the source package as usual and since the distribution field is no longer UNRELEASED it will also:
- add a Git tag for the release version to the repository
- rebuild the release source package
- store the release sources in the pristine-lfs-source branch
- upload the release source package to the main project (for instance apertis:v2020dev0:target)

If the apertis/$RELEASE-updates or apertis/$RELEASE-security branches for published stable releases do not exist yet, they should be created from the GitLab web UI since their protected status makes pushing forbidden.

For trivial changes it is also possible to combine the release commit in the same MR as the changes. Again, developers need to be careful to ensure the changelog entries are kept up-to-date when the commit messages get changed via rebase.

Pulling updates or security fixes from upstream distributions

A separate set of pipeline steps are configured on the debian/$RELEASE-gitlab-update-job branches (for instance, debian/buster-gitlab-update-job) of each package.

The pipeline will check the Debian archive for updates, pull them in the debian/$RELEASE branch (for instance, debian/buster), try to merge the new contents with the matching apertis/* branches and, if successful, push a proposed updates branch while creating a Merge Request for each apertis/* branches it should be landed on.

The upstream update pipeline is scheduled to run automatically each weekend, but can be manually triggered from the GitLab web UI by selecting the Run Pipeline button in the Pipelines page of each repository under pkg/* and selecting the debian/buster-gitlab-update-job branch as the reference.

Reviewers can then force-push to the proposed update branch in the Merge Request to fixup any issue caused by the automated merge, and ultimately land the MRs to the apertis/* branches.

In some situations the automated merge machinery may ask to PLEASE SUMMARIZE remaining Apertis Changes, and in that case reviewers should:

check out the proposed update branch
edit the changelog to list all the downstream changes the package still ships compared to the newly merged upstream package and their reason, describing the purpose of each downstream patch and of any other change shown by git diff against the debian/* branch
amend the merge commit
force-push to the proposed update branch
land the associated Merge Requests as described above

Remember to check that the updated package gets included in the next daily reference image build and wait for its QA test results to catch regressions timely and act accordingly.

Backporting updates or security fixes

Often downstream fixes, upstream updates or security fixes need to be applied to multiple active releases.

Changes should be introduced in the most recent development release where they can be tested and regression detected with little impact, following the instructions in the Landing downstream changes to the main archive and Pulling updates or security fixes from upstream distributions sections.

Once the changes have been thoroughly tested paying close attention to the QA test results, they can then be propagated to the more stable releases, where any mistake can impact the product teams using Apertis in the field.

For instance, once a fix is landed to apertis/v2021dev0 and no regressions are found in the subsequent QA test results, a MR should be create to land the changes to the stable releases.

If there is no divergence between the packages in the different releases and the backport can be done with a fast-forward, a MR should be created to submit the changes from for instance apertis/v2021dev0 to apertis/v2020-updates or apertis/v2020-security, following the Landing downstream changes to the main archive steps, choosing the destination depending on the nature and impact of the fix.

If package diverged across releases, a separate branch has to be created where the fixes are cherry-picked appropriately before creating the MR. See the Diverging release branches section for further details about versioning divergent packages.

Diverging release branches

Sometimes different downstream patches need to be applied in the different Apertis release branches. A clear case of that is the base-files package which ships the release name in /etc/os-release.

In such situation it is crucial to use different version identifiers in each branch: the version for a given package needs to be globally unique across the whole archive since uploading different package sources with the same name/version would lead to errors difficult to diagnose.

When targeting a specific release, ~${RELEASE}.${COUNTER} needs to be appended to the version identifier after the local build suffix:

0.42 → append co1~v2020pre.0 → 0.42co1~v2020pre.0
0.42co3 → bump to co4 and append ~v2020pre.0→ 0.42co4~v2020pre.0
0.42co4~v2020pre.0 → increase the release-specific counter → 0.42co4~v2020pre.1

This uses the fact that ~ in Debian package numbers sorts before anything, see the Debian Policy §5.6.12 for more details. Adding ~ is necessary so that if a new upstream version 0.42.1 or a new non-release-specific downstream version 0.42co4 is introduced, they will replace the release-specific package.

Note that dpkg considers 2020.0 to be newer than 2020pre.0, so the Apertis release identifiers can be used with no modification (if in doubt, check with dpkg --compare-versions 2020pre.0 '<<' 2020.0 && echo ok).

Adding new packages from Debian

This is the process to import a new package from Debian to Apertis:

locally create a new git repository
invoke import-debian-package from the packaging-tools repository to populate the local git repository:
- fetch a specific version: import-debian-package --upstream buster --downstream apertis/v2020dev0 --create-ci-branches --package hello --version 2.10-2
- fetch the latest version: import-debian-package --upstream buster --downstream apertis/v2020dev0 --create-ci-branches --package hello
- multiple downstream branches can be specified, in which case all of them will be updated to point to the newly imported package version
- don't use import-debian-package on existing repositories, it does not attempt to merge apertis/* branches and instead it re-sets them to new branches based on the freshly imported Debian package
Add a debian/apertis/component file reflecting the repository component it is part of (for instance, target)
- check out the apertis repository: git checkout apertis/v2021dev0
- add the component file: echo target > debian/apertis/component
- add the component file to git: git add debian/apertis/component
- commit the file: git commit -m "Add debian/apertis/component file pointing to target" debian/apertis/component
create an empty project on GitLab under the pkg/* namespaces (for instance, pkg/target/hello)
configure the origin remote on your local git: git remote add origin git@gitlab.apertis.org:pkg/target/hello
push your local git contents to the newly created GitLab project: git push --all --follow-tags origin
set it up with gitlab-rulez apply rulez.yaml --filter pkg/target/hello from the gitlab-rulez repository
- sets the CI config path to debian/apertis/gitlab-ci.yml
- changes the merge request settings:
  - only allow fast-forward merges
  - ensure merges are only allowed if pipelines succeed
- adds a schedule on the debian/buster-gitlab-update-job branch to run weekly
- marks the apertis/* and debian/* branches as protected
follow the process described in the section about landing downstream changes to the main archive above to publish the package on OBS.

Adding updates from a non-default upstream repository of a distribution

There are circumstances, when we deviate from the default upstream. This usually happens when:

Packages are not available in the default distribution repository
Packages in the default distribution repository are outdated
Newer version of package, available in the non-default repository, is needed

For example, Debian Buster ships an older version of the Linux kernel (4.9.x) than what we have in Apertis (> 5.4). In such cases, special care needs to be taken to update packages from their respective upstreams.

Below are a set of steps which can be adapted to such exception packages. Let us assume that for such repository, the package was picked from Debian Unstable instead

Clone your repository

  $ git clone git@gitlab.apertis.org:ritesh/libgpiod.git
  Cloning into 'libgpiod'...
  remote: Enumerating objects: 114, done.
  remote: Counting objects: 100% (114/114), done.
  remote: Compressing objects: 100% (85/85), done.
  remote: Total 114 (delta 18), reused 110 (delta 18), pack-reused 0
  Receiving objects: 100% (114/114), 110.99 KiB | 360.00 KiB/s, done.
  Resolving deltas: 100% (18/18), done.

  $ cd libgpiod/

Ensure you have a branch against your deviated upstream. If you are tracking changes from a deviated upstream like Debian Unstable, it needs to be ensured that the package's packaging repository has the corresponding branches available. This is needed because the automated machinery tools expect respective branches to be available.

For example, if you picked a package from Debian Unstable, ensure to have a branch named debian/unstable in your git repository.

$ git checkout -b debian/unstable origin/debian/buster
Branch 'debian/unstable' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/unstable'

$ git checkout debian/buster
Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
Switched to a new branch 'debian/buster'

Similarly, ensure to have a branch named upstream/unstable in your git repository.

$ git checkout upstream/buster
Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
Switched to a new branch 'upstream/buster'

$ git checkout -b upstream/unstable upstream/buster
Switched to a new branch 'upstream/unstable'

Ensure that these branches are pushed to your remote origin. It is important that these branches are pushed and in sync with the default remote.

  $ git push -u origin --all
  Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
  remote:
  remote: To create a merge request for debian/unstable, visit:
  remote:   https://gitlab.apertis.org/ritesh/libgpiod/-/merge_requests/new?merge_request%5Bsource_branch%5D=debian%2Funstable
  remote:
  To gitlab.apertis.org:ritesh/libgpiod.git
   * [new branch]      debian/unstable -> debian/unstable
  Branch 'apertis/v2019' set up to track remote branch 'apertis/v2019' from 'origin'.
  Branch 'debian/buster' set up to track remote branch 'debian/buster' from 'origin'.
  Branch 'upstream/buster' set up to track remote branch 'upstream/buster' from 'origin'.
  Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.

Pull in new updates using the apertis-pkg-pull-updates script, instructing it with the deviated upstream

Eg. apertis-pkg-pull-updates --package PKGNAME --upstream unstable --mirror http://deb.debian.org/debian

$ git checkout apertis/v2021dev2
Switched to a new branch 'apertis/v2021dev2'

$ ../apertis-pkg-pull-updates --package libgpiod --upstream unstable --mirror http://deb.debian.org/debian
source package libgpiod
running git branch --track -f debian/unstable origin/debian/unstable
Branch 'debian/unstable' set up to track remote branch 'debian/unstable' from 'origin'.
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
running git branch --track -f upstream/unstable origin/upstream/unstable
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-security
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable-proposed-updates
local version: 1.2-3
fetch https://qa.debian.org/madison.php?package=libgpiod&yaml=on&s=unstable
remote version: 1.4.1-4
update to 1.4.1-4
fetch https://snapshot.debian.org/mr/package/libgpiod/1.4.1-4/srcfiles?fileinfo=1
download http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
running dget --download-only --allow-unauthenticated http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.dsc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   332  100   332    0     0    449      0 --:--:-- --:--:-- --:--:--   448
100  2294  100  2294    0     0   1381      0  0:00:01  0:00:01 --:--:--  5474
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1.orig.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0    637      0 --:--:-- --:--:-- --:--:--   636
100  307k  100  307k    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k
dget: retrieving http://deb.debian.org//debian/pool/main/libg/libgpiod/libgpiod_1.4.1-4.debian.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   342  100   342    0     0    677      0 --:--:-- --:--:-- --:--:--   677
100  6132  100  6132    0     0   9922      0 --:--:-- --:--:-- --:--:--  9922
Moving branch debian/unstable to debian/, was: 713dadc
running gbp import-dsc /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc --author-is-committer --author-date-is-committer-date --upstream-branch=upstream/unstable --debian-branch=debian/unstable '--debian-tag=debian/%(version)s' --no-sign-tags --no-pristine-tar
gbp:info: Version '1.4.1-4' imported under '/home/rrs/NoBackup/Gitlab_Packages/packaging-tools/libgpiod'
running ./import-tarballs /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1-4.dsc
Importing /tmp/pull-updatesywrcg0_m/libgpiod_1.4.1.orig.tar.xz

Adding updates from distribution development repositories

This is another scenario, wherein the user may need updates which are not yet released into the Upstream Distributions's repositories.

For example, for Apertis, we may need a very newer version of libgpiod, which may not yet have been released into any of Debian development releases (Unstable, Testing). Under such cases, where the changes may only be available in the packaging repositories, we need to take extra care when pulling in such updates.

Let us assume libgpiod 1.4.2 has been available in Debian's libgpiod Packaging repository but is not released into any of the Debian releases. In such case, we can try:

Clone the remote libgpiod git packaging repository from Debian.
Generate a source package out of the packaging repository using gbp buildpackage -S
- If successful, this will give us a proper libgpiod source package.
Clone the Apertis libgpiod git packaging repostiory
- Use the import-tarballs tool to import the source package generated from the Debian repository into Apertis packaging repository. Eg. import-tarballs libgpiod-1.4.2-1.dsc
- Note: The import-tarballs script imports the new tarball into the git repository and commits it to the pristine-lfs branch. While, a user can commit to the branch manually by-hand, we recommend the use of the import-tarballs tool to import new tarballs and commiting them to the packaging repository

License scans

As merge requests to packaged software are submitted, the CI pipeline performs license scans on the package. The scans are performed on all files in the package, not just the new submission. The pipeline fails or emits a warning (depending on the configuration) when if finds files with unknown or unclear licensing terms, or files under licenses not allowed in the package. When such situation arises, it is the responsibility of the submitter to perform the review of the license scan results and make updates to the package if necessary.

When the license scan mistakenly identifies a file as being under an incorrect license, or fails to process it correctly, there are three ways to fix this:

Specify the correct copyright and the license in debian/apertis/copyright.yml. The format of the file is specified in the Dpkg::Copyright::Scanner manpage. In short, it’s a YAML file mapping paths to their licensing information:
```
debian:
  copyright: 2015, Marcel
  license: Expat
src/:
  copyright: 2016, Joe
  license: Expat
.*/NOTICE:
  skip: 1
src/garbled/:
  'override-copyright': 2016 Marcel MeXzigue
```
Patterns follow the Perl regular expression rules.

Please also verify debian/copyright specifies the correct license, and if it doesn’t, submit a patch to Debian.
Add the file to the list of ignored files. debian/apertis/copyright.whitelist is formatted the same way as gitignore, please refer to the gitignore manpage for more information.
If the file is under a license not suitable for Apertis, it can be removed from the package by either repackaging the tarball or patching it out, in which case the scanner will not take it into account.

The license scanner will store the automatically generated copyright report file under debian/apertis/copyright, updating the merge request when necessary.

Internals

Main components:

ci-package-builder: centralized location of the GitLab-to-OBS and Debian-to-GitLab pipeline definitions
debian/apertis/gitlab-ci.yaml: imports the ci-package-builder pipelines from each packaging repository
apertis-package-source-builder: Docker environment for the GitLab pipelines
pristine-lfs: stores upstream original tarballs and packaging source tarballs using Git-LFS, as a more robust replacement for pristine-tar

Branches:

pristine-lfs: stores references to the Git-LFS-hosted original tarballs
debian/$DEBIAN_RELEASE (for instance, debian/buster): contains the extracted upstream sources and packaging information from Debian
pristine-lfs-source: stores references to the Git-LFS-hosted packaging tarballs, mainly to ensure that each (package, version) tuple is built only once and no conflicts can arise
apertis/$APERTIS_RELEASE (for intance, apertis/v2020dev0): contains the extracted upstream sources and possibly patched packaging information for Apertis, including the debian/apertis/gitlab-ci.yaml to set up the GitLab-to-OBS pipeline
apertis/$APERTIS_RELEASE-security and apertis/$APERTIS_RELEASE-updates (for intance, apertis/v2019-updates): similar to ``apertis/$APERTIS_RELEASE` but respectively target the Security and Updates repositories for published stable releases as described in Process after a product release
debian/$DEBIAN_RELEASE-gitlab-update-job (for instance, debian/buster-gitlab-update-job): hosts the debian/apertis/gitlab-ci.yaml file to configure the Debian-to-GitLab pipeline

Tags:

debian/*: tags for Debian releases in the debian/* branches
apertis/*: tags for the Apertis releases in the apertis/* branches

Git:

Gitattributes is a feature that enables modificaitons to be automatically made by git when certain operations are done (such as checkout, check-in and clone) to enforce certain project standards. but this has the potential effect of modifying files and that's not what we want when packaging. More details about gitattributes.

Many upstream projects ship a .gitattributes file in their source repositories, which are tightly tied to their development workflows. At times, certain settings in .gitattributes can lead to problems in our packaging workflow, as summarized above. Thus, it is advised to execute the following command, which will override any such inherited settings from the .gitattributes file:
1

echo "* -text -eol -crlf -ident -filter -working-tree-encoding -export-subst" > .git/info/attributes
- This will override the settings in the .gitattributes file from the repository, for all files and for the mentioned features