FreeBSD updating a port: Twisted python

Introduction

FreeBSD is a wonderful Operating System, one of its many interesting ideas is a division between the base system and other pieces of software, which are called ports.

Since FreeBSD is an operating system, this also means that there are exactly two source trees or repositories: the main tree (base system) and the ports tree.

The base system is developed together and therefore is kept consistent, whenever an API or a behaviour changes, all pieces of the base system must be updated as well.

This makes changes somewhat seamless and easy to deal with.

On the other hand, ports provide a description of how to compile third-party software under FreeBSD and how to install it manually or how to build an installable binary package.

However, FreeBSD development, including maintaining the ports tree, is a voluntary-based effort; which means sometimes it lacks human power.

Being a good open source user sometimes goes through being aware that people’s time is limited, and helping out when things fall through the cracks.

Here I document a bit how the general process of updating a port you detect as being outdated is.

Table of Contents

Communication and coordination

Contrary to “common wisdom” in the industry, it is not “the technical bits”, that is difficult when trying to help with a project; it is communication and coordination that are difficult things when many people are involved. And nothing big or useful enough gets built and maintained without people.

In a nutshell, when interacting with a project of this kind:

• Be kind
• Respect and appreciate other people’s time and energy
• Don’t assume incompetence or unwillingness, when it can be explained by lack of resources (usually: time, human, money, infra)
• Make sure you research before demanding time or effort from people

Interestingly, the FreeBSD project has the communication and coordination bits mostly quite well-defined.

Documentation

FreeBSD‘s documentation is really good, but there are multiple places to look for information, depending on the nature of the information.

• FreeBSD Handbook: General system information and administration
• Man pages: Information about specific bits of the system
• FreeBSD wiki: Information that is prone to change often

Chat – IRC

Following list of channels on irc.freenode.net are a great place to start, there are usually people who will point you out to the right resource, the right person or the right place to deal with things.

Just make sure you are patient since not everyone is online always (timezones!) and express your question in as much of an accurate fashion as possible.

• #freebsd: For general OS questions and support
• #freebsd-ports: For questions more specific to maintenance of the ports tree
• #freebsd-dev: For questions more specific to development of the OS

Bugzilla – Problem Reports (PRs)

FreeBSD’s Bugzilla is used to track problems with the base system or ports and also to document what is being worked on by whom.

It also serves a special purpose with the ports workflow: when a change is proposed to a specific port, if there is no feedback from the port maintainer within a reasonable time, that change times out and can be accepted and merged by anyone with permissions (commit bit) to the ports tree.

Phabricator – Code Reviews

Code reviews are wonderful. FreeBSD has a Phabricator instance that is used for precisely this purpose.

It requires getting used to it, but Phabricator is a very reasonable piece of software and the review workflow is incredibly useful.

Don’t be afraid, most people who might review your code will be kind and it means that the end result will be better.

Live example: Updating Twisted python

As of this writing Twisted python’s port is at version 18.9.0, because of the way Twisted manages versions, this means it dates back to September 2018.

In between, Twisted has gotten significantly better and has had some security fixes, as recently as a couple weeks ago: March 2020.

So we really should update the port to version 20.3.0.

Finding out more about the port

A good place to find out about ports is www.freshports.org.

If we go there and search for twisted, we’ll come to the port’s page.

Amongst other things, it tells us that the port is a build dependency for 4 other ports and a run dependency for 55 ports.

And it also tells us a very interesting bit, the full name of the port is: devel/py-twisted.

Each port has a main category, which is then followed by the port name, separated by a /.

Indeed, this is where we will find the code defining the FreeBSD port.

Telling people we’ll be updating the port

First things first, we search on FreeBSD’s Bugzilla for open Problem Reports (PRs) relating the port we want to update. Maybe someone is already working on it or there are subtleties that escape our uninitiated knowledge (like dependencies, external blockers, …).

In this case there are none, so we open a PR informing that we’ll be updating this port.

This is done on FreeBSD’s Bugzilla by filing a new bug against the Ports and packages “product”.

There is a very good write-up about this by koobs in the FreeBSD wiki.

Very important bits here are how to write the Summary field and how to manage the Flags of the PR.

In particular, this will be our Summary:

devel/py-twisted: WIP: Update to 20.3.0 (includes security updates)

Severity, hardware and OS are not that important here, so we leave the defaults.

Now, something will do magic and this PR will be assigned to the maintainer of the port, in this case that’s python@freebsd.org, which means there is no particular person, but a general Python team maintaining the port.

Let’s tell them about our intentions, this will be the body:

Hello, I noticed that FreeBSD's twisted version is a bit outdated, so I'll
try taking a go at it.

It is not entirely impossible that while at it I'll have to update other ports
like attrs, we'll see.

I'm also using the chance to document the whole process for myself (my future
self too) and hopefully people who would consider doing things like this.

Will be posting a patch over the next few days.

Cheers,

This way, if someone came wanting to do they same thing, they’d ask before if we are still working on it, and we won’t duplicate efforts.

Or maybe someone knows something we don’t, so they will use the chance to tell us in our PR #245252.

Getting the port tree

This can be done in multiple ways, the most convenient one is getting the code from the same repository that is used by developers over anonymous SVN:

svn checkout svn://svn.freebsd.org/ports ports

Another one is portsnap, which is in the base system, but instead of explaining that, I’ll go a bit into my favourite: poudriere.

Poudriere

Poudriere makes it simple to manage a pkg repository and to build customised packages, it also makes it simple to work on the ports tree.

There is a wonderful Poudriere guide on the FreeBSD wiki, so I’ll just do the TL;DR:

# Install poudriere from pre-built ports with pkg
> pkg install poudriere
# Create a builder jail for amd64 based on 12-1-RELEASE
> poudriere jail -c -j 121-amd64 -v 12.1-RELEASE
# Only if you have something under /usr/ports and are completely sure you never
# messed with it, clean that up with:
# rm -rf /usr/ports
# Create the ports tree
> poudriere ports -c -p default
# Or if you prefer to have it somewhere else:
# poudriere ports -c -p default -M /poudriere/ports/default

Testing that the port builds currently

Before making any modifications, let’s test that the port builds as it is, to discard that there is an issue somewhere else.

For that we can execute:

poudriere bulk -j 121-amd64 -p default devel/py-twisted

Now Poudriere will build the devel/py-twisted port and everything it depends on. The first time this will take a while, but afterwards only the ports that have changed will be rebuilt.

Actually updating the port

Since this is a Python port, the Python Ports Policy has precedence over the Porter’s handbook, both are worth a read.

We go to the place where the ports tree is located and navigate to the devel/py-twisted subdirectory.

All ports have at least following structure:

• Makefile: Describes what is going to be built and how
• distinfo: Contains the checksums of any needed external files (like source code!)
• pkg-descr: Is what users see when they search for this port/package.

Since Twisted’s is a simple port, we just have to edit its Makefile:

Changing the version

The most obvious change is going to be… The version, that’s following diff:

 PORTNAME=  twisted
-PORTVERSION=   18.9.0
-PORTREVISION=  1
+PORTVERSION=   20.3.0
 CATEGORIES=    devel net python

We change the PORTVERSION and since there now is no PORTREVISION, because this is a new version, we remove that line.

This might be enough, but also, Twisted is not an island, it has dependencies, and they might have changed in between.

Dependencies

Port dependencies are also listed in the port’s Makefile, in particular in the BUILD_DEPENDS variable.

How we update this depends a bit on how comfortable we are with the project’s tooling.

A perfectly valid option could be just to eyeball the code from a web interface, since I contribute to Twisted when time allows, I have the code locally and can use following:

git diff twisted-18.9.0..twisted-20.3.0 -- src/twisted/python/_setup.py

Which tells me exactly how the dependencies have changed between these two versions. Resulting in following diff:

 BUILD_DEPENDS= ${PYTHON_PKGNAMEPREFIX}constantly>=15.1:devel/py-constantly@${PY_FLAVOR} \
-       ${PYTHON_PKGNAMEPREFIX}attrs>17.4.0:devel/py-attrs@${PY_FLAVOR} \
+       ${PYTHON_PKGNAMEPREFIX}attrs>19.2.0:devel/py-attrs@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}hyperlink>=17.1.1:www/py-hyperlink@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}incremental>=16.10.1:devel/py-incremental@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}PyHamcrest>=1.9.0:textproc/py-pyhamcrest@${PY_FLAVOR} \
@@ -23,7 +22,7 @@
        ${PYTHON_PKGNAMEPREFIX}zope.interface>=4.4.0:devel/py-zope.interface@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}Automat>=0.3.0:devel/py-Automat@${PY_FLAVOR}
 RUN_DEPENDS:=  ${BUILD_DEPENDS}
-TEST_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}service_identity>0:security/py-service_identity@${PY_FLAVOR}
+TEST_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}service_identity>=18.1.0:security/py-service_identity@${PY_FLAVOR}

 USES=      python tar:bzip2
 USE_PYTHON=    autoplist concurrent distutils

We might need to update devel/py-attrs and sercurity/py-service_identity as well. Time will tell.

Since the distfiles for the port have changed, because we now are building a different version, we also have to update the checksums file.

This is easily done by running:

make makesum

Which downloads all needed distfiles, calculates the checksums and updates the corresponding file.

Building the new version

Is the same as before our changes, Poudriere will notice that the port’s definition has changed and trigger a rebuild:

poudriere bulk -j 121-amd64 -p default devel/py-twisted

Turns out, devel/py-attrs is already at version 19.3.0, and security/py-service_identity at version 18.1.0 both of which satisfy Twisted’s requirements. So the port builds without issues \o/.

Time to party now? Well, not quite. We still have to make sure the port still works as intended.

Testing the port

We can test the port with make test, which runs upstream’s testing suite.

But also, if we are caring enough to update this port, odds are that we are using it in way or another, that’s the best kind of test.

In my case, I’ll also keep developing my custom software, which depends on Twisted, but using the new version.

Submitting the patch

After we have tested the newly built port, we should send the patch; this is done by adding an attachment to our PR with the diff (obtained by svn diff).

For a more complex patch than this one, we might want to use the Code Review flow as well as this one, and keep the patch in Bugzilla updated as it evolves.

This is because Bugzilla is where maintainer timeouts and triage happen and in general things are better tracked.

When we add the attachment, we’ll request maintainer feedback by setting the flag with the same name to ?, the patch keyword will be added automatically, and we’ll also remove WIP: from the PR’s title and add the security keyword.

To make things easier for maintainers / reviewers / ports security team, we’ll also add more information in the comments:

Changelog:
  https://github.com/twisted/twisted/blob/twisted-20.3.0/NEWS.rst

QA:

  * portlint: OK (looks fine.)
  * testport: OK (poudriere: 3.3.3, amd64)


Related Security issues:

  CVE-2020-10108
  CVE-2019-9512
  CVE-2019-9514
  CVE-2019-9515
  CVE-2019-12387
  CVE-2019-12855
  https://twistedmatrix.com/trac/ticket/9420

And now someone will hopefully check our PR soon and commit it.

Conclusions

Sometimes the quickest way to have something be done is doing it yourself :-D. This doesn’t even take that long, so get into the habit of considering updating ports and proposing patches yourself.

Also, these things have effects, e.g. I know for a fact that Matrix Synapse depends on Twisted! This was probably being a blocker :-p.