Many readers will be aware that an unusually high number of features
have been reverted from the PostgreSQL 19 branch after the beta period
began. Having some reverts is not unusual, maybe one per cycle could
be expected. But this time around, about 8 to 10 significant
features, depending on how you count, have been reverted, and some of
them quite late in the beta period. I have been on the receiving end
of some of that, as the developer or committer of some of those
features, and have gotten some questions about it, and so I want to
take a moment to reflect on this. I’ll try to extrapolate a little
bit to similarly affected work that I was not actively involved in,
but these are just my opinions at this point.
So what happened?
I don’t think we, meaning all the developers affected, have had enough
time to fully reflect on all this. But here are some thoughts. It’s
probably a combination of these, and a different combination in the
case of each reverted feature.
-
Maybe we just didn’t do a good enough job. Maybe the code was just
not good enough. We’ll learn and improve. It’s probably some of
that, and I’m mentioning it here to not give the impression that it
was only the other factors that I’m going to mention. -
It could also be a coincidence. Maybe we had been on such a good
roll that we attempted to tackle several features that turned out to
be too hard, or at least too hard to finish in time.
Photo: Brett Jordan / Pexels -
Some of the reverted features had significant architectural defects
that would have been hard to fix quickly and during beta. But there
was also a long tail of relatively harmless issues involving various
edge cases. In past times, many of these would not have surfaced
until much later and we would have fixed them bit by bit over the
subsequent five years. But with LLM-assisted code review, these
kinds of things can get found much faster, and then you’re staring
at a list of like forty defect reports. And even if each of them
requires only a three-line fix on average, there is a fixed overhead
to dealing with each of these, and then you run out of time. Most
if not all of the affected reverted features were written a long
time ago when LLM-assisted code review was not as useful as it is
now, meaning that the effective quality bar has been raised after
the time the feature was written and committed. Unless the
capabilities of LLM-assisted code review keep improving at the
current rate, this situation should sort itself out for future
development cycles. -
A corollary of that is that, at least for the most part, as far as I
can tell, this situation is not due to “vibe coding”. This code
was written well before that was a possibility.
-
Because of the Vulnpocalypse, many senior developers were busy
between, say, April (the start of feature freeze for PostgreSQL 19,
and also the start of the Vulnpocalypse) and August (the most recent
PostgreSQL security release) with working on fixes for security
issues. Because of that, the usual additional review and testing
that happens during the beta period was reduced, and then picked up
again in August. That’s possibly why a lot of new issues were
discovered quite late in the beta period and why so many reverts
happened so late, which is what made all of this so notable. Again,
some issues were more fundamental but there was also a long tail of
smaller issues, and I feel with more time, say, two months, these
could have been addressed (given adequate focus, but note that the
Vulnpocalypse is ongoing). But the community preferred trying to
stick to the release schedule rather than delaying to get more fixes
in. (And you never know how long a long tail really is.)
-
The complexity of the system has made some kinds of features
extremely complex to get right, and there is not enough scaffolding
in the code to support that. Consider, as an abstracted example,
some SQL-level query language feature. Does it work with domains?
How about with domains over a composite type? Where one of the
fields is also a domain? And the domain has a not null constraint?
And one of the columns was dropped and re-added? And this is part
of a table partition that has been detached and reattached with a
different column order? And this is part of a view that is called
from a security-definer function from a trigger? And so on. No one
can manually test this or even enumerate these test cases. But
LLM-assisted fuzzing can find problems with this really quickly. So
this probably needs to be part of feature development in the future.
But we should also think about making the internal interfaces more
robust so that each different combination of contexts doesn’t open
up so many new possibilities of interactions.I like to say, in PostgreSQL, “everything works with everything”.
This is part of the appeal, that you can construct cases like this
and expect them to work. I don’t want to give up on that at all.
But it does impose a significant penalty on feature development that
everyone needs to be aware of.
Photo: Brett Sayles / Pexels -
As a final point, the PostgreSQL development community has been
contemplating for some time the possibility of committing features
with some kind of “experimental” status and then maturing them in
the main tree. This would also be one possibility to address the
previous point. This would have been a helpful tool for at least
some of the reverted features, but we haven’t yet figured out a way
to do this, and some of the work suffered from misunderstandings
related to this. This is a concrete point to work on.
So what’s next?
First of all, from my personal perspective, thank you to everyone who
has reached out in person or online to express support and
encouragement. That’s the most important thing. I’m confident that
many of the reverted features will come back before too long.
PostgreSQL 19 will still be a great release. REPACK CONCURRENTLY,
logical replication of sequences (itself a previously reverted
feature), and planner hinting with pg_plan_advice, just to pick a few
headline items, will make this release very attractive. And it will
be very robust, after all the additional scrutiny.
It’s difficult to predict things now. A few months ago I said,
PostgreSQL 20 might have no new features, just bug fixes. It turned
out that things moved even faster, and PostgreSQL 19 now, kind of, has
half as many major features as then expected, but now we also already
have a few nice 80%-ready features queued up for PostgreSQL 20 that
just need those remaining 80% to get done!
