VOOZH about

URL: https://phabricator.wikimedia.org/T380127

⇱ ⚓ T380127 [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble

Maniphest T380127

[builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble
Open, In Progress, MediumPublicFeature

Description

The Ubuntu 2024.04 noble base would allow access to newer apt packages. Heroku themselves uses a system of opt-in upgrades to transition from one stack to another. We do not currently have a mechanism for doing something similar. Making a direct move from Ubuntu 2022.04 to Ubuntu 2024.04 should probably require some mechanism for folks to test rebuilding their images while preserving the ability to run the prior image.

https://github.com/heroku/cnb-builder-images/tree/main/builder-24

Things that changed/testing (WIP):

  • Apt buildpack was replaced by deb-packages T394466: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack
  • Procfile changed behavior, no extra arguments are passed anymore at all (can't pass custom args at runtime to procfile defined entries) - still testing to make sure
  • Golang and procfile now behave correctly (procfile can override golang defined processes)
  • Test php (+nodejs)
  • Test ruby (+nodejs)
  • Test python (+nodejs)
  • Test nodejs
  • Test static
  • Replace injected dotnet with built-in dotnet (test compatibility with existing dotnet project)
  • Test java/scala - jvm
  • Test clojure injected buildpack for compatibility
  • Poetry + requirements.txt compatibility (I think it will not allow poetry.lock + requirements.txt at the same time)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions

Much looking forward to this change, thanks so much for doing it! Can't wait :-)

Comment Actions

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/775

builds-api: bump to 0.0.190-20250512093109-f2cdc829

Comment Actions

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/914

components-api: bump to 0.0.141-20250811094820-170ea067

Comment Actions

@dcaro Is it possible to test this new heroku stack? I'm trying to migrate to openjdk 25, it's supposed to work with heroku-22 but it doesn't, so I'm hoping it would work with the new one.

EDIT: I tried to add but it seems to still pick heroku-22 build runner.

EDIT2: no, it uses heroku-24 but I still cannot use jdk 25, is it possible to refresh the heroku images? Here's my error:

[step-set-stack-toml] 2025-09-23T18:19:40.402757880Z Using run image: tools-harbor.wmcloud.org/toolforge/heroku-runner:24
[step-detect] 2025-09-23T18:19:43.665626150Z 3 of 6 buildpacks participating
[step-detect] 2025-09-23T18:19:43.666898836Z heroku/jvm 6.2.0
[step-detect] 2025-09-23T18:19:43.666946250Z heroku/maven 6.2.0
[step-detect] 2025-09-23T18:19:43.666960346Z heroku/procfile 4.2.1
[step-build] 2025-09-23T18:19:44.380870594Z
[step-build] 2025-09-23T18:19:44.380966600Z ## Heroku OpenJDK Buildpack
[step-build] 2025-09-23T18:19:44.380985154Z
[step-build] 2025-09-23T18:19:44.385525182Z - OpenJDK version resolution
[step-build] 2025-09-23T18:19:44.385565043Z - Using version string provided in system.properties
[step-build] 2025-09-23T18:19:44.390233334Z - Done (finished in < 0.1s)
[step-build] 2025-09-23T18:19:44.397278222Z ! ERROR: Unsupported OpenJDK version
[step-build] 2025-09-23T18:19:44.397344059Z !
[step-build] 2025-09-23T18:19:44.397363553Z ! The OpenJDK major version 25 you specified in your system.properties file is not supported.
[step-build] 2025-09-23T18:19:44.397378698Z ! Please specify a supported major version in your system.properties file.
[step-build] 2025-09-23T18:19:44.397388712Z !
[step-build] 2025-09-23T18:19:44.399329026Z ERROR: failed to build: exit status 1

EDIT3: Sorry for the noise, I created T405415 for the problem above.

Comment Actions

I am not sure whether my comment is related to this, but my campwiz-nxt build (which used to be build using go 1.24 properly) suddenly is broken (See the logs using ). it is showing that heroku/go:0.1.13 is being used which only supports upto go1.21. Before the Push-and-Deploy, I used to build the backend using go1.24. I am getting the following:

[step-detect] 2026-02-06T17:56:14.471730657Z 2 of 4 buildpacks participating
[step-detect] 2026-02-06T17:56:14.471846552Z heroku/go 0.1.13
[step-detect] 2026-02-06T17:56:14.471863050Z heroku/procfile 2.0.2
[step-restore] 2026-02-06T17:56:03.647139396Z 2026/02/06 17:56:03 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-build] 2026-02-06T17:56:15.253735558Z 
[step-build] 2026-02-06T17:56:15.253795469Z [Reading build configuration]
[step-build] 2026-02-06T17:56:15.261732071Z Detected Go version requirement: =1.24.1
[step-build] 2026-02-06T17:56:15.262765305Z 
[step-build] 2026-02-06T17:56:15.262802661Z [Error: Heroku Go Buildpack version resolution error]
[step-build] 2026-02-06T17:56:15.262807802Z Couldn't resolve go version for: =1.24.1
[step-build] 2026-02-06T17:56:15.264415401Z ERROR: failed to build: exit status 1
Comment Actions

Merged the wrong way :facepalm:

Comment Actions

I propose having a process like:

  • Regular operation:
    • default gives version X
    • gives X+1
    • there's no
  • We announce that the buildpacks are going to be upgraded and give a date so people can start testing with (say 2 weeks?)
  • 2 weeks later we roll out a new version:
    • is enabled and gives version X
    • default gives X+1
    • uses X+2
    • we send an announcement that the deprecated version is available but will go away in a month, we show the warning message (see below).
  • A month after:
    • goes away
    • default gives X+1
    • gives X+2

The key point of adding and removing the (or failing when passed) is there to avoid people on relying on that one so in regular operation they use the default, so on the next upgrade, they still have a buffer to migrate instead of start failing right away.
If they were using all the time, on the next upgrade they will probably ignore the warnings and start trying to migrate only when the image for that flag went away (and the build started breaking without possibility of buffer).

The warning would be something like:

Warning: using the deprecated versions will stop being supported on XX-XX-XX, please adapt to use the default versions before that time or you will not be able to rebuild your tool.
Comment Actions

I propose having a process like:

Looks good to me!

Comment Actions

Sent the email to announce: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/EMYTA32EV2V5SQ2JIEOD2CL66YFIZEKV/

Will start working on testing the newer versions (that will be the in two weeks).

aputhin changed the task status from Open to In Progress.Tue, Mar 17, 5:10 PM
Comment Actions

@dcaro, I see the version of heroku/deb-packages is still v0.1.3 from April 2025. Is there any chance we could get that bumped up to the latest v0.3.0? The big win there would be support for alternate apt repos.

Comment Actions

@dcaro, I see the version of heroku/deb-packages is still v0.1.3 from April 2025. Is there any chance we could get that bumped up to the latest v0.3.0? The big win there would be support for alternate apt repos.

That will happen in two weeks yep, the default version (when no flags passed) will be the current and the will be the one at the version 0.21.5 of the builder (pulled a week or so back, has the deb packages 0.3.0, the one you ask for yep ;) ).

Comment Actions

Due to T420547, the release of the newer buildpacks has been pushed forward (already done), so @bd808 you can get the latest heroku/deb-packages already with , let me know if you have any issues ;) (will become the default in a month and one week)

Comment Actions

Due to T420547, the release of the newer buildpacks has been pushed forward (already done), so @bd808 you can get the latest heroku/deb-packages already with , let me know if you have any issues ;) (will become the default in a month and one week)

👍

tools.bd808-test@tools-bastion-14:~$ toolforge build start --use-latest-versions --image-name perl --envvar BP_LOG_LEVEL=DEBUG https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion
...
[step-detect] 2026-03-19T20:56:17.047208365Z 3 of 6 buildpacks participating
[step-detect] 2026-03-19T20:56:17.047319124Z heroku/deb-packages 0.3.0
[step-detect] 2026-03-19T20:56:17.047406505Z heroku/python 6.0.1
[step-detect] 2026-03-19T20:56:17.047506495Z heroku/procfile 4.2.1
...
Comment Actions

All of the cluebot tools are running with the latest builder image (also in CI), the only small issue that came up was opcache wanting an existing path with the newer version of php, otherwise everything is working as expected.

Comment Actions

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1182

builds-builder: bump to 0.0.143-20260324172359-c67d1d81

Comment Actions

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1183

builds-api: bump to 0.0.210-20260326093139-b2f70662

Comment Actions

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1188

components-api: bump to 0.0.187-20260401094330-1b5c8837

Comment Actions

Default buildpack version now is the newer one, latest has a slightly newer one (0.21.7)

Comment Actions

The flag is available, and will start failing on the 27-04-2026, so all tools should have migrated by then.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits