pak internals

The embedded library

From version 0.7.0, the pak source repo and the pak source package includes all of the dependent packages in src/library. They are installed during R CMD INSTALL. in the compilation step. It is started from Makevars (generaged from Makevars.in), and it is implemented via the src/install-embedded.R script.

install-embedded.R istalls all embedded dependencies in the correct order, into /library in the package we are building.

Advantages

Much simpler installation. No need to download anything. Hopefully it works fine on CRAN.
It is now possible to write real tests for pak, that can also run on CRAN, if we want.
A pak version always has the same dependencies. In the past the versions of the embedded packages could be different.
Better development. No need to manually install dependencies before calling load_all().
We can install dev pak from GitHub or other sources now seemlessly, using pak itself (in another library), or remotes, or from a local git clone, etc.

Disadvantages

Starting another installation during the installation is slightly tricky. We need to unset a number of environment variables (see them in Makevars.in). Hopefully this won’t be too fragile.
The source package is bigger. Still not very big.
pak always needs a compiler now when it is being installed from source.

Undating embedded packages

The embed.R file has some tools for * listing the versions of the embedded packages, * listing the versions available on CRAN, * checking if we need to update a dependency, * updating a dependency, etc.

pkgload::load_all()
embed$check_update()
embed$update()
...

`pkgload::load_all()`

We do not embed dependencies in this case, but we create a separate library in the user cache directory. This happens from .onLoad(). .onLoad() regcognizes load_all() and calls install-embedded.R again, with a special argument, to install dependencies into the user cache library. If the same version of a dependency is already installed, then that dependency is skipped, to make load_all() fairly quick.

Cross compilation

pak supports cross-compilation now. You need to pass --host=<platform> to ./configure via the --configure-args argument of R CMD INSTALL to do cross-compilation. You also need to set up the compilers, e.g. via ~/.R/Makevars.

Currently you need to install all of pak’s dependencies (ideally the vendored ones) before cross-compiling. This is because the dependencies have to be able to load their dependencies during cross compilation. This is not great, and I am planning to improve it in the near future. In particular, PPM cannot build pak binaries now for macOS because of this, because the PPM build system only installs the declared dependencies before cross compilation, but of course it does not know about the embedded dependencies.

install-embedded.R handles cross-compoilation specially. (It has to.) It installs each dependency into its own library, and then moves them into the common library at the end. This is so that they can load their dependencies from the native library during their installation.

install-embedded.R updates the Built fields in the metadata of the embedded packages, and also for the pak package itself, after compilation.

Cross-compilation is currently used

to build the macOS arm64 binaries on x86_64 macOS (see tools/build/macos),
to build static aarch64 Linux binaries on x86_64 Linux (see tools/build/Linux/Dockerfile-aarch64 and other files there),
on PPM, to cross compile (x86_64 and arm64) macOS binaries on x86_64 Linux. (This does not work currently.)

Our repository of pak builds

R versions

We usually support at least one more R version than the tidyverse, so the last six release and R-devel. This is to give people time to update their GitHub actions.

Platforms

Currently:

Windows x86_64.
macOS x86_64.
macOS arm64 (cross compiled on x86_64 macOS).
Linux x86_64 static.
Linux aarch64 static (cross compiled on x86_64 Linux).

Hopefully soon:

Windows arm64. (The challenge is that we either need to cross-compile or have a self-hosted runner to build packages.)

Streams

Stable

Latest CRAN release, with a version number with three components.

RC

Release condidate, typically shortly before a release, with a version number with four or more components, the fourth one is 9999.

At other times it is the same as the stable stream.

Devel

Build daily from the main branch of https://github.com/r-lib/pak. It has a version number with four or more components, the fourth one is 9000 or larger, but it is not 9999 (as that would be an RC version).

Update procedure

Package binaries are built on GitHub Actions, in the nightly.yaml workflow.
They are pushed to GitHub Packages, to the pak package, in the r-lib/pak repo: https://github.com/r-lib/pak/pkgs/container/pak.
For each successful push, the manifest file on the packages branch is updated: https://github.com/r-lib/pak/tree/packages.
They are deployed to GitHub Packages, to this repo: https://github.com/r-lib/r-lib.github.io.

The `nightly.yaml` workflow

Can be also run manually, on a subset of all platforms, if needed. Othewise it runs daily: https://github.com/r-lib/pak/actions/workflows/nightly.yaml.

GitHub Packages

GitHub Pages

The challenge is that the site is very close to the 1GB limit, sometimes it is slightly over, which still works, but we won’t be able to add new platforms to it.

An alternative would be to keep the package in a repository, and use https://raw.githubusercontent.com. We would need to create a repo branch, and have https://raw.githubusercontent.com/r-lib/pak/repo as the repo URL. So the Windows and macOS binaries would go into

https://raw.githubusercontent.com/r-lib/pak/repo/bin/windows/...
https://raw.githubusercontent.com/r-lib/pak/repo/bin/macosx/...

etc.

Challenges

TODO