Building Singularity images in layers

Recently I have been learning about Singularity and have been experimenting with building Singularity images. However it seems that there is no layering in Singularity, which exists with Docker and Dockerfiles. This is quite convenient to have because building will resume from the point where the last step failed. However the comment:

You can build a Singularity container from an existing container on disk. So you could build your base container and save it and then modify the def file to build from the existing container to save time while you prototype.

gave me the idea to use a Makefile to mimic the layering behaviour. I have a blog post on Makefiles and a Makefile I use for testing, for those of us that need a refresher in Makefiles (like myself!).

Here's a simple example of building a Singularity image in layers starting with the Makefile.

.PHONY: all clean

assert-command-present = $(if $(shell command -v $1),,$(error '$1' missing and needed for this build))
$(call assert-command-present,singularity)

NAME = verse
RVER = 4.3.2
IMG = rocker/$(NAME):$(RVER)
FROM = docker://
BASE = $(NAME)_$(RVER)

all: $(BASE).sif $(BASE)_updated.sif $(BASE)_updated_seurat.sif

$(BASE).sif:
        singularity pull $(FROM)$(IMG)

$(BASE)_updated.sif: $(BASE).sif base.def
        singularity build --fakeroot $@ $(word 2,$^)

$(BASE)_updated_seurat.sif: $(BASE)_updated.sif seurat.def
        singularity build --fakeroot $@ $(word 2,$^)

clean:
        rm -rf $(BASE).sif $(BASE)_updated.sif $(BASE)_updated_seurat.sif

The file base.def installs some dependencies.

Bootstrap: localimage
From: verse_4.3.2.sif

%post
    # Update the image
    apt update
    apt upgrade -y

    # for igraph
    apt install -y glpk-utils libglpk-dev

    # for sctransform
    apt install -y libicu-dev

    # for BPCells
    apt install -y libhdf5-dev

    # Install R packages
    R -e 'install.packages(c("BiocManager", "remotes"))'

    apt clean

The file seurat.def installs Seurat.

Bootstrap: localimage
From: verse_4.3.2_updated.sif

%post
    # Install R packages
    R -e 'install.packages("Seurat", repos="https://cran.ism.ac.jp/")'

There should be a better (and automatic way) to populate the From: field in the definition file, which will come in handy when more "layers" are added.

If you put the three files above into the same directory and run make (assuming you have Singularity installed and Internet connectivity), it should create verse_4.3.2.sif, verse_4.3.2_updated.sif, and verse_4.3.2_updated_seurat.sif, which correspond to the different "layers"!

And there you have it: mimicking the layering behaviour of Docker in Singularity by using a Makefile!