Building a Bioinformatics Base Environment Image Based on Docker

This article is an upgraded version of "Setting Up an Analysis Environment for Transcriptome RNA-Seq with docker+bioconda", which addresses the following issues:

Q: Why not use a locally deployed docker container mode? Instead, why use a docker image with an SSH service?
A: A single machine has limited capacity, resources, and computing power, making extensive scaling impossible. Docker deployment with SSH connects over the network, which offers better scalability.

Q: Why upgrade the docker usage from the previous article?
A: After all, I have been using docker for a while and made some improvements. Submitting images via local docker commit inflates the image size, which hinders portability; moreover, it is not the officially recommended approach—workable, but not recommended. Having some spare time recently, I wanted to refine it further.

Q: How to install docker and docker-compose?
A: Install docker and docker-compose:

#Ultra-fast docker installation, quick and time-saving
curl  -sSfL get.docker.io -o get_docker.sh
bash  get_docker.sh --mirror Aliyun

#Newer versions of Docker: the installation script above installs docker-compose-plugin directly, which replaces the early docker-compose; it can also be installed manually
sudo apt-get install docker-compose-plugin
#Usage is similar to docker-compose, with the command replaced by docker compose; the remaining parameters are basically the same
docker compose up -d

#Install docker-compose 1.x
#Download docker-compose from the URL below, place the docker-compose file in a directory on the PATH, e.g. /usr/local/bin, and add executable permission
sudo chmod +x /usr/local/bin/docker-compose
https://github.com/docker/compose/releases

The Dockerfile is as follows:

#Build from the Ubuntu 20.04 image; Ubuntu 22.04 reportedly kills processes that consume too many resources from time to time
FROM        ubuntu:20.04
#1. Set the account character encoding to C.UTF-8 for better compatibility; set the time zone; the ssh login password is 20201110 (account: root); the connection port is 9022
ENV         LANG=C.UTF-8 TZ=Asia/Shanghai PS=20201110 port=9022
#Set the time zone, update the image software, and install aria2 (a download tool replacing wget and curl for faster downloads, with automatic retry on failure/download)
#Install the openssh service and update the config file so that the root account can log in, and set the root account password to the configured value
RUN         ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
            apt update && apt upgrade -y && apt install openssh-server aria2 -y && apt autoremove -y
RUN         sed -i "s/#PermitEmptyPasswords no/PermitEmptyPasswords no/g" /etc/ssh/sshd_config && \
            sed -i "s/#PermitRootLogin prohibit-password/PermitRootLogin yes/g" /etc/ssh/sshd_config && \
            sed -i "s/#Port 22/Port ${port}/g" /etc/ssh/sshd_config && \
            sed -i "s/#ListenAddress 0.0.0.0/ListenAddress 0.0.0.0/g" /etc/ssh/sshd_config && \
            sed -i "s/#LoginGraceTime 2m/LoginGraceTime 2m/g" /etc/ssh/sshd_config && \
            echo root:${PS} | chpasswd
#Download miniconda with aria2, install it, add channels, delete the installer (to reduce image size), and initialize miniconda
RUN         aria2c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -d ~ && \
            bash ~/Miniconda3-latest-Linux-x86_64.sh -b && \
            ~/miniconda3/bin/conda config --add channels defaults    && \
            ~/miniconda3/bin/conda config --add channels bioconda    && \
            ~/miniconda3/bin/conda config --add channels conda-forge && \
            rm -f ~/Miniconda3-latest-Linux-x86_64.sh && \
            ~/miniconda3/bin/conda init
#Copy this file into the image's root directory; condarc is the Tsinghua mirror config file; uncomment this line to speed things up in China
#COPY       --chown=root:root ./condarc /root/.condarc
#Expose the ssh connection port
EXPOSE      $port
#Initialize the image at runtime: change the root password according to the PS config variable; the password can be reset at runtime during initialization; finally start the ssh service
ENTRYPOINT  ["/bin/bash", "-c", "echo root:${PS} | chpasswd && service ssh start -D"]

Build the image or pull it from Docker Hub

#Place the condarc file and the dockerfile in the same directory, then build the image
docker     build -t doujiangbaozi/sliverworkspace:latest .

#Or use the pre-built image and pull it directly to the local machine
docker     pull     doujiangbaozi/sliverworkspace:latest

condarc file (optional; the Tsinghua mirror is used here):

channels:
  - conda-forge
  - bioconda
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

How to use this image:

docker-compose is recommended; the docker-compose.yml config file is as follows

version: "3"
services:
  TumorOnly:
    image: doujiangbaozi/sliverworkspace:latest
    container_name: TumorOnly
    volumes:
      - /media/sliver/Data/data:/opt/data:rw                               #Mount the input data directory
      - /media/sliver/Manufacture/tumor-only/envs:/root/miniconda3/envs:rw #Mount the envs directory
      - /media/sliver/Manufacture/tumor-only/config:/opt/config:rw         #Mount the config directory
      - /media/sliver/Manufacture/sliver/ref:/opt/ref:rw                   #Mount the reference directory
      - /media/sliver/Manufacture/tumor-only/result:/opt/result:rw         #Mount the intermediate files and results directory
    ports:
      - "9022:9022"                                                        #ssh connection port; can be mapped to another port here
    environment:
      - TZ=Asia/Shanghai                                                   #Set the time zone
      - PS=20191124                                                        #Set the ssh password

Run docker-compose up -d in the directory containing the docker-compose.yml file to start it;

Check the docker running status

#Check the running status of docker containers
docker ps

#Or run in the docker-compose.yml directory
docker-compose ps

How to build a pipeline with this docker?

Log in to the container's ssh account, and you can start bioinformatics analysis in the environment

Following the Snakemake style, one yaml file is created per analysis step, containing the software and versions used. On first run, the environment for that step is checked; if it does not exist, the software is installed to initialize it.

#Environment variables used, taking the simplest fastqc/multiqc as an example
export env=/root/miniconda3/envs  #conda environment software installation directory, best mounted as a host volume
export conf=/opt/config           #conda environment config file directory
export sn=RD1703007FFP            #Sample number, sample number
export pn=TumorOnly               #Project number, project number
export result=/opt/result         #Intermediate file output directory

#conda checks whether the environment exists; on first run, if it does not exist, the environment is created and software installed
if [ ! -d "${envs}/${pn}.fastqc" ]; then
  conda env create -f ${conf}/fastqc.yaml
fi
#Switch to the environment and run fastqc and multiqc
conda activate "${pn}.fastqc"
mkdir  -p ${result}/${sn}/qc
fastqc -o ${result}/${sn}/qc \
    ${result}/${sn}/trimmed/${sn}_trimmed_R1.fastq.gz \
    ${result}/${sn}/trimmed/${sn}_trimmed_R2.fastq.gz
multiqc ${result}/${sn}/ -f -o ${result}/${sn}/qc
#Exit the environment
conda deactivate

The fastqc.yaml file is as follows; it is located in the /opt/config directory

name: TumorOnly.fastqc
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - fastqc=0.11.9
  - multiqc=1.13a

This makes it easier to build the entire analysis workflow afterwards. To be continued....