Python Handbook
Published:
Conda
Install mamba:
conda install mamba -n base -c conda-forge
Remove environment:
conda remove --name r_env --all -y
For MacOS, override your default platform osx-arm64 and replace it with osx-64:
CONDA_SUBDIR=osx-64 mamba create -n renv
mamba activate renv
conda config --env --set subdir osx-64
Tensorflow-gpu in conda (Reference):
export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX
Create a conda environment from an environment.yml file:
conda env create -f environment.yml
Setup of R environment
mamba create -n r_env -y
conda activate r_env
conda config --add channels bioconda
conda config --add channels conda-forge
# conda config --set channel_priority strict
# conda config --set channel_priority false
mamba install r-base=4.1.1 -y
mamba install -c conda-forge -c bioconda r-seurat r-seuratdisk r-signac -y
mamba install -c bioconda bioconductor-genomeinfodbdata -y
mamba install -c bioconda bioconductor-genomeinfodbdata --force-reinstall
Jupyter Notebook
Python kernel
Add a new kernel kernel_name from a conda environment env (Reference):
conda activate env
mamba install ipykernel -y
python -m ipykernel install --name kernel_name
conda deactivate
R kernel
Add a new R kernel kernel_name from a conda environment r_env to another conda environment env:
conda activate r_env
mamba install r-irkernel -y
conda activate env
/raid6/Jinhong/anaconda3/envs/r_env/bin/R -e 'IRkernel::installspec(name="kernel_name", displayname="kernel_name")'
Bash commands
Running scripts
nohup python -u foo.py 0 > foo.out 2>&1 &
nohup Rscript foo.R 0 > foo.out 2>&1 &
Python dependencies
Print the versions of Python packages in the current conda environment:
conda list -f "python|numpy|numba|scipy|pandas|scikit-learn|h5py|anndata|scanpy|statsmodels|matplotlib|joblib|cvxpy|tqdm" | awk '{ print $1" | "$2 }'
R dependencies
Linux
System information:
# for basic system info
landscape-sysinfo
# for detailed system info
top
htop
Delete all files with a certain pattern in the current directory and subdirectories (Reference):
find . -type f -name '*.o' -delete
find . -type f -name '*.o' -exec rm {} \;
rm -- **/*.o
Compress all files in the current directory and subdirectories with a certain pattern:
zip -r archive.zip results/* -i "*gears*"
conda/mamba environment for multiple users:
sudo groupadd micromamba
sudo chgrp -R micromamba /data/miniforge3
sudo chmod 770 -R /data/miniforge3
# Each user who wants to use anaconda must be added to that group. Note that you need to log out or otherwise refresh your shell to see your new membership.
sudo adduser USER_TO_BE_ADDED micromamba
# Last, each user who wants to use conda needs to add it to their path. This can be done following the instructions in the installation doc:
source /data/miniforge3/bin/activate
conda init
Slurm
Check GPU usage on the node n01:
ssh n01 'nvidia-smi -l 1'
Check the status of the jobs:
# all jobs
squeue -u jinhongd
# pending jobs
squeue --start -u jinhongd
Check the CPU and memory usage of a job:
# for running jobs
sacct -e
sacct --format="CPUTime,MaxRSS"
sacct -a -j <job_id> --format=user%10,jobname%10,node%10,start%10,end%10,elapsed%10,MaxRS
# for finished jobs
seff <job_id>