Part III: Additional package install in Windows Subsystem for Linux (WSL)

This is Part III of a 3-part blog post on how I configured my system. Here I’ll explain how to install additional packages in Windows Subsystem for Linux (WSL)

  • Part I: Setup Dev Environment in Windows 10
  • Part II: Windows Subsystem for Linux (WSL) and Docker setup
  • Part III: Installation of additional packages and tools in WSL

Miniconda3

Open WSL Bash (anywhere), then:

  • download Miniconda3 installer for Linux with wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

  • bash Miniconda3-latest-Linux-x86_64.sh

  • “Do you wish the installer to initialize Miniconda3 by running conda init?” yes

  • afterwards: conda update -n base conda -c defaults

  • optionally, move the conda envs directory to a persistent location

Create an env, e.g. for jupyter lab:

  • conda create -n jupyter_env

Add conda-forge only to specific envs, e.g.:

  • conda activate jupyter_env
  • conda config --add channels conda-forge
  • Otionally add conda config --set channel_priority strict

Install Jupyter Lab to jupyter_env

  • conda activate jupyter_env
  • conda install -c conda-forge jupyterlab (conda-forge flag is not necessary if conda-forge has been added to default channels)

To update all packages in a specific env to latest possible versions:

  • conda activate jupyter_env
  • conda update --all -c conda-forge

Test Jupyter Lab with commands:

  • conda activate jupyter_env
  • jupyter lab

Visual Studio Code

To use Visual Studio Code in WSL and connect from it using Windows, you need the Insider Version of Visual Studio Code.

Visual Studio Code Insiders is not available on Choco, download installer directly from https://code.visualstudio.com/insiders/

After installation, open WSL bash in any folder and type code-insiders .

This will open the folder in Visual Studio Code and let VS Code connect to WSL.

Git

For git, the following instructions are taken from the official docs:

sudo apt update
sudo apt install git
Latest version of git? *Click*

If you want the latest “upstream” stable version from git for Ubuntu, use [1]:

sudo apt remove git
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

You can confirm that you have installed Git correctly by running the following command:

git --version
Output
git version 2.17.1

Line endings fix:

Some pograms in Windows (such as Notepad) will convert LF line-endings to CRLF without notice, rendering those files unusable in Linux WSL.

The following command will tell git to always change (accidental) CRLF (Windows) line-endings to Unix Style LF line-endings. You can still use LF line-endings in Windows with programs like VS Code or Notepad++, which understand LF.

git config --global core.autocrlf input

File permissions fix:

When you look up this problem, people will usually recommend doing the following:

git config --global core.filemode false

Sadly, git init and git clone always set filemode to true, no matter what is set with git config --global. Therefore, you need to set this manually for each repository you’re working on with

git config --local core.filemode false

If you once forgot to set this, and you’re seeing a staged change in git with 644 -> 755, you can undo those changes by typing:

chmod -x file.xyz

There’s also the following command:

git update-index --chmod=-x file.xyz

.. or the alternative for newer git versions:

> `git add --chmod=-x` / `--chmod=+x`

Here’s an explanation:

The executable bit will not be detected (and therefore will not be set) for paths in a repository with core.filemode set to false, though the users may still wish to add files as executable for compatibility with other users who do have core.filemode functionality.
For example, Windows users adding shell scripts may wish to add them as executable for compatibility with users on non-Windows.

Although this can be done with a plumbing command (git update-index --add --chmod=+x foo), teaching the git-add command allows users to set a file executable with a command that they’re already familiar with.

Git branch display

This tipp comes from Marc: it will display the current branch you’re on, if the folder is a git repo.

nano ~/.bashrc

Copy the following with paste (right-click) or CTRL+Shift+V, depending on your terminal:

# Git branch in prompt
# https://stackoverflow.com/a/35218509/1329744
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
}
# create the prompt if not root / else
    PS1="\t \u@\h \w\$(parse_git_branch) "

Symbolic links in Git for Windows: see

Git Aliases and further config settings

Follow the steps described here to further simplify git usage in WSL:

https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases

Also, I like adding git rerere by default:

git config --global rerere.enabled true

This means that if any merge conflicts appear again, git will remember how they’re solved.

These are some other aliases, added to ~/.bashrc, that I frequently use. Have a look at this post.

SSH

When you’re using git, chances are high you’re also authenticating with SSH (e.g. to push/pull from and to a remote repository).

How SSH is supposed to be used:

  • each physical device you’re authenticating from should have its own key-pair
  • that means: it is recommended to not use private keys across several computers (e.g. home and work)
  • if one of your physical devices got compromised, you can remove that computers public key from remote servers/ websites etc. as a means to disable the compromised private key
  • commonly, only one key is needed for each device you use and this is used across all serves/targets; creating multiple key-pairs does not increase security

Generate a key-pair

You can generate SSH Key-pairs in Windows (with e.g. Puttygen) or in WSL.

Here I’ll describe the Windows route. Have a look at this post that describes how to create an ssh-key-pair in WSL/Linux.

If you create your ssh-key with Puttygen, there’s a pitfall because there are two ways to export the private id_rsa from Puttygen:

  1. The first option is using Save private key in puttygen

(Note: of course, this is a sample key generated, not any of my personal keys)

Keys exported this way can only be used in Windows, e.g. copy to %USERPROFILE%\.ssh\id_rsa.

If this key is saved to ~/.ssh/ in Linux (e.g. Windows Subsystem for Linux), it won’t work because of the wrong format.

  1. The correct way is to export key for WSL/Linux from Windows Puttygen is from the menu Conversions/Export OpenSSH key in puttygen

  1. Copy the public key (see image from puttygen, above, the field “Public key for pasting..”) to

e.g. Gitlab:

https://gitlab.vgiscience.de/profile/keys

ssh-key-gitlab

e.g. an Ubuntu Server:

sudo adduser john
sudo mkdir /home/john/.ssh/
sudo bash -c 'echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAl8hZ0nZRDeFGflhmtuKFCCkxif58z11L+21daz3pglyVg7IZF5HiDBtyNj6Y9wNFef013cOgim5rjRz5D+sco+9zXwxaurM6/cRnffDNiC9bsfMPnAv63AZ7qu8Locb9PC41JH0gDkXuLzuXlKoge2Mjuoe4Ux3SuIdcz9V++P+OJ55QtAcDMC5nQLKiUA1wTiTwEoOmfhw3vlTA083UHk52LCQQ6CufvpjeEFgZbHhDPdY7HgX6jT9XnwpqqlbUvr0dpfOwr8ujxm4xA7gxyBvEx/SlcNchXmExfTOq+eW05ah6uIDdoMywL1/M2qb/sbDgd4KumoqPg7agOYLUBQ== rsa-key-20200110" >> /home/john/.ssh/authorized_keys'

SSH-Agent Setup

You’ve generated your key-pair, but you haven’t setup an SSH Agent that manages your keys and is used for authentication.

You’ve several options:

  • Option A: Use Linux SSH Agent in WSL
    • easy to setup
    • caveat: you can’t use the SSH Agent from WSL to authenticate in Windows (e.g. cmd/git-for-windows)
  • Option B: Use Pageant in Windows 10:
  • Option C: Use Windows 10 OpenSSH Agent
    • same caveats as B
  • Option D: Use wsl-ssh-pageant or other solutions to connect from WSL to your Windows SSH Agent
    • convenient once it is setup properly
    • there’s a list of steps that need to be taken

For a long time, I was happy with a combination of Option A and B. This required unlocking SSH Keys with the password each time I opened a new WSL Window. In windows, I had to unlock my KeePass Store and SSH Keys were available through KeeAgent (but I rarely use SSH in Windows these days).

I never used Option C (the native Windows 10 OpenSSH Agent) since I got used to Pageant and didn’t trust Microsoft to get the ssh-agent right the first time.

I recently discovered Option D, which I am using now because I only need to unlock my ssh-key once during my whole Windows Session.

This is my current setup:

  • Putty & Pageant installed with choco
  • there’s one private (password protected) keyfile for Git/Repo/Github in C:\Users\[User]\.ssh, which will be unlocked on startup and is available for all authentications while the system is running
  • there’s an additional key that is made available through KeeAgent plugin for Keepass that I used to login to remote servers (increased security because I need to unlock Keepass first)
  • Pageant-Keys from Windows are made available to WSL through wsl-ssh-pageant bridge

Option A: Setup WSL ssh agent

If you generated a private id_rsa with the above instructions, copy it to the linux ssh folder with:

cp /c/temp/id_rsa ~/.ssh/

If the file was generated in Windows, you also need to update permissions, e.g.:

cd ~/.ssh/
chmod 700 id_rsa

In case your .ssh folder doesn’t exist:

mkdir ~/.ssh/

Optional: auto ssh

Now, each time you authenticate, SSH will ask for the ssh key password. If you want to authenticate once for the whole session (that is: as long as you keep the WSL Window open), follow these steps:

nano ~/.bashrc

Add anywhere the following line:

alias autossh='eval `ssh-agent -s`;ssh-add'

Hit CTRL+O (save), then reload .bashrc:

source ~/.bashrc

When you open a WSL window now, simply type autossh. It will ask you for a password. You will stay authenticated during the full WSL session. Another tip: if you do this in a byobu session, you will stay authenticated even when shutting down your computer (this may or may not be a security risk).

Option B: Setup Windows ssh agent with Pageant

Optionally unlock your ssh key on Windows Startup:

  • Copy your private key (that is password protected) to %USERPROFILE%\.ssh\id_rsa
  • Use ⊞ + R and run shell:startup to open your startup folder
  • right-click / Add shortcut and name it “load_ssh_key”
    • Target:
    C:\ProgramData\chocolatey\lib\putty.portable\tools\PAGEANT.EXE "C:\Users\[User]\.ssh\id_rsa.ppk"
    
    • replace [User] by your username - you can get the full path from explorer by using %USERPROFILE%\.ssh and Shift + Right Click on id_raw.ppk (“Copy as Path”)
    • use the path to your pageant executable, example above shows path to PAGEANT.EXE if you installed putty with choco
    • Start in: C:\ProgramData\chocolatey\lib\putty.portable\tools

Keepass and KeeAgent

An alternative (or complementary) option is to use KeePass with the KeeAgent plugin.

Everytime you unlock your KeePass password store, selected keys are made available through KeeAgent in Pageant SSH Agent, there’s no need to enter the password for the key. This allows me to make keys available that I do not use as often (e.g. my Home SSH key, when I’m at work).

  • install KeePass with choco
  • install KeeAgent plugin
    • in KeePass Options / KeeAgent Tab, add (replace [User] with your user or use any other path):
      • Cygwin socket file C:\Users\[User]\Documents\workspace\cyglockfile
      • Msysgit socket file C:\Users\[User]\Documents\syslockfile
    • if you are using pageant in Windows, select the KeeAgent Mode Client, otherwise Agent

I also set the following options for KeeAgent:

  • Add your ssh-key to KeePass, enable KeeAgent availability for each key you want to make available on KeePass unlock
  • it is a good practice to have only one or two keys made available, otherwise your login may fail due to too many login attempts
  • Check which keys are unlocked in KeePass under Tools/KeeAgent
    • you can also use this Window to manually remove/add keys form the SSH Agent

Option D: Use Windows SSH-Agent in WSL

Make sure you’ve followed B first if you want auto-unlock of your key on windows startup.

Prepare and test setup:

  • open windows command line (shift + right click, e.g. in your windows documents folder)
  • create folder workspace with mkdir workspace (e.g. in windows documents folder)
  • create another folder for the SSH_SOCK_FILE somewhere, e.g. C:\wsl-ssh-pageant
  • Download the latest release of
  • cd workspace in your cmd from above
  • run
    wsl-ssh-pageant-amd64.exe ^
      -force -systray -wsl  C:\wsl-ssh-pageant\ssh-agent.sock  ^
      -winssh ssh-pageant -verbose
    
  • test your connection:
    • in Windows cmd:
      • set SSH_AUTH_SOCK=\\.\pipe\ssh-pageant
      • ssh -vvvv -T john@111.11.11.1 (use a server and username of your own)
    • in WSL:
      • export SSH_AUTH_SOCK=/c/wsl-ssh-pageant/ssh-agent.sock
      • ssh-add -l should now list your keys available in Windows Pageant

Make it automatically available in WSL:

In WSL, run nano ~/.bashrc and add export SSH_AUTH_SOCK=/c/wsl-ssh-pageant/ssh-agent.sock.

In Windows, go to your task scheduler (⊞ + R and run taskschd.msc)

In the folder Task Scheduler Library, on the right side, click Create Task..

Go to Actions Tab and click New

Copy the full path to your wsl-ssh-pageant-amd64-gui.exe, including the arguments from above (without -verbose), e.g.:

C:\Users\[User]\Documents\workspace\wsl-ssh-pageant-amd64-gui.exe -force -systray -wsl  C:\wsl-ssh-pageant\ssh-agent.sock  -winssh ssh-pageant

We’re using the gui version because it’ll run in the background inside your task items.

  • Optional: Add /noshow to the field “Add arguments” if you don’t want the icon to be visible in your taskbar

Click OK, confirm with yes.

Under General, provide a name for the task such as wsl-ssh-bridge

Optional: check “Hidden” and “Windows 10” if you don’t want the task to be visible

Under Triggers Click New.. and chose At log on (select your user).

Restart your computer or logout and login again.

After you’ve unlocked your key, open WSL session and test if your key is available with:

ssh-add -l

Optional: If you don’t want to start this task automatically, you can also add a shortcut to start the task with the target wsl-ssh-bridge (the name you’ve given your task):

C:\Windows\System32\schtasks.exe /run /tn "wsl-ssh-bridge"

.. and an alterative shortcut to stop the task:

C:\Windows\System32\schtasks.exe /end /tn "wsl-ssh-bridge"

Done!

Option D: Use Windows SSH-Agent in WSL2

The instructions below are based on rupor-github/wsl-ssh-agent and benpye/wsl-ssh-pageant/issues/33.

Install dependencies:

sudo apt update && sudo apt install socat p7zip-full

Get npiperelay.exe, store it on the Windows side and make it available in Linux with correct permissions.

  • replace {username} with your Linux username (e.g. whoami)
  • replace [USER] with your Windows username
cd /tmp
wget https://github.com/jstarks/npiperelay/releases/download/v0.1.0/npiperelay_windows_386.zip
7z e -y /tmp/npiperelay_windows_386.zip
sudo cp npiperelay.exe /c/Users/[USER]
sudo chmod +x /c/Users/[USER]/npiperelay.exe
sudo chown {username}:{username} /c/Users/[USER]/npiperelay.exe

Get wsl-ssh-agent-relay and store it correct permissions:

cd /tmp
wget https://raw.githubusercontent.com/rupor-github/wsl-ssh-agent/master/docs/wsl-ssh-agent-relay
sudo cp wsl-ssh-agent-relay ~/.local/bin/
sudo chmod +x ~/.local/bin/wsl-ssh-agent-relay
sudo chown {username}:{username} ~/.local/bin/wsl-ssh-agent-relay

Edit wsl-ssh-agent-relay and modify paths to npiperelay.exe and wsl-ssh-agent.sock:

nano ~/.local/bin/wsl-ssh-agent-relay

Edit:

  • replace [USER] with your Windows username
RELAY_BIN="/c/Users/[USER]/npiperelay.exe"
WSL_AGENT_SSH_SOCK="${HOME}/.ssh/wsl-ssh-agent.sock"

Prepare SSH directory:

mkdir -m 700 -p ~/.ssh

Update ~/.bashrc to auto-start relay and socat pipe:

nano ~/.bashrc
  • replace [USER] with your Windows username
${HOME}/.local/bin/wsl-ssh-agent-relay start
export SSH_AUTH_SOCK=${HOME}/.ssh/wsl-ssh-agent.sock
setsid nohup socat EXEC:"/c/Users/[USER]/npiperelay.exe //./pipe/\ssh-pageant" \
    UNIX-LISTEN:${SSH_AUTH_SOCK},unlink-close,unlink-early,fork \
    >/dev/null 2>&1 &

Reload ~/.bashrc bashrc:

source ~/.bashrc

Test:

ssh-add -l
> 2048 SHA256:
> ...

PHP & Composer

Instructions as per this guide. Watch out for version variables within the following instructions - all php versions must match - e.g. replace 7.2 with your version of choice.

Install Apache and PHP

For PHP development in WSL, we need to get a web server and PHP. We’re using Apache here, but feel free to install the web server of your preference.

sudo apt-get install php

Install Composer inside Ubuntu in WSL

Following the official instructions for downloading and installing Composer, copy and paste this command into the CLI:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === 'a5c698ffe4b8e849a443b120cd5ba38043260d5c4023dbf93e1558871f1f07f58274fc6f4c93bcfd858c6bd0775cd8d1') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"

To make Composer easier to use, run the following command to move Composer into your global path:

sudo mv composer.phar /usr/local/bin/composer

Now you can run composer

Finally, to start a local webserver running php in the current dictionary, enter the following command:

php -S 127.0.0.1:8000

..and open http://127.0.0.1:8000/ in your browser of choice.

NVM, NPM and NodeJS

.. with JS, things can get confusing:

Nvm is a nodejs version manager. It let’s you easily install and switch between versions. It retains globally installed packages for each version.

Npm is a package manager. It let’s you install software (libraries, plugins, frameworks and applications). Typically this software is installed to build Node applications. Sometimes it isn’t.

That means, NPM and NodeJS can be installed with NVM.

And Yarn? Yarn and NPM are both package managers for JS - if you have NPM, you don’t need Yarn. In other words, these two commands are interchangeable:

# install local dependencies with yarn
yarn install .
# install local dependencies with npm
npm install .

However, sometimes packages are only available with yarn.lock - for those situations, you installing yarn and npm in parallel is possible.

Install NVM

Check newest release and download, following the instructions:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
  • Check install with nvm --version

Install Node and NPM

Install Node and NPM with NVM (see)

nvm install node

Check with node --version

This will automatically install NPM, too.

With NVM, you can install multiple versions of node in parallel, e.g.:

nvm install 6.14.4 # or 10.10.0, 8.9.1, etc
nvm use 6.14.4

Using Node and NPM with NVM makes it much easier managing JS environments.

To use NPM packages globally, use prefix npx, e.g.:

npx markdown-toc -i markdown.md

Install yarn

We first install node with nvm and then use npm (which comes with node) to install yarn.

nvm install node
npm install --global yarn
yarn --version

Then install dependencies of your project with:

yarn install

Install RVM, Ruby, Gem, Bundler

e.g., for locally testing running jekyll-reveal web presentations.

Similar to NVM, RVM is the Ruby Version Manager.

e.g.:

gpg --keyserver keyserver.ubuntu.com --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
curl -sSL https://get.rvm.io | bash -s stable --ruby

Afterwards, restart WSL and install ruby with RVM:

rvm install ruby-3.0.0
Ruby/RVM issues

Many issues with RVM and Ruby installation are related to minimal versions required by distros. For instance, Ubuntu 22.04 ships with OpenSSL 3.0.0, which limits the lower Ruby version to 3.1.0.

You can check this with:

curl -Lks 'https://git.io/rg-ssl' | ruby

If you see OpenSSL issues during installation, try explicitly setting openssl dir during installation (update [user] below, watch for the path output by the previous command):

rvm pkg install openssl
rvm install ruby-3.0.0 --with-openssl-dir=/home/[user]/.rvm/usr

There is also a wider range of recommendations available from bundler.io.

Check if ruby is installed:

ruby --version

The recommended way is to install ruby in your home directory. Below, replace with your version:

rvm use 3.0.0 --default

The output should be similar to

Using /home/[user]/.rvm/gems/ruby-3.0.0

Update gems once:

gem update
gem update --system

Install bundler and jekyll:

gem install bundler jekyll
bundle update --bundler

For jekyll on ruby 3.09, I also had to manually install webrick:

bundle add webrick

Then prepare and run your presentation, e.g.:

bundle install
bundle exec jekyll serve -o

If your rvm install fills up after a while, use the following to cleanup:

[rvmsudo] rvm cleanup all

Troubleshooting

  • If you see a message like
/usr/share/rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/ext/builder.rb:76: warning: Insecure world writable dir /home/userxxx/.rvm/gems in PATH, mode 040777

Fix with:

sudo chmod go-w /usr/share/rvm/
  • If you see a message like
/usr/share/rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/fileutils.rb:1656: warning: already initialized constant FileUtils::METHODS

run

uninstall fileutils

to fix it.

Hugo

For building static sites, I often use Hugo (in fact, this page is generated from markdown with Hugo).

To install Hugo in WSL (for local testing), get the latest release from Github Release page (check the version number):

wget https://github.com/gohugoio/hugo/releases/download/v0.79.0/hugo_extended_0.79.0_Linux-64bit.deb

.. and install to user folder (/usr/local/bin/) with:

sudo apt-get install -f ./hugo_extended_0.79.0_Linux-64bit.deb

Now build and serve your site locally with:

hugo serve

byobu

byobu is a window manager and terminal multiplexer. I like to use it when I have long running sessions on remote computers, where I want to detach and continue letting scripts run.

  1. it allows opening multiple windows next to each other and
  2. running commands in background.

It can be installed with:

apt-get install byobu

Start it with byobu. Here’s a list with the most frequent shortcuts I use:

  • F2 - Create a new window
  • F3/F4 - Move focus among windows
  • F6 - Detach session and then logout (byobu window keeps running, even if WSL is closed)
  • Ctrl-F6 - Kill current Window in focus (window closed)
  • F7 - Enter scrollback history
  • F8 - Rename the current window

When you close WSL and reopen it later, type byobu and you’ll have all your Sessions still running.

To enable byobu automatically on startup:

byobu-enable

SSH-Agent forwarding:

It can be challenging to get SSH Agend forwarding (-A) to work with byobu (in case you want to use ssh from within a byobu session).

This worked for me, slightly adapted for byobu:

  1. In ~/.ssh/rc, add
if test "$SSH_AUTH_SOCK"; then
	ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

This will create a symlink from the auth socket to a fixed path, so that we can refer to it in the next step.

  1. In ~/.byobu/.tmux.conf, add
setenv -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
set -g update-environment -r

On attaching to a tmux session, this will set SSH_AUTH_SOCK to the path we just created.

Afterwards:

  • reload byobu with F5
  • logout and login again
  • test with ssh-add