Part III: Additional package install in Windows Subsystem for Linux (WSL)

This is Part III of a 3-part blog post on how I configured my system. Here I’ll explain how to install additional packages in Windows Subsystem for Linux (WSL)

• Part I: Setup Dev Environment in Windows 10
• Part II: Windows Subsystem for Linux (WSL) and Docker setup
• Part III: Installation of additional packages and tools in WSL

Miniconda3

Open WSL Bash (anywhere), then:

• download Miniconda3 installer for Linux with wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

• bash Miniconda3-latest-Linux-x86_64.sh

• “Do you wish the installer to initialize Miniconda3 by running conda init?” yes

• afterwards: conda update -n base conda -c defaults

• optionally, move the conda envs directory to a persistent location

Create an env, e.g. for jupyter lab:

• conda create -n jupyter_env

Add conda-forge only to specific envs, e.g.:

• conda activate jupyter_env
• conda config --add channels conda-forge
• Optionally add conda config --set channel_priority strict

Install Jupyter Lab to jupyter_env

• conda activate jupyter_env
• conda install -c conda-forge jupyterlab (conda-forge flag is not necessary if conda-forge has been added to default channels)

To update all packages in a specific env to latest possible versions:

• conda activate jupyter_env
• conda update --all -c conda-forge

Test Jupyter Lab with commands:

• conda activate jupyter_env
• jupyter lab

Visual Studio Code

To use Visual Studio Code in WSL and connect to it using Windows as a frontend.

Visual Studio Code Insiders is available in Choco, install with choco install vscode

After installation, open WSL bash in any folder and type code .

This will open the folder in Visual Studio Code and let VS Code connect to WSL.

Git

For git, the following instructions are taken from the official docs:

sudo apt update
sudo apt install git

sudo apt remove git
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

You can confirm that you have installed Git correctly by running the following command:

git --version

Output
git version 2.17.1

Line endings fix:

Some pograms in Windows (such as Notepad) will convert LF line-endings to CRLF without notice, rendering those files unusable in Linux WSL.

The following command will tell git to always change (accidental) CRLF (Windows) line-endings to Unix Style LF line-endings. You can still use LF line-endings in Windows with programs like VS Code or Notepad++, which understand LF.

git config --global core.autocrlf input

File permissions fix:

When you look up this problem, people will usually recommend doing the following:

git config --global core.filemode false

Sadly, git init and git clone always set filemode to true, no matter what is set with git config --global. Therefore, you need to set this manually for each repository you’re working on with

git config --local core.filemode false

If you once forgot to set this, and you’re seeing a staged change in git with 644 -> 755, you can undo those changes by typing:

chmod -x file.xyz

There’s also the following command:

git update-index --chmod=-x file.xyz

.. or the alternative for newer git versions:

> `git add --chmod=-x` / `--chmod=+x`

Here’s an explanation:

The executable bit will not be detected (and therefore will not be set) for paths in a repository with core.filemode set to false, though the users may still wish to add files as executable for compatibility with other users who do have core.filemode functionality.
For example, Windows users adding shell scripts may wish to add them as executable for compatibility with users on non-Windows.

Although this can be done with a plumbing command (git update-index --add --chmod=+x foo), teaching the git-add command allows users to set a file executable with a command that they’re already familiar with.

Git branch display

This tipp comes from Marc: it will display the current branch you’re on, if the folder is a git repo.

nano ~/.bashrc

Copy the following with paste (right-click) or CTRL+Shift+V, depending on your terminal:

# Git branch in prompt
# https://stackoverflow.com/a/35218509/1329744
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/- \(.*\)/ (\1)/'
}
# create the prompt if not root / else
    PS1="\t \u@\h \w\$(parse_git_branch) "

Symbolic links in Git for Windows: see

Git Aliases and further config settings

Follow the steps described here to further simplify git usage in WSL:

https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases

Also, I like adding git rerere by default:

git config --global rerere.enabled true

This means that if any merge conflicts appear again, git will remember how they’re solved.

These are some other aliases, added to ~/.bashrc, that I frequently use. Have a look at this post.

gpg

I added a dedicated blog post for using git with GPG

SSH

When you’re using git, chances are high you’re also authenticating with SSH (e.g. to push/pull from and to a remote repository).

How SSH is supposed to be used:

• each physical device you’re authenticating from should have its own key-pair
• that means: it is recommended to not use private keys across several computers (e.g. home and work)
• if one of your physical devices got compromised, you can remove that computers public key from remote servers/ websites etc. as a means to disable the compromised private key
• commonly, only one key is needed for each device you use and this is used across all serves/targets; creating multiple key-pairs does not increase security

Generate a key-pair

You can generate SSH Key-pairs in Windows (with e.g. Puttygen) or in WSL.

Here I’ll describe the Windows route. Have a look at this post that describes how to create an ssh-key-pair in WSL/Linux.

If you create your ssh-key with Puttygen, there’s a pitfall because there are two ways to export the private id_rsa from Puttygen:

1. The first option is using *Save private key- in puttygen

(Note: of course, this is a sample key generated, not any of my personal keys)

Keys exported this way can only be used in Windows, e.g. copy to %USERPROFILE%\.ssh\id_rsa.

If this key is saved to ~/.ssh/ in Linux (e.g. Windows Subsystem for Linux), it won’t work because of the wrong format.

2. The correct way is to export key for WSL/Linux from Windows Puttygen is from the menu *Conversions/Export OpenSSH key- in puttygen

3. Copy the public key (see image from puttygen, above, the field “Public key for pasting..”) to

e.g. Gitlab:

https://gitlab.vgiscience.de/profile/keys

e.g. an Ubuntu Server:

sudo adduser john
sudo mkdir /home/john/.ssh/
sudo bash -c 'echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAl8hZ0nZRDeFGflhmtuKFCCkxif58z11L+21daz3pglyVg7IZF5HiDBtyNj6Y9wNFef013cOgim5rjRz5D+sco+9zXwxaurM6/cRnffDNiC9bsfMPnAv63AZ7qu8Locb9PC41JH0gDkXuLzuXlKoge2Mjuoe4Ux3SuIdcz9V++P+OJ55QtAcDMC5nQLKiUA1wTiTwEoOmfhw3vlTA083UHk52LCQQ6CufvpjeEFgZbHhDPdY7HgX6jT9XnwpqqlbUvr0dpfOwr8ujxm4xA7gxyBvEx/SlcNchXmExfTOq+eW05ah6uIDdoMywL1/M2qb/sbDgd4KumoqPg7agOYLUBQ== rsa-key-20200110" >> /home/john/.ssh/authorized_keys'

SSH-Agent Setup Overview

You’ve generated your key-pair, but you haven’t setup an SSH Agent that manages your keys and is used for authentication.

You’ve several options:

• Use Linux SSH Agent in WSL
  • easy to setup
  • caveat: you can’t use the SSH Agent from WSL to authenticate in Windows (e.g. cmd/git-for-windows)
• Use Pageant in Windows 10:
  • easy to setup (use choco)
  • you can use Keepass and the KeeAgent Plugin to manage your ssh keys in windows
  • caveat: you can’t use pageant-keys in your WSL without extra work
• Use Windows 10 OpenSSH Agent
  • supported out of the box
  • difficult to integrate with Keepass/KeeAgent
• Use wsl-ssh-pageant or other solutions to connect from WSL to your Windows SSH Agent
  • convenient once it is setup properly
  • there’s a list of steps that need to be taken

This is my current setup:

• Putty & Pageant installed with choco
• Keepass with KeeAgent plugin, KeeAgend in Agent Mode, adds SSH Keys to Putty
• Pageant-Keys from Windows are made available to WSL through a socket (WSL1 & WSL2)

Basic SSH Setup in WSL

If you generated a private id_rsa with the above instructions, copy it to the linux ssh folder with:

cp /c/temp/id_rsa ~/.ssh/

If the file was generated in Windows, you also need to update permissions, e.g.:

cd ~/.ssh/
chmod 700 id_rsa

In case your .ssh folder doesn’t exist:

mkdir ~/.ssh/

Optional: auto ssh

Now, each time you authenticate, SSH will ask for the ssh key password. If you want to authenticate once for the whole session (that is: as long as you keep the WSL Window open), follow these steps:

nano ~/.bashrc

Add anywhere the following line:

alias autossh='eval `ssh-agent -s`;ssh-add'

Hit CTRL+O (save), then reload .bashrc:

source ~/.bashrc

When you open a WSL window now, simply type autossh. It will ask you for a password. You will stay authenticated during the full WSL session. Another tip: if you do this in a byobu session, you will stay authenticated even when shutting down your computer (this may or may not be a security risk).

Setup Windows ssh agent with Pageant

• install Putty with Choco
• optionally, install and setup Keepass and the KeeAgent Plugin to manage your keys with KeePass

Optionally unlock your ssh key on Windows Startup:

• Copy your private key (that is password protected) to %USERPROFILE%\.ssh\id_rsa
• Use ⊞ + R and run shell:startup to open your startup folder
• right-click / Add shortcut and name it “load_ssh_key”
  • Target:

  C:\ProgramData\chocolatey\lib\putty.portable\tools\PAGEANT.EXE "C:\Users\[User]\.ssh\id_rsa.ppk"
  

  • replace [User] by your username - you can get the full path from explorer by using %USERPROFILE%\.ssh and Shift + Right Click on id_raw.ppk (“Copy as Path”)
  • use the path to your pageant executable, example above shows path to PAGEANT.EXE if you installed putty with choco
  • Start in: C:\ProgramData\chocolatey\lib\putty.portable\tools

Keepass and KeeAgent

An alternative (or complementary) option is to use KeePass with the KeeAgent plugin.

Everytime you unlock your KeePass password store, selected keys are made available through KeeAgent in Pageant SSH Agent, there’s no need to enter the password for the key. This allows me to make keys available that I do not use as often (e.g. my Home SSH key, when I’m at work).

• install KeePass with choco
• install KeeAgent plugin
  • in KeePass Options / KeeAgent Tab, add (replace [User] with your user or use any other path):
    • Cygwin socket file C:\Users\[User]\Documents\workspace\cyglockfile
    • Msysgit socket file C:\Users\[User]\Documents\syslockfile
    • Check “Create WSL1 compatible Socket file”
  • select the KeeAgent Mode Agent

Screenshot:

I also set the following options for KeeAgent:

• Add your ssh-key to KeePass, enable KeeAgent availability for each key you want to make available on KeePass unlock
• it is a good practice to have only one or two keys made available, otherwise your login may fail due to too many login attempts
• Check which keys are unlocked in KeePass under Tools/KeeAgent
  • you can also use this Window to manually remove/add keys form the SSH Agent

2024: WSL2 & Windows SSH Agent

WSL2 kept me busy with SSH issues. After a few months of trying to fix these, I found a pretty nice solution for connecting from WSL2 to my SSH Agent in Windows (KeePass with KeeAgent).

This was described by zoredache on Github. It is based on the idea to connect to WSL2 via OpenSSH from the Windows side to symlink the Windows SSH authentication socket into WSL2.

1. First, check that you can actually see your windows keys on windows. Use cmd:

ssh-add.exe -L
> ssh-rsa ...

If not, install OpenSSH in Windows.

2. Install OpenSSH Server in WSL2

sudo apt-get install openssh-server

3. Change the port openssh is listening in WSL2

sudo nano /etc/ssh/sshd_config

Add:

Port 2222

4. Enable systemd support in WSL2

sudo nano /etc/wsl.conf

Add:

[boot]
systemd=true

For completeness, my full wsl.conf looks like this:

[automount]
root = /
options = "metadata"

[network]
generateResolvConf = false

[boot]
systemd=true

5. Shutdown WSL2

In CMD, use:

wsl.exe --shutdown -n WSL_UF2

..where WSL_UF2 must be replaced with your WSL name. Use >wsl --list -v to get a list of names.

6. Add your public SSH key to WSL2

In order for Windows to connect to WSL2, you must add the public key part to WSL2 authorized_keys.

nano ~/.ssh/authorized_keys

Add:

ssh-ed25519 ...

7. Test SSH connection from Windows to WSL2

In CMD:

cd C:\Windows\System32\OpenSSH
ssh.exe alex@localhost -p 2222
> Last login: Fri Aug  9 10:02:22 2024
> ...

..where alex must be replaced with your WSL2 username.

8. Create a systemd service to update SSH_AUTH_SOCK in WSL2

sudo nano /etc/systemd/system/ssh_agent.service

Add:

[Unit]
Description=SSH Agent
After=ssh.service
ConditionPathExists=/c/Windows/System32/OpenSSH/ssh.exe

[Service]
ExecStart=/c/Windows/System32/OpenSSH/ssh.exe \
    alex@localhost -p 2222 -A -t -t bash -c \
    ': ; ln -sf "$SSH_AUTH_SOCK" ~/.ssh/agent ; exec sleep 1d'
KillMode=process
Restart=on-failure
Type=simple

[Install]
WantedBy=multi-user.target

9. Start the agent

sudo systemctl start ssh_agent

Check that it is active:

systemctl is-active ssh_agent

10. Test to get SSH keys from WSL2

ssh-add -l
> 2048 SHA256: ...

11. Update your ~/.bashrc to start the daemon automatically

if [ -L ~/.ssh/agent -a -z ${SSH_AUTH_SOCK+x} ]; then
    # wait a bit for systemd to start sshd and ssh_agent units.
    wait_tries=20
    until test $((wait_tries--)) -eq 0 -o -S ~/.ssh/agent ; do sleep 0.5; done
    unset wait_seconds
fi
# set our socket
export SSH_AUTH_SOCK=~/.ssh/agent

I still had problems that the ssh_agent service would not start automatically. I solved this by adding:

sudo systemctl start ssh_agent

before the above in ~/.bashrc.

To not always enter my sudo password, I updated visudo for an exception for user alex and the systemctl command.

sudo visudo

Add:

alex ALL=NOPASSWD: /bin/systemctl

.. and replace user alex with your username.

Finally I can skip WSL1 and move all my work to WSL2.

PHP & Composer

Instructions as per this guide. Watch out for version variables within the following instructions - all php versions must match - e.g. replace 7.2 with your version of choice.

Install Apache and PHP

For PHP development in WSL, we need to get a web server and PHP. We’re using *Apache- here, but feel free to install the web server of your preference.

sudo apt-get install php
sudo apt-get install php-dom # optional

Install Composer inside Ubuntu in WSL

Following the official instructions for downloading and installing Composer, copy and paste this command into the CLI:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === 'a5c698ffe4b8e849a443b120cd5ba38043260d5c4023dbf93e1558871f1f07f58274fc6f4c93bcfd858c6bd0775cd8d1') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"

To make Composer easier to use, run the following command to move Composer into your global path:

sudo mv composer.phar /usr/local/bin/composer

Now you can run composer

Finally, to start a local webserver running php in the current dictionary, enter the following command:

php -S 127.0.0.1:8000

..and open http://127.0.0.1:8000/ in your browser of choice.

NVM, NPM and NodeJS

.. with JS, things can get confusing:

Nvm is a nodejs version manager. It let’s you easily install and switch between versions. It retains globally installed packages for each version.

Npm is a package manager. It let’s you install software (libraries, plugins, frameworks and applications). Typically this software is installed to build Node applications. Sometimes it isn’t.

That means, NPM and NodeJS can be installed with NVM.

And Yarn? Yarn and NPM are both package managers for JS - if you have NPM, you don’t need Yarn. In other words, these two commands are interchangeable:

# install local dependencies with yarn
yarn install .
# install local dependencies with npm
npm install .

However, sometimes packages are only available with yarn.lock - for those situations, you installing yarn and npm in parallel is possible.

Install NVM

Check newest release and download, following the instructions:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

• Check install with nvm --version

Install Node and NPM

Install Node and NPM with NVM (see)

nvm install node

Check with node --version

This will automatically install NPM, too.

With NVM, you can install multiple versions of node in parallel, e.g.:

nvm install 6.14.4 # or 10.10.0, 8.9.1, etc
nvm use 6.14.4

Using Node and NPM with NVM makes it much easier managing JS environments.

To use NPM packages globally, use prefix npx, e.g.:

npx markdown-toc -i markdown.md

Install yarn

We first install node with nvm and then use npm (which comes with node) to install yarn.

nvm install node
npm install --global yarn
yarn --version

Then install dependencies of your project with:

yarn install

Install RVM, Ruby, Gem, Bundler

E.g., for locally testing running jekyll-reveal web presentations.

Similar to NVM, RVM is the Ruby Version Manager.

• First, follow the official instructions to install RVM in Linux

e.g.:

gpg --keyserver keyserver.ubuntu.com --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
curl -sSL https://get.rvm.io | bash -s stable --ruby

Afterwards, restart WSL and install ruby with RVM:

rvm install ruby-3.0.0

curl -Lks 'https://git.io/rg-ssl' | ruby

rvm pkg install openssl
rvm install ruby-3.0.0 --with-openssl-dir=/home/[user]/.rvm/usr

Check if ruby is installed:

ruby --version

The recommended way is to install ruby in your home directory. Below, replace with your version:

rvm use 3.0.0 --default

The output should be similar to

Using /home/[user]/.rvm/gems/ruby-3.0.0

Update gems once:

gem update
gem update --system

Install bundler and jekyll:

gem install bundler jekyll
bundle update --bundler

For jekyll on ruby 3.09, I also had to manually install webrick:

bundle add webrick

Then prepare and run your presentation, e.g.:

bundle install
bundle exec jekyll serve -o

If your rvm install fills up after a while, use the following to cleanup:

[rvmsudo] rvm cleanup all

Troubleshooting

• If you see a message like

/usr/share/rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/ext/builder.rb:76: warning: Insecure world writable dir /home/userxxx/.rvm/gems in PATH, mode 040777

Fix with:

sudo chmod go-w /usr/share/rvm/

• If you see a message like

/usr/share/rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/fileutils.rb:1656: warning: already initialized constant FileUtils::METHODS

run

uninstall fileutils

to fix it.

Hugo

For building static sites, I often use Hugo (in fact, this page is generated from markdown with Hugo).

To install Hugo in WSL (for local testing), get the latest release from Github Release page (check the version number):

wget https://github.com/gohugoio/hugo/releases/download/v0.131.0/hugo_extended_0.131.0_linux-amd64.deb

.. and install to user folder (/usr/local/bin/) with:

sudo apt-get install -f ./hugo_extended_0.131.0_linux-amd64.deb

Now build and serve your site locally with:

hugo serve

byobu

**byobu*- is a window manager and terminal multiplexer. I like to use it when I have long running sessions on remote computers, where I want to detach and continue letting scripts run.

1. it allows opening multiple windows next to each other and
2. running commands in background.

It can be installed with:

apt-get install byobu

Start it with byobu. Here’s a list with the most frequent shortcuts I use:

• F2 - Create a new window
• F3/F4 - Move focus among windows
• F6 - Detach session and then logout (byobu window keeps running, even if WSL is closed)
• Ctrl-F6 - Kill current Window in focus (window closed)
• F7 - Enter scrollback history
• F8 - Rename the current window

When you close WSL and reopen it later, type byobu and you’ll have all your Sessions still running.

To enable byobu automatically on startup:

byobu-enable

SSH-Agent forwarding:

It can be challenging to get SSH Agend forwarding (-A) to work with byobu (in case you want to use ssh from within a byobu session).

This worked for me, slightly adapted for byobu:

1. In ~/.ssh/rc, add

if test "$SSH_AUTH_SOCK"; then
	ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

This will create a symlink from the auth socket to a fixed path, so that we can refer to it in the next step.

2. In ~/.byobu/.tmux.conf, add

setenv -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
set -g update-environment -r

On attaching to a tmux session, this will set SSH_AUTH_SOCK to the path we just created.

Afterwards:

• reload byobu with F5
• logout and login again
• test with ssh-add