Various different types of Virtual Private Networks (VPN) exist since the early days of computers. Wireguard is one of those technologies and the one described here, with all it’s benefits and pitfalls.

Wireguard

Wireguard is a protocol similar to IPSec. It allows traffic to flow between 2 machines which are connected together either directly, over multiple hops eg. the internet, or any other mechanism transporting UDP packages. The development team of Wireguard in addition to the core implementation of the protocol also provides a tools that allows easy managing it, one of those is wg-quick.

Notes on the terminology: there is no differenciation configuration wise between server and client in the Wireguard protocl. Both are configured the same way and will only differ in the connected-to devices or by having a more complex network routing setup internally.

Base configuration per tunnel always contains:

private key of the server (client1)
public key to corresponding private key (pub1)
listen port

Additional settings contain one or more clients that are connected to this one, given:

public key of client2

This allows those writing a configuration for those 2 seperate clients that trust each other. The definition of trust in this form of communication between 2 entities is based on a cryptographic mechanism called public-key-cryptography, allowing one part of the system to encrypt a message agains a public key that only the holder of the corresponding private key can decrypt. Further reading can be done here

To actually connect the 2 clients, we need a way for one of them to talk to the other. Networks are based on addresses, at the moment the most common used ones are IPv4 (some providers allowing IPv6 as well). As long as normal package flow via UDP between those 2 clients is possible on the defined ports, encrypted communication will be possible once configration is set up.

TLDR; Create private key on clients, exchange the public key and reachable address of at least 1 client, create configuration for them, see Wireguard website for a config example.

Pitfalls

Actually this is about debugging, and most of the time either the confiuration has a copy & paste error, or connectivity between the clients didnt work to begin with. In the output of wg, one will always see the listen port of the client, using networking base default debugging tools like tcpdump -nettvi any port $wireguardport on the client, one can see if the client receives a package from the remote client, same works for seeing if the local client is actually sending a package to the remote client. Having verified that packages are sent by the client and taking the correct path and arriving at the remote client, one can check the configured publickey of the remote client corresponds to the local private key (doing the same for the remote client as well). Having verified the network flows and cryptographic configuration is correct as well, outstanding internal config is the AllowedIPs (that have not been mentioned so far :p). AllowedIPs being one of the most crucial parts of Wireguard, as they define internally 2 things:

To which client (if multiple are configured on 1 interface) to actually send the package (the best fitting one will be chosen, so x.x.x.42/32 is taken, even when x.x.x.0/24 is configured for another peer)
What IP source packages to allow from that peer (eg. when AllowedIPs is set to x.x.x.42/32, but package comes from x.x.x.21, it will only be visible in the external tcpdump, but the wireguard interface will silently drop it)

Having verified all above working, the next problem that often comes up is the MTU, and needs to be set to maximum 1420 as of this mail thread that contains lots of useful information on this topic. Symptoms of MTU mismatch are: some packages (eg. ping) make it, but once actual traffic is applied through the tunnel, packages are getting dropped.

Routing

Routing is something that everyone uses and too few people experiment with. Most Linux based systems will allow simple set of route via ip route add 10.127.0.0/24 dev netw1, where netw1 can be any type of network that supports transmission of IP packages, eg. a Wireguard tunnel. This concludes the routing part.

Dynamic routing is a form of routing that has software defined rules behind it that allow the running tool (eg. bird to change routing tables dynmically during runtime based on certain events like an keep-alive ping to fail or upstream connected devices reporting a route table update. A good learning platform and experimentation suite is available by setting up a virtual network at home (eg. via gns3) or by joining an experimental networking communite like dn42.