Etikettarkiv: linux

Sleep resume in Ubuntu screws up mouse pad

I found that my Lenovo laptops did not always get the mousepad right when coming out of sleep or hibernate. After a bit of research I found that a modprobe remove and insert of the psmouse kernel module did the trick.

To automatize this you can insert a file in the systemd control structure to fix the problem yourself (if you are experiencing it). Below is a block of code. Save this to a file in /lib/systemd/system-sleep/touchpad


#!/bin/bash
# 2019 Täpp-Anders Sikvall
# Reinsert kernel module for mouse pad on lenovo after waking up
# from a sleep or a suspend so that things like gestures work 
# properly
# bugs to anders@sikvall.se
 
case "$1" in
   pre)
     exit 0                  # Do nothing just return
     ;;
   post)
     sleep 3                 # Wait for system to stabilize
     modprobe -r psmouse     # Remove psmouse from kernel
     modprobe psmouse        # Reinsert psmouse to kernel
     exit 0                  # Return no error
     ;;
   *)
     exit 1                  # Normally we should not be here
     ;;                      # but if we are, return error
 esac

Borg Backup

The Borg

A very useful utility is the Borg Backup system, or just Borg. It’s a deduplicating backup system meaning that is scans the files and when it finds data that is already in the backup the data in the second and all other subsequent files are replaced with a reference to the first instance of that data.

The idea is that the same data is only stored once. All the backups you take after the initial one stores only the differences and the new data that has been accumulated since the last backup. This means that backing up after the initial backup is done is very fast, efficient, saves bandwidth and storage space.

Traditional backups have usually a full backup every month and then they take increments daily or so. If you need to restore a file you need to take the latest full backup, then apply each increment that was taken after. With the borg backup that is not necessary as you can view the file system exactly as it looked upon each and every backup point taken.

In fact you can mount the whole backup as a file system and then traverse it from there. It’s very effective. So let’s get started because face it, you don’t back up as much as you should do!

Borg can be used for multiple platforms but my commands here will be for linux.

The first step is to create a repository, this may sit on a different machine, NAS, attached USB drive or even on the same machine, of course you want multiple backups really so you can take the borg backup locally and then rsync it to as many locations as you feel is necessary.

Take the backup

The first step is to create the dir of the the backup repo and then we need to initialize it for being used with borg. This is quite simply done as:

$ sudo mkdir /bup
$ sudo borg init /bup

When the repository has thus been created it is time for the first initial backup. The format should be clear in a bit, it’s not complicated and can look like this:

$ sudo borg create --progress --info --stats /bup::lenovo-170202_163423 /home /root /boot /etc /var

The command above should be a single line. The first thing we give to borg is the command, in this case it is create to create a new backup set for us. Then we have some flags, –progress shows a progress indicator while borg is working that details also the number of bytes being read, backed, compressed and deduplicated. The next –info sets the information level borg presents to us and –stats lets borg summarize the operation with some statistics.

The next part of the command the /bup::lenovo-170202_163423 specifies the backup location and backup name. The name is given after the double colon :: mark. In this case its composed of the date yymmdd and time hhmmss of when the backup is started, doing that makes it easy to find the right set of data later when a restore is needed.

Why did I prefix it with lenovo? Well my main linux laptop is a lenovo and I also have other computers, like an ASUS laptop etc.  The beauty with deduplicating backups is that I backup multiple machines to the same repo. By doing that it will deduplicate across the machines and if I have the same files and data in multiple places it will just be replaced by references to the data that is already in the backup.

The final part of the command is just all paths I want to include in this backup. They can vary from time to time. I might backup /home daily but /root only once a week if I want. No problem at all with borg.

Restoring a backup

No system of backup is actually deployed before you have attempted and successfully retrieved data from it so that you know what to do in an emergency as well as being able to extract old data mistakenly erased or restore a full system after a hard drive crash.

Restoring a borg backup works a little different from what you may be used to. First of all you can of course extract the data fully or just single files if you know their paths just like with any other backup system. The restore command is called extract in borg.

$ sudo borg extract /bup::lenovo-170202_163423

This will extract the entire archive and then you can move the files into their respective locations. You can also extract for example only the etc folder from the archive:

$ sudo borg extract /bup::lenovo-170202_163423 etc

Extraction always writes in the current working directory. Therefore you should first extract then move the files into their correct location in your file system or if  all the backups are taken from the root of the file system / then you can cd there before extracting but I recommend extracting on a different volume first and then restoring from there. The reason is that there is usually a lot of stuff in a backup that you may not always want to restore.

Mounting the backup as a file system

So borg actually offers another way also. You can mount the backup as a volume, or you can mount the whole repo and see all the backup points made, select which one you want and then just copy the files from there to the live system.

$ sudo borg mount /bup::lenovo-170202_163423 /mnt

This will mount the backup lenovo-170202_163423 in the file system at /mnt. You can then cd to /mnt and then use cp etc to copy the files to their right places.

When done you can dismount it (otherwise other processes can’t backup, the repo is locked while mounted)

$ sudo umount /mnt

Borg uses fuserfs to mount local directories.

You may also mount the whole repository:

$ sudo borg mount /bup /mnt

Now when you go into the /mnt folder you will see all your backup names as directories:

$ ls
161204_040001 170101_203409 170113_040001 170117_040001 170121_040001 170125_010344 170128_030332
161206_040001 170108_040001 170114_040001 170118_040001 170122_214910 170125_040001 170128_040001
161218_174848 170111_040001 170115_040002 170119_040001 170123_040001 170126_040001 170129_040001
161225_040001 170112_040001 170116_040001 170120_040001 170124_040002 170127_040001 170201_082851

As you can see I generally name my backups with YYMMDD_HHMMSS just so it’s easy for me to find a specific date.

I can then cd to one of them

$ cd 170112_040001
$ ls
boot etc home root var vmlinuz vmlinuz.old

When done, don’t forget to unmount the archive as no new backups can be taken while it is mounted.

There you go. Start using.

 

More effective CIDR-blocking

Previously we have talked about how to block certain addresses using the firewall in Linux (iptables) but if you have a large number of CIDR blocks, say whole countries like China (about 7000 blocks) this will not be keen on the CPU in the server.

Especially the script that inserts it by repeatedly calling iptables. The first few hundred calls will be quit but then is slows down as the kernel won’t process so many insertions in the iptables lists.

There is another way that is just as effective called blackholing the ip ranges you wish to block from your server. This is done by adding routes for those packest that leads nowhere.

# ip route add blackhole <ip address>

This works quite beautifully with tens of thousands of addresses of course. As before we should read the CIDR files we want in order to create the null routes that is needed.

Here is a script that will read a directory of CIDR files and null route all of them.

for f in /etc/iptables/hosts-banned/*
do
    LINES=$(wc -l $f | awk '{print $1}')
    echo -n "Processing k-line file $f with $LINES blocks... "
    date +"%H:%M:%S"
    while read p
    do
        ip route add blackhole $p
    done < $f
done

The CIDR files in this case resides in /etc/iptables/hosts-banned/ an they can be gotten from online or you may add any address ranges you want, perhaps based on automatic firewalling.

To remove a certain blacholed range or ip you can do the same thing again changin the ip route add to an ip route del command instead.

ip route del <ip address>

You can produce a script that removes them by doing the following:

ip route | grep blackhoe | awk '{ print "ip route del " $2 }' >unblock
chmod 700 unblock
./unblock

That’s it, they are all now cleared.

 

Ubuntu persistent network interface names

In Ubuntu 16.x the systemd is used more than in the previous versions. This also means it is now responsible for setting up your network cards. Many people have been sort of surprised that their eth0 have changed to something like enp0s25. This is of course an improvement from before, there was no real telling in which order NICs would be assigned names so potentially a hardware change could offset eth0 and eth1 and so on.

The new way is actually not too bad but if you like me do a lot of manual configurations on the fly to the network interfaces their names can be tedious to type and also remember. But of course there is a rather simple mechanism to change this so you can select your own names for the interfaces such as lan0 and dmz1 or why not wifi plain and simple if there is never to be any more than one wifi card in the computer.

This is a step-by step guide that was tested under Ubuntu 16.10 and worked for me. Please leave your comments if you have problems, improvements or any such things to add.

Getting the names

First of all we need to find out what the names of the NICs we have in the system actually are. Here is a dump from my laptop using the ifconfig command to list all interfaces:

root@kraken:~# ifconfig -a
enp0s25: flags=4098<BROADCAST,MULTICAST> mtu 1500
 ether f0:de:f1:8d:89:fe txqueuelen 1000 (Ethernet)
 RX packets 0 bytes 0 (0.0 B)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 0 bytes 0 (0.0 B)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
 device interrupt 20 memory 0xf2a00000-f2a20000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
 inet 127.0.0.1 netmask 255.0.0.0
 inet6 ::1 prefixlen 128 scopeid 0x10<host>
 loop txqueuelen 1 (Local Loopback)
 RX packets 3143 bytes 204307 (204.3 KB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 3143 bytes 204307 (204.3 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 192.168.1.3 netmask 255.255.255.0 broadcast 192.168.1.255
 inet6 fe80::846f:cc3d:2984:d240 prefixlen 64 scopeid 0x20<link>
 ether 00:24:d7:f0:a3:a4 txqueuelen 1000 (Ethernet)
 RX packets 4600 bytes 5069857 (5.0 MB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 3348 bytes 592050 (592.0 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wwp0s29u1u4i6: flags=4098<BROADCAST,MULTICAST> mtu 1500
 ether 02:80:37:ec:02:00 txqueuelen 1000 (Ethernet)
 RX packets 0 bytes 0 (0.0 B)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 0 bytes 0 (0.0 B)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

We are looking for two things in the above, the MAC address and the name of the network interface card we want to rename. The NICs we have here are named after the type of card, the bus it is attached to etc. What used to be called eth0 is now referred to as enp0s25 and wlan0 is wlp3s0 and there is also a WAN card in the machine called wwp0s29u1u4i6 which definitely is a mouthful.

Okay, so we would like to rename these to more sensible names. First we pick the names such as eth0, wlan0, wan0 etc. Then we note down the MAC address of each card. You find this highlighted in red in the above dump next to the keywork ”ether”. Once we have that we can tell the systemd to rename the cards in the way we want. By connecting the name to the MAC address it should also be persistent and not affected by inserting a new card into the computer system.

In directory /etc/systemd/network we will create the following files:

root@kraken:/etc/systemd/network# ll
 total 20
 drwxr-xr-x 2 root root 4096 Dec 11 04:28 ./
 drwxr-xr-x 5 root root 4096 Nov 24 15:03 ../
 -rw-r--r-- 1 root root 55 Dec 6 23:44 01-eth0.link
 -rw-r--r-- 1 root root 56 Dec 6 23:39 02-wifi.link
 -rw-r--r-- 1 root root 55 Dec 6 23:40 03-wan.link

These link files can be used to match a device and then change its parameters. So they consists of a matching section and then a link section. The first one called 01-eth0.link contains the following lines:

[Match]
  MACAddress=f0:de:f1:8d:89:fe

[Link]
  Name=eth0

We can then create the other ones in the same way. When we are done with that we need to do two things. First we need to update the initial ram file system in boot because some of these may already be up during boot time (such as eth0). This is done with the following command:

root@kraken:/etc/systemd/network# update-initramfs -u
 update-initramfs: Generating /boot/initrd.img-4.8.0-30-generic
 W: Possible missing firmware /lib/firmware/i915/kbl_guc_ver9_14.bin for module i915
 W: Possible missing firmware /lib/firmware/i915/bxt_guc_ver8_7.bin for module i915
 W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.

Once we have done this we can reboot our computer.

When up again we can check the network names again:

anders@kraken:~$ ifconfig -a
eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
 ether f0:de:f1:8d:89:fe txqueuelen 1000 (Ethernet)
 RX packets 0 bytes 0 (0.0 B)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 0 bytes 0 (0.0 B)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
 device interrupt 20 memory 0xf2a00000-f2a20000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
 inet 127.0.0.1 netmask 255.0.0.0
 inet6 ::1 prefixlen 128 scopeid 0x10<host>
 loop txqueuelen 1 (Local Loopback)
 RX packets 1732 bytes 110296 (110.2 KB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 1732 bytes 110296 (110.2 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wan0: flags=4098<BROADCAST,MULTICAST> mtu 1500
 ether 02:80:37:ec:02:00 txqueuelen 1000 (Ethernet)
 RX packets 0 bytes 0 (0.0 B)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 0 bytes 0 (0.0 B)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
 inet 192.168.1.3 netmask 255.255.255.0 broadcast 192.168.1.255
 inet6 fe80::1ed7:d5ac:433d:70c5 prefixlen 64 scopeid 0x20<link>
 ether 00:24:d7:f0:a3:a4 txqueuelen 1000 (Ethernet)
 RX packets 93 bytes 71048 (71.0 KB)
 RX errors 0 dropped 0 overruns 0 frame 0
 TX packets 137 bytes 18113 (18.1 KB)
 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

As you can see we now have eth0, wlan0 and wan0 instead of the default names. So if you like me work from the command line mainly you will be happy that ifconfig eth0 now works just like it did before the systemd entered the scene and if you have firewall scripts you can of course rename your interface to something that is useful to you such as lan, wan and dmz or whatever makes sense.

IP-Tables control of traffic by Country

Recently I have had quite a few bouts with people trying all sorts of nefarious things with my VPS from various places in the world. I realized the best way is to take whole countries out of the equation and I wanted to make this a nice easy way of doing this.

First of all, I soon realized that I needed to be a little bit restrictive but since some of the domains that runs on the server is probably legitimately accessed I wanted to divide all countries up in three categories:

0 – No restrictions
1 – Restricted access, basically just allowing ICMP and http on port 80
2 – Complete blocking, drop all packets

So countries in the 0 class would be Sweden and all countries where there is a reasonable legal system and where I have not yet seen too many attempts on the security from. Class 1 would be places like Russia, where there are legit traffic but also a lot of crap coming from and Class 2 would be china where there is likely no legit traffic and still a lot of attempts on the security.

First of all, you need to get hold of a zone file divided on country by country. This is also called a CIDR file (Classless Internet Domain Routing) where all the IP blocks assigned on an international level are put in the right file. You can find this here. Just download the file with all the blocks in and keep it somewhere. You may want to refresh this now and then, say on a montly basis or so.

I am assuming you already have an iptables script file and that you are just looking at adding this functionality. Open the script file and add the following to it

# Loop over all lists of banned networks
# Any rules below this will not work on these ranges as they will
# drop before they reach any other rule. If you want to open some ports
# even for banned countries, then you need to put those rules in front
# of this rule!
echo "Kill line certain CIDR, one way of blocking suspect countries!"
for f in /etc/iptables/banned-hosts/*
do
  echo "Processing k-line file $f..."
  while read p
  do
    $IPTABLES -A INPUT -s $p -j CBLK
  done < $f
done

This script should be placed before any rules that will allow any traffic what so ever!

After this part you place the rules that allows any traffic you wish to allow from the restricted countries and then you place this after those rules:

# Restricted hosts here from CIDR files in the restricted session
# these guys will only be able to do ICMP and http, nothing else
# and that should be quite a few countries
echo "Restrict line certain CIDR, one way of blocking suspect countries!"
for f in /etc/iptables/restricted-hosts/*
do
  echo "Processing r-line file $f..."
  while read p
  do
    $IPTABLES -A INPUT -s $p -j CBLK
  done < $f
done

Now you should create the following directories:

sudo mkdir /etc/iptables
sudo mkdir /etc/iptables/banned-hosts/
sudo mkdir /etc/iptables/restricted-hosts/

Explode the file you downloaded with all the IP Blocks in country by country into the /etc/iptables/banned-hosts/ directory and you should get a bunch of files called af.zone, al.zone and so on. Each of these referrs to a ISO 2 letter country code.

Do not run the iptables script at this point. Start by removing the file for your own country. In my case that would be se.zone for Sweden. Your mileage may vary here. Refer to this page if you do not know the country codes (which are the same as these countries internet domains).

Delete the files that you do not wish to impose any restrictions on.

Move the files for the countries you want to restrict to the /etc/iptables/restricted-hosts/ dir.

Anything remaining when you are done in the /etc/iptables/banned-hosts/ will be denied access when you run your iptables script.

So run the script now, it may take some time.

When you are done run the command iptables-save > /etc/iptables/tables to save your iptables then add the line in /etc/rc.local or some other similar place iptables-restore < /etc/iptables/tables in order to automatically load your tables on boot time.

Full example

echo "### IP-tables ###"

IPTABLES=/sbin/iptables

echo "Default policies."
$IPTABLES -P INPUT ACCEPT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT

echo "Flushing old rules"
$IPTABLES -F
$IPTABLES -X

echo "Create LOGDROP chain"
$IPTABLES -N LOGDROP
$IPTABLES -A LOGDROP -j LOG --log-prefix "IPT: DROP " --log-level 7
$IPTABLES -A LOGDROP -j DROP

echo "Create LOGACCEPT chain"
$IPTABLES -N LOGACCEPT
$IPTABLES -A LOGACCEPT -j LOG --log-prefix "IPT: ACCEPT " --log-level 7
$IPTABLES -A LOGACCEPT -j ACCEPT

echo "Create INVALIDDROP chain"
$IPTABLES -N INVALIDDROP
$IPTABLES -A INVALIDDROP -j LOG --log-prefix "IPT: INVALID " --log-level 7
$IPTABLES -A INVALIDDROP -j DROP

echo "Killfile certain IP chain"
$IPTABLES -N BANNED
$IPTABLES -A BANNED -j LOG --log-prefix "IPT: BANNED " --log-level 7
$IPTABLES -A BANNED -j DROP

echo "Create a country block chain"
$IPTABLES -N CBLK
$IPTABLES -A CBLK -j LOG --log-prefix "IPT: CBLK " --log-level 7
$IPTABLES -A CBLK -j DROP

# Loop over all lists of banned networks
# Any rules below this will not work on these ranges as they will
# drop before they reach any other rule. If you want to open some ports
# even for banned countries, then you need to put those rules in front
# of this rule!

echo "Kill line certain CIDR, one way of blocking suspect countries!"
for f in /etc/iptables/banned-hosts/*
do
 echo "Processing k-line file $f..."
  while read p
  do
    $IPTABLES -A INPUT -s $p -j CBLK
  done < $f
done

echo "Enabling ICMP"
$IPTABLES -A INPUT  -p icmp -j LOGACCEPT
$IPTABLES -A OUTPUT -p icmp -j LOGACCEPT

echo "Enabling http on standard port"
$IPTABLES -A INPUT  -p tcp --dport http -m state --state NEW         -j LOGACCEPT
$IPTABLES -A INPUT  -p tcp --dport http -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --sport http -m state --state ESTABLISHED -j ACCEPT

# Restricted hosts here from CIDR files in the restricted session
# these guys will only be able to do ICMP and http, nothing else
# and that should be quite a few countries

echo "Restrict line certain CIDR, one way of blocking suspect countries!"
for f in /etc/iptables/restricted-hosts/*
do
 echo "Processing r-line file $f..."
  while read p
  do
    $IPTABLES -A INPUT -s $p -j CBLK
  done < $f
done

echo "Dropping invalid packets"
$IPTABLES -A INPUT -m state --state INVALID -j INVALIDDROP

echo "Enabling DNS server connections."
$IPTABLES -A INPUT  -p tcp --sport domain -j ACCEPT
$IPTABLES -A INPUT  -p udp --sport domain -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --dport domain -j ACCEPT
$IPTABLES -A OUTPUT -p udp --dport domain -j ACCEPT

echo "Enabling NTP server connections."
$IPTABLES -A INPUT  -p udp --sport 123 -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p udp --dport 123 -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p udp --dport 123 -m state --state NEW         -j LOGACCEPT

echo "Applying rules for inbound and outbound ssh"
$IPTABLES -A INPUT  -p tcp --dport gopher -m state --state NEW         -j LOGACCEPT
$IPTABLES -A INPUT  -p tcp --dport gopher -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --sport gopher -m state --state ESTABLISHED -j ACCEPT

$IPTABLES -A INPUT  -p tcp --sport gopher -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --dport gopher -m state --state NEW         -j LOGACCEPT
$IPTABLES -A OUTPUT -p tcp --dport gopher -m state --state ESTABLISHED -j ACCEPT

echo "Applying rules for outbound ssh standard port"
$IPTABLES -A INPUT  -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --dport 22 -m state --state NEW         -j LOGACCEPT
$IPTABLES -A OUTPUT -p tcp --dport 22 -m state --state ESTABLISHED -j ACCEPT

echo "Enabling https traffic out from this machine"
$IPTABLES -A INPUT  -p tcp --sport 443  -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT  -p tcp --sport 443  -m state --state NEW         -j LOGACCEPT
$IPTABLES -A OUTPUT -p tcp --dport 443  -m state --state ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT -p tcp --dport 443  -m state --state NEW         -j LOGACCEPT

echo "Dropping all other input packets."
$IPTABLES -A INPUT -j LOGDROP

echo "Done."

echo "IPV6 setting policy"

/sbin/ip6tables -P INPUT DROP
/sbin/ip6tables -P OUTPUT DROP
/sbin/ip6tables -P FORWARD DROP

echo "IPV6 flushing tables"

/sbin/ip6tables -F

echo "IPV6 Done."