Author: darkskiez

Making Shared Libraries with Go – Versioned Symbols

darkskiez Leave a comment

Having previously made u2f-luks tools and scripts to allow me to use my surplus U2F (FIDO1) keys as a second factor for disk encryption, it was with some excitement that I read about the newly merged cryptsetup loadable plugin feature. The script used a custom keyscript for cryptsetup, but keyscripts will not be supported by systemd. This meant that the required volumes need to be unlocked and mounted in initramfs with the original cryptsetup tooling, which has its own pitfalls (removable devices etc)

Lets convert my keyscript into a cryptsetup plugin!

How do I convert my u2-luks tool, written in Go, into a shared library for a C program? cgo (godoc) is the Go and C interface layer, and can create shared libraries as nicely documented by other bloggers. The cryptsetup API (libcryptsetup.h) is fairly simple, only one function is required “cryptsetup_token_open”, so this sounded like a perfect evening project, but I ended up spending a few evenings and implementing most of them.

The required function signature from libcryptsetup.h:

typedef int (*crypt_token_open_func) (
	struct crypt_device *cd,
	int token,
	char **buffer,
	size_t *buffer_len,
	void *usrptr);

What does this look like in Go?

package main

// #cgo pkg-config: libcryptsetup
// #cgo LDFLAGS: -Wl,--version-script=cryptsetup_token.map
// #include <errno.h>
// #include <stdlib.h>
// #include <libcryptsetup.h>
import "C"

// ... snip ... 

//export cryptsetup_token_open
func cryptsetup_token_open(cd *C.struct_crypt_device, token C.int, 
    password **C.char, password_len *C.size_t, usrptr *C.char) C.int {

    if cerr := C.crypt_token_json_get(cd, token, &cjson); cerr < 0 {
        C.crypt_log(cd, C.CRYPT_LOG_ERROR,
            C.CString(fmt.Sprintf("token_json_get failed: errno %v\n", -cerr)))
        return -C.EINVAL
    }
    // ... snip ... 
}

The main takeaways I’d like to point out from this:

  • cgo shared libraries are built from the main package
  • C include and #cgo pragmas must appear directly before the import “C”
  • cgo exported functions are lowercase named
  • cgo exported functions have an export comment
  • C structs appear in the C package (with a struct_ prefix), that match types from the included headers.
  • There isn’t a void type directly, its equivalent in Go is unsafe.Pointer, but I didn’t want to use unsafe package, so I left that with a char pointer type, though I don’t plan to use it.
  • C likes negative Enum values as errors (-C.EINVAL), which are easy to use but look a bit weird in Go.
  • I can pass the referenced struct back into other calls in the API to get other needed data than provided in the function signature.
  • Go Strings need converted to C Strings – for both the format and to protect them from garbage collection by the Go Runtime, only pointers created by the C package can be passed into cgo calls.

The most difficult nut to crack was figuring out how to version the exported symbols, I tried not bothering to set any versions, but the module failed loading with symbol not found messages:

$ cryptsetup open --type luks  --debug ./testcontainer testcontainer
[snip]
Trying to load /lib/x86_64-linux-gnu/cryptsetup/libcryptsetup-token-u2f.so
/lib/x86_64-linux-gnu/cryptsetup/libcryptsetup-token-u2f.so: undefined symbol: cryptsetup_token_open, version CRYPTSETUP_TOKEN_1.0
[snip]

I thought I could export the function with a new name:

//export cryptsetup_token_open@CRYPTSETUP_TOKEN_1.0

But the compiler complained:

./main.go:28:1: export comment has wrong name "cryptsetup_token_open@CRYPTSETUP_TOKEN_1.0", want "cryptsetup_token_open"

So, time to read up on how linking works with versioned symbols. Eventually I ended up at GNU ld Manual – Version Scripts that explains how to configure the linker to add the versions, so lets try it out, I created the following file.

CRYPTSETUP_TOKEN_1.0 {
   global:
    cryptsetup_token_open;
    cryptsetup_token_open_pin;
    cryptsetup_token_validate;
    cryptsetup_token_dump;
    cryptsetup_token_version;
};

Adding the #cgo LDFLAGS pragma to say: tell the compiler when linking (LDFLAGS), to pass the linker a flag (-Wl) setting (--version-script) to this file cryptsetup_token.map

// #cgo LDFLAGS: -Wl,--version-script=cryptsetup_token.map

This however results in the following compiler error

go build github.com/darkskiez/u2f-luks/lukstoken/plugin: invalid flag in #cgo LDFLAGS: -Wl,--version-script=cryptsetup_token.map

Which is fixed by setting CGO_LDFLAGS_ALLOW environment to an appropriate regex, so now the Makefile looks like:

libcrypt-token-u2f.so: main.go cryptsetup_token.map
    CGO_LDFLAGS_ALLOW='-Wl,--version-script=.*' go build -x \
    -buildmode c-shared -o libcrypt-token-u2f.so

Success!

The module loads and works, supports a presence only mode – for unlocking with just a touch, or with a passphrase/pin and presence. I had hoped to make cryptsetup automatically prompt the user when a password was required in addition to the token, but this does not yet work – cryptsetup issue #670 – there are unresolved UX implications that are amplified in severity when using tokens that may self-erase after incorrect guesses (eg. FIDO2 tokens lock out after 8 attempts.)

Unresolved issues: I’d like to use the crypt_safe_memzero or a comparable go function to make sure the key data is securely erased from the go runtime memory, but you cant (safely) pass go memory into C calls (cgo – Passing Pointers), so I’m not sure what would be best here.

My next steps will be investigating how systemd integrates fido2 tokens with cryptsetup, which wasn’t released at the time I wrote this module, and seeing if I can improve the UX with a similar system.

The rest of the code is available on github.com/darkskiez/u2f-luks/tree/master/lukstoken

Solar Powered Temperature & Humidity BLE Beacon Fun With Linux

darkskiez Leave a comment

Overview

Solar Powered temperature and humidity sensing beacon with super capacitor storage for nighttime operation.

http://www.cypress.com/documentation/development-kitsboards/cyalkit-e02-solar-powered-ble-sensor-beacon-reference-design

http://www.cypress.com/documentation/development-kitsboards/cyalkit-e03-solar-powered-ble-sensor-5-pack

Another low effort blog post with mainly notes to myself that others might find useful.

TODO: pics

TODO: screenshot of app

TODO: Crude Linux dumping commands

sudo hcitool lescan --duplicates --passive >/dev/null &
sudo hcidump --raw|grep -A2 "50 A0 00" 

> 04 3E 2A 02 01 03 00 28 20 03 50 A0 00 1E 02 01 04 1A FF 4C 
  00 02 15 00 05 00 01 00 00 10 00 80 00 00 80 5F 9B 01 31 00 
  01 6F 68 C3 B9
--
> 04 3E 2A 02 01 03 00 1A 1B 17 50 A0 00 1E 02 01 04 1A FF 4C 
  00 02 15 00 05 00 01 00 00 10 00 80 00 00 80 5F 9B 01 31 00 
  02 6C 66 C3 C5 

# Temperature Sensor
TEMP [℃] = 175.72 x (TEMP x 256) / 65536 – 46.85
Example: TEMP = 0x67 [hex] = 103[dec]
TEMP [℃] = 175.72 x (103 x 256) / 65536 – 46.85 = 23.85[℃]

# Humidity Sensor
RH[%] = 125 x (HUMIDITY x 256) / 65536 – 6
Example: HUMIDITY = 0x74 [hex] = 116[dec]
RH[%] = 125 x ( 116 x 256) / 65536 – 6 = 50.64[%]

Sometimes errors when starting lescan

sudo hciconfig hci0 reset

USB Debug Stick

The supplied debug stick for programming the beacons is partially usable with linux with no special drivers. The stick has a mode switch for BLE scanning mode and interfacing with the beacons serial interface / charging it.

$ sudo dmesg
[132354.828122] usb 1-1: New USB device found, idVendor=04b4, idProduct=f139
[132354.828133] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=128
[132354.828140] usb 1-1: Product: Cypress KitProg
[132354.828147] usb 1-1: Manufacturer: Cypress Semiconductor
[132354.828153] usb 1-1: SerialNumber: BLE1304041XXXXXXXXX
[132354.837228] hid-generic 0003:04B4:F139.000C: hiddev0,hidraw0: USB HID v1.11 Device [Cypress Semiconductor Cypress KitProg] on usb-0000:00:14.0-1/input0
[132354.838599] cdc_acm 1-1:1.2: ttyACM0: USB ACM device

# With debian or probably ubuntu
sudo apt install ckermit

cat < cyalkit.kermit
set line /dev/ttyACM0
set speed 115200
set handshake none
set flow-control none
set carrier-watch off
set local-echo on
set terminal newline-mode on
connect
quit
EOF

kermit -y cyalkit.kermit
# Press ctrl-\ and then q to quit

TODO: device config

help
Get uuid, input:uuid+Enter
Set uuid, input:uuid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx+Enter
Get major, input:major+Enter
Set major, input:major xxxx+Enter
Get minor, input:minor+Enter
Set minor, input:minor xxxx+Enter
Get txpwr, input:txpwr+Enter
Set txpwr, input:txpwr *+Enter
Get rssi, input:rssi+Enter
Set rssi, input:rssi *+Enter
Get itrvl, input:itrvl+Enter
Set itrvl, input:itrvl *+Enter
Get coid, input:coid+Enter
Set coid, input:coid xxxx+Enter

Get nid, input: ednid+Enter
Set nid, input: ednid x...x(20 char)+Enter
Get bid, input: edbid+Enter
Set bid, input: edbid x...x(12 char)+Enter
Get edtxpwr, input:edtxpwr+Enter
Set edtxpwr, input:edtxpwr * * * *+Enter
Get edframe, input:edframe+Enter
Set edframe, input:edframe + one or some of uid/url/tlm+Enter
Get edstate, input: edstate+Enter
Set edlock, input: edlock xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx+Enter
Set edunlock, input: edunlock xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx+Enter
Get eduri, input:eduri+Enter
Set eduri, input:eduri (url or hex string)+Enter
Get edadpwr, input:edadpwr+Enter
Set edadpwr, input:edadpwr * * * *+Enter
Get edtxmode, input:edtxmode+Enter
Set edtxmode, input:edtxmode LOWEST/LOW/MEDIUM/HIGH+Enter
Geblue 0Command format error!!er
Set editrvl, input:editrvl *+Enter
edreset, input: edreset+Enter

Get sensor, input:sensor+Enter
Set sensor, input:sensor on/off+Enter
Get mode,input: mode+Enter
Set mode,input: mode BLEBeacon/Eddystone/EDTest+Enter
Initialize, input: init+Enter
Get ver, input: ver+Enter
Exit, input: exit+Enter

Unplug USB Stick.

Attach sensor to the stick.

Push switch towards Usb port

TODO: advert dumping with usb stick

Push switch away from usb port

cat the tty

TODO: linux parsing

TODO: iBeacon format
Different advertisment flags might break simpler softer – link to beacontools
Major is beacon id
Minor is temp/humidity – Equations
Shame its not eddystone with sensor values, thought programmable firmwares from cypress community forums might help

Wireshark Bluetooth Dumping

$ sudo hcitool lescan --duplicates --passive >/dev/null &
$ xhost +local:
$ sudo wireshark
$ xhost -

TODO: Wireshare dump example
00:A0:50:XX:XX:XX mac

TODO: Crude Range analysis

Can I change tx power in a useful way?

Does itrvl setting in DM allow a different timer / battery usage ?

I added support to beacontools for this format (https://github.com/citruz/beacontools/pull/8)

sudo apt install libbluetooth-dev
sudo pip install beacontools[scan]

Backup Pi

darkskiez Leave a comment

My online backup provider Crashplan cancelled home users accounts to focus on business users.. Now what?

Spoilers: This is unfinished at time of publishing. I’ll update this post as I progress towards a solution I’m happy with.

There aren’t any obvious Linux compatible unlimited (priced per computer rather than per GB) online backup solutions left, and I didnt fancy running one in a VM or any other jiggery pokery like that.

Cloud providers would be ideal, BackBlaze B2 is a good candidate https://www.backblaze.com/b2/cloud-storage-pricing.html, but Wasabi cheapest I’ve found, https://wasabi.com/pricing/. It still works out for 8TiB over 3 Years with one full restore to cost $1468. I suspect any cheaper providers will change/increase their pricing model or close in a few years and I don’t want to find a new solution every two years, but this is still too expensive for my needs.

Potential Solution: run my own offsite backup system.

Hardware List

$35 Raspberry Pi 3

$16 SD Card

$30 USB Dual HDD Dock

$275 2 * WD RED 8TiB

Total $666

Onsite backup seeding saves normal ~month of initial backup nonsense.

Configuring the Pi

Power

Works on normal usb power as not doing anything too fancy to need the extra watts.

Operating system

Raspian no-gui whatever was latest

Basic WiFi Networking

Wifi setup at my location has public WiFi, that will do

#/etc/network/interfaces.d/wlan0
auto wlan0
iface wlan0 inet dhcp
wireless-essid GuestWifiNoCrypto

VPN

My pi is installed in a firewalled location and can’t act as a server that my backup machine can reach.

Simplest VPN: ssh tunnel!

However, using ssh to create a tunnel is very temporary, autossh is a solution that will start ssh on boot and restart it if the connection fails, but it needs some encouragement to have the right / most robust config. The flag that bit me that was missing from other examples is the ExitOnForwardFailure, until I had that the sshvpn was reconnecting, but not establishing the reverse tunnel when an old stale one hadn’t quite cleaned up.

homemachine$ sudo adduser --system pitunnel
# First step, create and install a ssh key for the inbound tunnel:
pi$ ssh-keygen -t rsa -b 4096
# Install key into my home machine.
pi$ ssh-copy-id [email protected]
# Test the key, this should connect without a password
pi$ ssh [email protected]

Here is my autossh setup.

# /lib/systemd/system/autossh.service 
[Unit]
Description=tunnel
After=network.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
User=pi
Restart=always
RestartSec=5
Environment=AUTOSSH_GATETIME=0
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" -p 22 -l pitunnel my.dyndns.address -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_rsa

[Install]
WantedBy=multi-user.target

Now Enable it / Start it and check it is running ok.

$sudo systemctl enable autossh
$sudo systemctl start autossh
$sudo systemctl status autossh
● autossh.service - tunnel
   Loaded: loaded (/lib/systemd/system/autossh.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-10-xx 01:01:38 UTC; 0 days ago
 Main PID: 739 (autossh)
   CGroup: /system.slice/autossh.service
           ├─ 739 /usr/lib/autossh/autossh -M 0 -N -q -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -o ExitOnForwardFailure yes -p 22 -l pitunnel my.dyndns.address -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_
           └─2748 /usr/bin/ssh -N -q -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -o ExitOnForwardFailure yes -p 22 -l pitunnel -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_rsa my.dyndns.address

Set up convenience ssh host target, so you dont need all the usernames and ports in every ssh and scp you want to use:

# /root/.ssh/config 
Host pitunnel
    Hostname 127.0.0.1
    User borg
    port 2200
    IdentityFile ~/.ssh/borg

Test it with 

ssh pitunnel

It should connect with no password prompt.

Installing the disks

TODO I’ve not really finished this section.

I had initially used a btrfs raid1 mirror instead of a linux mdadm raid, but one of the drives in my dock had connectivity issues.

  • Running a btrfs balance took far far too long (days). I have no idea why btrfs takes so long to rebalance, it is not IO speed limited. I’d like someone to benchmark btrfs rebalance in ram and see how slow it still is.
  • It would crash when trying to mount after a bad reset, needing to be mounted from my PC with more ram / different kernel / whatever else, in order to clean up the filesystem state. This happened more than once.
  • It would force the filesystem read-only after one degraded mount after I took a disk out, this is supposedly fixed in Linux 4.14: http://lkml.iu.edu/hypermail/linux/kernel/1709.1/00609.html

I refuse to use a non check-summing filesystem, and zfs is out of the question for me right now (not open for discussion here, may work for you), so I’m sticking with btrfs, but ignoring the raid modes.

$EDITOR /etc/fstab

/dev/disk/by-label/borgbackup /backup btrfs defaults,noatime,nofail 0 2

linux raid jiggery, I’ve done this a squillion times, but havent actually set it up yet.

$ sudo apt install smartmontools

TODO: You should probably configure a mail relay so emailing from the pi works.

watchdog

The systemd watchdog looks an interesting framework, but didn’t have an immediately obvious way of checking net connection was ok, so I opted for the lazy/easy watchdog package..

sudo apt install watchdog

$EDITOR /etc/watchdog.conf
# set the following lines
ping = 8.8.8.8 # google dns
interface = wlan0 # check wifi interface
watchdog-timeout = 10 # largest pi watchdog value, default  of 60 errors

Misc Config

I had the Pi plugged into a spare HDMI port on a nearby monitor to be able to check in on it from time to time and plugged into a usb switcher to swap keyboard/mouse over.

aliexpress $6 USB KVM Switcher was the cheapest I found, and I just ignore the vga ports

Annoyingly when my machine was locked, the Pi would occasionally reboot or steal input via HDMI to the monitor..

#/boot/config.txt
hdmi_blanking=1

This setting is supposed to fix it, at the cost of breaking media player software horribly.. which is ok, as I dont use it on this. I’ll update my experience.

Backup software

restic

I first tried with restic, it ticked most of my boxes, golang, encrypted, authenticated, de-duplicating, incremental. I however had a few problems, restic seems to have a few problems with pruning over slow links, and it ran out of memory when I tried to prune from the pi itself. It looks like it’s got scope to grow into a solid backup tool, but doesn’t quite meet my low memory / high latency needs at the moment.

Issues I ran into:

No snapshot usage stats https://github.com/restic/restic/issues/693

Random Crash: https://github.com/restic/restic/issues/1251

Poor errors with SFTP backend: https://github.com/restic/restic/issues/1323

Very little progress output on prune: https://github.com/restic/restic/issues/1322

No network stats https://github.com/restic/restic/issues/1033

The restic author is maintaining a detailed feature list of alternative backup software, which I can highly recommend for those exploring backup systems

The most mature alternative candidate seems to be borgbackup. I didn’t really need compression that borgbackup adds, and I didn’t want the overhead that python has, but it seems to tick most boxes.

borgbackup

I used the borgbackup 1.1.1 release from pip, as it fixes some issues I had with clearing out stale locks (after the watchdog resets the pi) and raspian hasn’t updated yet..
Version 1.1.0 managed to crash my host machine due to a bug with it opening every block device, and possibly a secondary bug in my kernel, this needs further investigation for the kernel side – https://github.com/borgbackup/borg/issues/3213

$ sudo pip3 install borgbackup

I dropped the script from the borg backup Quick Start – Automating Backups into /etc/cron.daily/borgbackup I added the -x flag for one filesystem and then added the paths to the mounts I wanted

 export BORG_REPO=ssh://pitunnel/~/repo

Oops I slightly corrupted my repo with the watchdog resets and lock fudgery, the prune was crashing with a missing directory.

Python single threaded, cpu bound, multiple day long check…awarghgjbwakwffkawkfkafhgh, it’s still not finished, I have little idea about its progress (it’s spent all the time Analyzing archive “hostname-2017-10-18T11:44:25.checkpoint (3/19)”, and I have 15 idle cores.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                |
22288 root      20   0 1000796 897456   6984 R 100.0  5.5   3844:24 /usr/bin/python3 /usr/bin/borg check --verbose --repair

Day 7: This breached 10000 minutes 7 days of runtime so I’ve shut it down..

Day 14: I forgot to check up on it for a while, miraculously its been running for a week making backups every day! Hooray

But the repo isnt “healthy”..

$ borg delete $BORG_REPO::darkskiez-2017-10-23T23:05:06.checkpoint
borg.repository.Repository.ObjectNotFound: Object with key fe00c10f6ce4b733d013ee30a60f47284368e4fa47164cb0d8af507dc0acedf0 not found in repository ssh://pitunnel/~/reopo::darkskiez-2017-10-23T23:05:06.checkpoint.

Sod it, I’ll force it and nuke the lot of them.

borg list --short|grep checkpoint|xargs -i borg delete --force $BORG_REPO::{}
forced deletion succeeded, but the deleted archive was corrupted.
borg check --repair is required to free all space.

Argh! borg list still shows a checkpoint file… and borg delete –force is erroring with the same not found error, how hard is it to delete something that isn’t there?!!?

We know how well borg check worked last time, but I really didn’t want it to repair these checkpoints before, maybe now it will work 😐 I don’t have high hopes

using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
SSH command line: ['ssh', 'pitunnel', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 0.72 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/~/repo'
Remote: Resolved repository path to '/backup/repo'
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES
Remote: Starting repository check
Remote: Verified integrity of /backup/repo/index.179252
Remote: Read committed index of transaction 179252
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 179252).
Remote: Segment transaction is    179252
Remote: Determined transaction is 179252
Remote: Found 166446 segments
Remote: Checking segments 100%
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /backup/repo/index.179253
TAM-verified manifest
Analyzing archive darkskiez-2017-10-12T11:01:00 (1/9)
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 179253).
Remote: Verified integrity of /backup/repo/hints.179253
Analyzing archive darkskiez-2017-10-18T10:22:54.checkpoint (2/9)
Archive metadata block is missing!
Analyzing archive darkskiez-2017-10-24T10:02:14 (3/9)
/srv/photos/IMG_2445.JPG: New missing file chunk detected (Byte 0-888561). Replacing with all-zero chunk.
/srv/photos/IMG_2445.JPG: New missing file chunk detected (Byte 888561-1153253). Replacing with all-zero chunk.
item metadata chunk missing [chunk: 000528_fe00c10f6ce4b733d013ee30a60f47284368e4fa47164cb0d8af507dc0acedf0]

It got here before in the 7 day run, so I assume it doesn’t do the fixing so well.

I’m going to delete this specific archive.

Day 21: For various reasons this check couldnt finish either, still takes too long..

I am very sad. I am resisting the strong urge to write my own backup system. Perhaps I should retry restic but with much smaller repos. Perhaps I should consider trying another system… knoxite looks better than restic on the box, but has nearly zero development activity / users. I refused to believe it is mature with this evidence, but then again, I was wholly wrong about borg backup it seems.

btrfs snapshots

Ok, maybe all this backup software is too fancy.. Maybe btrfs is fancy enough a filesystem that I don’t need the extra things..

There are a few btrfs backup scripts, based on the principals in: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

So I have quite a lot of data I wish to “exclude” from my backup, and reorganizing my filesystem to have different (sub)volumes doesn’t sound like fun, especially when we are talking about lots of ‘cache’ dirs and what not.

I wonder if btrfs snapshot diffing is intelligent enough that if I snapshot fs A into S1 and S2 in different times, prune the files I don’t want in S1 and S2, and do a btrfs send of the deltas S1-S2 if it will work at all, and be intelligent enough to not include the deleted files.

Crude test time.

btrfs subvol create /home/snaptest
# Create a bigfile1 /home/snaptest/b1
btrfs subvolume snapshot /home/snaptest /home/snaptest-s1
# Create a bigfile2 /home/snaptest/b2
# Create a small file /home/snaptest/s
btrfs subvolume snapshot /home/snaptest /home/snaptest-s2
# Delete b2 from  /home/snaptest-s1/b2 and /home/snaptest-s2/b2

btrfs send -p /home/snaptest-s1 /home/snaptest-s2|wc -c
ERROR: subvolume /home/snaptest-s1 is not read-only
# Woops
btrfs property set -ts /home/snaptest-s1 ro true
btrfs property set -ts /home/snaptest-s2 ro true

btrfs send -p /home/snaptest-s1 /home/snaptest-s2|wc -c
# Wahoo: only a bit bigger than smallfile
btrfs sub del /home/snaptest*

There may be some hope here yet.

If I encrypt the data that is being sent with ‘btrfs send’ that is secure, but, very awkward to remotely use in an emergency when I cant restore the snaphots..
Perhaps layering it on something like https://nuetzlich.net/gocryptfs/ will do the job. If I really need to I can use that to mount the encrypted snapshots remotely.

My concerns mostly revolve around how to recover from partial data loss. If my btrfs base snapshot becomes corrupt, is there a way to recover data from a serialized snapshot diff? I’d like a btrfs receive –ignore-inconsistencies option, so I could apply a partial snapshot diff to a blank snapshot and get any new files that were included in the delta.

Use Google Authenticator / OATH for two-factor SSH access with SSH-Keys

darkskiez Leave a comment

I wanted to use two factor auth combined with ssh keys to restrict access to some of the production machines at work, however this wasn’t entirely straight forward as google authenticator pam module would be entirely bypassed with ssh-keys, and only supports one key per account, (so shared accounts like root would be a problem)

I’ve discovered a lovely little simple tool that lets you work around it ssh-opt

+ You can use a different OATH token for each ssh key!
+ You can choose to not require tokens for some keys, eg. for automated systems
+ You can use it along with google authenticator pam for single password + single token access
+ You can install it on machines you don’t have root access to

– It doesn’t support scratch emergency codes or replay protection [yet, it wouldnt be that hard to add]
– It leaks your token key to other users via ps [easily fixable]
– It breaks scp! not sure why yet, it just hangs for me.

Its dead simple to use too, just prefix the key in .ssh/authorized_keys:

command="/usr/bin/ssh-otp OATHTOKEN" ssh-dss AAAAB3...

Fix Linux Flash Player “Error #2046” on iceweasel / debian

darkskiez 1 Comment

This error had plagued me for some time, but the chrome plugin worked… so I didnt mind until I updated to a full 64bit OS, and the chrome plugin does not exist for 64bit

Turns out that the plugin was trying to read the firefox preferences but fails on two accounts, it apparently doesnt handle spaces or relative paths..

Change ~/.mozilla/firefox/profiles.ini something like this:

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=0
Path=/home/user/.firefox/default/187x6ax2.slt

to

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=1
Path=187x6ax2.slt

and then

ln -s ~/.firefox/default/187x6ax2.slt ~/.mozilla/firefox/

How to add an additional Google Authenticator Device

darkskiez 5 Comments

You will need a rooted device to do this!

I’ve obtained more than one android device, and I was looking at installing the google authenticator app on it as well, however, google wanted me to delete my existing key and create a new one if I was going to configure other devices. This means disabling 2 factor authentication, and I’m not sure if that would mean recreating all of my application specific passwords…but I didnt fancy that..

$ adb shell
# sqlite3 /data/data/com.google.android.apps.authenticator/databases/databases
sqlite> select * from accounts;
1|[email protected]|your2factorkey|0|0
sqlite> .quit
#exit

Now open google authenticator on your new device and choose manually add account, put in your email and key as read above. bish bash bosh, done. Validate this is working by running authenticator on both devices, they should have the same id.

If you dont have a rooted device, you will probably just need to disable and re-setup two-factor authentication to discover your new key.

As a side note, I enjoyed discovering the existence of the following packages:


http://code.google.com/p/mod-authn-otp/ – for adding google two factor auth to your webserver, not sure that this supports scratch codes.


http://code.google.com/p/google-authenticator/ – for adding google two factor auth to your linux machine/services, available on debian as libpam-google-authenticator. It has terminal based ascii-art QR-codes, cool! You can probably just update your ~/.google_authenticator with your key you extracted and also manually enter your scratch codes into this file.

Stenaline Public Wifi Very Insecure – SSL MITM Attacks

darkskiez Leave a comment

Update: Stenaline have responded and disabled email inspection

Fabulous, they have disabled the configuration that was causing me the most problems, within 24hrs of my original post. Congratulations stena line and I look forward to my return journey!

As a direct action on your mail we immediately notified our service provider regarding this issue and the problem is now located and solved.

IMAP protocol inspection is now disabled. The previous setting was an unintentional mistake by our service provider. .

Original Post:

Stena Line provide free wifi, which is awesome, however, their egregious content filtering system massively compromises the passengers online safety far more than normal public wifi hotspots.

They claim on their captive portal login screen on their on-ship wifi:

 

Privacy protection
All open wireless networks are by nature insecure. Please, take the necessary precautions to protect your privacy and data communications.

What kind of security, does the network provide?
The network security level is basically the same as you find on public hotspots.

However, this is clearly not the case, because they go out of their way to invalidate “the necessary precautions” that I normally take which is checking my email over SSL protected connections..

It appears stenaline think that its OK for them to snoop on my SSL secured private and work emails some fortinet/fortigate snooping appliance they seem to have. This appliance proxies SSL connections and re-signs the certificates with its own keys, effectively performing an SSL MITM (Man In The Middle) Attack. SSL is designed to prevent this sort of thing, which is why your browser throws up an error message when somethings gone wrong. All in all, this however means nobody can tell if its them “protecting” me or if it is infact a rogue hacker running a fake access point, a so called “Evil Twin” network, collecting peoples google, and corporate credentials, or snooping on my emails.. Thanks very much guys.

Update: It appears that It’s not limited to email or non-https web ports, lots of websites that have https connections blow up too, even common ones like google plus which loads content from https://fls.doubleclick.net (a google owned site). They appear to whitelist certain popular websites to allow https directly, but this is unmanageable and unmaintainable, and wholly ridiculous. I cant safely sign into my work SSL VPN with this either.

Heres an example where they have stripped the regular CA and added the Self-Signed Fortinet CA Certificate for imap.google.com

$ openssl s_client -connect imap.gmail.com:993
CONNECTED(00000003)
depth=1 C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = FortiGate CA, emailAddress = [email protected]
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
i:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/[email protected]
1 s:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/[email protected]
i:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/[email protected]
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
issuer=/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/[email protected]

it should look like

$ openssl s_client -connect imap.gmail.com:993 -showcerts
CONNECTED(00000003)
depth=1 C = US, O = Google Inc, CN = Google Internet Authority
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority

All in all, the standard advice of when using a public wifi connection, use a VPN still stands…

As most regular folk dont have a VPN (and even SSL VPNs are compromised here), the best you can hope for is that the sites you use will use SSL, and that you dont just willy-nilly click “OK” on Browser SSL Warning Pages, sadly, this standard web-safety advice means you just cant browse the web safely on stena line ferries.

Google take a very dim view on this sort of thing: Google security blog: Attempted man in the middle attacks

Another interesting thing is that this may also be illegal in the UK within the terms of the computer misuse act or telecommunication acts, as they are attempting to decrypt an encrypted communication, though I’m not a lawyer. If anyone can shed any light on this I’d like to know.

I’ve emailed stena a link to this post to see if they respond, and hopefully set a timeframe for removing this egregious device/configuration from their network.

Automating Debian preseed installs with raid and LVM

darkskiez 3 Comments

There are few problems I needed to address with debian preseeding to install onto multiple different machines

  • Testing my preseed in virtualbox lead to it pulling in half of X as it detected I needed virtualbox client extensions, when i wanted a clean small base install.
  • Support installing onto all machines including to boot from USB hdd and slow raid controllers that take a few seconds to initialise
  • For some reason the debian preseed file for requires you to specify /dev/sda or /dev/hda and does not allow you to specify in the devfs independent formats /dev/discs/disc0/disc
  • I wanted to always set up software raid mirror, even if there was just one hard disk (so later a second hdd could be added to mirror quickly)
  • They should be able to install with any debian installer, but the system should default to using network install for updates afterwards

Full sample config and example usage

Sample Config: http://www.darkskiez.co.uk/d-i/squeeze/preseed.cfg

Boot Debian Squeeze Install CD/DVD/Netinst/etc

Choose Advanced, Automated Install

Answer keyboard / network setup prompts

Realise that after typing the next line in your VM or Computer will be fully erased without warning:

Preseed Location: www.darkskiez.co.uk

The full location is of that above, but d-i/squeeze/preseed.cfg path is added automatically by the installer if none is specified.

Make a cup of strong tea.

login as root/changemenow

passwd

#For some reason lvremove hangs when I invoked it from the preseed script, so tidy up now.
lvremove /dev/vg0/dummy

Relevant Config Snippets

Stop debian preseed from installing virtualbox stuff


# Disable Debian installer from installing hardware specific packages
# Warning: This could break preseeding to some hardware I dont know or care about.
d-i preseed/early_command string rm /usr/lib/pre-pkgsel.d/20install-hwpackages

Use network install for updates, but cd/dvd for install


d-i preseed/early_command string anna-install net-retriever;
### Mirror settings
# If you select ftp, the mirror/country string does not need to be set.
#d-i mirror/protocol string ftp
d-i mirror/country string manual
d-i mirror/http/hostname string ftp.uk.debian.org
d-i mirror/http/hostname seen true
d-i mirror/http/directory string /debian
d-i mirror/http/directory seen true
d-i mirror/http/proxy string

# Suite to install.
d-i mirror/suite string squeeze
# Suite to use for loading installer components (optional).
d-i mirror/udeb/suite string squeeze

# disable the cdrom path from the apt soruces.list automatically
d-i preseed/late_command in-target sed -i 's/^deb cdrom/# DISABLED deb/' /etc/apt/sources.list

Work around USB boot drives and slow controllers


# Work around for slow raid controllers on some machines
# and usb bootable hdds
d-i debian-installer/add-kernel-opts string rootdelay=7

Automatic Debian raid and LVM Setup


### Partitioning
# http://d-i.alioth.debian.org/svn/debian-installer/installer/doc/devel/partman-auto-recipe.txt
# has moved to:
# http://anonscm.debian.org/gitweb/?p=d-i/debian-installer.git;a=blob;f=doc/devel/partman-auto-recipe.txt

# Next you need to specify the physical partitions that will be used.
# The numbers aren't pretty as I experimented till
# I got something approaching what i wanted
#
# This example creates:
# /dev/md0 - 100 to 256Mb - /boot
# /dev/md1 - 900 to 4000Mb(ignored rest of disk used) - LVM vg0
# /dev/vg0/root - 700 to 5000Mb - / ext4
# /dev/vg0/swap_1 - 300% of RAM size - swap
# /dev/vg0/home - 64 to 4096Mb - /home ext4
# /dev/vg0/var+log - 64 to 4096Mb - /var/log ext4
# /dev/vg0/dummy - rest of space - to be deleted and others resized.

d-i partman-auto/expert_recipe string \
multiraid :: \
100 512 256 raid \
$lvmignore{ } \
$primary{ } \
method{ raid } \
. \
900 5000 4000 raid \
$lvmignore{ } \
method{ raid } \
. \
700 5000 5000 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ / } \
. \
64 512 300% linux-swap \
$defaultignore{ } \
$lvmok{ } \
method{ swap } \
format{ } \
. \
64 1024 4096 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /home } \
. \
64 1024 4096 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var/log } \
. \
64 2048 1000000000 ext3 \
$defaultignore{ } \
$lvmok{ } \
lv_name{ dummy } \
use_filesystem{ } \
filesystem{ ext3 } \
method{ keep } \
.
d-i partman-auto/method string regular

# If one of the disks that are going to be automatically partitioned
# contains an old LVM configuration, the user will normally receive a
# warning. This can be preseeded away...
d-i partman-lvm/device_remove_lvm boolean true
# The same applies to pre-existing software RAID array:
d-i partman-md/device_remove_md boolean true
# And the same goes for the confirmation to write the lvm partitions.
d-i partman-lvm/confirm boolean true

#Name default volume group vg0
d-i partman-auto-lvm/new_vg_name string vg0

# You can choose one of the three predefined partitioning recipes:
# - atomic: all files in one partition
# - home: separate /home partition
# - multi: separate /home, /usr, /var, and /tmp partitions
d-i partman-auto/choose_recipe select expert_recipe

# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
d-i partman/early_command string \
DISKA=$(list-devices disk|head -n1);\
DISKB=$(list-devices disk|head -n2|tail -1);\
if [ "${DISKA#/dev/cciss}" != "$DISKA" ]; then DISKAP="p"; fi;\
if [ "${DISKB#/dev/cciss}" != "$DISKB" ]; then DISKBP="p"; fi;\
if [ "$DISKA" = "$DISKB" ]; then\
debconf-set partman-auto/disk "$DISKA";\
debconf-set partman-auto-raid/recipe "1 2 0 ext3 /boot ${DISKA}${DISKAP}1 . 1 2 0 lvm - ${DISKA}${DISKAP}5 .";\
debconf-set grub-installer/bootdev "$DISKA";\
else\
debconf-set partman-auto/disk "$DISKA $DISKB";\
debconf-set partman-auto-raid/recipe "1 2 0 ext3 /boot ${DISKA}${DISKAP}1#${DISKB}${DISKBP}1 . 1 2 0 lvm - ${DISKA}${DISKAP}5#${DISKB}${DISKBP}5 .";\
debconf-set grub-installer/bootdev "$DISKA $DISKB";\
fi;

# lvremove hangs.. weird
#d-i preseed/late_command string in-target lvremove -f /dev/vg0/dummy

## Partitioning using RAID
# The method should be set to "raid".
d-i partman-auto/method string raid
# This makes partman automatically repartition without confirmation.
d-i partman-md/confirm boolean true
d-i partman-md/confirm_nooverwrite boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-basicfilesystems/no_mount_point yes

## Controlling how partitions are mounted
# The default is to mount by UUID, but you can also choose "traditional" to
# use traditional device names, or "label" to try filesystem labels before
# falling back to UUIDs.
d-i partman/mount_style select label

 

Creating a debian native package from scratch with git-buildpackage

darkskiez 1 Comment

A missing snippet from the manual of http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html was how to create a debian native package from scratch, I did something like this:


mkdir newdeb
cd newdeb
git init
git commit --allow-blank -m "Empty upstream, native Debian package"
# or with newer git (thanks marco)
git commit --allow-empty -m "Empty upstream, native Debian package"
git branch upstream
dh_make -p newdeb_1.0 -n -e [email protected] -s
git add debian
git commit -m "Initial blank Debian package template"

Dear gizoo.co.uk

darkskiez Leave a comment

RE: Anysharp emails.

You’ve sent me emails promoting the LIMITED AVAILABILITY SPECIAL OFFER anysharp on:

12th Aug 2010
21st Aug 2010
12th Oct 2010
16th Nov 2010
4th Jan 2011
25th Jan 2011
29th Jan 2011

And its always, and currently still is, cheaper direct from the manufacturers site than from you.

Please find a better offer.

Also the submit comment on your site was broken, “error ‘8004020e’ /Scripts/email/sendMail.asp, line 119” so i posted this on my blog 🙂