Backup Pi

My online backup provider Crashplan cancelled home users accounts to focus on business users.. Now what?

Spoilers: This is unfinished at time of publishing. I’ll update this post as I progress towards a solution I’m happy with.

There aren’t any obvious Linux compatible unlimited (priced per computer rather than per GB) online backup solutions left, and I didnt fancy running one in a VM or any other jiggery pokery like that.

Cloud providers would be ideal, BackBlaze B2 is a good candidate https://www.backblaze.com/b2/cloud-storage-pricing.html, but Wasabi cheapest I’ve found, https://wasabi.com/pricing/. It still works out for 8TiB over 3 Years with one full restore to cost $1468. I suspect any cheaper providers will change/increase their pricing model or close in a few years and I don’t want to find a new solution every two years, but this is still too expensive for my needs.

Potential Solution: run my own offsite backup system.

Hardware List

$35 Raspberry Pi 3

$16 SD Card

$30 USB Dual HDD Dock

$275 2 * WD RED 8TiB

Total $666

Onsite backup seeding saves normal ~month of initial backup nonsense.

Configuring the Pi

Power

Works on normal usb power as not doing anything too fancy to need the extra watts.

Operating system

Raspian no-gui whatever was latest

Basic WiFi Networking

Wifi setup at my location has public WiFi, that will do

#/etc/network/interfaces.d/wlan0
auto wlan0
iface wlan0 inet dhcp
wireless-essid GuestWifiNoCrypto

VPN

My pi is installed in a firewalled location and can’t act as a server that my backup machine can reach.

Simplest VPN: ssh tunnel!

However, using ssh to create a tunnel is very temporary, autossh is a solution that will start ssh on boot and restart it if the connection fails, but it needs some encouragement to have the right / most robust config. The flag that bit me that was missing from other examples is the ExitOnForwardFailure, until I had that the sshvpn was reconnecting, but not establishing the reverse tunnel when an old stale one hadn’t quite cleaned up.

homemachine$ sudo adduser --system pitunnel
# First step, create and install a ssh key for the inbound tunnel:
pi$ ssh-keygen -t rsa -b 4096
# Install key into my home machine.
pi$ ssh-copy-id pitunnel@my.dyndns.address
# Test the key, this should connect without a password
pi$ ssh pitunnel@my.dyndns.address

Here is my autossh setup.

# /lib/systemd/system/autossh.service 
[Unit]
Description=tunnel
After=network.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
User=pi
Restart=always
RestartSec=5
Environment=AUTOSSH_GATETIME=0
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" -p 22 -l pitunnel my.dyndns.address -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_rsa

[Install]
WantedBy=multi-user.target

Now Enable it / Start it and check it is running ok.

$sudo systemctl enable autossh
$sudo systemctl start autossh
$sudo systemctl status autossh
● autossh.service - tunnel
   Loaded: loaded (/lib/systemd/system/autossh.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-10-xx 01:01:38 UTC; 0 days ago
 Main PID: 739 (autossh)
   CGroup: /system.slice/autossh.service
           ├─ 739 /usr/lib/autossh/autossh -M 0 -N -q -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -o ExitOnForwardFailure yes -p 22 -l pitunnel my.dyndns.address -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_
           └─2748 /usr/bin/ssh -N -q -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -o ExitOnForwardFailure yes -p 22 -l pitunnel -R 2200:127.0.0.1:22 -i /home/pi/.ssh/id_rsa my.dyndns.address

Set up convenience ssh host target, so you dont need all the usernames and ports in every ssh and scp you want to use:

# /root/.ssh/config 
Host pitunnel
    Hostname 127.0.0.1
    User borg
    port 2200
    IdentityFile ~/.ssh/borg

Test it with

ssh pitunnel

It should connect with no password prompt.

Installing the disks

TODO I’ve not really finished this section.

I had initially used a btrfs raid1 mirror instead of a linux mdadm raid, but one of the drives in my dock had connectivity issues.

  • Running a btrfs balance took far far too long (days). I have no idea why btrfs takes so long to rebalance, it is not IO speed limited. I’d like someone to benchmark btrfs rebalance in ram and see how slow it still is.
  • It would crash when trying to mount after a bad reset, needing to be mounted from my PC with more ram / different kernel / whatever else, in order to clean up the filesystem state. This happened more than once.
  • It would force the filesystem read-only after one degraded mount after I took a disk out, this is supposedly fixed in Linux 4.14: http://lkml.iu.edu/hypermail/linux/kernel/1709.1/00609.html

I refuse to use a non check-summing filesystem, and zfs is out of the question for me right now (not open for discussion here, may work for you), so I’m sticking with btrfs, but ignoring the raid modes.

$EDITOR /etc/fstab

/dev/disk/by-label/borgbackup /backup btrfs defaults,noatime,nofail 0 2

linux raid jiggery, I’ve done this a squillion times, but havent actually set it up yet.

$ sudo apt install smartmontools

TODO: You should probably configure a mail relay so emailing from the pi works.

watchdog

The systemd watchdog looks an interesting framework, but didn’t have an immediately obvious way of checking net connection was ok, so I opted for the lazy/easy watchdog package..

sudo apt install watchdog

$EDITOR /etc/watchdog.conf
# set the following lines
ping = 8.8.8.8 # google dns
interface = wlan0 # check wifi interface
watchdog-timeout = 10 # largest pi watchdog value, default  of 60 errors

Misc Config

I had the Pi plugged into a spare HDMI port on a nearby monitor to be able to check in on it from time to time and plugged into a usb switcher to swap keyboard/mouse over.

aliexpress $6 USB KVM Switcher was the cheapest I found, and I just ignore the vga ports

Annoyingly when my machine was locked, the Pi would occasionally reboot or steal input via HDMI to the monitor..

#/boot/config.txt
hdmi_blanking=1

This setting is supposed to fix it, at the cost of breaking media player software horribly.. which is ok, as I dont use it on this. I’ll update my experience.

Backup software

restic

I first tried with restic, it ticked most of my boxes, golang, encrypted, authenticated, de-duplicating, incremental. I however had a few problems, restic seems to have a few problems with pruning over slow links, and it ran out of memory when I tried to prune from the pi itself. It looks like it’s got scope to grow into a solid backup tool, but doesn’t quite meet my low memory / high latency needs at the moment.

Issues I ran into:

No snapshot usage stats https://github.com/restic/restic/issues/693

Random Crash: https://github.com/restic/restic/issues/1251

Poor errors with SFTP backend: https://github.com/restic/restic/issues/1323

Very little progress output on prune: https://github.com/restic/restic/issues/1322

No network stats https://github.com/restic/restic/issues/1033

The restic author is maintaining a detailed feature list of alternative backup software, which I can highly recommend for those exploring backup systems

The most mature alternative candidate seems to be borgbackup. I didn’t really need compression that borgbackup adds, and I didn’t want the overhead that python has, but it seems to tick most boxes.

borgbackup

I used the borgbackup 1.1.1 release from pip, as it fixes some issues I had with clearing out stale locks (after the watchdog resets the pi) and raspian hasn’t updated yet..
Version 1.1.0 managed to crash my host machine due to a bug with it opening every block device, and possibly a secondary bug in my kernel, this needs further investigation for the kernel side – https://github.com/borgbackup/borg/issues/3213

$ sudo pip3 install borgbackup

I dropped the script from the borg backup Quick Start – Automating Backups into /etc/cron.daily/borgbackup I added the -x flag for one filesystem and then added the paths to the mounts I wanted

 export BORG_REPO=ssh://pitunnel/~/repo

Oops I slightly corrupted my repo with the watchdog resets and lock fudgery, the prune was crashing with a missing directory.

Python single threaded, cpu bound, multiple day long check…awarghgjbwakwffkawkfkafhgh, it’s still not finished, I have little idea about its progress (it’s spent all the time Analyzing archive “hostname-2017-10-18T11:44:25.checkpoint (3/19)”, and I have 15 idle cores.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                |
22288 root      20   0 1000796 897456   6984 R 100.0  5.5   3844:24 /usr/bin/python3 /usr/bin/borg check --verbose --repair               

Day 7: This breached 10000 minutes 7 days of runtime so I’ve shut it down..

Day 14: I forgot to check up on it for a while, miraculously its been running for a week making backups every day! Hooray

But the repo isnt “healthy”..

$ borg delete $BORG_REPO::darkskiez-2017-10-23T23:05:06.checkpoint
borg.repository.Repository.ObjectNotFound: Object with key fe00c10f6ce4b733d013ee30a60f47284368e4fa47164cb0d8af507dc0acedf0 not found in repository ssh://pitunnel/~/reopo::darkskiez-2017-10-23T23:05:06.checkpoint.

Sod it, I’ll force it and nuke the lot of them.

borg list --short|grep checkpoint|xargs -i borg delete --force $BORG_REPO::{}
forced deletion succeeded, but the deleted archive was corrupted.
borg check --repair is required to free all space. 

Argh! borg list still shows a checkpoint file… and borg delete –force is erroring with the same not found error, how hard is it to delete something that isn’t there?!!?

We know how well borg check worked last time, but I really didn’t want it to repair these checkpoints before, maybe now it will work 😐 I don’t have high hopes

using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
SSH command line: ['ssh', 'pitunnel', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 0.72 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/~/repo'
Remote: Resolved repository path to '/backup/repo'
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES
Remote: Starting repository check
Remote: Verified integrity of /backup/repo/index.179252
Remote: Read committed index of transaction 179252
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 179252).
Remote: Segment transaction is    179252
Remote: Determined transaction is 179252
Remote: Found 166446 segments
Remote: Checking segments 100%
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Remote: Verified integrity of /backup/repo/index.179253
TAM-verified manifest
Analyzing archive darkskiez-2017-10-12T11:01:00 (1/9)
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 179253).
Remote: Verified integrity of /backup/repo/hints.179253
Analyzing archive darkskiez-2017-10-18T10:22:54.checkpoint (2/9)
Archive metadata block is missing!
Analyzing archive darkskiez-2017-10-24T10:02:14 (3/9)
/srv/photos/IMG_2445.JPG: New missing file chunk detected (Byte 0-888561). Replacing with all-zero chunk.
/srv/photos/IMG_2445.JPG: New missing file chunk detected (Byte 888561-1153253). Replacing with all-zero chunk.
item metadata chunk missing [chunk: 000528_fe00c10f6ce4b733d013ee30a60f47284368e4fa47164cb0d8af507dc0acedf0]

It got here before in the 7 day run, so I assume it doesn’t do the fixing so well.

I’m going to delete this specific archive.

Day 21: For various reasons this check couldnt finish either, still takes too long..

I am very sad. I am resisting the strong urge to write my own backup system. Perhaps I should retry restic but with much smaller repos. Perhaps I should consider trying another system… knoxite looks better than restic on the box, but has nearly zero development activity / users. I refused to believe it is mature with this evidence, but then again, I was wholly wrong about borg backup it seems.

btrfs snapshots

Ok, maybe all this backup software is too fancy.. Maybe btrfs is fancy enough a filesystem that I don’t need the extra things..

There are a few btrfs backup scripts, based on the principals in: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

So I have quite a lot of data I wish to “exclude” from my backup, and reorganizing my filesystem to have different (sub)volumes doesn’t sound like fun, especially when we are talking about lots of ‘cache’ dirs and what not.

I wonder if btrfs snapshot diffing is intelligent enough that if I snapshot fs A into S1 and S2 in different times, prune the files I don’t want in S1 and S2, and do a btrfs send of the deltas S1-S2 if it will work at all, and be intelligent enough to not include the deleted files.

Crude test time.

btrfs subvol create /home/snaptest
# Create a bigfile1 /home/snaptest/b1
btrfs subvolume snapshot /home/snaptest /home/snaptest-s1
# Create a bigfile2 /home/snaptest/b2
# Create a small file /home/snaptest/s
btrfs subvolume snapshot /home/snaptest /home/snaptest-s2
# Delete b2 from  /home/snaptest-s1/b2 and /home/snaptest-s2/b2

btrfs send -p /home/snaptest-s1 /home/snaptest-s2|wc -c
ERROR: subvolume /home/snaptest-s1 is not read-only
# Woops
btrfs property set -ts /home/snaptest-s1 ro true
btrfs property set -ts /home/snaptest-s2 ro true

btrfs send -p /home/snaptest-s1 /home/snaptest-s2|wc -c
# Wahoo: only a bit bigger than smallfile
btrfs sub del /home/snaptest*

There may be some hope here yet.

If I encrypt the data that is being sent with ‘btrfs send’ that is secure, but, very awkward to remotely use in an emergency when I cant restore the snaphots..
Perhaps layering it on something like https://nuetzlich.net/gocryptfs/ will do the job. If I really need to I can use that to mount the encrypted snapshots remotely.

My concerns mostly revolve around how to recover from partial data loss. If my btrfs base snapshot becomes corrupt, is there a way to recover data from a serialized snapshot diff? I’d like a btrfs receive –ignore-inconsistencies option, so I could apply a partial snapshot diff to a blank snapshot and get any new files that were included in the delta.

, , , ,

No Comments

Use Google Authenticator / OATH for two-factor SSH access with SSH-Keys

I wanted to use two factor auth combined with ssh keys to restrict access to some of the production machines at work, however this wasn’t entirely straight forward as google authenticator pam module would be entirely bypassed with ssh-keys, and only supports one key per account, (so shared accounts like root would be a problem)

I’ve discovered a lovely little simple tool that lets you work around it ssh-opt

+ You can use a different OATH token for each ssh key!
+ You can choose to not require tokens for some keys, eg. for automated systems
+ You can use it along with google authenticator pam for single password + single token access
+ You can install it on machines you don’t have root access to

– It doesn’t support scratch emergency codes or replay protection [yet, it wouldnt be that hard to add]
– It leaks your token key to other users via ps [easily fixable]
– It breaks scp! not sure why yet, it just hangs for me.

Its dead simple to use too, just prefix the key in .ssh/authorized_keys:

command="/usr/bin/ssh-otp OATHTOKEN" ssh-dss AAAAB3...

No Comments

Fix Linux Flash Player “Error #2046” on iceweasel / debian

This error had plagued me for some time, but the chrome plugin worked… so I didnt mind until I updated to a full 64bit OS, and the chrome plugin does not exist for 64bit

Turns out that the plugin was trying to read the firefox preferences but fails on two accounts, it apparently doesnt handle spaces or relative paths..

Change ~/.mozilla/firefox/profiles.ini something like this:

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=0
Path=/home/user/.firefox/default/187x6ax2.slt

to

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=1
Path=187x6ax2.slt

and then

ln -s ~/.firefox/default/187x6ax2.slt ~/.mozilla/firefox/

1 Comment

How to add an additional Google Authenticator Device

You will need a rooted device to do this!

I’ve obtained more than one android device, and I was looking at installing the google authenticator app on it as well, however, google wanted me to delete my existing key and create a new one if I was going to configure other devices. This means disabling 2 factor authentication, and I’m not sure if that would mean recreating all of my application specific passwords…but I didnt fancy that..

$ adb shell
# sqlite3 /data/data/com.google.android.apps.authenticator/databases/databases
sqlite> select * from accounts;
1|your@email.address|your2factorkey|0|0
sqlite> .quit
#exit

Now open google authenticator on your new device and choose manually add account, put in your email and key as read above. bish bash bosh, done. Validate this is working by running authenticator on both devices, they should have the same id.

If you dont have a rooted device, you will probably just need to disable and re-setup two-factor authentication to discover your new key.

As a side note, I enjoyed discovering the existence of the following packages:


http://code.google.com/p/mod-authn-otp/
– for adding google two factor auth to your webserver, not sure that this supports scratch codes.


http://code.google.com/p/google-authenticator/
– for adding google two factor auth to your linux machine/services, available on debian as libpam-google-authenticator. It has terminal based ascii-art QR-codes, cool! You can probably just update your ~/.google_authenticator with your key you extracted and also manually enter your scratch codes into this file.

5 Comments

Stenaline Public Wifi Very Insecure – SSL MITM Attacks

Update: Stenaline have responded and disabled email inspection

Fabulous, they have disabled the configuration that was causing me the most problems, within 24hrs of my original post. Congratulations stena line and I look forward to my return journey!

As a direct action on your mail we immediately notified our service provider regarding this issue and the problem is now located and solved.

IMAP protocol inspection is now disabled. The previous setting was an unintentional mistake by our service provider. .

Original Post:

Stena Line provide free wifi, which is awesome, however, their egregious content filtering system massively compromises the passengers online safety far more than normal public wifi hotspots.

They claim on their captive portal login screen on their on-ship wifi:

 

Privacy protection
All open wireless networks are by nature insecure. Please, take the necessary precautions to protect your privacy and data communications.

What kind of security, does the network provide?
The network security level is basically the same as you find on public hotspots.

However, this is clearly not the case, because they go out of their way to invalidate “the necessary precautions” that I normally take which is checking my email over SSL protected connections..

It appears stenaline think that its OK for them to snoop on my SSL secured private and work emails some fortinet/fortigate snooping appliance they seem to have. This appliance proxies SSL connections and re-signs the certificates with its own keys, effectively performing an SSL MITM (Man In The Middle) Attack. SSL is designed to prevent this sort of thing, which is why your browser throws up an error message when somethings gone wrong. All in all, this however means nobody can tell if its them “protecting” me or if it is infact a rogue hacker running a fake access point, a so called “Evil Twin” network, collecting peoples google, and corporate credentials, or snooping on my emails.. Thanks very much guys.

Update: It appears that It’s not limited to email or non-https web ports, lots of websites that have https connections blow up too, even common ones like google plus which loads content from https://fls.doubleclick.net (a google owned site). They appear to whitelist certain popular websites to allow https directly, but this is unmanageable and unmaintainable, and wholly ridiculous. I cant safely sign into my work SSL VPN with this either.

Heres an example where they have stripped the regular CA and added the Self-Signed Fortinet CA Certificate for imap.google.com

$ openssl s_client -connect imap.gmail.com:993
CONNECTED(00000003)
depth=1 C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = FortiGate CA, emailAddress = support@fortinet.com
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
i:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/emailAddress=support@fortinet.com
1 s:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/emailAddress=support@fortinet.com
i:/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/emailAddress=support@fortinet.com
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
issuer=/C=US/ST=California/L=Sunnyvale/O=Fortinet/OU=Certificate Authority/CN=FortiGate CA/emailAddress=support@fortinet.com

it should look like

$ openssl s_client -connect imap.gmail.com:993 -showcerts
CONNECTED(00000003)
depth=1 C = US, O = Google Inc, CN = Google Internet Authority
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority

All in all, the standard advice of when using a public wifi connection, use a VPN still stands…

As most regular folk dont have a VPN (and even SSL VPNs are compromised here), the best you can hope for is that the sites you use will use SSL, and that you dont just willy-nilly click “OK” on Browser SSL Warning Pages, sadly, this standard web-safety advice means you just cant browse the web safely on stena line ferries.

Google take a very dim view on this sort of thing: Google security blog: Attempted man in the middle attacks

Another interesting thing is that this may also be illegal in the UK within the terms of the computer misuse act or telecommunication acts, as they are attempting to decrypt an encrypted communication, though I’m not a lawyer. If anyone can shed any light on this I’d like to know.

I’ve emailed stena a link to this post to see if they respond, and hopefully set a timeframe for removing this egregious device/configuration from their network.

No Comments

Automating Debian preseed installs with raid and LVM

There are few problems I needed to address with debian preseeding to install onto multiple different machines

  • Testing my preseed in virtualbox lead to it pulling in half of X as it detected I needed virtualbox client extensions, when i wanted a clean small base install.
  • Support installing onto all machines including to boot from USB hdd and slow raid controllers that take a few seconds to initialise
  • For some reason the debian preseed file for requires you to specify /dev/sda or /dev/hda and does not allow you to specify in the devfs independent formats /dev/discs/disc0/disc
  • I wanted to always set up software raid mirror, even if there was just one hard disk (so later a second hdd could be added to mirror quickly)
  • They should be able to install with any debian installer, but the system should default to using network install for updates afterwards

Full sample config and example usage

Sample Config: http://www.darkskiez.co.uk/d-i/squeeze/preseed.cfg

Boot Debian Squeeze Install CD/DVD/Netinst/etc

Choose Advanced, Automated Install

Answer keyboard / network setup prompts

Realise that after typing the next line in your VM or Computer will be fully erased without warning:

Preseed Location: www.darkskiez.co.uk

The full location is of that above, but d-i/squeeze/preseed.cfg path is added automatically by the installer if none is specified.

Make a cup of strong tea.

login as root/changemenow

passwd

#For some reason lvremove hangs when I invoked it from the preseed script, so tidy up now.
lvremove /dev/vg0/dummy

Relevant Config Snippets

Stop debian preseed from installing virtualbox stuff


# Disable Debian installer from installing hardware specific packages
# Warning: This could break preseeding to some hardware I dont know or care about.
d-i preseed/early_command string rm /usr/lib/pre-pkgsel.d/20install-hwpackages

Use network install for updates, but cd/dvd for install


d-i preseed/early_command string anna-install net-retriever;
### Mirror settings
# If you select ftp, the mirror/country string does not need to be set.
#d-i mirror/protocol string ftp
d-i mirror/country string manual
d-i mirror/http/hostname string ftp.uk.debian.org
d-i mirror/http/hostname seen true
d-i mirror/http/directory string /debian
d-i mirror/http/directory seen true
d-i mirror/http/proxy string

# Suite to install.
d-i mirror/suite string squeeze
# Suite to use for loading installer components (optional).
d-i mirror/udeb/suite string squeeze

# disable the cdrom path from the apt soruces.list automatically
d-i preseed/late_command in-target sed -i 's/^deb cdrom/# DISABLED deb/' /etc/apt/sources.list

Work around USB boot drives and slow controllers


# Work around for slow raid controllers on some machines
# and usb bootable hdds
d-i debian-installer/add-kernel-opts string rootdelay=7

Automatic Debian raid and LVM Setup


### Partitioning
# http://d-i.alioth.debian.org/svn/debian-installer/installer/doc/devel/partman-auto-recipe.txt
# has moved to:
# http://anonscm.debian.org/gitweb/?p=d-i/debian-installer.git;a=blob;f=doc/devel/partman-auto-recipe.txt

# Next you need to specify the physical partitions that will be used.
# The numbers aren't pretty as I experimented till
# I got something approaching what i wanted
#
# This example creates:
# /dev/md0 - 100 to 256Mb - /boot
# /dev/md1 - 900 to 4000Mb(ignored rest of disk used) - LVM vg0
# /dev/vg0/root - 700 to 5000Mb - / ext4
# /dev/vg0/swap_1 - 300% of RAM size - swap
# /dev/vg0/home - 64 to 4096Mb - /home ext4
# /dev/vg0/var+log - 64 to 4096Mb - /var/log ext4
# /dev/vg0/dummy - rest of space - to be deleted and others resized.

d-i partman-auto/expert_recipe string \
multiraid :: \
100 512 256 raid \
$lvmignore{ } \
$primary{ } \
method{ raid } \
. \
900 5000 4000 raid \
$lvmignore{ } \
method{ raid } \
. \
700 5000 5000 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ / } \
. \
64 512 300% linux-swap \
$defaultignore{ } \
$lvmok{ } \
method{ swap } \
format{ } \
. \
64 1024 4096 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /home } \
. \
64 1024 4096 ext4 \
$defaultignore{ } \
$lvmok{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ /var/log } \
. \
64 2048 1000000000 ext3 \
$defaultignore{ } \
$lvmok{ } \
lv_name{ dummy } \
use_filesystem{ } \
filesystem{ ext3 } \
method{ keep } \
.
d-i partman-auto/method string regular

# If one of the disks that are going to be automatically partitioned
# contains an old LVM configuration, the user will normally receive a
# warning. This can be preseeded away...
d-i partman-lvm/device_remove_lvm boolean true
# The same applies to pre-existing software RAID array:
d-i partman-md/device_remove_md boolean true
# And the same goes for the confirmation to write the lvm partitions.
d-i partman-lvm/confirm boolean true

#Name default volume group vg0
d-i partman-auto-lvm/new_vg_name string vg0

# You can choose one of the three predefined partitioning recipes:
# - atomic: all files in one partition
# - home: separate /home partition
# - multi: separate /home, /usr, /var, and /tmp partitions
d-i partman-auto/choose_recipe select expert_recipe

# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
d-i partman/early_command string \
DISKA=$(list-devices disk|head -n1);\
DISKB=$(list-devices disk|head -n2|tail -1);\
if [ "${DISKA#/dev/cciss}" != "$DISKA" ]; then DISKAP="p"; fi;\
if [ "${DISKB#/dev/cciss}" != "$DISKB" ]; then DISKBP="p"; fi;\
if [ "$DISKA" = "$DISKB" ]; then\
debconf-set partman-auto/disk "$DISKA";\
debconf-set partman-auto-raid/recipe "1 2 0 ext3 /boot ${DISKA}${DISKAP}1 . 1 2 0 lvm - ${DISKA}${DISKAP}5 .";\
debconf-set grub-installer/bootdev "$DISKA";\
else\
debconf-set partman-auto/disk "$DISKA $DISKB";\
debconf-set partman-auto-raid/recipe "1 2 0 ext3 /boot ${DISKA}${DISKAP}1#${DISKB}${DISKBP}1 . 1 2 0 lvm - ${DISKA}${DISKAP}5#${DISKB}${DISKBP}5 .";\
debconf-set grub-installer/bootdev "$DISKA $DISKB";\
fi;

# lvremove hangs.. weird
#d-i preseed/late_command string in-target lvremove -f /dev/vg0/dummy

## Partitioning using RAID
# The method should be set to "raid".
d-i partman-auto/method string raid
# This makes partman automatically repartition without confirmation.
d-i partman-md/confirm boolean true
d-i partman-md/confirm_nooverwrite boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-basicfilesystems/no_mount_point yes

## Controlling how partitions are mounted
# The default is to mount by UUID, but you can also choose "traditional" to
# use traditional device names, or "label" to try filesystem labels before
# falling back to UUIDs.
d-i partman/mount_style select label

 

3 Comments

Creating a debian native package from scratch with git-buildpackage

A missing snippet from the manual of http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html was how to create a debian native package from scratch, I did something like this:


mkdir newdeb
cd newdeb
git init
git commit --allow-blank -m "Empty upstream, native Debian package"
# or with newer git (thanks marco)
git commit --allow-empty -m "Empty upstream, native Debian package"
git branch upstream
dh_make -p newdeb_1.0 -n -e joe@bloggs.com -s
git add debian
git commit -m "Initial blank Debian package template"

1 Comment

Dear gizoo.co.uk

RE: Anysharp emails.

You’ve sent me emails promoting the LIMITED AVAILABILITY SPECIAL OFFER anysharp on:

12th Aug 2010
21st Aug 2010
12th Oct 2010
16th Nov 2010
4th Jan 2011
25th Jan 2011
29th Jan 2011

And its always, and currently still is, cheaper direct from the manufacturers site than from you.

Please find a better offer.

Also the submit comment on your site was broken, “error ‘8004020e’ /Scripts/email/sendMail.asp, line 119” so i posted this on my blog 🙂

No Comments

How to reduce the threshold for “Low on Space” Android Warnings

Low on Space – Phone storage space is getting low.

Its a cursed message on my Android HTC Hero, but there is 16MB free on /data partition! I want my email to sync a bit more and I want to receive text messages and I dont want to delete any apps.

You need to have rooted your android device and have the android sdk installed and debugging enabled on your phone. I might package this recipe up into an apk for easy installation.

The default limit is 10% of free space, i’ve reduced mine to 5%, I don’t know if there are any terrible side effects. As you’ve already rooted your phone you’ve already probably voided your warranty 🙂

To reduce from 10% to 5% warning from your “adb shell”:

sqlite3 /data/data/com.android.providers.settings/databases/settings.db
insert into secure (name, value) VALUES('sys_storage_threshold_percentage','5');
insert into gservices (name, value) VALUES('sys_storage_threshold_percentage','5');
.quit

Then reboot your phone.

Some firmwares seem to look for the setting in gservices but the latest android source looks like it looks for it in the secure settings, so i’ve included both for good measure.

7 Comments

How to get rid of GPG NO_PUBKEY errors when doing apt-get update

When doing apt-get update you might see a lot of errors like

W: GPG error: http://ceph.newdream.net lenny Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY DA4420ED288995C8
W: GPG error: http://download.opensuse.org Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 85753AA5EEFEFDE9
W: GPG error: http://ppa.launchpad.net karmic Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 28577FE31F882273
W: GPG error: http://download.virtualbox.org lenny Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54422A4B98AB5139

For the best part you should install the apropriate keyrings

apt-cache search keyring$

should list most of them, sometimes they dont exist for some third party repositories, so try this one liner, split for (a little) clarity

for KEY in `apt-get update 2>&1 |grep NO_PUBKEY|awk  '{print $NF}'`; do
 gpg --keyserver subkeys.pgp.net --recv $KEY; gpg --export --armor $KEY|apt-key add -;
done

Caveat, this is insecure, but more secure than disabling validation. Please be aware for full security you should validate the key signatures you are importing via private quantumly secured links to the originator obtained at your own cost, etc etc.

,

1 Comment