= *Nix related tidbits. = This is a page on various tools and techniques related to administration, programming, disaster recovery - basically, odds and ends that come in handy once every so often on a UNIX-like system. Anything based on a link is reiterated just for completeness, and just in case a link dies. This page is a compilation of things done with Ubuntu and FreeBSD, the OSes that the author of this page use/break on a regular basis^1^. Although syntax will inevitably be different amongst various systems, the general ideas of what a UNIX-like OS can do /should/ be there. Additions/corrections to the page are very much welcome. [#shell Shell-based] Command-line tools/tricks. So far, this section includes: * `extundelete` - Linux file system recovery * `cdrecord` - burning bootable ISO's * process I/O redirection (`gdb`) * merging pdfs (`gs`) [#dev Development-related] Tools and workarounds * fixing garbled text (`gcc`) * `git` server (bare repo) setup * various `git` commands * importing non-Eclipse projects to Eclipse * FreeBSD, mininet (`mn`), and `qemu` [#net Network] Various networking-related things. * Disabling SSH timeout * NAT boxes * with `ufw` (Linux) * with `pf` (*BSD) * Wireless with wpa_supplicant * FreeBSD `ifconfig` shinaniganries. [#print Quick Printing] Printing under *nix, relatively quickly * CUPS * `lpd` [#fortune one-liners] miscellaneous single-sentence tips. [[BR]] ---- = Shell-based tools and techniques = #shell == File recovery with Extundelete == Extundelete lets you recover files (directories) that you've accidentally deleted on an ext file system. The best way to use the tool is to have a live CD with the tool on it, although it does not seem to come packaged with Linux by default. 1. Boot the machine off of live CD. [[BR]] 2. Install `extundelete`. On Ubuntu, you need to edit sources.list, since it is not part of the standard repositories. [[BR]] 3. Restore the files on the partition: {{{ # sudo extundelete --restore-all /dev/sda1 }}} This restores everything that has been deleted in /dev/sda1, given the tool finds them. All recovered files are placed in a recovery directory that you can package and back up before rebooting the machine. == Command-line CD burner == source : http://www.yolinux.com/TUTORIALS/LinuxTutorialCDBurn.html Many CD burners are fairly dependency-heavy, mostly thanks to their GUI. `cdrecord` is a command-line based CD/DVD burner with just two dependencies at ~2.3kB: {{{ $ sudo apt-get install cdrecord ... The following NEW packages will be installed: genisoimage wodim ... After this operation, 2,306 kB of additional disk space will be used. }}} Now, to use: 1. Check for supported devices. You'll need to specify what to use when burning the image. `cdrecord` seems to just be a wrapper plus then some for `woodim`, and the two can be used interchangeably for this step. {{{ $ wodim --devices wodim: Overview of accessible drives (2 found) : ------------------------------------------------------------------------- 0 dev='/dev/scd0' rwrw-- : 'SONY' 'DVD-ROM DDU1615' 1 dev='/dev/scd1' rwrw-- : 'TSSTcorp' 'CDDVDW SE-208AB' ------------------------------------------------------------------------- }}} 2. Burn. The meaning of the options can be listed with 'cdrecord --help'. {{{ $ sudo cdrecord -v -eject speed=16 dev='/dev/scd1' Downloads/ubuntu-12.04-server-amd64.iso }}} This will spew a bunch of output, but complete with something like below: {{{ Starting new track at sector: 0 Track 01: 684 of 684 MB written (fifo 100%) [buf 99%] 16.8x. Track 01: Total bytes read/written: 717533184/717533184 (350358 sectors). Writing time: 327.327s Average write speed 14.6x. Min drive buffer fill was 99% Fixating... Fixating time: 24.715s BURN-Free was never needed. wodim: fifo had 11302 puts and 11302 gets. wodim: fifo was 0 times empty and 11100 times full, min fill was 97%. }}} And eject the CD. == Redirecting the output of a live process. == source: http://etbe.coker.com.au/2008/02/27/redirecting-output-from-a-running-process/ Say that you have a running process whose outputs are either being piped to /dev/null or a terminal that you can't get a hold of, but you'd like to be able to see its output. Conversely, you may want to redirect something's output to /dev/null. You can force the process to change its output using `gdb`, the GNU debugger. This is a powerful tool that allows you to attach to a live process and inspect and manipulate the contents of its address space. Here I have a script that I had backgrounded but is still outputting to my terminal (annoying). I will redirect the program's output to a file. 1. Find the PID of your process. Mine is called arrival_collector.rb, so I look for it using `ps`. {{{ # ps -ef | grep collector root 17793 11276 0 18:39 pts/1 00:00:00 ruby arrival_collector.rb -r a,h,ee,f,wknd1,wknd2 -s scott,busch_a,pubsafs,foodsci,scott,scott -v }}} /proc/ shows that its STDOUT is my terminal, /dev/pts/1: {{{ # ls -l /proc/17793/fd total 0 lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1 lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /dev/pts/1 lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1 lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742] }}} 2. Attach to the process with GDB. {{{ # gdb -p [PID] [path/to/executable] }}} It wil spew a bunch of output but eventually give you a prompt: {{{ # gdb -p 17793 /opt/grailrtls/grail3_ruby/grail3protocols/arrival_collector.rb GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. ... Loaded symbols for /lib/libnss_files.so.2 0xb78c7424 in __kernel_vsyscall () (gdb) }}} 3. Close STDOUT: {{{ (gdb) p close(1) $1 = 0 }}} 4. Point STDOUT to a file. Like in the link, I will use creat() to point STDOUT to a log file in /tmp/. creat() takes the path to the file and its permissions as the two arguments. {{{ (gdb) p creat("/tmp/output.log", 0600) $2 = 1 }}} 5. Exit gdb. Choose yes when asked if you want to quit. {{{ (gdb) quit A debugging session is active. Inferior 1 [process 17793] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/bin/ruby1.8, process 17793 }}} Now when you check /proc/ you should see that your STDOUT is directed att the file you created: {{{ # ls -l /proc/17793/fd total 0 lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1 lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /tmp/output.log lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1 lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742] }}} you can confirm this with `tail -f` (or the fact that your program has stopped outputting to terminal). == Merging PDFs. == This is a long one-liner with Ghostscript (`gs`), a tool that comes with the Ubuntu base install (even server). [[BR]] reference (which references other tools for this purpose) : http://www.linux.com/news/software/applications/8229-putting-together-pdf-files {{{ gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=finished.pdf file1.pdf file2.pdf }}} == Fixing garbled gcc and man page output. == On some machines, `gcc` and man pages might produce garbled text. This is usually caused by xterm not supporting UTF-8, or from mismatch in locale information if the garbling is happening when you are working on a remote machine. In either case (for people working with US English), setting LANG to `en_US` or `C` fixes things: {{{ $ export LANG=en_US }}} Set it the same for both local and remote machines if it's happening over SSH. ---- = Development-related = #dev Various odds-and-ends related to tools and workarounds in coding sessions. == git server setup == A central git repository which things can be pushed to/pulled from, but no changes are directly made to, is a bare repository. These are good as central coordination/synchronization points for larger projects and/or if you work on a project from many locations. Here we assume that you have a git repo that you wish to make a bare repo from, and a host for your bare repo. The host should be running `git`, and you should have SSH access to it. A things to note before beginning is the structure of a Git repository: Within your working directory is a hidden directory called `.git`. This is where Git stores all of the configurations, logs, states, etc. of your repository. Within `.git` is a file named `config`, which stores configurations global to your repository. We will be working with these two components. 1. Copy the .git directory from your working git repo, and rename it. If your repo is named "project", the practice is to rename the .git taken from it to "project.git": {{{ $ cp -Rf project/.git project.git }}} 1. change "bare" to "true" in file `config` in project.git: {{{ [core] repositoryformatversion = 0 filemode = true bare = true logallrefupdates = true ... }}} 1. copy the new bare repo to your server. {{{ $ scp -r project.git [bare repo host]:[path to destination] }}} You should now be able to pull from/push to the host with the bare repository as usual. For example, with `git clone`: {{{ $ git clone ssh+git://[bare repo host]/path/to/project.git }}} You will be asked to authenticate with your SSH password. If you already have a local working directory, and you wish to start using your bare repo, just edit .git/config in your local git repo to point it to your server: {{{ ... [remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = ssh+git://[bare repo host]/path/to/project.git ... }}} Your changes to `config` will be applied starting with the next the git operation you do. Alternatively, the `--bare` flag for `git init` lets you if you start a bare repository from scratch: {{{ $ git init --bare my-repo.git }}} === Miscellaneous Git ops. === * deleting branches: local - `git branch -D [branchname]` remote - `git push origin :[branchname]` (note the colon prepended to branch name) * "rewinding" to a previous commit: `git rebase -i [hash ID]^^` (note the two carets) * undoing a rebase: `git reset --hard [tag]`, where [tag] is the "HEAD@{n}" key for the commit from `git reflog` * cloning into a renamed directory, from another user's repo, with ssh: `git clone ssh+git://user@repolocation/repo.git [newname]` - the cloned repo will be named [newname] == FreeBSD , mininet, and `qemu/kvm`.== [http://mininet.org Mininet] is an SDN prototyping platform/data plane emulator. It uses Linux network namespaces, so it can only be natively installed on Linux. However, they provide a pre-installed VM that can be run with some virtualization. `qemu` is one such platform that can be used to run this VM. We need to diverge a bit from the Mininet page tutorials to get this working on FreeBSD. Note,we do need a machine that meets the prereqs for running `qemu` for this to work. === references === * https://wiki.freebsd.org/qemu - basic configs/setup * http://www.cyberciti.biz/faq/howto-configure-freebsd-vlans-with-ifconfig-command/ - figuring out some of the configs in the link above do * http://mininet.org/vm-setup-notes/ - starting the mininet VM === the steps. === Installation. [[BR]] Repeating the steps on the first link above: 1. Install `qemu` and `kqemu-kmod`. The first is qemu itself. the latter is the accelerator. 2. Add to /boot/loader.conf: {{{ if_bridge_load="YES" if_tap_load="YES" aio_load="YES" kqemu_load="YES" }}} 3. add `kqemu_enable="YES"` to /etc/rc.conf so `aio` and `kqemu` are loaded at startup (optional). 4. force re-reading of configs by rebooting or dropping to single-user mode and returning. Network configuration. [[BR]] We need to configure network settings so the VM can communicate with the outside world. 1. add/set sysctl's in /etc/sysctl.conf : {{{ echo net.link.tap.user_open=1 >> /etc/sysctl.conf echo net.link.tap.up_on_open=1 >> /etc/sysctl.conf }}} 2. force read by service restart: `/etc/rc.d/sysctl start` 3. create/bridge interfaces: {{{ ifconfig bridge0 create ifconfig tap0 create ifconfig bridge0 addm $iface addm tap0 up chmod 0660 /dev/tap0 }}} where $iface is your main networked interface. Convert/run Mininet VM. [[BR]] 1. Acquire VM from mininet site. 2. convert .vmdk to .qcow2" {{{ qemu-img convert -O qcow2 mininet-vm-disk1.vmdk mininet-vm-disk1.qcow2 }}} 3. start VM (we don't use an ifup/ifdown script here): {{{ qemu -m 1024 mininet-vm-disk1.qcow2 -net nic,model=virtio -net tap,name=tap0,script=no }}} Configure guest. [[BR]] 1. To the guest's eth0, assign an address in the same IP block as your host's interface bound to tap0. For example, if tap0 is bound to fxp0 with address 192.168.1.100/24, assign something like 192.168.1.200. 2. set default gateway. If my host's gateway is 192.168.1.1: {{{ sudo route add default gw 192.168.1.1 }}} 3. fix /etc/resolv.conf to match your network's DNS settings. You should be able to reach the VM from the network now/vice versa. ---- = Networking-related odds and ends = #net Various non-experimental network setups, usually done for convenience. == Disabling SSH timeout == source: http://docs.oseems.com/application/ssh/disable-timeout You can essentially prevent SSH from timing out after a long idle period by adding the following to /etc/ssh/ssh_config : {{{ ServerAliveInterval 100 }}} The above will make the client send a keepalive signal once every 100 seconds. Alternatively, you can modify server-side configs by adding the following to /etc/ssh/sshd_config : {{{ ClientAliveInterval 30 TCPKeepAlive yes ClientAliveCountMax 99999 }}} And restarting sshd: {{{ /etc/init.d/ssh restart }}} == NAT boxes. == NAT boxes are handy if you want to build a gateway. First and foremost, you need to enable IP packet forwarding on your to-be-NAT box. [[BR]] On Linux, the command {{{ sudo sysctl -e net.ipv4.ip_forward=1 }}} at the shell enables it. For persistence, add the following line to /etc/default/ufw: {{{ net.ipv4.ip_forward=1 }}} [[BR]] Similarly for FreeBSD (assuming you have `sudo` installed), {{{ sudo sysctl net.inet.ip.forwarding=1 }}} enables it, and the following line in /etc/sysctl.conf makes it persistent: {{{ net.inet.ip.forwarding=1 }}} === with `ufw` === source: https://nowhere.dk/articles/tip_nat_with_ubuntus_ufw_firewall `ufw` is your standard Linux firewall, and comes with Ubuntu server edition. Turning a multi-interface Linux box into a router is a matter of the following steps: 1. configure IP forwarding edit /etc/default/ufw : {{{ DEFAULT_FORWARD_POLICY="ACCEPT" }}} 2. set up IP masquerading in `ufw` edit /etc/ufw/before.rules, just after the header : {{{ # nat Table rules *nat :POSTROUTING ACCEPT [0:0] # Forward traffic through ppp0 - Change to match you out-interface -A POSTROUTING -s 192.168.1.0/24 -o ppp0 -j MASQUERADE # don't delete the 'COMMIT' line or these nat table rules won't # be processed COMMIT }}} The address block after -s should match the address block behind the NAT firewall. 3. restart ufw: {{{ sudo ufw disable && sudo ufw enable }}} === with `pf` === `pf` is the OpenBSD packet filter, a piece of software intended for heavy-duty packet filtering/firewalls and comes with some Berkeley UNIX derivatives. Assuming you have IP forwarding enabled, the following configuration in /etc/pf.conf should give you a NAT firewall: {{{ ext_if="bge0" int_if="em0" external_addr="192.168.203.155" internal_net="192.168.1.0/24" nat on $ext_if from $internal_net to any -> ($ext_if) pass in all pass out all }}} `ext_if` is the interface facing the external network, and `int_if` is the interface connected to your NATed net. Once saved, start `pf`: {{{ sudo pfctl -e -f /etc/pf.conf }}} If it throws errors, make sure that the kernel module (pf.ko or something similar) is loaded. == FreeBSD `ifconfig` shinanigans. == FreeBSD's `ifconfig` combines the features of `ifconfig` and `iwconfig` in Linux (and probably more). For example, you can get a list of AP's, you can do: {{{ $ ifconfig wlan0 list scan SSID/MESH ID BSSID CHAN RATE S:N INT CAPS kitchen 08:86:3b:a0:20:f2 6 54M 8:0 100 EP HTCAP WPA RSN WME WPS WINMAIN c4:7d:4f:37:2d:f0 6 54M 15:0 102 ES HTCAP WME front.door 08:86:3b:d7:f4:6c 6 54M 5:0 100 EP RSN HTCAP WME WPS }}} And so on. Another thing that you can do (unintentionally or otherwise) is to assign multiple network addresses to a single interface. The `add` keyword lets you do this intentionally: {{{ $ sudo ifconfig wlan0 inet 192.168.206.1 add $ ifconfig wlan0 wlan0: flags=8843 metric 0 mtu 1500 ether 00:1c:bf:9a:61:c7 inet 192.168.206.120 netmask 0xffffff00 broadcast 192.168.206.255 inet 192.168.206.1 netmask 0xffffff00 broadcast 192.168.206.255 nd6 options=29 media: IEEE 802.11 Wireless Ethernet OFDM/36Mbps mode 11g status: associated ssid WINMAIN channel 6 (2437 MHz 11g) bssid c4:7d:4f:37:2d:f0 country US authmode OPEN privacy OFF txpower 0 bmiss 7 scanvalid 60 protmode CTS bintval 102 }}} On wired interfaces, assigning addresses with `ifconfig [iface] inet [address]` will have the same effect, which may not be what you want, especially when you expect the old address to just be replaced by your new one^2^. In this case, you can remove the unwanted address with `-alias`: {{{ $ sudo ifconfig wlan0 inet 192.168.206.1 -alias $ ifconfig wlan0 wlan0: flags=8843 metric 0 mtu 1500 ether 00:1c:bf:9a:61:c7 inet 192.168.206.120 netmask 0xffffff00 broadcast 192.168.206.255 nd6 options=29 media: IEEE 802.11 Wireless Ethernet OFDM/36Mbps mode 11g status: associated ssid WINMAIN channel 6 (2437 MHz 11g) bssid c4:7d:4f:37:2d:f0 country US authmode OPEN privacy OFF txpower 0 bmiss 7 scanvalid 60 protmode CTS bintval 102 }}} Otherwise, this feature is pretty handy for quickly adding a IP interface to an interface for, say (as a silly example), creating a carted-off piece of network for a host you don't want to put on the main network, but you still need to access. ---- == Quick Printing Setup. == #print This section contains info that should let you get up and printing from a network printer relatively quickly. === with CUPS === * ref: http://linuxcourse.rutgers.edu/howto/cups_printing.html CUPS stands for Common UNIX Printing System, and is fairly standard as a means to print from UNIX and UNIX-like things. We'll describe the steps needed for quick CUPS printing setup (bare-bones) in Ubuntu 11.04 [[BR]] 1. install packages. {{{ cups cupsys-driver-gutenprint libcupsys2 libcupsimage2 }}} 2. The default port that CUPS serves its admin GUI on is tcp:631: {{{ $ netstat -na -4 | grep 631 tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN }}} If you see it, point your browser to localhost:631. You should get a shiny-white welcome page. Yes, it was made by Apple. 3. Add your printer from the Administration tab -> Add Printer. It will ask for the root user and password (OK if you are on sudoers). You will be asked to provide info on the printer, and choose the drivers. [[BR]] If all goes well, you will be able to see the printer's status with `lpstat`. {{{ $ lpstat -p -d printer Phaser_5500DT is idle. enabled since Sat 28 Apr 2012 01:14:59 AM EDT no system default destination }}} If you want to print, you can do so by invoking `lp`: {{{ $ lp -o sides=two-sided-long-edge -d Phaser_5500DT [filename] }}} === with `lpd` === * ref: http://www.wonkity.com/~wblock/docs/html/lpdprinting.html `lpd` comes packaged with FreeBSD, and is therefore extensively covered in its handbook. The above link lets you get going relatively quickly; here parts of the link are repeated. 1. (1) set up the spool directory (where printed files go temporarily), with (2) modified perms to keep jobs private: {{{ mkdir /var/spool/lpd/lp (1) chown daemon:daemon /var/spool/lpd/lp chmod 770 /var/spool/lpd/lp (2) }}} 2. enable `lpd`. Add the following to /etc/rc.conf: {{{ lpd_enable="YES" }}} to make `lpd` start at boot-up. "lpd" at the shell as root will also start it. 3. edit /etc/printcap : {{{ lp:\ :lp=9100@192.168.200.32:\ :sh:\ :mx#0:\ :sd=/var/spool/lpd/lp:\ :lf=/var/log/lpd-errs: }}} * lp : the printer (port@machine IP), specifically Phaser_5500DT (the WINLAB printer in the large cubicle room) * sh : suppress page header * mx : maximum file size set to 0 (unlimited) * sd : spool directory * lf : file for error logging [[BR]] You can test if your setup works by piping a text string to `lpr`: {{{ printf "printer test page\r\n\f" | lpr }}} You also may have to do some extra work to print certain file formats. The WINLAB printer doesn't support direct printing of PDFs, so a quick work-around is to (1) convert, and (2) print it as a Postscript file: {{{ pdf2ps /tmp/myfile.pdf /tmp/myfile.ps (1) lpr /tmp/myfile.ps (2) }}} ---- == one-liners. == #fortune * in bash, if `seq` doesn't exist, try `gseq`. * options for mounting a *BSD filesystem in Linux: `-t ufs -o ufstype=ufs2` (not '44bsd' like the man pages say) * a dependency fix not requiring '`-f install`' with apt-get: '`build-dep [package]`', then install as usual. * `[sudo] ip addr flush [iface]` resets the IP configurations for an interface on Ubuntu. * `[sudo] ip route flush` resets the route table on Ubuntu; `[sudo] route flush` does the same on FreeBSD. * tar -xvf [archive.tgz] [filename] lets you extract a single file from an archive and -tf lists contents. * the --exclude flag for tar can be used to ignore certain files when packing whole directories into tarballs. * find -x [dir] -size +10000 -exec du -h {} \; finds files over 10kB, including hidden files. * common native serial devices: /dev/ttyS0 for Linux, /dev/cuau0 for FreeBSD. * `ctrl-l` clears a terminal and `ctrl-u` clears the current line, including password prompts. ---- [[BR]] ^1.That being said, Ubuntu is the official sanctioned OS of ORBIT.^ [[BR]] ^2.Apparently Mac OSX inherits this behavior. However I don't know if the same (-)alias parameter applies.^