Context Navigation

← Previous Version
View Latest Version
Next Version →

Version 35 (modified by akoshibe, 11 years ago) ( diff )
—

*Nix related tidbits.

This is a page on various tools and techniques related to administration, programming, disaster recovery - basically, odds and ends that come in handy once every so often on a UNIX-like system. Anything based on a link is reiterated just for completeness, and just in case a link dies.

This page is a compilation of things done with Ubuntu and FreeBSD, the OSes that the author of this page use/break on a regular basis¹. Although syntax will inevitably be different amongst various systems, the general ideas of what a UNIX-like OS can do /should/ be there. Additions/corrections to the page are very much welcome.

Shell-based Command-line tools/tricks. So far, this section includes:

extundelete - Linux file system recovery
cdrecord - burning bootable ISO's
process I/O redirection (gdb)
merging pdfs (gs)

Development-related Tools and workarounds

fixing garbled text (gcc)
git server (bare repo) setup
various git commands
importing non-Eclipse projects to Eclipse
FreeBSD, mininet (mn), and qemu

Network Various networking-related things.

Disabling SSH timeout
NAT boxes
- with ufw (Linux)
- with pf (*BSD)
Wireless with wpa_supplicant
FreeBSD ifconfig shinaniganries.

Quick Printing Printing under *nix, relatively quickly

CUPS
lpd

one-liners miscellaneous single-sentence tips.

Shell-based tools and techniques

File recovery with Extundelete

Extundelete lets you recover files (directories) that you've accidentally deleted on an ext file system. The best way to use the tool is to have a live CD with the tool on it, although it does not seem to come packaged with Linux by default.

Boot the machine off of live CD.
Install extundelete. On Ubuntu, you need to edit sources.list, since it is not part of the standard repositories.

Restore the files on the partition:

# sudo extundelete --restore-all /dev/sda1

This restores everything that has been deleted in /dev/sda1, given the tool finds them. All recovered files are placed in a recovery directory that you can package and back up before rebooting the machine.

Command-line CD burner

source : http://www.yolinux.com/TUTORIALS/LinuxTutorialCDBurn.html

Many CD burners are fairly dependency-heavy, mostly thanks to their GUI. cdrecord is a command-line based CD/DVD burner with just two dependencies at ~2.3kB:

$ sudo apt-get install cdrecord
...
The following NEW packages will be installed:
  genisoimage wodim
...
After this operation, 2,306 kB of additional disk space will be used.

Now, to use:

Check for supported devices. You'll need to specify what to use when burning the image. cdrecord seems to just be a wrapper plus then some for woodim, and the two can be used interchangeably for this step.

$ wodim --devices
wodim: Overview of accessible drives (2 found) :
-------------------------------------------------------------------------
 0  dev='/dev/scd0'     rwrw-- : 'SONY' 'DVD-ROM DDU1615'
 1  dev='/dev/scd1'     rwrw-- : 'TSSTcorp' 'CDDVDW SE-208AB'
-------------------------------------------------------------------------

Burn. The meaning of the options can be listed with 'cdrecord —help'.

$ sudo cdrecord -v -eject speed=16 dev='/dev/scd1' Downloads/ubuntu-12.04-server-amd64.iso

This will spew a bunch of output, but complete with something like below:

Starting new track at sector: 0
Track 01:  684 of  684 MB written (fifo 100%) [buf  99%]  16.8x.
Track 01: Total bytes read/written: 717533184/717533184 (350358 sectors).
Writing  time:  327.327s
Average write speed  14.6x.
Min drive buffer fill was 99%
Fixating...
Fixating time:   24.715s
BURN-Free was never needed.
wodim: fifo had 11302 puts and 11302 gets.
wodim: fifo was 0 times empty and 11100 times full, min fill was 97%.

And eject the CD.

Redirecting the output of a live process.

source: http://etbe.coker.com.au/2008/02/27/redirecting-output-from-a-running-process/

Say that you have a running process whose outputs are either being piped to /dev/null or a terminal that you can't get a hold of, but you'd like to be able to see its output. Conversely, you may want to redirect something's output to /dev/null. You can force the process to change its output using gdb, the GNU debugger. This is a powerful tool that allows you to attach to a live process and inspect and manipulate the contents of its address space.

Here I have a script that I had backgrounded but is still outputting to my terminal (annoying). I will redirect the program's output to a file.

Find the PID of your process. Mine is called arrival_collector.rb, so I look for it using ps.

# ps -ef | grep collector
root     17793 11276  0 18:39 pts/1    00:00:00 ruby arrival_collector.rb -r a,h,ee,f,wknd1,wknd2 -s scott,busch_a,pubsafs,foodsci,scott,scott -v

/proc/ shows that its STDOUT is my terminal, /dev/pts/1:

# ls -l /proc/17793/fd
total 0
lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742]

Attach to the process with GDB.
```
# gdb -p [PID] [path/to/executable]
```

It wil spew a bunch of output but eventually give you a prompt:

# gdb -p 17793 /opt/grailrtls/grail3_ruby/grail3protocols/arrival_collector.rb
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
...
Loaded symbols for /lib/libnss_files.so.2
0xb78c7424 in __kernel_vsyscall ()
(gdb)

Close STDOUT:
```
(gdb) p close(1)
$1 = 0
```

Point STDOUT to a file. Like in the link, I will use creat() to point STDOUT to a log file in /tmp/. creat() takes the path to the file and its permissions as the two arguments.
```
(gdb) p creat("/tmp/output.log", 0600)
$2 = 1
```

Exit gdb. Choose yes when asked if you want to quit.

(gdb) quit
A debugging session is active.

        Inferior 1 [process 17793] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/ruby1.8, process 17793

Now when you check /proc/ you should see that your STDOUT is directed att the file you created:

# ls -l /proc/17793/fd
total 0
lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /tmp/output.log
lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742]

you can confirm this with tail -f (or the fact that your program has stopped outputting to terminal).

Merging PDFs.

This is a long one-liner with Ghostscript (gs), a tool that comes with the Ubuntu base install (even server).
reference (which references other tools for this purpose) : http://www.linux.com/news/software/applications/8229-putting-together-pdf-files

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=finished.pdf file1.pdf file2.pdf

Fixing garbled gcc and man page output.

On some machines, gcc and man pages might produce garbled text. This is usually caused by xterm not supporting UTF-8, or from mismatch in locale information if the garbling is happening when you are working on a remote machine. In either case (for people working with US English), setting LANG to en_US or C fixes things:

$ export LANG=en_US

Set it the same for both local and remote machines if it's happening over SSH.

Development-related

Various odds-and-ends related to tools and workarounds in coding sessions.

git server setup

A central git repository which things can be pushed to/pulled from, but no changes are directly made to, is a bare repository. These are good as central coordination/synchronization points for larger projects and/or if you work on a project from many locations.

Here we assume that you have a git repo that you wish to make a bare repo from, and a host for your bare repo. The host should be running git, and you should have SSH access to it.

A things to note before beginning is the structure of a Git repository: Within your working directory is a hidden directory called .git. This is where Git stores all of the configurations, logs, states, etc. of your repository. Within .git is a file named config, which stores configurations global to your repository. We will be working with these two components.

Copy the .git directory from your working git repo, and rename it. If your repo is named "project", the practice is to rename the .git taken from it to "project.git":
```
 $ cp -Rf project/.git project.git
```

change "bare" to "true" in file config in project.git:

 [core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        logallrefupdates = true
 ...

copy the new bare repo to your server.
```
 $ scp -r project.git [bare repo host]:[path to destination]
```
You should now be able to pull from/push to the host with the bare repository as usual. For example, with git clone:
```
 $ git clone ssh+git://[bare repo host]/path/to/project.git
```
You will be asked to authenticate with your SSH password.

If you already have a local working directory, and you wish to start using your bare repo, just edit .git/config in your local git repo to point it to your server:
 ...

 [remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = ssh+git://[bare repo host]/path/to/project.git
 ...
Your changes to config will be applied starting with the next the git operation you do.

Alternatively, the --bare flag for git init lets you if you start a bare repository from scratch:

 $ git init --bare my-repo.git

Miscellaneous Git ops.

deleting branches: local - git branch -D [branchname] remote - git push origin :[branchname] (note the colon prepended to branch name)
"rewinding" to a previous commit: git rebase -i [hash ID]^^ (note the two carets)
undoing a rebase: git reset --hard [tag], where [tag] is the "HEAD@{n}" key for the commit from git reflog
cloning into a renamed directory, from another user's repo, with ssh: git clone ssh+git://user@repolocation/repo.git [newname] - the cloned repo will be named [newname]

FreeBSD , mininet, and `qemu/kvm`.

Mininet is an SDN prototyping platform/data plane emulator. It uses Linux network namespaces, so it can only be natively installed on Linux. However, they provide a pre-installed VM that can be run with some virtualization. qemu is one such platform that can be used to run this VM. We need to diverge a bit from the Mininet page tutorials to get this working on FreeBSD.

Note,we do need a machine that meets the prereqs for running qemu for this to work.

references

https://wiki.freebsd.org/qemu - basic configs/setup
http://www.cyberciti.biz/faq/howto-configure-freebsd-vlans-with-ifconfig-command/ - figuring out some of the configs in the link above do
http://mininet.org/vm-setup-notes/ - starting the mininet VM

the steps.

Installation.
Repeating the steps on the first link above:

Install qemu and kqemu-kmod. The first is qemu itself. the latter is the accelerator.

Add to /boot/loader.conf:

if_bridge_load="YES"
if_tap_load="YES"
aio_load="YES"
kqemu_load="YES"

add kqemu_enable="YES" to /etc/rc.conf so aio and kqemu are loaded at startup (optional).
force re-reading of configs by rebooting or dropping to single-user mode and returning.

Network configuration.
We need to configure network settings so the VM can communicate with the outside world.

add/set sysctl's in /etc/sysctl.conf :

echo net.link.tap.user_open=1 >> /etc/sysctl.conf
echo net.link.tap.up_on_open=1 >> /etc/sysctl.conf

force read by service restart: /etc/rc.d/sysctl start

create/bridge interfaces:

ifconfig bridge0 create
ifconfig tap0 create
ifconfig bridge0 addm $iface addm tap0 up
chmod 0660 /dev/tap0

where $iface is your main networked interface.

Convert/run Mininet VM.

Acquire VM from mininet site.

convert .vmdk to .qcow2"

qemu-img convert -O qcow2 mininet-vm-disk1.vmdk mininet-vm-disk1.qcow2

start VM (we don't use an ifup/ifdown script here):

qemu -m 1024 mininet-vm-disk1.qcow2 -net nic,model=virtio -net tap,name=tap0,script=no

Configure guest.

To the guest's eth0, assign an address in the same IP block as your host's interface bound to tap0. For example, if tap0 is bound to fxp0 with address 192.168.1.100/24, assign something like 192.168.1.200.
set default gateway. If my host's gateway is 192.168.1.1:
```
sudo route add default gw 192.168.1.1
```
fix /etc/resolv.conf to match your network's DNS settings.

You should be able to reach the VM from the network now/vice versa.

Networking-related odds and ends

Various non-experimental network setups, usually done for convenience.

Disabling SSH timeout

source: http://docs.oseems.com/application/ssh/disable-timeout

You can essentially prevent SSH from timing out after a long idle period by adding the following to /etc/ssh/ssh_config :

ServerAliveInterval 100

The above will make the client send a keepalive signal once every 100 seconds. Alternatively, you can modify server-side configs by adding the following to /etc/ssh/sshd_config :

ClientAliveInterval 30
TCPKeepAlive yes 
ClientAliveCountMax 99999

And restarting sshd:

/etc/init.d/ssh restart

NAT boxes.

NAT boxes are handy if you want to build a gateway. First and foremost, you need to enable IP packet forwarding on your to-be-NAT box.
On Linux, the command

sudo sysctl -e net.ipv4.ip_forward=1

at the shell enables it. For persistence, add the following line to /etc/default/ufw:

net.ipv4.ip_forward=1

Similarly for FreeBSD (assuming you have sudo installed),

sudo sysctl net.inet.ip.forwarding=1

enables it, and the following line in /etc/sysctl.conf makes it persistent:

net.inet.ip.forwarding=1

with `ufw`

source: https://nowhere.dk/articles/tip_nat_with_ubuntus_ufw_firewall

ufw is your standard Linux firewall, and comes with Ubuntu server edition. Turning a multi-interface Linux box into a router is a matter of the following steps:

configure IP forwarding

edit /etc/default/ufw :

DEFAULT_FORWARD_POLICY="ACCEPT"

set up IP masquerading in ufw

edit /etc/ufw/before.rules, just after the header :

# nat Table rules
 *nat
 :POSTROUTING ACCEPT [0:0]

 # Forward traffic through ppp0 - Change to match you out-interface
 -A POSTROUTING -s 192.168.1.0/24 -o ppp0 -j MASQUERADE

 # don't delete the 'COMMIT' line or these nat table rules won't
 # be processed
 COMMIT

The address block after -s should match the address block behind the NAT firewall.

restart ufw:
```
sudo ufw disable && sudo ufw enable
```

with `pf`

pf is the OpenBSD packet filter, a piece of software intended for heavy-duty packet filtering/firewalls and comes with some Berkeley UNIX derivatives. Assuming you have IP forwarding enabled, the following configuration in /etc/pf.conf should give you a NAT firewall:

ext_if="bge0"
int_if="em0"
external_addr="192.168.203.155"
internal_net="192.168.1.0/24"
nat on $ext_if from $internal_net to any -> ($ext_if)
pass in all 
pass out all

ext_if is the interface facing the external network, and int_if is the interface connected to your NATed net. Once saved, start pf:

sudo pfctl -e -f /etc/pf.conf

If it throws errors, make sure that the kernel module (pf.ko or something similar) is loaded.

FreeBSD `ifconfig` shinanigans.

FreeBSD's ifconfig combines the features of ifconfig and iwconfig in Linux (and probably more). For example, you can get a list of AP's, you can do:

$ ifconfig wlan0 list scan
SSID/MESH ID    BSSID              CHAN RATE   S:N     INT CAPS
kitchen         08:86:3b:a0:20:f2    6   54M   8:0    100 EP   HTCAP WPA RSN WME WPS
WINMAIN         c4:7d:4f:37:2d:f0    6   54M  15:0    102 ES   HTCAP WME
front.door      08:86:3b:d7:f4:6c    6   54M   5:0    100 EP   RSN HTCAP WME WPS

And so on.

Another thing that you can do (unintentionally or otherwise) is to assign multiple network addresses to a single interface. The add keyword lets you do this intentionally:

$ sudo ifconfig wlan0 inet 192.168.206.1 add
$ ifconfig wlan0 
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:1c:bf:9a:61:c7
        inet 192.168.206.120 netmask 0xffffff00 broadcast 192.168.206.255
        inet 192.168.206.1 netmask 0xffffff00 broadcast 192.168.206.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: IEEE 802.11 Wireless Ethernet OFDM/36Mbps mode 11g
        status: associated
        ssid WINMAIN channel 6 (2437 MHz 11g) bssid c4:7d:4f:37:2d:f0
        country US authmode OPEN privacy OFF txpower 0 bmiss 7 scanvalid 60
        protmode CTS bintval 102

On wired interfaces, assigning addresses with ifconfig [iface] inet [address] will have the same effect, which may not be what you want, especially when you expect the old address to just be replaced by your new one². In this case, you can remove the unwanted address with -alias:

$ sudo ifconfig wlan0 inet 192.168.206.1 -alias
$ ifconfig wlan0 
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:1c:bf:9a:61:c7
        inet 192.168.206.120 netmask 0xffffff00 broadcast 192.168.206.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: IEEE 802.11 Wireless Ethernet OFDM/36Mbps mode 11g
        status: associated
        ssid WINMAIN channel 6 (2437 MHz 11g) bssid c4:7d:4f:37:2d:f0
        country US authmode OPEN privacy OFF txpower 0 bmiss 7 scanvalid 60
        protmode CTS bintval 102

Otherwise, this feature is pretty handy for quickly adding a IP interface to an interface for, say (as a silly example), creating a carted-off piece of network for a host you don't want to put on the main network, but you still need to access.

Quick Printing Setup.

This section contains info that should let you get up and printing from a network printer relatively quickly.

with CUPS

ref: http://linuxcourse.rutgers.edu/howto/cups_printing.html

CUPS stands for Common UNIX Printing System, and is fairly standard as a means to print from UNIX and UNIX-like things. We'll describe the steps needed for quick CUPS printing setup (bare-bones) in Ubuntu 11.04

install packages.

cups cupsys-driver-gutenprint libcupsys2 libcupsimage2

The default port that CUPS serves its admin GUI on is tcp:631:
```
$ netstat -na -4 | grep 631
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
```
If you see it, point your browser to localhost:631. You should get a shiny-white welcome page. Yes, it was made by Apple.

Add your printer from the Administration tab → Add Printer. It will ask for the root user and password (OK if you are on sudoers). You will be asked to provide info on the printer, and choose the drivers.

If all goes well, you will be able to see the printer's status with lpstat.
$ lpstat -p -d
printer Phaser_5500DT is idle.  enabled since Sat 28 Apr 2012 01:14:59 AM EDT
no system default destination
If you want to print, you can do so by invoking lp:
$ lp -o sides=two-sided-long-edge -d Phaser_5500DT [filename]

with `lpd`

ref: http://www.wonkity.com/~wblock/docs/html/lpdprinting.html

lpd comes packaged with FreeBSD, and is therefore extensively covered in its handbook. The above link lets you get going relatively quickly; here parts of the link are repeated.

(1) set up the spool directory (where printed files go temporarily), with (2) modified perms to keep jobs private:

mkdir /var/spool/lpd/lp                      (1)
chown daemon:daemon /var/spool/lpd/lp         
chmod 770 /var/spool/lpd/lp                  (2)

enable lpd. Add the following to /etc/rc.conf:
```
lpd_enable="YES"
```
to make lpd start at boot-up. "lpd" at the shell as root will also start it.

edit /etc/printcap :

lp:\
        :lp=9100@192.168.200.32:\
        :sh:\
        :mx#0:\
        :sd=/var/spool/lpd/lp:\
        :lf=/var/log/lpd-errs:

lp : the printer (port@machine IP), specifically Phaser_5500DT (the WINLAB printer in the large cubicle room)
sh : suppress page header
mx : maximum file size set to 0 (unlimited)
sd : spool directory
lf : file for error logging

You can test if your setup works by piping a text string to lpr:

printf "printer test page\r\n\f" | lpr

You also may have to do some extra work to print certain file formats. The WINLAB printer doesn't support direct printing of PDFs, so a quick work-around is to (1) convert, and (2) print it as a Postscript file:

pdf2ps /tmp/myfile.pdf /tmp/myfile.ps       (1)
lpr /tmp/myfile.ps                          (2)

one-liners.

in bash, if seq doesn't exist, try gseq.
options for mounting a *BSD filesystem in Linux: -t ufs -o ufstype=ufs2 (not '44bsd' like the man pages say)
a dependency fix not requiring '-f install' with apt-get: 'build-dep [package]', then install as usual.
[sudo] ip addr flush [iface] resets the IP configurations for an interface on Ubuntu.
[sudo] ip route flush resets the route table on Ubuntu; [sudo] route flush does the same on FreeBSD.
tar -xvf [archive.tgz] [filename] lets you extract a single file from an archive and -tf lists contents.
the —exclude flag for tar can be used to ignore certain files when packing whole directories into tarballs.
find -x [dir] -size +10000 -exec du -h {} \; finds files over 10kB, including hidden files.
common native serial devices: /dev/ttyS0 for Linux, /dev/cuau0 for FreeBSD.
ctrl-l clears a terminal and ctrl-u clears the current line, including password prompts.

^{1.That being said, Ubuntu is the official sanctioned OS of ORBIT.}
^{2.Apparently Mac OSX inherits this behavior. However I don't know if the same (-)alias parameter applies.}

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text